CodeRunner: Should students be penalised when the JOBE server is offline?

We just did a test of our CodeRunner set-up. We have 2 JOBE servers sitting behind a load-balancer, and what we did is to put the system under some load, using a Jmeter script we had made, and then we restarted one of the two JOBE servers.

The kind of load we were simulating was about 100 responses being submitted per minute, and when the server restart happened, 8 qiuestion submissions got an error. I am pretty happy with that.

However, the point I wanted to raise was that when this happened, the student got marked wrong, as if they had submitted an incorrect response. It seems a bit unfair to penalise the students for this:

Would it not be fairer to treat this case differently? For example, in Moodle question system terms, we could treat this as an 'invalid' response, rather than an incorrect one. That way, students would not get a penalty, and they could just try again later when the server comes back.

This might be something to think about for CodeRunner version 3.2?

(It might also be good to re-word the message to suggest students try again a bit later.)

Re: Should students be penalised when the JOBE server is offline?

de Richard Lobb - viernes, 24 de febrero de 2017, 10:11

Thanks Tim. Yes, I agree that the student shouldn't be penalised when the Jobe server fails. I was vaguely aware that I should fix that some time but it never got to be a priority, since Jobe never fails :-)

I'll bump it up the priority list for the next version, and think about the error message too. In the meantime, a "regrade all", or regrade of affected students, does of course fix any such problems.

Richard

Re: Should students be penalised when the JOBE server is offline?

de Tim Hunt - viernes, 24 de febrero de 2017, 20:34

Thanks Richard.

Re: Should students be penalised when the JOBE server is offline?

de Martin Zwerschke - sábado, 6 de enero de 2018, 01:35

At least students should be informed, if CodeRunner (to be exact: the jobe server) is online and working correctly.

For this reason I put a java script into the announcements of my Moodle course:

News and announcements
<br>

<script type="text/javascript" language="javascript">

function CR_Probe() 
{
    var xhr = new XMLHttpRequest();
     xhr.timeout=3000;
     xhr.ontimeout= function() {
      document.getElementById('cr_check').style.color = "red";  
      document.getElementById('cr_check').innerHTML="Coderunner not active!";
    };
    
   xhr.onerror= function() {     
      document.getElementById('cr_check').style.color = "red"; 
      document.getElementById('cr_check').innerHTML="Coderunner-Error!"; 
    };

    xhr.onload= function() {
      if( xhr.status==200 && xhr.responseText.includes("python3") ) 
      {        
        document.getElementById('cr_check').style.color = "green";
        document.getElementById('cr_check').innerHTML="Coderunner is active"; 
      }
      else
      {
        document.getElementById('cr_check').style.color = "red";
        document.getElementById('cr_check').innerHTML="Coderunner-Error!";
      }
       
    };
   xhr.open("GET", "http://OUR_JOBE_URL/jobe/index.php/restapi/languages", true);
   xhr.setRequestHeader("Content-type", "application/jsonp");
   xhr.send();
}

setInterval(CR_Probe, 6000);
//window.onload = CR_Probe;

</script>

<button type="button" id="cr_check">Checking if CodeRunner is running...</button>
<br>
<br>
You can only check in "CodeRunner"-questions, if "CodeRunner" is active !

The script is not optimal, because I don't know how to access the moodle settings in javascript, to retrieve the Jobe-URL (and API-Key if necessary).

Moreover it would be better to integrate such a green/red light into the CodeRunner plugin itself.

Re: Should students be penalised when the JOBE server is offline?

de Richard Lobb - domingo, 7 de enero de 2018, 14:59

Hi Martin

I agree that the student shouldn't be penalised if the Jobe server is down, although I would also hope that such a situation is very rare. I can't recall any unplanning jobe outages on our production server for some years.

I thought I had dealt with the problem Tim raised, at least to the point of ensuring students don't get penalised. I just did a bit of brief testing and here's what I found:

If the question type (or a customised version of it) explicitly specifies in the Advanced Customisation section that the sandbox to use is jobesandbox, then if the Jobe server doesn't respond the student should see the something like the following, and they should not be penalised.
If instead the sandbox is specified as DEFAULT then things are not so pretty. CodeRunner attempts to ask each sandbox in turn what languages it supports, in order to find the first sandbox that works with the current language. [Historically CodeRunner had several different sandboxes available, but Jobe has now replaced all but the deprecated IdeoneSandbox]. If the Jobe sandbox fails to respond to that request, an exception is thrown. This results in somewhat ugly output, but it shouldn't result in penalising the student.

So ... are students actually been penalised when your Jobe server is down, Martin? If so, can you clarify exactly what error messages are being generated. I'm also curious to know why you're having issues with the Jobe server - is this something I should know about it? I haven't heard of any significant stability issues with it.

Richard

Re: Should students be penalised when the JOBE server is offline?

de Martin Zwerschke - lunes, 8 de enero de 2018, 00:08

Hi Richard,

There have been no unexpected outages (save the time in my holidays, when I shut down the server, that's located in my home office by now consuming my privately paid electricity :-) ).

I will move the jobe server to our school now, when school starts again.
But there may be times, the jobe will be down or not reachable, e.g. the DSL-wire or the router/proxy in our school gets stuck or the mains is down for electricians working on the installations or the DDNS service has issues.

There may also be situations, when I shut down the server myself expecting no student to use my courses at the moment, but some do use them anyway - who knows (hope dies last).

I know, that if you select jobe as the sandbox, the red message "Unexpected error..." is displayed. But it appears after the student has tried to check in his solution. He could be disappointed or frustrated this way. Better if he knows the outage before and does not try.
(Also I'm not sure, if my students will really read or understand the english message...)

That's why I created the small script (messages are German)

On start:

If jobe responds:

If connection fails: