We just did a test of our CodeRunner set-up. We have 2 JOBE servers sitting behind a load-balancer, and what we did is to put the system under some load, using a Jmeter script we had made, and then we restarted one of the two JOBE servers.
The kind of load we were simulating was about 100 responses being submitted per minute, and when the server restart happened, 8 qiuestion submissions got an error. I am pretty happy with that.
However, the point I wanted to raise was that when this happened, the student got marked wrong, as if they had submitted an incorrect response. It seems a bit unfair to penalise the students for this:
Would it not be fairer to treat this case differently? For example, in Moodle question system terms, we could treat this as an 'invalid' response, rather than an incorrect one. That way, students would not get a penalty, and they could just try again later when the server comes back.
This might be something to think about for CodeRunner version 3.2?
(It might also be good to re-word the message to suggest students try again a bit later.)