CodeRunner question giving three different error messages for staff/student on live/dev servers

CodeRunner question giving three different error messages for staff/student on live/dev servers

by Chris Nelson -
Number of replies: 5

Hi folks,

Bit of a weird one, and it's happened for two students in our M250 Computing exam over three years now (800-900 student a year = 2/~2500 students affected rate). But it's non-reproducible. 

I'm having to be very mindful of not accidently sharing student info, so I can't share a nice explanatory screencast.

Essentially, we occasionally get a student write something invalid, but rather than a validation error, we (staff) get:


(Which we interpret as Jobe was down/busy at the time of the regrade).

But the student sees the following when reviewing their attempt:


Which is closer to "Your answer was invalid", but because it's an exam (Deferred Feedback behaviour for the quiz, 'Check' button disabled for CodeRunner questions in the quiz), it caused the more recent student to panic as they thought the human exam marker was disregarding it, so that led to a complaint. 

Anyway, if we try to reproduce it on a test server, we instead get this as staff:


The question itself (running /question/type/coderunner/questiontestrun.php?cmid=N&questionid=N) is valid. 

Regrading the question doesn't do anything to the initial staff error message. 

Because we initially felt that the data was cached/stuck, we re-ran a regrade for this student on the question again after the 14-day data clear, but it came back with the same error.

We also tried /question/type/coderunner/management.php and running Runs-Cache purge, to force through any cached data.

So now we're wondering if it's perhaps a bug in the feedback or CodeRunner needs more precise error messages. But we're not 100% sure. 

We don't think we should end up with three different outputs: for staff on the live server, the student on the live server, and the staff view on our test server. 

Would be grateful for any thoughts here!

Versions

  • CodeRunner: 2026041301
  • Moodle: 5.1.4

In reply to Chris Nelson

Re: CodeRunner question giving three different error messages for staff/student on live/dev servers

by Richard Lobb -
Thanks for the report. There are some very complicated things happening here and indeed you're exposing a couple of bugs in the handling of Jobe errors. But it's the apparently recurring Jobe error that is at the heart of the problem and we need to focus on that.
 
The initial grading reports "HTTP Response from Jobe was 0: null". This can occur for quite a few different reasons, such as misconfigured Jobe servers, intermittent network failures or a failure of the connection to Jobe during the run. Critically, though, you say "Regrading the question doesn't do anything to the initial staff error message."  I assume you mean that the HTTP response from Jobe is still 0: null regardless. Misconfiguration or intermittent network errors are ruled out. And all forms of caching should be bypassed during a regrade. CodeRunner's own runs cache doesn't store failed runs and the _testoutcome record from an earlier run is bypassed on a regrade. So it looks like that particular student answer is crashing on Jobe, so a response never gets back to Moodle.
 
You say that when you attempt to reproduce the problem on a test server you get a result table with compile errors. But - the critical question - is that test server using the same Jobe server as the production Moodle server? If not, please switch it across to the same Jobe server and retry. I'm predicting you'll get the bad HTTP Response from Jobe again.
 
It that's the case, then we need to find why that job crashes Jobe. Most simple crashes like resource exhaustion are caught by Jobe's Runguard environment, but not all. In particular, OOM kills take down the entire job, including Runguard.  If you find that sIf you've turned off Runguard's memory limit (i.e. set it to 0) relying on the JVM to manage memory adequately (which should prevent the OOM killer ever getting in on the act) then you should carefully check your Java settings (probably the javac parameters, since this appears to be a job that never compiles anyway).
 
I'll look into whether we can handle Jobe crashes better but I think we first need to confirm that that's what's happening.
 
PS: it seems from what you've said that you've disabled student viewing of the specific feedback even when the student is reviewing the quiz. If they were seeing the specific feedback too, they'd have seen the Jobe error themselves. Why have you disabled specific feedback at the end of an exam?
 
In reply to Richard Lobb

Re: CodeRunner question giving three different error messages for staff/student on live/dev servers

by Chris Nelson -
Thanks for the steer, Richard :-) We'll see if we can work this out yet then! The test and live servers should be 1:1, but as I say, it's hard to replicate.

Yes, the "null" message just won't change, despite other attempts being graded/regraded fine, even other CodeRunner questions for the student.

Regarding the 'PS': the Moodle Quiz-based exams are set as per The Open University's exam departments' policy; possibly around students not necessarily taking the exam concurrently so set as to not give later-starters an undue advantage.
In reply to Richard Lobb

Re: CodeRunner question giving three different error messages for staff/student on live/dev servers

by Chris Nelson -
Hi Richard,

Here's the live server logs.

I tried:
  • Regrading Q5 (null/ we don't think it even reached JOBE).
  • Regrading Q7 (also CR, that returned the same mark).
  • Regrading Q9 (also CR, that returned the same mark).
  • I then tried a full student attempt dry run regrade (so as to not change any marks). Q5 still null.

JOBE logs were:

172.28.56.27 - - [14/Aug/2026:10:00:07 +0100] "POST /jobe/index.php/restapi/runs HTTP/1.1" 200 765 "-" "MoodleBot/5.1 (+https://[server])"
172.28.56.27 - - [14/Aug/2026:10:00:09 +0100] "POST /jobe/index.php/restapi/runs HTTP/1.1" 200 765 "-" "MoodleBot/5.1 (+https://[server])"
172.28.56.27 - - [14/Aug/2026:10:00:10 +0100] "POST /jobe/index.php/restapi/runs HTTP/1.1" 200 539 "-" "MoodleBot/5.1 (+https://[server])"

172.28.56.27 - - [14/Aug/2026:10:00:13 +0100] "POST /jobe/index.php/restapi/runs HTTP/1.1" 200 1789 "-" "MoodleBot/5.1 (+https://[server])"

172.28.56.27 - - [14/Aug/2026:10:00:15 +0100] "POST /jobe/index.php/restapi/runs HTTP/1.1" 200 1789 "-" "MoodleBot/5.1 (+https://[server])"

In reply to Chris Nelson

Re: CodeRunner question giving three different error messages for staff/student on live/dev servers

by Richard Lobb -
I don't have any way to map from the regrading of a particular question to a JOBE log entry. There are two runs with a 765 byte payload, one with a 539 byte payload and two with a 1789 byte payload. I don't know if that's 3 questions, two with retries, or 5 questions with no retries.

The critical question here is whether the regrade of Q5 does or does not reach JOBE. That needs to be resolved definitively.

There are two possibilities:
  1. The request never reaches Jobe.
  2. The response never gets back to Moodle.

The OOM-killer theory is ruled out if it is guaranteed that one of those JOBE log entries corresponds to Q5, since all returned with a 200 OK response. So what might stop a request getting to JOBE or a response getting back? Is there a firewall/WAF/reverse proxy/load balancer in front of production Jobe that isn't in front of your test one? If so, worth checking its logs (not Jobe's) for both inbound and outbound traffic during a Q5 regrade— particularly anything about a reset, timeout, or oversized/blocked response.

Some things to possibly try:

  1. Customise Q5 and in Advanced customisation > Sandbox parameters set {"debug": 1} . Regrade Q5 for that one student (and maybe another student for comparison). Then on the Jobe server look in /home/jobe/runs (which you should also check before starting). The complete record of the run - code, compile output, run output etc - will be there if the job made it to Jobe. If there's nothing there, your theory that the job never makes it to Jobe is confirmed (though please double-check with a regrade of a question that works).
  2. Are you able to pull the current Development branch from GitHub ? If so, you'd get the exact cURL error message rather than the unhelpful 0:null response. But I'm guessing that's not possible?
  3. Look in /var/www/html/jobe/writable/logs for any error reports during regrades. 
  4. Try to carry out the test I suggested earlier of switching your test server to use the same Jobe as the production server and see if you then get the 0: null response.
In reply to Richard Lobb

Re: CodeRunner question giving three different error messages for staff/student on live/dev servers

by Chris Nelson -
Hi Richard,

We tried adding the debug in #1, but cannot move the student to the latest question version through any method, and we're not allowed to regrade the entire exam on our live system. And as stated, we're unable to replicate this issue in our testing servers. Curiously, all three CodeRunner questions in the exam don't move to the latest version (usually methods are to go to Quiz Settings and press 'Save', or regrade the students' attempt). I had a brief chat with Tim Hunt about this, and he's not sure why that has happened either.

For #2 No, sorry. Next time we upgrade (for our December release in late November), I can try running the regrade and see if anything changes.

#3 I don't have sight of those, but my developers have said they can't see anything obvious.

#4 Unfortunately our environments are heavily locked down and we can't mix and match the live and test servers/sub-servers :-(

I think we've probably spent as much time troubleshooting this as we can our side, but at least this problem is documented for the future in case anyone else has a student response get stuck in the same sort of state. Hopefully #2 will shed some more light on it after our next upgrade hits the live environment.