Failed to run tests: how to override mark?

Failed to run tests: how to override mark?

by Anton Tremetzberger -
Number of replies: 2

Dear community

we are testing Coderunner in our University with Moodle 4.1.9+ and the latest Coderunner plugin. Our hoster installed the coderunner server (newest version). First small tests was successfull. Last week we had the following error message during a test with a few participants (2 out of 60 students):

Failed to run tests
Job could not be run due to server overload. Perhaps try again shortly?
The submission was invalid, and has been disregarded without penalty.


1. Most of the students din't have this problem. It seems to be that the server was overloaded for this 2 students. Can I change something in the plugin/question settings or should I change the settings on the Jobe server (e.g. increase RAM, CPU, ... or other settings)?
2. The professor tried to override the mark for manual grading ("make comment or override mark"), but it wasn't possible (in all other questions override was possible). Are there any possibilities to override the mark?

thanks, best regards
Anton

In reply to Anton Tremetzberger

Re: Failed to run tests: how to override mark?

by Richard Lobb -
I've never seen Moodle refuse to allow a grade override. I could look further into it if you can provide a bit more detail about exactly what happened (e.g. any error messages, or missing links etc). However, I think the first thing to try is to regrade the quiz submissions by the two students who had problems. That has always worked for me in the very rare cases of a Jobe server failure. Please report back if that doesn't solve the problem.

60 students isn't a big class - we regularly run exams with 500 students. However, the load on the Jobe server depends on many different factors and you should start by estimating the average rate of Jobe server runs throughout the test/exam. Then make sure you have enough Jobe server capacity to handle at least 5 times that rate in order to handle statistical peak submission rates. 

[I am assuming you're running the exam in the usual adaptive mode, so students can submit on-the-fly during the exam. If you hide the Check button so students can't submit during the exam, you'll get a huge load on the Jobe server at the end of the exam when all questions of all students hit the server at once; I strongly advise against that mode unless you have a compelling reason and have done some careful load calculations.]

The average submission rate depends on:
  • the number of students,
  • the number of questions in the quiz
  • the quiz duration 
  • the number of expected attempts on each question by an average student
  • (less obviously) it might also depend on the number of test cases - questions that have per-test-case standard input do a separate run for each test case. 
  • If you're using the new Scratchpad UI, which gives students a way to test their code, you will have lots of extra submissions, too.
Once you know the average submission rate you need to know the average run time for each of your quiz questions. This varies hugely with language, with Java being way more expensive than Python or C. 

You can test the maximum throughput of your Jobe server in your chosen language using the testsubmit.py program supplied with Jobe. See here. But if the questions being asked in the exam require non-trivial computation, the actual throughput might be much less. You need to measure the CPU and clock time for a typical question to run on Jobe.

You can configure multiple Jobe servers for your Moodle server as a semicolon separated list - that's the easiest way to scale up your Jobe server capacity. A server is selected at random from the list for each run. Our current production Moodle server, which handles Python exams with 500 students has eight 8-core Jobe servers, plus extra ones for the Scratchpad UI! [We recently doubled the number because we were required by central IT to install some resource-hungry security software, which cut the throughput by more than a factor of two.]  But this is probably at least twice as much capacity as needed - our Jobe servers are only lightly loaded during exams. 

If you had just two students with failed submissions in a class of 60, you should be fine with just two Jobe servers.
In reply to Richard Lobb

Re: Failed to run tests: how to override mark?

by Anton Tremetzberger -
Hello Richard,

Thank you very much for the detailed and quick answer.

I have now made a re-grading. That was the solution for the problem. We were now able to make a manual grading.

Regarding the adaptive mode: to be sure, I ask you – does this mean the test settings in Moodle (Question behavior – adaptive mode with/without penalty)? We still had the setting to "deferred feedback" in this test. We had overlooked the setting and we will do it in adaptive mode in the future. Perhaps this could be a reason for the problem with the two students.

Regarding scratchpad: at the moment we don't use it, but we are considering using it.

Regarding the server capacity: All our test cases based on Java, so as you said it is much more expensive than other languages. Thank your for this “formula”, we will calculate this and do a test run with testsubmit.py. Then we will see if it the server settings are ok. If necessary, we will add a second server.

Thank you very much for your help.

best regards
Anton