Regrading Issue in CodeRunner questions

Regrading Issue in CodeRunner questions

by Nithya Mallichetti -
Number of replies: 1

Hello everyone,

I'm experiencing an issue with the regrading option for CodeRunner questions. While regrading works well with a small number of questions (e.g., one or two), it does not function properly when I have around 10 questions. Could someone please help me resolve this?

Additionally, when regrading was working, I noticed that after updating the test cases in a question and regrading a selected attempt, if a previous submission was marked as incorrect and the code now works with the updated test cases, the marks are updated properly. However, if the code doesn't work with the new test cases, the marks from the previous attempt are not reduced, even though the code is incorrect.

I need some clarity on this behavior.

Thank you in advance.

In reply to Nithya Mallichetti

Re: Regrading Issue in CodeRunner questions

by Richard Lobb -
Can you clarify what you mean by "does not function properly when I have around 10 questions", please? Please describe the exact behaviour you're observing including such information as the number of students you're regrading, the version of CodeRunner you're running and the version of Moodle. We might also need to know whether your Moodle server is running Apache or nginx.

We have a class of over 900 students with quizzes containing up to 40 CodeRunner questions. Students can have multiple attempts on quizzes. A regrade of such a quiz can take well over 10 hours. Often the browser times out and disconnects from the regrade, but regrading usually continues in the background, although there's no easy way to know it's still going except when you spot updated grades. We have also had a problem with an nginx webserver having too low a timeout on its request to the PHP server, which aborts the regrade immediately - this is a webserver configuration problem that requires sysadmin help.

It's very unlikely that a regrade is raising a student's grade but not lowering it. The most likely explanation here is that the student has made multiple submissions to the question and that during the regrade an earlier submission got marked right but their final submission got marked wrong. Since Moodle never allows a mark for a quiz question to go down, the earlier correct answer will win out over the final wrong answer, which I'm guessing is the one you're looking at?

FWIW: the cripplingly long regrade times with large classes and quizzes are likely to be greatly reduced by two major upcoming enhancements:
  1. The Moodle framework will allow you to select only a subset of the questions in a quiz for regrading.
  2. CodeRunner will provide a configuration option that caches the results of all Jobe-server runs so that only questions that have been materially altered (i.e. where the question template, test cases or prototype have been changed) will get regraded. This will allow for much more efficient pushing out of changes to question wording, for which no actual code execution is required.