CodeRunner: Debugging the template

Is there a combination of settings that will display to the question developer -exactly- what (json.dump'd) dictionary the marker sent back to CodeRunner, no matter how bizarre it might be?

It would help (me) a lot in getting better at using this (very nice!) system. At the moment, however, I can't tell whether I have a Python-syntax error in my grader (at one extreme) or whether I have accidentally sent back the wrong combination of values for testresults, showoutputonly... (at the other extreme)

I have tried various combinations of "template debugging" and "validate on save", but don't seem to got the above "accept anything" combination for now.

Indeed one thing that would be really handy is a key whose value (of type string) is printed as text no matter what that text might be and no matter what else went wrong --- a kind of "debug write".

Thanks!

Re: Debugging the template

von Richard Lobb - Dienstag, 2. November 2021, 10:11

You can see what output your template is generating by turning on template debugging and then just copying the generated code from the browser window into a Python IDE and running it.

You should also be able to inspect the output just by replacing the final template line of the form "print(json.dumps(outcome))" with something that makes it invalid JSON, such as "print("My outcome is:", json.dumps(outcome))". If you have validate-on-save turned on then when you try to save the question you should then get output starting with something like:

Exception: Bad JSON output from combinator grader output. Output was: My outcome is: {"fraction": 1.0, "testresults": [["iscorrect", "Test", "Expected", "Got", "iscorrect", "ishidden"], [true, "print_one()", "1.0", "1.0", true, false]], "showdifferences": true}

If you are generating output that isn't behaving as you expect, feel free to export the question as an XML and post it to the forum. Combinator template grading is non-trivial.

Re: Debugging the template

von Carroll Morgan - Dienstag, 2. November 2021, 16:18

Thanks for the help. I post below a special "grader" that allows me to simulate and experiment with grader output. When I simply press "check", it shows the code and the table etc. as expected. But if I change for example "fraction" to "friction", ie correct except that I have used the wrong key, all that returns is

"Stored test results could not be deserialised. Perhaps try regrading?
Your code must pass all tests to earn any marks. Try again."

Is there any way to get more specific information, like "friction not recognised" or similar?
I'm running in a local "sandpit", and the question is being previewed from the question bank.

Thanks!

<?xml version="1.0" encoding="UTF-8"?>

<quiz>

<name>

<text>Quiz CS6721 Result tester 211102</text>

</name>

<p>Test results passed back to Moodle.</p>

<table>

<tbody>

<tr>

<td>STUDENT_ANSWER</td>

</tr>

<tr>

<td>comment</td>

<td>arbitrary</td>

</tr>

<tr>

<td>fraction</td>

</tr>

<tr>

<td>awarded</td>

<td>integer</td>

</tr>

<tr>

<td>abort</td>

<td>Boolean</td>

</tr>

<tr>

<td>prologuehtml</td>

</tr>

<tr>

<td>epilogue</td>

</tr>

<tr>

<td>testresults</td>

<td>list of lists</td>

</tr>

<tr>

<td>showoutputonly</td>

<td>Boolean</td>

</tr>

</tbody>

</table>

</div>]]></text>

</questiontext>

</generalfeedback>

<coderunnertype>python3</coderunnertype>

<p>Processing stops when the name field is empty.</p>

<p>

<table>

<tr>

</tr><tr>

</tr><tr>

</tr><tr>

</tr><tr>

</tr><tr>

</tr><tr>

</tr><tr>

</tr><tr>

</tr><tr>

</tr></table>

</p>

<template><![CDATA[import json

# Get answer.

answerString= """{{ STUDENT_ANSWER | e('py') }}"""

answerDict= json.loads(answerString)

results,n= {},0

while True:

dd= '{:02d}'.format(n)

name= answerDict['N'+dd][0]

if name=='': break

value= answerDict['V'+dd][0]

results[name]= json.loads(value)

n+= 1

print(json.dumps(results))

]]></template>

<grader>TemplateGrader</grader>

</testcases>

</question>

</quiz>

ResultsTester.xml

Re: Debugging the template

von Richard Lobb - Dienstag, 2. November 2021, 20:57

That's interesting. If I import your question into my current development system, change the word 'fraction' to 'friction' and click Check I get the following:

The complete text of the error message is:


Exception: Bad or missing fraction in output from template grader. Output was: {"showoutputonly": false, "testresults": [["L", "R"], [0, 1]], "friction": 0.5, "prologuehtml": "prologue", "epiloguehtml": "epilogue"}
 in /var/www/html/moodle/question/type/coderunner/classes/jobrunner.php:275

Am I right in thinking that if you were getting the same output as I'm getting you'd be happy? Or happier, anyway?

So why aren't you seeing the same output?

I've pored over my source code and I think I see part of the problem. I've set the error message within the TestOutcome object to be the Exception object that is thrown by the grader, rather than setting it to just the message text (a string) within the exception. That's why I see a full traceback when I eventually print the message. That was OK when developing but isn't really appropriate in a production system.

The reason I think this is the cause of the problem on your system is that the test outcome, which contains the error message (or, currently, an Exception object) gets serialised for storing in the database then gets unserialised again for rendering. You're getting an exception on the unserialise. It seems that on your system PHP is unable to serialise then unserialise an Exception object. That shouldn't happen! In my understanding, PHP should be able to serialize anything and unserialize it again, albeit with a few provisos relating to user-defined classes, which shouldn't apply in this case. The relevant code in CodeRunner has been around for many years, through several versions of PHP, and I've never seen the problem that you're having. So it makes me wonder what OS and version of PHP your Moodle server is running. Would you be able to find that out from your sysadmins, please?

I've now changed the source code on my development system so that only the message is stored within the test outcome rather than the Exception object. I think (hope) that that would fix the problem on your system, but it'll be a few weeks before I push the next release of CodeRunner. And furthermore since your Moodle system has an old version of CodeRunner without stepinfo, I'm guessing it'll be a lot longer again before the fix appears on your system.

Thanks for reporting the problem, but I'm afraid I don't have a simple fix for you right now. You'll have to work on the assumption that a "Can't unserialise" error means the combinator grader threw an Exception. Hopefully once you'd had a bit more practice with combinator template graders these problems will stop occurring.

Re: Debugging the template

von Carroll Morgan - Mittwoch, 3. November 2021, 16:34

Hi Richard --- Below is the reply from our IT people. Just while I'm here: is it PHP that is unable to serialise (for my purposes) an Exception object, but is able to for you? On my own platform (MacOS; Python3, import json) in fact even json cannot serialise an Exception object. (I tried it out.) So there's further (de)serialisation going on back on the Moodle host, with a different tool (ie PHP)?

So json and PHP both play a role, and it's our UNSW Moodle-installation PHP that is the issue here?
===
And yes :-) I would be much happier with the exception report rather that just the simple "Sorry"! But I do understand the difficulties; and I much appreciate your taking the time to look through the code to see what is going on.

Carroll

===
Hi Carroll,

I don't think the OS should matter much, but I believe Moodle is hosted on servers running CentOS Linux, not sure the version though. The following details are current and accurate:

Moodle - 3.9.9+ (Build: 20210812)
PHP - 7.4.12
Postgres - 11.9

Hope that helps.

Regards,
John Paul

John Paul Posada
Educational Technologist, UNSW Engineering
===

Re: Debugging the template

von Richard Lobb - Freitag, 5. November 2021, 12:47

Thanks Carroll. Certainly the problem isn't with the Moodle version or PHP version.

The serialise/unserialise operations I'm using here are pure PHP. And yes, they work fine for me on multiple versions of Moodle and PHP over many years.

My current guess at why you're getting a different result from me is that Postgres is behaving differently from mysql/mariaDB. I serialise the outcome of a test run so it can be stored in the database. Reading the fine print on the PHP serialize documentation I see that the output is a byte sequence that can include null bytes. The serialised testoutcome is being stored in a text field in the database, not a blob. So, my theory is that the serialisation of an Exception object yields null bytes which cause problems for postgres but not for mysql. I could check this theory out, but I believe I've already fixed the root cause of the problem, which was to try serialising the exception in the first place (a mistake).

If you and your IT staff are feeling adventurous and don't want to wait until the next push you could try patching the source code. The change is in the file <moodlehome>/question/type/coderunner/classes/jobrunner.php. The relevant line numbers in my file are 300/301, but they may be different in your version. The critical lines are at the end of the function do_combinator_grading().

Before, the function ended with:

        } catch (Exception $error) {
            $outcome->set_status(qtype_coderunner_testing_outcome::STATUS_BAD_COMBINATOR, $error);
        }
        return $outcome;
    }

After patching, it is now:

        } catch (Exception $except) {
            $outcome->set_status(qtype_coderunner_testing_outcome::STATUS_BAD_COMBINATOR, $except->getMessage());
        }
        return $outcome;
    }

I'd rather like confirmation that this does in fact fix the problem on your system. At very least it improves the error output to the author on mysql-based systems like ours by suppressing all the extra stack-trace info.

Re: Debugging the template

von Carroll Morgan - Freitag, 5. November 2021, 19:36

Hi Richard: I have heard back from our IT. Thank you for your analysis! Summarising:

UNSW's Moodle is hosted by OpenLMS, and so patching it would
mean fitting in with their schedule and procedures for
Moodle updates. In particular, from September to December
there is a code freeze on Moodle for the yearly upgrade,
where they test and fix any issues that come up in the
version they upgrade to. This year the upgrading is to Moodle
3.11 which should be done on December 16.

An estimate for the earliest change to Moodle that could be
made (ie a change to CodeRunner within it) is next February.
However our IT will see whether in the meantime we can get
into a test environment still running Moodle 3.9. They will
let me know how that proposal goes, and I will get back to
you.

Carroll