CodeRunner: Twig parameter not working properly

Hi,
I managed to create a nice set of template to work in Javascript with Coderunner, using prompt and a fixed random sequence. I shared my prototypes in another thread.

I am now trying to make the template seed the random number differently per test.

This is the starting code:

const randomValues = [ 0.15, 0.23, 0.52, 0.34, 0.75, 0.38, 0.62, 0.01, 0.29, 0.40, 0.92, 0.54, 0.42, 0.09, 0.10, 0.07, 0.88, 0.46, 0.01, 0.21,  0.17, 0.64, 0.06, 0.57, 0.25, 0.29, 0.81, 0.57, 0.02, 0.62,0.03, 0.38, 0.64, 0.75, 0.22, 0.49, 0.89, 0.92, 0.13, 0.72 ];
let randomIndex = 0;
Math.random = () => randomValues[randomIndex++ % randomValues.length];

Doing it this way, each test starts with the first element of the array (0.15). I would like for the template to give different starting index for every test case. In my mind a great solution is to use the test number. So i did:

const randomValues = [ 0.15, 0.23, 0.52, 0.34, 0.75, 0.38, 0.62, 0.01, 0.29, 0.40, 0.92, 0.54, 0.42, 0.09, 0.10, 0.07, 0.88, 0.46, 0.01, 0.21,  0.17, 0.64, 0.06, 0.57, 0.25, 0.29, 0.81, 0.57, 0.02, 0.62,0.03, 0.38, 0.64, 0.75, 0.22, 0.49, 0.89, 0.92, 0.13, 0.72 ];
let randomIndex = {{ TEST.rownum }};
Math.random = () => randomValues[randomIndex++ % randomValues.length];

This works, but only while I'm editing the question. The code correctly evaluates to the test number.
If I try to test it in preview mode, it stops working because {{ TEST.rownum }} stop existing and evaluates to nothing!

In TWIG that line evaluates to let randomIndex = ;
I tried some debugging, I printed the TEST object as a JSON and, just as I thought, the rownum field does not exist!

What am I doing wrong?

This is my complete template for reference. Starting from nodejs, with is_combinator disabled:

const inputs = `{{ TEST.stdin }}`.split('\n')
let inputIndex = 0;
global.prompt = () => inputs[inputIndex++]

const randomValues = [ 0.15, 0.23, 0.52, 0.34, 0.75, 0.38, 0.62, 0.01, 0.29, 0.40, 0.92, 0.54, 0.42, 0.09, 0.10, 0.07, 0.88, 0.46, 0.01, 0.21,  0.17, 0.64, 0.06, 0.57, 0.25, 0.29, 0.81, 0.57, 0.02, 0.62,0.03, 0.38, 0.64, 0.75, 0.22, 0.49, 0.89, 0.92, 0.13, 0.72 ];
let randomIndex = 0;
Math.random = () => randomValues[randomIndex++ % randomValues.length];  

{{ STUDENT_ANSWER }}
var SEPARATOR = "#<ab@17943918#@>#";
{{ TEST.testcode }}


https://coderunner.org.nz/pluginfile.php/1564/mod_forum/post/3073/PROTOTYPE_JavascriptWithPromptAndRandom.xml

Re: Twig parameter not working properly

by Richard Lobb - Saturday, 29 March 2025, 10:54 AM

Certainly you're not doing anything wrong. This is my bad - the rowNum attribute of the TEST variable should never have been documented. I should have read my own source code which says:

    // [when validating] we need to add an extra
    // rownum attribute to the testcase to allow failed test case results
    // to be copied back to the form with a mouse click.
    ...
    if ($validation) {
        $testcase->rownum = $i;  // The row number in the edit form - relevant only when validating.
    }

Sorry to have misled you.

In fact rownum isn't a test sequence number at all, because when the question is saved, the tests are reordered according to the value of the ordering attribute. And, as an aside, there's a bug relating to rownum when you're using combinator template graders and try to copy Got back into Expected after a test failure with a single mouse click. It turns out to be hard to fix, but it's on my TODO list.

Neither rownum nor ordering are stored in the database as attributes of a test, so shouldn't have been documented. I have deleted mention of them in my development README.md.

Can you instead use TEST.extra, setting it to a (different) starting index for each test? That has the advantage over a row-numbering approach that you can safely reorder tests without changing the expected outputs. OTOH it is a bit of extra overhead when writing the question.

Re: Twig parameter not working properly

by Matteo Candita - Saturday, 29 March 2025, 7:23 PM

Oh, I see! Thank you very much.
I already thought of (and tried) using TEST.extra, but as you said it's extra work when writing a question. I prefer to have it change automatically, if I manage to.

I ended up using "is combinator" in order to get all the TESTCASES and using a for cycle to keep track of the test number myself. I threw in there the option to further customize the randomindex using TEST.extra.

Can I ask you for a sanity check? The template seems to be working as intended, but I am no expert and maybe there is something I'm not taking into consideration. For example, is using eval() a bad idea?

//Get the student answer as text
let studentanswer = "{{ STUDENT_ANSWER | e('js') }}" ;

//Get the tests as a JSON
let tests = {{ TESTCASES | json_encode }};
let numTest=Object.keys(tests).length;
let currentTest=0;

//run the tests one by one
for(let test of tests){
    //for randomIndex we use either TEST.extra (if there is one) or the current test number
    randomIndex = test['extra']!=="" ? test['extra'] : currentTest;
    
    //combine studentanswer and testcode and evaluate them.
    let testcode = test['testcode'];
    let code=studentanswer + testcode;
    eval(code);
    
    currentTest++;
    if(currentTest<numTest){//we don't need the SEPARATOR on the last test
        console.log(SEPARATOR);
    }
}

Thank you so so much for all your work! This plugin is amazing! Using coderunner has dramatically improved our students' learning process and their understanding of what they are doing.

Re: Twig parameter not working properly

by Richard Lobb - Sunday, 30 March 2025, 7:51 PM

Firstly, I should have said many thanks for sharing your JavaScript questions types in your earlier thread. I don't teach JavaScript myself, but lots of people do and I'm sure your prototypes will be useful. Also, I'm pleased to hear you're finding CodeRunner useful yourself.

Now to this thread ...

Using a Combinator template is a great idea. It's certainly much more efficient and it does potentially give you a way to auto-generate the test numbers. But unfortunately there are a couple of issues with your approach as it stands.

The first problem is rather subtle and relates to how CodeRunner handles runtime errors, by which I mean any sort of runtime exception or stderr output. When using a combinator template, CodeRunner needs to present the student with all test results up to the point of the error. Otherwise the student has no way of knowing what went wrong. Imagine, for example, that the student code goes into an endless loop on the fourth test case. The Jobe task will timeout and it's quite likely (e.g. due to buffering) that no useful test output at all will get back to Moodle. To deal with this, if a combinator run fails with any sort of runtime error (including stderr output), CodeRunner switches to running all the tests again, one test at a time. It does this using the combinator template, but the TESTCASES variable is now a (different) singleton each time through. This will break your model, because on the re-run each test will be numbered 0 and will get the same random seed. The only way to avoid that problem is to write a template which absolutely guarantees never to crash or produce stderr output.

The second problem is with the use of eval. I'm no security guru but there are bound to be ways a student could exploit the use of eval. For example, they could output the tests variable to the console to get the complete set of tests, including any you were trying to hide. Other more devious exploits are likely possible too.

I think it's possible, though difficult, to deal with both these problems by using the child_process module in nodejs. This would allow you to run the student code in its own environment. It could capture all stdout and stderr output from the child and write all output to stdout. It could have a timeout, set less than the question's Jobe timeout, to ensure the parent regained control if the student code loops. BUT ... this is quite a lot of work. It's over to you whether you think it's worth it. But - for what it's worth - nearly all our in-house question types work that way. And additionally they use combinator template graders for the ultimate in flexibility :-)

Re: Twig parameter not working properly

by Matteo Candita - Tuesday, 1 April 2025, 11:41 AM

Wow, that's a lot!

I think I'll get back to it when (if) I will have more spare time. All things considered, for now I'll stick with the TEST.extra approach. I'll definitely look more into how the combinator works. There are still so many options left to explore! :D
Combinator template graders are on my list.

I'll leave it like this. I'll be sure to post on the forum when I'm fully satisfied with my prototype!
Again, many thanks!

{% if TEST.extra is not empty %}
let randomIndex = {{ TEST.extra }};
{% else %}
let randomIndex=0;
{% endif %}