Testing random (Java) questions

Testing random (Java) questions

by Jan Derriks -
Number of replies: 1

Generating (java) programming questions with random numbers is nice but testing the answer is a bit hard with TWIG.

An example with three random template params:

{ "divisor": "{{ random([3,4,5,6,7,8,9]) }}", 
   "min": "{{ 10+random(10) }}",
   "max": "{{ 100+random(100) }}"
}

And the question text:

Write a program to calculate the sum of all integer numbers that are:
  • larger or equal to {{min}} AND
  • smaller or equal to  {{max}} AND 
  • dividable by {{divisor}} 

I managed to have a testcase that calculates the answer with a TWIG FOR loop: in the expected output window:

{% set som = 0 %}
{% for i in min..max%}
{% if (i % divisor) == 0 %}{% set som = som + i %}
{%endif%}
{%endfor%}
{{som}}

But my questions are:

a. should I use a python template instead of twig? Would that be better, easier, faster?

b. why must I use 10+random(10) when the twig documentation says random(10,20) should work?

In reply to Jan Derriks

Re: Testing random (Java) questions

by Richard Lobb -

To answer your second question first: the version of Twig in CodeRunner is currently 1.42.3 (2019-08-24). I've held back on updating to a newer version because Twig 2.x needs at least PHP 7.1.3 to run. Many Moodle sites are running on older versions of PHP. The second parameter (max) to the random function was added in Twig 2.7.

Indeed, testing in Twig is a bit of a pain. But with practice you can usually find workarounds. In this particular example you could precompute a table of (min, max, divisor, result) tuples and select one of those at random. See https://github.com/trampgeek/moodle-qtype_coderunner#miscellaneous-tips.

If precomputed parameter sets aren't possible, you might be able to write a solution to the problem in either the template language or the student's answer language (if different). A template grader can then run your answer to get the 'Expected' value of the test and the student's code to get the 'Got' value. But first you have to learn how to write a template grader.

For the maximum flexibility you can use a combinator template grader and run any code you like to construct a suitable result table. Our in-house Python3 question type has a "useanswerfortests" template parameter that, if true, runs the sample answer to fill in the expected field of any test case that doesn't already have a value filled in.

A problem with using template graders in conjunction with randomisation is that the template code isn't run until the student submits their answer. So the For example table can't contain values computed by the template. Even then, there can be workarounds, like using a Twig-computed result for an easy case in the For Example table, leaving all the harder cases for the template grader to compute. Or, you can dispense with the For Example table altogether and describe the problem in more general terms. For example, in your case:

If you have a range of numbers from some minimum min to some maximum max inclusive, and some divisor d, you could compute the sum of all the numbers in that range that are divisible by d using a loop. For example, the sum of the numbers from 3 to 11 divisible by 2 would be 4 + 6+ 8 + 10 = 28. You are to write a program to compute the sum of all the numbers from {{min}} to {{max}} that are exactly divisible by {{divisor}}.

Such an approach is probably better in your example question, anyway, to prevent students submitting an answer of the form (in Python):

print('28')
But in order to grade the student's answer, you would still need to either write a solution in Twig (as you did) or use a template grader that computed the expected result at run time.

Lastly, to answer the question "Should I use a Python template?" my answer would always be yes. But I'm not exactly impartial when it comes to Java versus Python :-)