CodeRunner Documentation (V3.2.1)

5.2 Combinator templates

The template for a question is by default defined by the code runner question type, which itself is defined by a special "prototype" question, to be explained later. You can inspect the template of any question by clicking the customise box in the question authoring form. You'll also find a checkbox labelled Is combinator. If this checkbox is checked the template is a combinator template. Such templates take the STUDENT_ANSWER template variable as shown above, but rather than taking just a single TEST variable, they take a TESTCASES variable, which is is a list of all the individual TEST objects.

The actual template used by the built-in C function question type is not actually a per-test template as suggested above, but is the following combinator template.

#include <stdio.h>
#include <stdlib.h>
#include <ctype.h>
#include <string.h>
#include <stdbool.h>
#include <math.h>
#define SEPARATOR "#<ab@17943918#@>#"

{{ STUDENT_ANSWER }}

int main() {
{% for TEST in TESTCASES %}
   {
    {{ TEST.testcode }};
   }
    {% if not loop.last %}printf("%s\n", SEPARATOR);{% endif %}
{% endfor %}
    return 0;
}

The Twig template language control structures are wrapped in {% and %}. If a C-function question had two three test cases, the above template might expand to something like the following:

#include <stdio.h>
#include <stdlib.h>
#include <ctype.h>
#include <string.h>
#include <stdbool.h>
#include <math.h>
#define SEPARATOR "#<ab@17943918#@>#"

int sqr(int n) {
    return n * n;
}

int main() {
    printf("%d\n", sqr(-9));
    printf("%s\n", SEPARATOR);
    printf("%d\n", sqr(11));
    printf("%s\n", SEPARATOR);
    printf("%d\n", sqr(-13));
    return 0;
}

The output from the execution is then the outputs from the three tests separated by a special separator string, which can be customised for each question if desired. On receiving the output back from the sandbox, CodeRunner then splits the output using the separator into three separate test outputs, exactly as if a per-test template had been used on each test case separately.

The use of a combinator template is problematic with questions that require standard input; if each test has its own standard input, and all tests are combined into a single program, what is the standard input for that program? The easiest resolution to this problem is simply to fall back to running each test separately. That is achieved by using the combinator template but feeding it a singleton list of testcases each time, i.e. the list [test[0]] on the first run, [test[1]] on the second and so on. The combinator template is then functioning just like a per-test template.

However, advanced combinator templates can actually manage the multiple runs themselves, e.g. using Python Subprocesses. To enable this, there is a checkbox "Allow multiple stdins" which, if checked, reverts to the usual combinator mode of passing all testcases to the combinator template in a single run.

The use of a combinator also becomes problematic if the student's code causes a premature abort due to a run error, such as a segmentation fault or a CPU time limit exceeded. In such cases, CodeRunner reruns the tests, using the combinator template in a per-test mode, as described above.