Question Authors' Forum

If you want to run all tests in a single run, you need to be using a combinator template. Such a template receives (via Twig) all the test cases in a Twig variable TESTCASES. 

Here's a relatively simple example, for a question type that runs a mono-C# program via Python. The complete prototype is attached. This question type is just for write-a-program questions, for which there is no extra per-test code, just per-test standard input. Note that this question type requires the Allow multiple stdins checkbox to be checked, otherwise it will still be called once per test, with TESTCASES set to a singleton test.

""" The template for a question type that compiles and runs a student-submitted
    mono C# program using a combinator template that tries to run all
    test cases in a single compile-and-execute Jobe run.
    This assumes a "write a program" type of question, with no test code to
    be incorporated.
"""

import subprocess, sys

# Write the student code to a file prog.cs
student_answer = """{{ STUDENT_ANSWER | e('py') }}"""
with open("prog.cs", "w") as src:
    print(student_answer, file=src)

# Compile
return_code = subprocess.call(['mcs', 'prog.cs'])
if return_code != 0:
    print("** Compilation failed. Testing aborted **", file=sys.stderr)

{% if not IS_PRECHECK %}
if return_code == 0:
    # If compile succeeded and it's not a precheck run all the tests.
{% for TEST in TESTCASES %}
    testcode = """{{ TEST.testcode | e('py') }}\n"""
    standard_input = """{{ TEST.stdin | e('py') }}\n""".rstrip() + '\n'

    try:
        output = subprocess.check_output(
            ["mono", "./prog.exe"],
            input=standard_input,
            universal_newlines=True
        )
        print(output)
    except subprocess.CalledProcessError as e:
        if e.returncode > 0:
            # Ignore non-zero positive return codes
            if e.output:
                print(e.output)
        else:
            # But negative return codes are signals - abort
            if e.output:
                print(e.output, file=sys.stderr)
            if e.returncode < 0:
                print("Task failed with signal", -e.returncode, file=sys.stderr)
            print("** Further testing aborted **", file=sys.stderr)
            sys.exit(1);
{% if not loop.last %}
    print('##');   # Testcase separator
{% endif %}
{% endfor %}
{% endif %}