CodeRunner Documentation (V3.0.0)

6 Advanced template use

It may not be obvious from the above that the template mechanism allows for almost any sort of question where the answer can be evaluated by a computer. In all the examples given so far, the student's code is executed as part of the test process but in fact there's no need for this to happen. The student's answer can be treated as data by the template code, which can then execute various tests on that data to determine its correctness. The Python pylint question type mentioned earlier is a simple example: the template code first writes the student's code to a file and runs pylint on that file before proceeding with any tests.

The per-test template for such a question type in its simplest form might be:

import subprocess
import os
import sys

def code_ok(prog_to_test):
    """Check prog_to_test with pylint. Return True if OK or False if not.
       Any output from the pylint check will be displayed by CodeRunner
        source = open('', 'w')
        env = os.environ.copy()
        env['HOME'] = os.getcwd()
        cmd = ['pylint', '']
        result = subprocess.check_output(cmd, 
            universal_newlines=True, stderr=subprocess.STDOUT, env=env)
    except Exception as e:
        result = e.output

    if result.strip():
        print("pylint doesn't approve of your program", file=sys.stderr)
        print(result, file=sys.stderr)
        print("Submission rejected", file=sys.stderr)
        return False
        return True

__student_answer__ = """{{ STUDENT_ANSWER | e('py') }}"""
if code_ok(__student_answer__):
    __student_answer__ += '\n' + """{{ TEST.testcode | e('py') }}"""

The Twig syntax {{ STUDENT_ANSWER | e('py') }} results in the student's submission being filtered by a Python escape function that escapes all double quote and backslash characters with an added backslash.

Note that any output written to stderr is interpreted by CodeRunner as a runtime error, which aborts the test sequence, so the student sees the error output only on the first test case.

The full Python3_pylint question type is a bit more complex than the above. It is given in full in the section on template parameters.

Some other more complex examples that we've used include:

  1. A Matlab question in which the template code (also Matlab) breaks down the student's code into functions, checking the length of each to make sure it's not too long, before proceeding with marking.

  2. A Python question where the student's code is actually a compiler for a simple language. The template code runs the student's compiler, passes its output through an assembler that generates a JVM class file, then runs that class with the JVM to check its correctness.

  3. A Python question where the students submission isn't code at all, but is a textual description of a Finite State Automaton for a given transition diagram; the template code evaluates the correctness of the supplied automaton.