CodeRunner Documentation (V2.4.2)

9 An advanced grading-template example

As an example of the use of a per-test-case template grader consider the following question:

"What single line of code can be inserted into the underlined blank space in the code below to make the function behave as specified? Your answer should be just the missing line of code, not the whole function. It doesn't matter if you indent your code or not in the answer box. For full marks your answer must be a single line of code. However, half marks will be awarded if you provide more than one line of code but it works correctly.

    def nums_in_range(nums, lo, hi):
        '''Given a non-empty list of numbers nums and two numbers
           lo and hi return True if and only if the minimum of the
           numbers in nums is greater than lo and the maximum of
           the numbers in nums is less than hi.'''
        ____________________________

The grader for this question, which needs to check both the number of lines of code submitted and the correctness, awarding marks and appropriate feedback accordingly, might be the following:

    import re
    __student_answer__ = """{{ STUDENT_ANSWER | e('py') }}"""
    if __student_answer__.strip().startswith('def'):
        raise Exception("You seem to have pasted the whole function " +
                        "definition. READ THE INSTRUCTIONS!")
    if re.search(r'print *\(.*\)', __student_answer__):
        mark = 0
        got = "BAD CODE: your function should not print anything"
    else:
        # Split the code into lines. Indent if necessary.
        __lines__ = __student_answer__.split('\n')
        __lines__ = ['    ' + line +
                '\n' for line in __lines__ if line.strip() != '']
        code = 'def nums_in_range(nums, lo, hi):\n' + ''.join(__lines__)
        exec(code)
        num_lines = len(__lines__)

        result = {{TEST.testcode}}
        if result == {{TEST.expected}}:
            if num_lines  1:
                mark = 0.5
                got = repr(result) + r"\n(but more than 1 line of code)"
            else:
                mark = 1
                got = repr(result)
        else:
            mark = 0
            if num_lines  1:
                got = repr(result) + r"\n(and more than one line of code)"
            else:
                got = repr(result)

    print('{"fraction":' + str(mark) + ',"got":"' + got + '"}')

If the student submits one line of code that behaves correctly their grading table looks normal, e.g. right answer image

If they submit multiple lines of code that behave correctly, their result table might instead be: wrong answer image

In both the above examples the result table has been customised to show the mark column. Result table customisation is covered in the next section.

Note that the "Got" column contains a customised message in addition to their output and the customised message varies according to whether their answer was right or wrong. Note too that the template performs various other checks on their code, such as whether it contains any print statements or whether they have pasted an entire function definition.

Obviously, writing questions using custom graders is much harder than using the normal built-in equality based grader. It is usually possible to ask the question in a different way that avoids the need for a custom grader. In the above example, the student could have been asked to submit their entire function twice, once to a question that evaluated its correctness and a second time to one that evaluated its correctness and its length. No custom grader is then required. That is somewhat clumsy from the student perspective but is much easier for the author.