running non-python tests

running non-python tests

by Jim Newton -
Number of replies: 4

Can you explain how the model works if I'm using some language other than python and I want to have the student submit one piece of code, but have several tests run? For example I have one unix command line which compiles the code and runs all the tests.

In reply to Jim Newton

Re: running non-python tests

by Richard Lobb -

If you want to run all tests in a single run, you need to be using a combinator template. Such a template receives (via Twig) all the test cases in a Twig variable TESTCASES. 

Here's a relatively simple example, for a question type that runs a mono-C# program via Python. The complete prototype is attached. This question type is just for write-a-program questions, for which there is no extra per-test code, just per-test standard input. Note that this question type requires the Allow multiple stdins checkbox to be checked, otherwise it will still be called once per test, with TESTCASES set to a singleton test.

""" The template for a question type that compiles and runs a student-submitted
    mono C# program using a combinator template that tries to run all
    test cases in a single compile-and-execute Jobe run.
    This assumes a "write a program" type of question, with no test code to
    be incorporated.
"""

import subprocess, sys

# Write the student code to a file prog.cs
student_answer = """{{ STUDENT_ANSWER | e('py') }}"""
with open("prog.cs", "w") as src:
    print(student_answer, file=src)

# Compile
return_code = subprocess.call(['mcs', 'prog.cs'])
if return_code != 0:
    print("** Compilation failed. Testing aborted **", file=sys.stderr)

{% if not IS_PRECHECK %}
if return_code == 0:
    # If compile succeeded and it's not a precheck run all the tests.
{% for TEST in TESTCASES %}
    testcode = """{{ TEST.testcode | e('py') }}\n"""
    standard_input = """{{ TEST.stdin | e('py') }}\n""".rstrip() + '\n'

    try:
        output = subprocess.check_output(
            ["mono", "./prog.exe"],
            input=standard_input,
            universal_newlines=True
        )
        print(output)
    except subprocess.CalledProcessError as e:
        if e.returncode > 0:
            # Ignore non-zero positive return codes
            if e.output:
                print(e.output)
        else:
            # But negative return codes are signals - abort
            if e.output:
                print(e.output, file=sys.stderr)
            if e.returncode < 0:
                print("Task failed with signal", -e.returncode, file=sys.stderr)
            print("** Further testing aborted **", file=sys.stderr)
            sys.exit(1);
{% if not loop.last %}
    print('##');   # Testcase separator
{% endif %}
{% endfor %}
{% endif %}
In reply to Richard Lobb

Re: running non-python tests

by Jim Newton -

Thanks Richard, this template is indeed helpful but there are still some questions in my mind.  The template mixes lots of different things together, and a debutante like me is easily confused.    Can you please comment on what some of the syntax in the template means?


  1. What does "| e('py')" mean?
  2. What does "STUDENT_ANSWER" mean?  It looks like some preprocess is going to substitute here what the student typed in a type-in widget?   Does this work independent of the escaping?  E.g., what if the student code contains """ somewhere? Or what if the student didn't close all parentheses?  Has the system manipulated the student input to add escape characters?  What if that escaping is not compatible with the target language, C# in your case, Scala in my case?
  3. Can I change the template so that the user "attaches/uploads" a file which he has written with his favorite editor?  How can I access the file from the template, and what can I do in the moodle question to allow/require the student to upload a file? test output with test/expected/got headers
  4. The template indicates that it runs ALL tests.  I have a script which compiles and runs all the tests.  What does my script need to do so that it will display separate tests as shown in the image above?
  5. This is perhaps a question about the moodle itself, maybe you can't answer, but my script to run the tests starts by calling "git update" (because from time to time I fix problems in the testing code).  How can the CodeRunner template access my personal reference data or testing code, which has not been installed by the system administrator?
  6. Using the template you provided, which executes my program using ```subprocess.call(...)```, it seems the script communicates with the system by what it prints to stdout and stderr?  How can it communicate the grade the student should get on the assignment.  Is it all or nothing, or can the script somehow communicate partial credit?

Attachment Screenshot 2020-01-20 at 22.21.13.png
In reply to Jim Newton

Re: running non-python tests

by Richard Lobb -

It sounds like you're trying to run before you can walk :)

I'd suggest you start by reading all the documentation. I think most of the questions are answered in there. 

In reply to Richard Lobb

Re: running non-python tests

by Jim Newton -

Could someone consider answering the questions I have posed either by pointing to the relavent section of documentation, or by commenting to the questions which are not answered in the documentation, or where the documentation is unclear?