How to make running time and memory usage two more requirements for the program to be correct in CodeRunner?

por Zhang Jeremy -

Hello there, I have met a problem when I am adding additional features to CodeRunner. I have noticed that it is possible to set the limit of running time and memory usage in the advanced settings of CodeRunner quiz, but I am just so confused about how to change the codes in order to make the program to be shown correct only when all the test cases have passed and it satisfies the requirements of running time and memory usage.

To be specific, the teacher can set the largest value for the running time and memory usage and then the program that the student has submitted will only be shown in green correct when all the test cases are right while the running time and memory usage all has not exceeded the limit number. If the program has exceeded the limt of running time or memory usage, even if all the test cases have passed, the feedback should still be red incorrect.

I just wonder how I can change the codes in some files like "render.php" in order to realize this extensions to CodeRunner? I guess some files in CodeRunner itself shoud be changed and probably some files in the Jobe server should also be editted.  I really need some help. Thank you so so much.

How to install CodeRunner locally in Moodle on win10?

por Zhang Jeremy -

Hello everyone, I have met a problem when I am installing coderunner on moodle. After I choose the "install plugin" in site administration, I upload the zip file that I download from the official coderunner plugin website of Moodle. However, it tells me that I have to choose a plugin type and I don't know what to do. Can anyone help me with that? Thanks a lot!

How to change the output result of CodeRunner?

por Zhang Jeremy -

Hello there, I am a complete new beginner of CodeRunner on Moodle. I have downloaded the folder of CodeRunner from Github, but can't really understand how it works because there are so many folders and different files in it.

Here i have a question that what if I want to change the output result of CodeRunner from "Passed all tests" to something like "Passed 5/5 tests" when the code that the student submmit has passed all the 5 tests cases. And what if I want to change the result to "Passed 4/5 tests" when the code only passes 4 of the 5 tests cases.

I just have no idea of how and where to change the code of CodeRunner, can you give some help on how to do it or where or how I can try to understand the code?

Check Function Exist

por Manoj Singh -

I want the student's program to be graded only if it has been created by the function.

Is it possible that we can check the program whether the function is available in the program or not.

Want to use Code runner

por vishali salgare -

Hi richard,

i have created my site on I want to design programming Based Assignment. So is it possible to use Code runner while doing so?

CodeRunner examples in a moodle course for download

por Aeneas Schwaiger -
Hello Richard,
I am an IT teacher from Germany and use your CodeRunner. I am currently trying to make your CodeRunner better known in Hessen (state in Germany).

I would like to thank you and all developers for your work on the CodeRunner. Thank you!

I created a moodle course with many CodeRunner examples (in Java, C ++ and SQL). The course is royalty-free and can be copied.

Course preview as video:

Course download:

Unfortunately the course and the videos are in German.
My English is very bad :(

Greetings from Germany
Aeneas Schwaiger

ACSL Problems

por Sharad Goyal -

I am also trying to setup ACSL problems which has six lines of input but only last 5 lines of has output. I am not sure how to setup this as well.

Attached problem

ACSLVille is a housing complex built around a circular road.  There are 4 gates to the complex, labeled A, B, C, and D.  Cars enter at one of the gates and must travel in a counterclockwise direction.


Houses are identified with a letter followed by a number, for example B124. The letter indicates the nearest gate to get to the house; for example, all houses starting with a B are between gates B and C. The numbers on the houses increase in a counterclockwise direction between gates. The even numbered houses are on the inside (left side of the car as driven counterclockwise), and odd numbered houses are on the other side.


The diagram at the right shows ACSLVille with its 5 original houses: A435, B84, B97, B124, and D4301.


Each day, the mail truck enters ACSLVille at one of the gates. The mail is delivered by driving counterclockwise (that’s the law in ACSLVille!), delivering to all the odd numbered houses followed by all the even numbered houses. For example, if the mail truck entered at Gate B, the delivery order would be B97, D4301, A435, B84, and finally, B124. If the mail truck entered at Gate A, the order would be A435, B97, D4301, B84, and finally, B124.


INPUT:  The first line of input contains N, the number of original houses in ACSLVille and their addresses.  The next lines contain the number of new houses built in ACSLVille that must be added to the delivery route, followed by their addresses.  This is followed by the letter of the gate where the mail truck enters, and a number, call it K.


OUTPUT:  For input lines 2 - 6, determine the order that the mail is delivered after the new house(s) are added. and print out the Kth house on the delivery route.


PROMISES: We promise that there won’t be more than 100 houses in ACSLVille, that the highest numbered house in ACSLVille is 9,999, and that all houses will have unique addresses.



  1. 5, A435, B84, B97, B124, D4301
  2. 1, C321, B, 4
  3. 2, C2, C84, A, 5
  4. 0, B, 7
  5. 1, D34, D, 9
  6. 3, A241, B510, D43, C, 10


  1. A435
  2. B84
  3. C2
  4. C84
  5. B84


USACO Problems

por Sharad Goyal -

I am trying to setup USACO problems in my Moodle Open source code runner. I do not know how to specify the file input which is given by one of the students in the solution. 

import java.util.*;

public class billboard {

	public static void main(String[] args) throws Exception {

		// Read the grid.
		Scanner stdin = new Scanner(new File(""));

		// Mark both seen billboards.
		boolean[][] see = new boolean[2001][2001];
		for (int i=0; i<2; i++) {

			int x1 = stdin.nextInt()+1000;
			int y1 = stdin.nextInt()+1000;
			int x2 = stdin.nextInt()+1000;
			int y2 = stdin.nextInt()+1000;

			for (int x=x1; x<x2; x++)
				for (int y=y1; y<y2; y++)
					see[x][y] = true;

		// Mark squares that can't be seen.
		int x1 = stdin.nextInt()+1000;
		int y1 = stdin.nextInt()+1000;
		int x2 = stdin.nextInt()+1000;
		int y2 = stdin.nextInt()+1000;
		for (int x=x1; x<x2; x++)
			for (int y=y1; y<y2; y++)
				see[x][y] = false;

		// Count which ones can be seen now.
		int res = 0;
		for (int i=0; i<=2000; i++)
			for (int j=0; j<2000; j++)
				if (see[i][j])

		// Ta da!
		PrintWriter out = new PrintWriter(new FileWriter("billboard.out"));
Can someone help me please to set up one general template in my code runner setup?

Displaying images in the CodeRunner feedback

por Richard Lobb -

Sometimes we want to ask students to write programs that display images. For example "Write a program that uses matplotlib to display a graph showing two cycles of a sine wave". Or "Write a function that uses the GraphViz "dot" program to plot the Huffman coding tree for a given set of character frequencies". What a shame we can't ask such questions in CodeRunner.

Well, actually, we can! For example:

Example question showing matplotlib image in feedback

In this post I'll show how you can author questions that capture and display images generated by student code. The two key ideas are:

  1. CodeRunner's combinator template graders give you complete control of the feedback, including the ability to include arbitrary html.
  2. Data URIs allow you to embed the entire contents of an image within the URL (which of course is itself within the HTML).

Combinator Template Graders

If you click the Customise checkbox in the question authoring form, a Customisation section appears as shown in the image below. This lets you edit the template (more on this later) and to set the way in which the question is graded, via a Grading drop down menu. If you select Template grader instead of the default Exact match, as shown in the image below, the role of the template changes. It must now grade the student's submission, rather than just run it. 

Question author form showing Customisation panel

There are two different types of template grader: per-test template graders and combinator template graders. The former, which is what you must write if the Is combinator checkbox is not checked, has to deal with only a single test case. It essentially defines one row of the result table for each run. The latter (combinator template graders), which are what we're concerned with here, are what you must write if Is combinator is checked. Combinator template graders are considerably more powerful but also more difficult to write. They are responsible for grading all test cases and defining the entire result table (if there is to be one) plus two optional additional HTML feedback sections, one before the result table and one after. These are called the prologuehtml and the epiloguehtml respectively. The image(s) we wish to return from the run can be placed in one or other (or both) of those.

The output from a combinator template grader must be a JSON string. The string is documented in the on-line help for the Grading element of the form:

If the template is a combinator, the JSON string output by the template grader should again contain a 'fraction' field, this time for the total mark, and may contain zero or more of 'prologuehtml', 'testresults', 'epiloguehtml', 'columnformats' and 'showdifferences'. The 'prologuehtml' and 'epiloguehtml' fields are html that is displayed respectively before and after the (optional) result table. The 'testresults' field, if given, is a list of lists used to display some sort of result table.

You can read further details in the on-line help itself; for now I wish to focus just on the epiloguehtml which is an html string, normally appearing after the result table. For now we'll assume that the only feedback we wish to give the student is the image itself. We'll further assume that the "grader" will mark any image correct. The output from the template grader then needs to have only the fraction and the epiloguehtml fields. For example, if the latter is simply a cheery message, the output from the template grader could be something of the form

    fraction: 1.0,
    epiloguehtml: "<h2>Well done!</h2>"

To see such a combinator template grader in action, create a Python3 question, customise it, select a Template grader, leave Is combinator checked and set the template to the code

import json
print(json.dumps({"fraction": 1.0, "epiloguehtml": "<h2>Well done!</h2>"}))

If you test drive this question, submitting any answer at all (it doesn't need to be code), you should see something like the following:

Trivial combinator template grader output

Now we'll look into how to include an image in that feedback.

Data URIs

Data URIs, also known as Data URLs, are URIs where the data that the URI "defines" is encoded within the URI itself rather than being pointed to by a traditional URL. We wish to encode image data into the URI and since the image data is (usually) binary we need to first encode it in base64. If we confine our attention to either png or jpeg images, the format of our data URIs will be either:

  1. data:image/png;base64,<filedatabase64> or
  2. data:image/jpeg;base64,<filedatabase64>

where <filedatabase64> is the contents of the file in base64.

Knowing that, a Python function to build a Data URI from a given png or jpeg image file is the following:

import base64
def make_data_uri(filename):
    """Given a png or jpeg image filename (which must end in .png or .jpg/.jpeg 
       resp.) return a data URI as a UTF-8 string.
    with open(filename, 'br') as fin:
        contents =
    contents_b64 = base64.b64encode(contents).decode('utf8')
    if filename.endswith('.png'):
        return "data:image/png;base64,{}".format(contents_b64)
    elif filename.endswith('.jpeg') or filename.endswith('.jpg'):
        return "data:image/jpeg;base64,{}".format(contents_b64)
        raise Exception("Unknown file type passed to make_data_uri")

Running the student code safely

The biggest complication with template graders in general, and particularly with combinator template graders, is that they must output the specified JSON record defining the feedback. They must do this even if the student code is faulty, or throws an exception or segment faults or times out, etc. This means the code has to be run separately from the template itself in some sort of secondary sandbox with a watchdog timer.

In Python, the subprocess module is usually the best tool for the job, with a piece of code like the following:

import subprocess
output = ''
failed = False
    outcome =
        ['python3', '-c', student_code],
        timeout = 2, # 2 second timeout MUST BE LESS THAN JOBE TIMEOUT
except subprocess.CalledProcessError as e:
    outcome = e
    output = "Task failed with return code = {}\n".format(outcome.returncode)
    failed = True
except subprocess.TimeoutExpired as e:
    outcome = e
    output = "Task timed out\n"
    failed = True
output += outcome.stdout
if outcome.stderr:
    output += "*** Error output ***\n"

Note that the watchdog timer (in this case the timeout parameter to must be less than the Jobe timeout for the task, which for Python defaults to 3 seconds. Otherwise an endless loop in the student code will result in the entire run being aborted by Jobe and the required JSON grader output won't happen.

Generating the JSON output

Suppose we are expecting the student code to generate an image file called outputimage.png. Then, having run the student code in a subprocess as above, the template grader can finish by picking up that image, encoding it into a data URI, and inserting that data URI into the epiloguehtml field of the final JSON output. For example:

html = ''
if output:
    html += "<pre>{}</pre>".format(output)
if not failed and os.path.isfile('outputimage.png'):
    data_uri = make_data_uri('outputimage.png')
    html += """<img class="data-uri-example" title="Output image" src="{}" alt="outputimage.png">
# Lastly print the JSON-encoded result required of a combinator grader
print(json.dumps({'epiloguehtml': html,
                  'fraction': 0.0 if failed else 1.0

Note that any textual output from the program precedes the image in the displayed feedback.

Picking up matplotlib images

The example at the start of this post shows a Python matplotlib question. Normally the student code for such a question will not explicitly generate a file; rather, the image will be displayed on screen when the code is run. To capture the matplotlib output into an image file when running on Jobe, we must prefix the student code with a request to matplotlib to use a non-interactive backend such as Agg, which writes figures to a png file. We also have to add suffix code to call plt.savefig.

The attached file matplotlibimagedemo.xml is an export of the question shown at the start of this posting. Also attached in the file is the template code for that question.

Grading the output

The astute reader will have noticed that while this question produces nice-looking output, the student gets given full marks for any code at all that runs correctly, regardless of whether it produces an appropriate output image or not. So ... it's useless, right?

Well, maybe not totally useless. That particular question does let students experiment with matplotlib code and see the output images in, say, a tutorial context. But it's certainly useless from an assessment point of view.

However, all is not lost. If what you're trying to grade is the matplotlib image properties, you can pick up the current axes (after the student code is run) and check various attributes such the title, axis labels, data plotted, etc against the expected ones and issue part marks accordingly.

But ... this blog has gone on long enough, so I'll finish with an example Python function that prints out the properties of the current matplotlib axes object (assumed to be a bar chart). You can use that as a starting point for your own questions.

def print_plot_info():
    """Output key attributes of current plot, as defined by plt.gca()"""
        current_axes = plt.gca()
        subplot = current_axes.axes   # The actual plot, assuming just one subplot
        print("Plot title: '{}'".format(current_axes.title.get_text()))
        print("X-axis label: '{}'".format(subplot.get_xlabel()))
        print("Y-axis label: '{}'".format(subplot.get_ylabel()))
        print("Number of bars:", len(subplot.patches))
        x_tick_labels = [label.get_text() for label in subplot.get_xticklabels()]
        if all(label.strip() == '' for label in x_tick_labels):
            x_tick_labels = [str(pos) for pos in subplot.get_xticks()]
        print("\nX-axis ticks:\n" + ', '.join(x_tick_labels))
        y_tick_labels = [label.get_text() for label in subplot.get_yticklabels()]
        if all(label.strip() == '' for label in y_tick_labels):
            y_tick_labels = [str(pos) for pos in subplot.get_yticks()]
        print("\nY-axis ticks:\n" + ', '.join(y_tick_labels))
        bar_height_strings = [format(bar.get_bbox().ymax, '.0f') for bar in subplot.patches]
        print("\nBar heights:\n" + ', '.join(bar_height_strings))
    except Exception as exception:
        print("Failed to get plot info:", str(exception))

Happy authoring!

-- Richard