CodeRunner: Issues with newlines in feedback

I have encountered an issue with displaying newlines (linebreaks) in the "expected" and "got" feedback shown to students. I am using a brand new installation of Moodle & Coderunner, but I have encountered a similar issue on an older installation. I have the following Java template grader:

public class Main {

	public static void main(String[] args) {
	    String studentAnswer = studentProgram();
	    String correctAnswer = correctProgram();
	    if (studentAnswer.equals(correctAnswer)) {
    		System.out.printf("{\"inputs\":\"No input\",");
    		System.out.printf("\"expected\":\"\",");
    		System.out.printf("\"got\":\"Passed!\",");
    		System.out.printf("\"fraction\":\"1.00\"}");
	    } else {
    		System.out.printf("{\"inputs\":\"No input\",");
    		System.out.printf("\"expected\":\"%s\",", correctAnswer);
    		System.out.printf("\"got\":\"%s\",", studentAnswer);
    		System.out.printf("\"fraction\":\"0.00\"}");
	    }
	}

	public static String studentProgram() {
		{{ STUDENT_ANSWER }}
	}

	public static String correctProgram() {
		{{ QUESTION.answer }}
	}

}

The answer contains

return "Hello, world!";

and it works fine for correct and incorrect student answers. Except, if the student were to input

return "Hello,\nworld!";

(or any text that contains a newline or "\t" or "\r"). Then it reports (for example):

Bad output from grader: {"inputs":"No input","expected":"Hello, world!","got":"Hello,
world!","fraction":"0.00"}. Your program execution may have aborted (e.g. a timeout or memory limit exceeded).

Note: I could filter the return value in Java, but there are cases where it is, in fact, correct and should be displayed. I have looked at the CodeRunner source code, but I cannot fathom what is going wrong. I have seen examples that show multiple lines in the output, but they do not really help me to resolve my issue. Any idea of what I could/should do?

Re: Issues with newlines in feedback

por Richard Lobb - viernes, 8 de junio de 2018, 10:45

The problem here is that JSON strings can't contain newline characters. So you need to convert any newline characters to the 2-character '\n' sequence. There are various other characters that need escaping, too, such as tabs and double-quote characters. Rather than trying to do all this yourself you should use a standard JSON encoder.

Template graders are tricky because you are required to output just a single valid JSON-encoded result, no matter how broken the student code is. This means you have to deal not only with unexpected characters in the student output but also exceptions and endless loops. The only really robust solution is to run the student code in a subprocess, with a timeout that's less than that of the question itself, and collect and analyse the output from that. Even in Python, which has good tools for running subprocesses with time-limits, JSON encoding etc, this isn't easy.

The CodeRunner documentation includes an example of a template grader for a student answer that's in Python. To grade Java answers you also need to include a compile step, such as in the last posting to this thread (which, however, isn't a template grader).

Nowadays I use Python combinator template graders for most languages I teach, because they give you total flexibility and control. However, the templates (which include style checking) run to many hundreds of lines. They're certainly not for the faint of heart. Are you sure you need a template grader?

Richard

Re: Issues with newlines in feedback

por Jaco Geldenhuys - viernes, 8 de junio de 2018, 19:15

Thank you very much for the response. I have tried to tried to "sanitize" the input by using

System.out.printf("\"got\":\"%s\",", studentAnswer.replaceAll("\n", "\\n"));

but this didn't work. Based on your feedback and a whim, I tried

System.out.printf("\"got\":\"%s\",", studentAnswer.replaceAll("\n", "\\\\n");

and this seems to work!

I appreciate your advice re Python, but my full Java template grader already handles timeouts and exceptions, and it works well (enough). Also, I like the control that a template grader provides, since in some cases I need to augment the students programs (by adding constants) or filter out "forbidden" constructs (e.g., for-loops). I have more than 150 small programs (a la CodingBat but even simpler) and I plan to double that. The thought of converting all of those and the (~100 tests in some cases) to a matching grader is a little daunting.

Re: Issues with newlines in feedback

por Richard Lobb - miércoles, 13 de junio de 2018, 21:19

Good to know it's solved. I admit that your solution had me puzzled, until I read in the spec for replaceAll:

Note that backslashes (\) and dollar signs ($) in the replacement string may cause the results to be different than if it were being treated as a literal replacement string; see Matcher.replaceAll.

A bit of a trap, indeed. Using replace rather than replaceAll is probably a simpler solution, as then you just have literal target and replacement strings.

Richard