Question Authors' Forum

Increase Java code answer testing speed

 
Picture of Richard Lobb
Re: Increase Java code answer testing speed
by Richard Lobb - Thursday, 17 November 2016, 4:11 PM
 
Hi Youming (Patrick)


I can't exactly test your question because I'm missing the prototype you're using, but I can see what you're trying to do. I made a similar prototype and got a 3.5 second response time on my laptop with a Jobe server on the same laptop and a 9 second response time when I switched to a remote Jobe server (jobe2.cosc.canterbury.ac.nz). Your 12 second response time corresponds to a slightly slower Jobe server. As you said, the cost is dominated by 6 separate compile+run submissions to the Jobe server.

When teaching with CodeRunner I try to minimise the use of standard input, at least at the start of a course. When setting questions it's worth asking yourself what is the particular skill you're trying to assess? In this question, are you trying to test whether students can use your Helper class to read doubles or whether they can use if statements + printing? If the latter, then a question like the following might be more satisfactory - it should run in under 3 seconds if you use the standard Java method question type.

Preview screenshot of Print biggest question

In addition to running 4 - 6 times faster, it avoids the complexity of having your own prototype and also avoids issues with prompt strings.

If you really do want to run multiple tests involving standard input, you either have to tolerate the slow response (that's Java for you!) or you have to use a much more complex question type that uses a combinator template to manage all the test cases in a single run.

I have written such a prototype to do roughly what I think your java_statement_rp type was intended to do, but with only one compile. I attach it as a proof of concept only (plus your question, slightly modified, and the question shown above). The prototype uses a Python3 script to compile the test program and then, if that succeeds, to run the compiled code with each of the different standard inputs. This version of your question runs in around 4 seconds when using a remote Jobe server. Note that with this prototype you have to enter the input data into the Extra Test Data field of each test case, not the standard input field. If you put it in the standard input field, CodeRunner will not even try to use the prototype template. [It doesn't usually make sense to try to roll all the test cases into a single file when there are multiple different standard input streams required. But this prototype shows that it can actually be done.]

Hope that helps.

Richard