Increase Java code answer testing speed

Increase Java code answer testing speed

by Patrick Wang -
Number of replies: 7

Dear all,

I recently wanted to increase the Java code answer testing speed on my Moodle server, so that the students could take less time to know whether their code is correct or not. So i upgraded my server from 1GB Memory/1 CPU to 2GB Memory/2 CPUs, but when I tested the code question, it seems that the time to get the feedback is more or less the same as before. For example, for a java question with 6 test cases, it took around 12 secs to finish the testing and get the feedback.

May I know any way I can increase the speed? like tuning some of the parameters for Moodle, CodeRunner or Jobe server?

Btw, the Jobe server is running on the same server as my Moodle server.

Thanks.


Patrick

In reply to Patrick Wang

Re: Increase Java code answer testing speed

by Richard Lobb -

Hi Patrick

12 seconds is certainly getting unpleasantly long.

I suspect what's happening here is that you're getting a separate compile-and-execute cycle for each test case and that you have a slowish server. However, it would be best to confirm that's the case before trying to fix it. I suggest you log on to your server and try the following 3 commands just to get a measure of performance (assuming you have some trivial Hello world program in a file Hello.java):

time javac Hello.java
time java Hello
time (javac Hello.java ; java Hello)

The "Real time" output from those commands is the one to note. My laptop development system takes about 0.5 seconds for a compile-and-execute, but our departmental Jobe server takes 1.1 secs. Hence my laptop gives about a 3 second response time for a Write a Program Java question with 6 test cases, while our departmental server takes around 7 seconds.

If it turns out that compile time is what's killing you here, we can discuss how to avoid recompiling the program for each test case.

Richard

In reply to Richard Lobb

Re: Increase Java code answer testing speed

by Patrick Wang -

Hi Richard,

I have done the tests you suggested and attached the result below. Compile time is indeed the problem. 

Thanks.


Regards,

Patrick


In reply to Patrick Wang

Re: Increase Java code answer testing speed

by Richard Lobb -

OK, those times are a bit slower than on our Jobe server, but not hugely. With those numbers, and if you're using a question type that doesn't make use of a combinator template, I'd expect a 6-test-case question to take around 7 - 8 seconds. This is long enough to be a bit tedious but isn't too bad, I think. However, you said you had a 12 second delay so ... are the tests themselves taking significant compute time too?

Perhaps you could tell us more about the question you're using: the question itself, the template(s) and the test cases? Is it fundamentally necessary to compile a different program for each test case or are you just running the same program over and over with different input?

Without knowing the specifics, the only suggestions I can make are:

  1. Reduce the number of test cases by packaging multiple tests into one.
  2. Use a question type that uses a combinator template, such as the supplied java_method type. Combinator templates package all the test cases into a single compile-and-run, so you'd then get a response time of just a couple of seconds. Note that if you edit the template of such a question and wish to get the speed-up from the combinator, you'll have to re-enable it (assuming you clicked OK when the dialog popped up warning you) and add your custom code to the combinator template as well as to the per-test template.
  3. As a variant on the above, you might be able to use a Python question (with a combinator template) as a script that compiles the Java program then runs it one test at a time. However, this requires advanced CodeRunner skills and is not recommended for newcomers. It's also important to know that the combinator template won't be used if the test cases have any standard input, so any such input would have to be placed in one of the other test case fields (e.g. the Extra Template Data field).
  4. Get a faster server :)


Richard

In reply to Richard Lobb

Re: Increase Java code answer testing speed

by Patrick Wang -

I measured the 12 sec by starting the timer when i clicked the "Check" button and stopped when the paged is refreshed and updated with the result. The question itself is pretty straightforwards, just test on simple if/else statements. So I think the HTTP request (sending, processing and receiving) itself takes around 3-4 sec, which I think is reasonable. 

Thanks for your advice, I will take your first one, reducing the test cases. In fact, most of the questions only need 1 or two test cases. I only need to use more test cases for questions using quite a few if/else branches with user input value.

In reply to Patrick Wang

Re: Increase Java code answer testing speed

by Richard Lobb -

The times do still seem excessive, Patrick. If you want to export your question as a Moodle XML and attach it to a posting, we can see if we can find any better solutions.

A colleague of mine (Kourosh Neshatian) uses the language Clojure, which is written in Java. Launching Clojure (which is a .jar file) is even slower than compiling and running Java, even though there's no Java compile stage. His solution to the performance problem was to use a combinator template, written in Python. Here's his template, in case it's of use to you or anyone else.

import subprocess

student_answer = """{{ ESCAPED_STUDENT_ANSWER }}"""
test_code = '(load-file "student_answer.clj")\n'
splitter = "##\n"
num_test_cases = 0

{% for TEST in TESTCASES %}
test_code += "\n" + """{{ TEST.testcode }}"""
{% if not loop.last %}
test_code += '\n(print "{}")'.format(splitter)
{% endif %}
num_test_cases += 1
{% endfor %}

with open('student_answer.clj', 'w') as file:
    file.write(student_answer)

with open('test_code.clj', 'w') as file:
    file.write(test_code)

command = "java -mx256m -cp /usr/share/java/clojure-1.6.0.jar:. clojure.main test_code.clj"

try:
    result = subprocess.check_output(command.split(), stderr=subprocess.STDOUT, universal_newlines=True)

except subprocess.CalledProcessError as cpe:
    result = cpe.output

outputs = result.split(splitter)
if len(outputs) < num_test_cases :
    outputs += ["\nNot evaluated due to an earlier error!"] * (num_test_cases - len(outputs))
    result = splitter.join(outputs)

print(result)@17943918#@>

Good luck!

Richard

In reply to Richard Lobb

Re: Increase Java code answer testing speed

by Patrick Wang -

Hi Richards,

I attached the Moodle XML file for your analysis.

Thank you very much.


Regards,

Youming

Attachment Image 1.png
In reply to Patrick Wang

Re: Increase Java code answer testing speed

by Richard Lobb -
Hi Youming (Patrick)


I can't exactly test your question because I'm missing the prototype you're using, but I can see what you're trying to do. I made a similar prototype and got a 3.5 second response time on my laptop with a Jobe server on the same laptop and a 9 second response time when I switched to a remote Jobe server (jobe2.cosc.canterbury.ac.nz). Your 12 second response time corresponds to a slightly slower Jobe server. As you said, the cost is dominated by 6 separate compile+run submissions to the Jobe server.

When teaching with CodeRunner I try to minimise the use of standard input, at least at the start of a course. When setting questions it's worth asking yourself what is the particular skill you're trying to assess? In this question, are you trying to test whether students can use your Helper class to read doubles or whether they can use if statements + printing? If the latter, then a question like the following might be more satisfactory - it should run in under 3 seconds if you use the standard Java method question type.

Preview screenshot of Print biggest question

In addition to running 4 - 6 times faster, it avoids the complexity of having your own prototype and also avoids issues with prompt strings.

If you really do want to run multiple tests involving standard input, you either have to tolerate the slow response (that's Java for you!) or you have to use a much more complex question type that uses a combinator template to manage all the test cases in a single run.

I have written such a prototype to do roughly what I think your java_statement_rp type was intended to do, but with only one compile. I attach it as a proof of concept only (plus your question, slightly modified, and the question shown above). The prototype uses a Python3 script to compile the test program and then, if that succeeds, to run the compiled code with each of the different standard inputs. This version of your question runs in around 4 seconds when using a remote Jobe server. Note that with this prototype you have to enter the input data into the Extra Test Data field of each test case, not the standard input field. If you put it in the standard input field, CodeRunner will not even try to use the prototype template. [It doesn't usually make sense to try to roll all the test cases into a single file when there are multiple different standard input streams required. But this prototype shows that it can actually be done.]

Hope that helps.

Richard