Using more than one JDK / Java prototype

Using more than one JDK / Java prototype

by Anton Dil -
Number of replies: 9

Hi Richard,

To avoid breaking lots of old questions that have (support file jar) incompatibilities with a new JDK, we installed JDK 21 alongside the old JDK and wanted to add a new question prototype for JDK 21.

Here's what the developer said:

"I am trying to use two versions of java in coderunner using the same jobe server. On the jobe server I have

Supported languages:
    c: 11.5.0
    cpp: 11.5.0
    java21: 21.0.7
    java: 1.8.0_452
    nodejs: 22.16.0
    php: 8.0.30
    python3: 3.11.11

 I created this by copying /application/libraries/java_task.php to /application/libraries/java21_task.php and changing /etc/alternatives/java to /etc/alternatives/java21

 In moodle I copied the BUILT_IN_PROTOTYPE_java_class to create a java21 version and built a question based on that. I am getting the below error where is seems coderunner is using the language name ‘java21’ for the file extension.

 Syntax Error(s)

error: Class names, '__tester__.java21', are only accepted if annotation processing is explicitly requested

1 error 

Am I using the wrong approach or can someone help with this approach to allow multiple java version questions on the same system so we can create new questions and maintain the old.

"

We don't seem to be linking correctly to the new JDK. If we make the question prototype (or old question) sandbox language java21 we get the same error.

Is what we're trying to do possible and would you have any insight as to what we're doing wrong?

Thank you,
Anton

In reply to Anton Dil

Re: Using more than one JDK / Java prototype

by Richard Lobb -

You've fallen over a "feature" of Jobe that language tasks always generate a file of the form "__tester__.<language>". Because you're trying to define a new language java21 you've landed up with a tester with the extension .java21, which Java objects to. [AFAIK, Java is the only language that incorporates filenames into the language definition - grrrrr].

I really don't recommend trying to introduce new languages into Jobe. It's much easier to script them directly in CodeRunner. You can have a question prototype that uses a Python script to run anything you like on Jobe, which gives you much more flexibility with things like compile and runtime options. And it's way more maintainable. You can still set the Ace editor to use Java syntax colouring even though the Jobe task is (initially at least) Python.

I attach the XML export of a question PROTOTYPE_java21 together with a question that uses that prototype. It assumes you want a Java Program question type but it's easy to change it to Java Class or Java Method. The prototype assumes that the Java21 compiler is javac21 and the runtime is java21 - you can edit those two commands in the prototype template if necessary.

The template for the prototype is:

import subprocess
import re
import sys
student_answer = """{{ STUDENT_ANSWER | e('py') }}"""

# Need to determine public class name in order to name output file. Sigh.
# The best I can be bothered to do is to use a regular expression match.
match = re.search(r'public\s+class\s+([_a-zA-Z][_a-zA-Z0-9]*)', student_answer, re.DOTALL | re.MULTILINE)
if match is None:
    raise Exception("Unable to determine class name. Does the file include 'public class name'?")
classname = match.group(1)
filename = classname + '.java'

# Write the file
with open(filename, "w") as src:
    print(student_answer, file=src)
    
# Compile and run 
return_code = subprocess.call(['javac21', "-J-Xss64m", "-J-Xmx4g", filename])
if return_code != 0:
    print("** Compilation failed. Testing aborted **", file=sys.stderr)
else:    
    # If compile succeeded, run it. (You may want to tune the Java parameters here.)
    exec_command = ["java21", "-Xss16m", "-Xmx500m", classname]
    try:
        result = subprocess.run(exec_command, text=True, check=True)
    except subprocess.CalledProcessError as e:
        if e.returncode > 0:
            print(f"Task failed with a return code of {e.returncode}", file=sys.stderr)
        else:
            # Negative return codes are signals.
            print("Task failed with signal", -e.returncode, file=sys.stderr)
        print("** Further testing aborted **", file=sys.stderr)

Caveats:

  • This has had minimal testing, so may need some tweaking. But the style of question - using Python to run other languages - is very well tested.
  • The question type has set the runguard sandbox to not impose any memory limits, since the Java VM should manage those.
    • You can adjust the memory limits in the Java run command.

Post back if you have any issues or need further help.

In reply to Richard Lobb

Re: Using more than one JDK / Java prototype

by Anton Dil -
Hi Richard,

Thank you for the speedy reply and useful example. I seem to have a per-test template working, although I am unsure about the best values for the various compiler and VM flags. I'm thinking of ~1000 students trying to run tests in an exam at once. So far, so good :-)

Regards,
Anton
In reply to Anton Dil

Re: Using more than one JDK / Java prototype

by Richard Lobb -
Hi Anton

1000 students running Java at the same time is a pretty frightning scenario! We run up to 1000 student in Python with a 16-core Moodle server, an 8-core psql DB server and a pool of several Jobe servers (the number of these varies). We use a staggered start to ease the load on the db server. But Java is much more expensive on the Jobe servers. Jobe on my laptop can maintain a steady run rate of 35 - 40 minimal-sized C or Python questions per second, but only 7 Java questions. That implies you'd need at least 5 times as many Jobe servers. I understand OU now uses Amazon elastic compute cloud (is that right?) - I sure hope it can stretch to hande the Jobe server load!

BTW, I don't have useful advice on the Java VM runtime parameters. I don't teach Java and the Java users in our department make only relatively light use of CodeRunner. But FWIW the default parameters used by "native" Jobe Java runs are "-Xrs -Xss8m -Xmx200m". The parameters in the template above double both the stack and heap memory allocations. But you may want to consider adding the -Xrs. Someone (presumably one of my collaborators) has added the -Xrs parameter to jobe with the comment "reduces usage signals by Java, because that generates extra debug output when program is terminated on timelimit exceeded". Hopefully, though, time limit exceeded is a relatively rare event so I doubt it will greatly change the average throughput.

One final comment. If you're running Java Class or Java Method questions, you'd probably want to switch to using a combinator template. Otherwise, each test case will require a separate jobe run, which means a separate compile. Java compiles are what chew up most of the CPU time. You can even use a combinator template with write-a-program questions but you need to set the allow-mutliple-stdins checkbox, and do your own management of stdin for each run. By default, CodeRunner falls back to one-jobe-task-per-test mode if there are multiple stdins.

Richard
In reply to Richard Lobb

Re: Using more than one JDK / Java prototype

by Richard Lobb -
A further thought. A collaborator added the following source code to Config/jobe.php on the Jobe server. Given you're planning on running a huge exam, you might want to consider adding these extra options to the template? Your call, of course - I have no experience with large Java classes in CodeRunner.

/*
|--------------------------------------------------------------------------
| Extra Java/Javac arguments [Thanks Marcus Klang
|--------------------------------------------------------------------------
|
| This section of the config file adds extra flags to java and javac
|
| Provided examples tells java/javac that there is only 1 core, which
| reduces the number of spawned threads when compiling. This option can be
| used to provide a better experience when many users are using jobe.
*/
public string $javac_extraflags = ''; //'-J-XX:ActiveProcessorCount=1';
public string $java_extraflags = ''; //'-XX:ActiveProcessorCount=1';
In reply to Richard Lobb

Re: Using more than one JDK / Java prototype

by Anton Dil -
Thanks again Richard, more to ponder on! I am exploring a combinator template now to deal with the multiple compilation issue.

@Tim I would like them to have Check buttons, even if only to compile, or say pass/fail on the tests without showing them which ones passed or failed.

We still have occasional Jobe server errors now without any check buttons being shown, but had many the first time a check button was made available.
In reply to Richard Lobb

Re: Using more than one JDK / Java prototype

by Tim Hunt -

Actually, we have run this exam for a few years now. Yes, it puts a big load on the server(s), but not more than they can handle - helped by the fact that this is an old-school exam moved online, so no check or pre-check, and there are only 3 CodeRunner questions.


There are two VMs like this. 8 cores, 40GB RAM. (We did increase that after the first year fo this exam.)


In reply to Richard Lobb

Re: Using more than one JDK / Java prototype

by Anton Dil -
I've been experimenting with the combinator and have it working up to a point, that is, when the answer is okay.

I'd like it to report just once if students' code doesn't compile, but I can't get this aspect working. I keep getting something similar to "Error in question Expected 3 test results, got 1. Perhaps excessive output or error in question?" when the answer doesn't compile.

I have been trying to compile the student's code separtely from the __tester__ class before compiling both together, but it doesn't seem to get that far in the template anyway.

Here's the template I am using (with some debugging output in it). When the student code doesn't compile I don't see that debugging output.

I could also compile the answer box code as a string, but I wasn't hoping to go that far, yet. Anyway t feels like something else is going on before the template is run.

import re
import subprocess
import sys

# 1. Get student answer and strip 'public' from class/interface/record
student_answer = """{{ STUDENT_ANSWER | e('py') }}"""

student_answer = re.sub(
r'\bpublic\s+(abstract\s+class|class|interface|record)\b',
r'\1',
student_answer
)

# 2. Extract classname from student code
match = re.search(
r'\b(?:abstract\s+)?(class|interface|record)\s+([_a-zA-Z][_a-zA-Z0-9]*)\b',
student_answer
)

if not match:
print("The answer didn't contain a class, interface or record.")
sys.exit(1)

classname = match.group(2)

# 3. Write student's Java code to file
with open(f"{classname}.java", "w") as f:
f.write(student_answer)

# 5. Compile student + tester files
cargs = ["javac21",
"-J-Xss512k",
"-J-Xmx64m",
"-J-XX:CompressedClassSpaceSize=32m"]

compile_cmd1 = cargs + [
f"{classname}.java"
]

comp = subprocess.run(compile_cmd1, capture_output=True, text=True)

if comp.returncode != 0: # first try compile by itself without template
print(comp.stderr)
sys.exit(2)
else:
print("Student code compiled")

# 4. Only now write __tester__.java using Twig loop syntax
with open("__tester__.java", "w") as f:

f.write("""
public class __tester__ {

private static final String SEPARATOR = "##";

public static void main(String[] args) {
__tester__ main = new __tester__();
main.runTests();
}

public void runTests() {
try {
{% for TEST in TESTCASES %}
{
{{ TEST.testcode }}
{% if not loop.last %}
System.out.print("##");
{% endif %}
}
{% endfor %}
}
catch (Throwable e) {
System.out.println("Exception: " + e);
}
}
}
""")

compile_cmd2 = cargs + [
f"{classname}.java",
"__tester__.java"
]

comp = subprocess.run(compile_cmd2, capture_output=True, text=True)

if comp.returncode != 0:
print(comp.stderr)
sys.exit(2)

# 6. Run compiled Java program
run_cmd = [
"java21",
"-Xms1m",
"-Xmx64m",
"-Xss512k",
"-XX:CompressedClassSpaceSize=32m",
"__tester__"
]

run = subprocess.run(run_cmd, capture_output=True, text=True)

print(run.stdout)
if run.returncode != 0:
print(run.stderr, file=sys.stderr)
sys.exit(3)
In reply to Anton Dil

Re: Using more than one JDK / Java prototype

by Richard Lobb -
Thanks Tim for the interesting info on your set up. Having only 3 questions is a big win - we have 30 in our typical Python exams. But not having any sort of Check or Precheck functionality rather horrifies me.

Anton: Well done on biting the bullet and writing a combinator Java Class question type! Something I never got around to, as I don't teach Java.
Well done - you're so close! And you have my sympathy/apology for missing just one subtlety that may or may not be buried somewhere in the documentation but is highly non-obvious. Namely, that if you wish to abort processing of testcases in a combinator, you need to generate output to the stderr stream. You've printed your error messages to stdout, which is taken to mean that the test "worked" in the sense of not being wrong enough to cause an abort.

Here's my updated version of your template.  Apart from changing your code to output all error messages to stderr, the only other change is removing your debugging message.

import re
import subprocess
import sys

# 1. Get student answer and strip 'public' from class/interface/record
student_answer = """{{ STUDENT_ANSWER | e('py') }}"""

student_answer = re.sub(
r'\bpublic\s+(abstract\s+class|class|interface|record)\b',
r'\1',
student_answer
)

# 2. Extract classname from student code
match = re.search(
r'\b(?:abstract\s+)?(class|interface|record)\s+([_a-zA-Z][_a-zA-Z0-9]*)\b',
student_answer
)

if not match:
    print("The answer didn't contain a class, interface or record.", file=sys.stderr)
    sys.exit(1)

classname = match.group(2)

# 3. Write student's Java code to file
with open(f"{classname}.java", "w") as f:
    f.write(student_answer)

# 5. Compile student + tester files
cargs = ["javac21",
"-J-Xss512k",
"-J-Xmx64m",
"-J-XX:CompressedClassSpaceSize=32m"]

compile_cmd1 = cargs + [
f"{classname}.java"
]

comp = subprocess.run(compile_cmd1, capture_output=True, text=True)

if comp.returncode != 0: # first try compile by itself without template
    print(comp.stderr, file=sys.stderr)
    sys.exit(2)

# 4. Only now write __tester__.java using Twig loop syntax
with open("__tester__.java", "w") as f:

    f.write("""
public class __tester__ {

  private static final String SEPARATOR = "##";

  public static void main(String[] args) {
    __tester__ main = new __tester__();
    main.runTests();
  }

  public void runTests() {
    try {
{% for TEST in TESTCASES %}
      {
        {{ TEST.testcode }}
{% if not loop.last %}
        System.out.println(SEPARATOR);
{% endif %}
      }
{% endfor %}
    }
    catch (Throwable e) {
      System.err.println("Exception: " + e);
    }
  }
}
""")

compile_cmd2 = cargs + [
f"{classname}.java",
"__tester__.java"
]

comp = subprocess.run(compile_cmd2, capture_output=True, text=True)

if comp.returncode != 0:
    print(comp.stderr, file=sys.stderr)
    sys.exit(2)

# 6. Run compiled Java program
run_cmd = [
    "java21",
    "-Xms1m",
    "-Xmx64m",
    "-Xss512k",
    "-XX:CompressedClassSpaceSize=32m",
    "__tester__"
]

run = subprocess.run(run_cmd, capture_output=True, text=True)

print(run.stdout)
if run.returncode != 0:
    print(run.stderr, file=sys.stderr)
    sys.exit(3)

I also attach an xml export of a silly test of this template.

When you've got it all working and tested to your satisfaction, please post the xml export of a prototype that I can include in the new unsupported question types folder. Or ... perhaps I should just throw away the existing Java Class question type and use this one instead? Let me know your thoughts.

I note that your new question type does two compiles rather than one, which is still a big win if you have greater than two tests. I assume you compile the student code separately to avoid confusing error messages being apparently in the test code, e.g. triggered by the student providing the wrong class name? You might be able to get away with a single compile by including a few syntax checks (in Python) on the supplied code? I'm not sure - it's a long time since I wrote much Java.

A final comment. If you want the ultimate in question types, you can use a combinator template grader, which gives you complete control of the output, though at a significant increase in complexity. For example, you can output compile error messages at the start, without any result table, rather than having them land up in the first test case. This is another area of CodeRunner that is lacking in public examples, unfortunately.
In reply to Richard Lobb

Re: Using more than one JDK / Java prototype

by Richard Lobb -
Another thought: you probably don't need to capture the output from the subprocess runs, then print it. I think you should be able to just remove the output-capturing and let the subprocess print it directly?

However, if you ever want to switch to a combinator template grader, you not only need to capture all the output, you need to package it all up into the JSON result that should be the only output from the run.