CodeRunner

OK, those times are a bit slower than on our Jobe server, but not hugely. With those numbers, and if you're using a question type that doesn't make use of a combinator template, I'd expect a 6-test-case question to take around 7 - 8 seconds. This is long enough to be a bit tedious but isn't too bad, I think. However, you said you had a 12 second delay so ... are the tests themselves taking significant compute time too?

Perhaps you could tell us more about the question you're using: the question itself, the template(s) and the test cases? Is it fundamentally necessary to compile a different program for each test case or are you just running the same program over and over with different input?

Without knowing the specifics, the only suggestions I can make are:

1. Reduce the number of test cases by packaging multiple tests into one.
2. Use a question type that uses a combinator template, such as the supplied java_method type. Combinator templates package all the test cases into a single compile-and-run, so you'd then get a response time of just a couple of seconds. Note that if you edit the template of such a question and wish to get the speed-up from the combinator, you'll have to re-enable it (assuming you clicked OK when the dialog popped up warning you) and add your custom code to the combinator template as well as to the per-test template.
3. As a variant on the above, you might be able to use a Python question (with a combinator template) as a script that compiles the Java program then runs it one test at a time. However, this requires advanced CodeRunner skills and is not recommended for newcomers. It's also important to know that the combinator template won't be used if the test cases have any standard input, so any such input would have to be placed in one of the other test case fields (e.g. the Extra Template Data field).
4. Get a faster server :)

Richard

Certainly not the right way, but by far the simplest (until the students catch on)...

In the question prototypes I've set up (in another forum thread), the only imported packages are java.util and java.util.stream. OTOH, nothing to stop them using java.net or java.nio or whatever classes by their complete name :^( . I suppose one could filter their answers for given package name occurrences in class names and raise a fuss.

Thanks for those two postings, Peter. I've just built the first of your java_code_test.py versions into a java_general question type to be included in a file 3rd_party_PROTOYPES.xml, which will be part of the next release. The question text includes all the explanatory comment at the start of your support file. I assume you're happy with this (it includes your Copyright note)?

I also assume that this new question type is intended to replace the earlier one you submitted, so I won't be including that.

That's an interesting approach, to move most of the template code into the support file. The disadvantage is that it's harder to make use of template parameters but certainly it makes it easier to edit and manage the code. 

It seems many teachers take the path of including code to do some sort of static checking of a student's submission before running it. In our first year course we run all code through the pylint Python style checker, with tight constraints on things like maximum function length, maximum line length, use of global variables, etc. We also have our own extra code to disallow or to require the use of certain constructs, as you've done in your enum example.

In my TODO list is a plan to include pre-run checking as a separate phase of the execution sequence so that you don't get the pre-run check failures coming up as Runtime Errors. That will however add a bit of overhead, as it will yet another Jobe submission at the start.

Thanks for contributing

Richard

Moving right along...

I also noticing from letting students loose on CodeRunner led to occasional unfortunate behaviour. For example, having given them code specs where they were to use enums in a switch, they might be tempted to just go with if-then-else-if-then-else... To CodeRunner's tester the output looks like what's wanted and the test case passes. Not what I wanted.

So I've cobbled up a few code-checker modules to actually do some static code analysis (extremely basic, it's not FindBugs but the students don't know that  :^)  ) to check whether their code is vaguely in-spec. If the code is out-of-spec CodeRunner pops an informative exception and they get to try again.

This works along the same principles as my last post - there's a module file java_code_checker.py included in the LOCAL_PROTOTYPE_java_checker question prototype which does the work of code-analysis.

For the enum checker, the template looks like:

from java_code_checker import *
from java_code_tester import *

student_answer = """{{ STUDENT_ANSWER | e('py') }}"""

student_answer = assemble_student_answer(student_answer, '{{QUESTION.parameters.enum}}')

# checks if enum used
check_for_enum(student_answer, '{{QUESTION.parameters.enum}}')

# loads and filters support files
support_files = assemble_support_files()

# assembles and saves test file from student answer and support files
assemble_tester(student_answer, support_files, '''{{TEST.testcode}}''')

# take it for a spin
compile_and_run()

These checks need a bit more configuration via the template parameters. For example, for the enums example I need to configure the name of the enum via

{"enum": "DayOfWeekend"}

because I need the name of the enum in order to include the possibility of a static import of its identifiers. One day I'll regex that bit of cruft away. One day...

This is nowhere near serious static code analysis, but it seems to keep the students a bit more honest.

Right, now that I've been using CodeRunner in anger in a Java-based course, I've had another go at making it workable. Instead of customizing templates, which quickly became unmanageable, I've put the code into modules which are then loaded as support files. This reduces the amount of customization of templates to an import statement and a couple of function invocations. For example, to test student answer functionality the customized template becomes:

from java_code_tester import *

student_answer = """{{ STUDENT_ANSWER | e('py') }}"""

# manipulates student answer
student_answer = assemble_student_answer(student_answer)

# loads and filters support files
support_files = assemble_support_files()

# assembles and saves test file from student answer and support files
assemble_tester(student_answer, support_files, '''{{TEST.testcode}}''')

# take it for a spin
compile_and_run()

I've worked this into a LOCAL_PROTOTYPE_java_tester to use as a question prototype and to be customize further as needed. Pre-loading the module file java_code_tester.py into the prototype as a support file means not having to remember to load it into every question; it's already available.

The java_code_tester.py file is attached below.

I've tweaked my java-in-a-packagetemplate to permit package names including a dot. So, eg, the following should work for the class foo.bar.Toto:

import sys, os, shutil, subprocess
__student_answer__ = """{{ STUDENT_ANSWER | e('py') }}"""
ncoding = '{{ QUESTION.parameters.encoding }}' or 'utf-8'
# Make package directory, put support files into it
package_dir = '{{ QUESTION.parameters.package }}'.replace('.', '/')
os.makedirs(package_dir)
[shutil.move(f, package_dir) for f in os.listdir() if f.endswith('.java')]
# Put student answer class into package directory
with open(package_dir + '/{{ QUESTION.parameters.class}}.java', mode='w', encoding=ncoding) as fd:
    print(__student_answer__, file=fd)
    fd.close()
    
# Build test class in package directory too
tester = '''package {{ QUESTION.parameters.package }};
public class __Tester__ {
    public static void main(String[] args) {
        {{TEST.testcode}}
    }
}
'''
with open(package_dir + '/__Tester__.java', mode='w', encoding=ncoding) as fd:
    print(tester, file=fd)
    fd.close()
# Compile and run
if (subprocess.call(['javac', '-encoding', ncoding, package_dir + '/__Tester__.java']) != 0 or
   subprocess.call(['java', package_dir + '/__Tester__']) != 0):
    print('** Further testing aborted **', file=sys.stderr)

Absolutely Ok.

Nice one! Thanks Peter. Are you happy for me to include that question type in the next release (with acknowledgements, of course)?

Richard

Ok, here's a slightly modified template to make it more generic. I've left in two template parameters for the name of the package and the student answer class, eg, {"package": "foobar", "class": "Foo"}. Basically I bailed on extracting this information from the student answer by trying to anticipate how students might construct their Java code file. Maybe later.

There's a limitation in that the student answer and supporting files must all be in the same package

import sys, os, shutil, subprocess
__student_answer__ = """{{ STUDENT_ANSWER | e('py') }}"""
# Make package directory, put support files into it
os.mkdir("{{ QUESTION.parameters.package }}")
[shutil.move(f, "{{ QUESTION.parameters.package }}") for f in os.listdir() if f.endswith(".java")]
# Put student answer class into package directory
with open("{{ QUESTION.parameters.package }}/{{ QUESTION.parameters.class}}.java", "w") as f:
    print(__student_answer__, file=f)
    
# Build test class in package directory too
tester = """package {{ QUESTION.parameters.package }};
public class __Tester__ {
    public static void main(String[] args) {
        {{TEST.testcode}}
    }
}
"""
with open("{{ QUESTION.parameters.package }}/__Tester__.java", "w") as f:
    print(tester, file=f)
# Compile and run
if (subprocess.call(["javac", "{{ QUESTION.parameters.package }}/__Tester__.java"]) != 0 or
   subprocess.call(["java", "{{ QUESTION.parameters.package }}/__Tester__"]) != 0):
    print("** Further testing aborted **", file=sys.stderr)

This is a prototype prototype (!) template for Java method questions, sort of like BUILT_IN_PROTOTYPE_java_method but with some differences:

• the major difference is that the method can depend on an object state by referencing instance attributes
• the method can refer to stuff in supporting classes
• the class can be in a Java package (see another discussion in this forum)

As an example, let's say you want the students to write their getHello method for the following class:

public class Toto {
    private String hello = "Yo!";
    public String getHello() {
        return hello;
    }
}

Replace the method with the string PLACEHOLDER so that the class looks like:

public class Toto {
    private String hello = "Yo!";
    PLACEHOLDER
}

The template uses convention over configuration, so the above class must be in a support file called placeholder. The template replaces PLACEHOLDER by the student answer and runs the resulting class against your tests.

Here's the template:

import sys, os, shutil, subprocess
__student_answer__ = """{{ STUDENT_ANSWER | e('py') }}"""
# First replace the placeholder with the student answer
placeholder = '{{QUESTION.parameters.placeholder}}' or 'placeholder'
ncoding = '{{QUESTION.parameters.encoding}}' or 'utf-8'
with open(placeholder, mode='r', encoding=ncoding) as fd:
    __student_answer__ = fd.read().replace('PLACEHOLDER', __student_answer__)
    fd.close()
with open('{{ QUESTION.parameters.class}}.java', mode='w', encoding=ncoding) as fd:
    print(__student_answer__, file=fd)
    fd.close()
# Make package directory, put support files into it
package_dir = '{{ QUESTION.parameters.package }}'.replace('.', '/')
os.makedirs(package_dir)
[shutil.move(f, package_dir) for f in os.listdir() if f.endswith('.java')]
    
# Build test class in package directory too
tester = '''package {{ QUESTION.parameters.package }};
public class __Tester__ {
    public static void main(String[] args) {
        {{TEST.testcode}}
    }
}
'''
with open(package_dir + '/__Tester__.java', mode='w', encoding='utf-8') as fd:
    print(tester, file=fd)
    fd.close()
# Compile and run
if (subprocess.call(['javac', '-encoding', ncoding, package_dir + '/__Tester__.java']) != 0 or
   subprocess.call(['java', package_dir + '/__Tester__']) != 0):
    print('** Further testing aborted **', file=sys.stderr)