CodeRunner: Prototype questions and customization

This sounds challenging.

We normally script our prototypes using Python but if you wish to make use of your existing question template code you'll have to use Java instead. I've never tried doing that, and my Java's very rusty anyway, so I'm not sure how easy it would be to copy the sort of design we use in our complex prototypes, but I'll describe it anyway and you can decide if any of it's useful to you.

We put as much as possible of the grader functionality into support files, one file per class. So for example we have a support file that provides us with a ResultTable class, with methods to set the column headers, add rows, manage images in columns, etc. That sounds like the sort of thing you're after?

All the Twig processing is done in boilerplate code in the template. There's a table of allowed template parameters and their default values, and the template extracts all the template parameters from {{ QUESTION.parameters }}, checking the names and types for validity. This is a great help in picking up authoring errors. A dictionary/map of parameter names -> values, PARAMS, is then passed to the constructors of all the support file classes, so they can make full use of the template parameters or other global Twig parameters like STUDENT_ANSWER, ALL_OR_NOTHING, GLOBAL_EXTRA etc.

Most of the rest of our structure, with classes for style checking, task compilation and execution, are probably not relevant to you. But if you want to look at what we do, our question type python3_stage1 is on github here. It's all somewhat messy and overcomplicated for your needs, but the ideas for doing all the Twig processing in one place (see template.py) and for managing the result table (see __resulttable.py in Files.zip) might be useful to you.

If you were to use such an approach you'd have to modify all existing templates by inserting the Twig parameter-processing code at the start and changing the template code to make calls to the ResultTable class. I'm not sure how feasible that is in Java.

You might also want to consider exporting all the questions to XML and using a program to translate the questions to a new format.

That may or may not help.

Re: Prototype questions and customization

by Tom Spander - Thursday, 13 April 2023, 6:56 AM

Hey, thank you very much for the extensive reply. Of course that wasn't quite as straightforward as I was hoping it could be but your response was still helpful.
I thought I would also check in to give an update on what we've ended up doing.

I've decided to develop the custom grader in a proper Java Maven project. This way I can use external libraries for serialization and bundle everything in one JAR file with all dependencies included, and add that as a support file to each question. (It's a much nicer developer experience too.)
If the custom grader is updated, that does mean I have to change out the file in all questions, but I believe this will still be the "minimum impact" solution for maintaining it in the future. I've resigned myself to having to update existing tests to use the new grader, but I think that is probably a good thing anyways since a lot of the tests could use a bit more structuring.

I've gone a bit further with the custom grader because we also wanted to catch compiler errors and display some more helpful tips to the students. I actually did not know this before, but it is possible to compile Java code from a string inside Java and run it in the same execution context. That means I can pass in the tester code to my custom grader as a string, try-catch the compilation and add custom behavior if it fails.

Best,
Tom