Prototype questions and customization

Prototype questions and customization

by Tom Spander -
Number of replies: 2

Hello there!

I am working on a custom wrapper/template grader API for our Java (class or method) CodeRunner quizzes, to simplify the generation of custom result tables (especially considering that JSON outputs are notoriously difficult to generate in Java). The exact API isn't really relevant, the point is that the question author can use it to define outcomes of dynamically generated test cases and how they should be presented.

I would like to build the API in such a way that I can migrate existing questions easily. Currently, very few of our questions use the built-in test cases, so most of the testing logic is in the question template code.

In my "ideal scenario", I would define the custom grader class in a prototype question, change the existing question to that new prototype without losing the customization, and have access to the custom grader in the question template, perhaps by including a new Twig variable. I realize that it probably isn't quite that easy.

The main point is that I would like to define the custom grader so I can share it across all questions with a prototype, but also have custom template code for each individual question. I've considered moving the existing template code into globalextra, but without the Ace editor that would make it quite tedious to edit question templates later on. I've also considered putting the custom grader in a support file, but those cannot use Twig variables (and couldn't be updated easily if necessary).

Is this at all realistic? Is there maybe an entirely different approach to accomplishing this that I should consider instead?

Thank you very much for any advice!

Best, Tom

In reply to Tom Spander

Re: Prototype questions and customization

by Richard Lobb -

This sounds challenging.

We normally script our prototypes using Python but if you wish to make use of your existing question template code you'll have to use Java instead. I've never tried doing that, and my Java's very rusty anyway, so I'm not sure how easy it would be to copy the sort of design we use in our complex prototypes, but I'll describe it anyway and you can decide if any of it's useful to you.

We put as much as possible of the grader functionality into support files, one file per class. So for example we have a support file that provides us with a ResultTable class, with methods to set the column headers, add rows, manage images in columns, etc. That sounds like the sort of thing you're after?

All the Twig processing is done in boilerplate code in the template. There's a table of allowed template parameters and their default values, and the template extracts all the template parameters from {{ QUESTION.parameters }}, checking the names and types for validity. This is a great help in picking up authoring errors. A dictionary/map of parameter names -> values, PARAMS, is then passed to the constructors of all the support file classes, so they can make full use of the template parameters or other global Twig parameters like STUDENT_ANSWER, ALL_OR_NOTHING, GLOBAL_EXTRA etc.

Most of the rest of our structure, with classes for style checking, task compilation and execution, are probably not relevant to you. But if you want to look at what we do, our question type python3_stage1 is on github here. It's all somewhat messy and overcomplicated for your needs, but the ideas for doing all the Twig processing in one place (see template.py) and for managing the result table (see __resulttable.py in Files.zip) might be useful to you.

If you were to use such an approach you'd have to modify all existing templates by inserting the Twig parameter-processing code at the start and changing the template code to make calls to the ResultTable class. I'm not sure how feasible that is in Java. 

You might also want to consider exporting all the questions to XML and using a program to translate the questions to a new format.

That may or may not help. 

In reply to Richard Lobb

Re: Prototype questions and customization

by Tom Spander -
Hey, thank you very much for the extensive reply. Of course that wasn't quite as straightforward as I was hoping it could be but your response was still helpful.
I thought I would also check in to give an update on what we've ended up doing.

I've decided to develop the custom grader in a proper Java Maven project. This way I can use external libraries for serialization and bundle everything in one JAR file with all dependencies included, and add that as a support file to each question. (It's a much nicer developer experience too.)
If the custom grader is updated, that does mean I have to change out the file in all questions, but I believe this will still be the "minimum impact" solution for maintaining it in the future. I've resigned myself to having to update existing tests to use the new grader, but I think that is probably a good thing anyways since a lot of the tests could use a bit more structuring.

I've gone a bit further with the custom grader because we also wanted to catch compiler errors and display some more helpful tips to the students. I actually did not know this before, but it is possible to compile Java code from a string inside Java and run it in the same execution context. That means I can pass in the tester code to my custom grader as a string, try-catch the compilation and add custom behavior if it fails.

Best,
Tom