With exams now over at the University of Canterbury (CodeRunner exams - no marking - yay!), I have a couple of months in which to play with CodeRunner development. Here's what I have in mind - alternative suggestions/discussions/comments welcome.
- Dispense with the complexity of having both a per-test template and a combinator template. Instead there will be a single template which is either one or the other, indicated by an is_combinator checkbox. [Can anyone think of a better name than 'combinator'?]. If the question author chooses to define a combinator template, and the run fails (usually because of a run error), the grader will attempt to run the tests one-at-a-time, still using the combinator template but passing it a test-list containing just the current test. The assumption is that question authors currently using combinator templates always make them consistent with the per-test templates when there's only one test. This change will significantly improve the Advanced part of the question authoring UI and will also simplify management of questions that currently have both a per-test template and a combinator template.
- Introduce a precheck capability as requested by Vladimir Ilyin. There will be a new Precheck dropdown in the question authoring interface which the user can set to one of Disabled (the default), Empty, Examples and Selected. If anything other than Disabled is selected, students will have an extra Precheck button to the left of the current Check button. Clicking Precheck will give the student a penalty-free submission in which the set of test cases will be one of the following:
- empty, which does a single run with empty test code. This will result in running any checks built-in to the template plus a compilation check
- examples, which run the subset of tests displayed in the "For example" table
- selected, which runs a specific subset of the tests chosen by the author.
- Provide an answer box Preload option. The question authoring interface will have a new UI field into which the author can specify text to be preloaded into the student's answer box. Like template text, the preload text will be subject to Twig expansion, with the QUESTION object available to it (and hence the template parameters). Further down the line I would like the author to be able to specify, probably by some form of Twig-style annotation, that portions of the preloaded text are ReadOnly, meaning they're locked so the student can not update them. I'm not yet sure how do-able this is, however.