Many thanks for the input. I agree with the principle that if there is a supplied right answer is should be graded as 100% and therefore checked before any data is saved. And indeed form-validation does seem like the appropriate place to do that, although as you say, there would need to be a checkbox to disable the checking, e.g. if the question has a large run time (some of my colleagues have written questions that take over 30 seconds to grade) or if the author wants to do a temporary save of an incomplete problem.
However, (and I may just be being cowardly) I find the prospect of implementing such a check in the form validation code rather daunting. At that stage all I have to work with is the data in the question-authoring form. The grade_response function is a method of the question object, so I need one of those for starters. Constructing this isn't as easy with CodeRunner questions as you might think because the question fields are only partly determined by the data in the form - most are inherited from the prototype, which must be loaded from the database first. Yes the code is all there, but it's distributed over various methods of the question-type superclass and my subclass, and I don't fully understand the sequence of operations while questions are being constructed - I just process the various callbacks you've supplied.
Then there's the problem of any test data files that might be associated with the question. These can come from the question instance and/or the prototype. In the current design data files are assumed to have all been loaded into the Moodle file system during question saving so are pulled from there during testing. I'd have to temporarily upload them and delete them later or do some other hack. Again, my understanding of the framework feels a bit fragile here.
Lastly, if I can solve all those problems, I have to send the run off to the Jobe server (or other sandbox) and untangle the response, feeding the errors back into the authoring form rather than the usual tabular result table display. Further complication comes from the fact that the author might be using a template grader so the results aren't necessarily just simple Yes/No booleans for each test case. I also need to be able to deal with the possibility that the Jobe server is down for maintenance.
I find myself thinking that all that sound and fury isn't really in keeping with the idea of form validation, is it?
All the same, if you still think it still sounds do-able and appropriate - and particularly if you're volunteering to help :-) - I'll move it higher up the TODO list. It won't happen for a while though - we're just starting our academic year here and I'm very busy.