Hi Anton
This is such a long thread that I'm losing track of the problem(s) you're trying to address. It may be time to start a new thread?
The decision to abort testing in the event of a runtime error was made way back in the first version of CodeRunner. My thinking at the time was that a runtime error is quite likely to occur in every test, not just one. In Python, for example, it might be something like an undefined variable. Or it might be an endless loop, in any language. Continuing to test after such runtime errors can be wasteful not only in test time but in the output generated - lots of tests all reporting the same error. If a runtime error occurs when using a combinator template, testing restarts the process, running each test separately until the first runtime error occurs. The student will have to fix the error to get any marks (assuming all-or-nothing grading), regardless.
Note that, as you observe, any output to stderr is taken to mean there's a runtime error.
I'm not keen on changing the existing default behaviour, but of course individual teachers' requirements vary. I encourage teachers to write their own question prototypes to fit their own needs - good to see you're doing that.
-- Richard
This is such a long thread that I'm losing track of the problem(s) you're trying to address. It may be time to start a new thread?
The decision to abort testing in the event of a runtime error was made way back in the first version of CodeRunner. My thinking at the time was that a runtime error is quite likely to occur in every test, not just one. In Python, for example, it might be something like an undefined variable. Or it might be an endless loop, in any language. Continuing to test after such runtime errors can be wasteful not only in test time but in the output generated - lots of tests all reporting the same error. If a runtime error occurs when using a combinator template, testing restarts the process, running each test separately until the first runtime error occurs. The student will have to fix the error to get any marks (assuming all-or-nothing grading), regardless.
Note that, as you observe, any output to stderr is taken to mean there's a runtime error.
I'm not keen on changing the existing default behaviour, but of course individual teachers' requirements vary. I encourage teachers to write their own question prototypes to fit their own needs - good to see you're doing that.
-- Richard