Hi Richard et al,
We currently only use coderunner for small formative-learning exercises, which works great. But I'm trying to work out if we can use it for larger programming assignments. This would help me automate marking workloads, and potentially give more impartial consistent feedback to students.
I'd like it so students submit their main-assignment programming work to coderunner, but they can't see all of the results of the coderunner unit-tests, until after the assignment deadline (or until after I click a button to instantly mark their work).
The reason I don't want students to be able to run ALL of the unit-tests themselves is that, even if a penalty scheme is implemented, the first student who fully passes the unit tests may then give their solutions to the entire class, and then the whole class may un-fairly score 100%. This is for a main assignment, so it's potentially worth a lot of marks, so it needs to be rigorous. Also, I don't want all students to repeatedly edit their submissions until they pass all unit tests always - I'd like to give them a vague specification, such as "your program must handle unexpected inputs gracefully", but then the exact choice of un-expected inputs I try are kept secret until after the assignment deadline. Otherwise students will just tailor their "unexpected inputs" to the ones I've chosen, defeating the objective of "unexpected"; and every student will score 100% again.
So ideally they would submit their largish program to coderunner, and click "check", and it would instantly check some perliminary features - such as the function signatures being correct, plus some very basic functionality, to steer the students in the right direction. But I would not want them to be able to click "check" on the main suite of unit tests, which mark the majority of the functionality of their work. Then, after the deadline, I would click "mark the rest of this assignment please", and all students would then get the results of some much more detailed unit tests, and my marking would be done in an instant, and all students would have detailed, instant feedback! So everyone would be happy.
If this is not provided already by coderunner, then maybe this could be achieved by adding one more feature into the user-interface for coderunner question specifications such as "DisableUnitTestTemporarily" which I could apply just to those more-thorough unit tests, and then I could remove this label after the assignment deadline and re-mark. Or maybe I could create two versions of the coderunner quiz, which are nearly identical, but where the second quiz includes the extra unit tests within it. Then I only give the students access to the first quiz. Then after the deadline I somehow copy all of their solutions over from the first quiz to the second one, and the I run the second one myself somehow???
Another solution I suppose I could use is to manually modify the quiz after the quiz deadline to add those extra more-thorough unit tests. Then I re-mark all the students work using Moodle. That would probably work nicely, but there is a known issue preventing efficient re-marking unfortunately https://github.com/trampgeek/moodle-qtype_coderunner/issues/44. I think if this issue was fixed, and if the coderunner interface was modified so that we could mark unit-tests with "DisableUnitTestTemporarily", then CodeRunner could be used to mark main assignments quite robustly, perhaps?
Thank you for any suggestions for this.