Marking main assignments with CodeRunner

Marking main assignments with CodeRunner

por Michael Fairbank -
Número de respuestas: 2

Hi Richard et al,

We currently only use coderunner for small formative-learning exercises, which works great.  But I'm trying to work out if we can use it for larger programming assignments.  This would help me automate marking workloads, and potentially give more impartial consistent feedback to students.

I'd like it so students submit their main-assignment programming work to coderunner, but they can't see all of the results of the coderunner unit-tests, until after the assignment deadline (or until after I click a button to instantly mark their work).

The reason I don't want students to be able to run ALL of the unit-tests themselves is that, even if a penalty scheme is implemented, the first student who fully passes the unit tests may then give their solutions to the entire class, and then the whole class may un-fairly score 100%.  This is for a main assignment, so it's potentially worth a lot of marks, so it needs to be rigorous.  Also, I don't want all students to repeatedly edit their submissions until they pass all unit tests always - I'd like to give them a vague specification, such as "your program must handle unexpected inputs gracefully", but then the exact choice of un-expected inputs I try are kept secret until after the assignment deadline.  Otherwise students will just tailor their "unexpected inputs" to the ones I've chosen, defeating the objective of "unexpected"; and every student will score 100% again.

So ideally they would submit their largish program to coderunner, and click "check", and it would instantly check some perliminary features - such as the function signatures being correct, plus some very basic functionality, to steer the students in the right direction.  But I would not want them to be able to click "check" on the main suite of unit tests, which mark the majority of the functionality of their work.  Then, after the deadline, I would click "mark the rest of this assignment please", and all students would then get the results of some much more detailed unit tests, and my marking would be done in an instant, and all students would have detailed, instant feedback!  So everyone would be happy.

If this is not provided already by coderunner, then maybe this could be achieved by adding one more feature into the user-interface for coderunner question specifications such as "DisableUnitTestTemporarily" which I could apply just to those more-thorough unit tests, and then I could remove this label after the assignment deadline and re-mark. Or maybe I could create two versions of the coderunner quiz, which are nearly identical, but where the second quiz includes the extra unit tests within it.  Then I only give the students access to the first quiz.  Then after the deadline I somehow copy all of their solutions over from the first quiz to the second one, and the I run the second one myself somehow???

Another solution I suppose I could use is to manually modify the quiz after the quiz deadline to add those extra more-thorough unit tests.  Then I re-mark all the students work using Moodle.  That would probably work nicely, but there is a known issue preventing efficient re-marking unfortunately https://github.com/trampgeek/moodle-qtype_coderunner/issues/44.  I think if this issue was fixed, and if the coderunner interface was modified so that we could mark unit-tests with "DisableUnitTestTemporarily", then CodeRunner could be used to mark main assignments quite robustly, perhaps?

Thank you for any suggestions for this.

En respuesta a Michael Fairbank

Re: Marking main assignments with CodeRunner

por Richard Lobb -
Hi Michael

This should be do-able out of the box - we do something similar in some exams where we wish to give students only a sanity check on their answer, holding back our full checks until they make heir final submissions (and we don't let them review then, either).

You can set up the sanity tests as specific test cases, set the Precheck to Selected, and select the subset of tests you want to let the student see. Also check the Hide check checkbox so student don't even get presented with a Check button.

Note that students don't get penalties for checking the Precheck button.

The full set of tests will be run when the student submits their quiz, so you also want to set the quiz review options so that the Specific Feedback  (namely the result table) is visible during the attempt (so they see the results of their precheck) but not visible again (nor the marks, whether correct, etc) until after the quiz closes. And perhaps not even then if you want to review first.

think that should work, though I've never used it for assignments. 
En respuesta a Richard Lobb

Re: Marking main assignments with CodeRunner

por Michael Fairbank -
Thank you Richard. I'll try that. I'll let you know how it goes but it might take a few months before I can try this out in earnest. But this is very exciting...