CodeRunner: How to update a test case after the quiz has started?

Hi firstly thanks for Coderunner I am hoping its going to be a big help teaching python to 12-18 year old's. It's lightning fast on Digital Ocean and I can use the Moodle books to write theory units in age appropriate language.

I have a question about quiz authoring workflow, I am authoring quizzes of about 10 questions using python3 and python3_w_input.

If students find a mistake in the questions (am I alone in that?) I have been successful in changing the question text to fit the published test cases. But when trying to change the test cases to fit the question text, the changes aren't appearing on the student accounts.

I have seen warning "You cannot add or remove questions because this quiz has been attempted. And, I read something about questions being 'instantiated.' So am I correct in assuming that no more editing of the mechanics of the question is possible after an attempt has been made on the quiz (eg. repagination, test cases, adding questions)?

I have resorted to adding "old" on the old quiz and creating a new quiz and inserting that into the Moodle course. Its a bit fiddly. And students may loose their work. Is this the best workaround?

Re: How to update a test case after the quiz has started?

de Matthew Toohey - martes, 14 de septiembre de 2021, 22:09

Hi Richard

You are certainly not alone in having a student find mistakes in questions, It is not uncommon for students or teachers to find mistakes in a question after the quiz has been published.

You are right that you can not modify a quiz once it has gone live. However, there is no issue in modifying the contents of the individual questions of the quiz.
You should have no issues modifying the test cases for an existing question. However, it may not show up for the student until they have "rechecked" their solution. Note that a small change might have to be made to the answer (such as introducing an extra space/newline at the end) in order for the question to re-run the tests as the results are cached if you submit the same answer in a row.

Additionally, if you go to the quiz and click 'Attempts' to view student attempts. There is a 'Regrade all' button. This button will rerun the tests (that is the most up to date ones) on all student submissions. Note that depending on how many students you have this may take some time. Alternatively, if the issue is only affecting one or a few students, you can select the students on the left and at the bottom of the page the page click 'Regrade selected attempts'. This will rerun the tests on just the students you select.

The regrade will grade every submission the student has made. This means if a student was penalized for submitted a correct answer that your tests were marking as wrong. The regrade will mark their original submission as correct and they would not loose any marks.

Hope that helps,

Matthew

Re: How to update a test case after the quiz has started?

de Richard Drake - miércoles, 15 de septiembre de 2021, 00:30

Thanks for the swift reply

Note that a small change might have to be made to the answer (such as introducing an extra space/newline at the end) in order for the question to re-run the tests as the results are cached if you submit the same answer in a row.

My test_student account was not showing the changes so I will investigate this method. I have just tried the the regrade method that you described and it works well.

If some students change their answer to fit the test case. Will they be marked wrong after the regrade?

Re: How to update a test case after the quiz has started?

de Matthew Toohey - miércoles, 15 de septiembre de 2021, 14:25

If students have changed their answer to fit the old (incorrect) test cases then their most resent submission would be marked as wrong. However, if they had an early submission that was correct (for the updated test cases) then they would get full marks for that submission. Once a student has a correct submission their mark for a question can not go down with subsequent submissions. This gives students the freedom to try different things once they have gotten the question right with out having to worry about losing marks.

However, the student will always see the result table for their most recent submission which may now show up as failing test cases if you have changed them and done a regrade. This may cause some confusion. For this reason, it can sometimes be best to run the regrade on a selection of students who are being affected by the mistake in the test case, if this is possible. But sometimes regrading all students will be the only option.

Re: How to update a test case after the quiz has started?

de Richard Lobb - martes, 14 de septiembre de 2021, 22:13

You can certainly change the test cases of a live question, including adding new test cases, but after doing that you need to bring up the table of all attempts (e.g. by clicking the blue link telling you how many attempts so far) and then click the Regrade All button. Students will then see their updated grading result as soon as they reload/refresh their quiz attempt.

The only thing you can't do on a live quiz is delete or add questions.

Re: How to update a test case after the quiz has started?

de Richard Drake - miércoles, 15 de septiembre de 2021, 00:40

Thanks for the precise feedback

The only thing you can't do on a live quiz is delete or add questions.

This is interesting/useful for me. My students work quite slowly. So I could create a quiz and make sure I put 10 questions in, even if the last few are 'placeholders' ie Titled, but not yet written. Then I could finish them off later.

Thanks for your work and your ongoing support Richard.