Error during check cases

Error during check cases

by miki dream -
Number of replies: 3

we are using coderunner for python3 under moodle.

We can see that all cases that comes after an error aren't being checked.

for example - in the following picture a student had an error in his code, and instead of checking ALL the cases, the coderunner stopped after the first error in the first case.

EXAMPLE-1

In following example the coderunner checked all the cases:

Example2


We would like to know if it is possible to continue checking the rest of the cases although we encounter an error?


We will appreciate any help,

Thanks a lot 


In reply to miki dream

Re: Error during check cases

by Richard Lobb -

It isn't usually appropriate to continue testing after a runtime error has occurred. The student code is probably badly broken and the same error is likely to occur on all subsequent tests too. There's not much point in repeating the same error over and over - the student already has 0 marks, at least with an all-or-nothing grading scheme, and will have to resubmit anyway. Worse still, if it's a timeout error, say because the student code is looping, it can lead to very long run times if there are lots of tests and each one has to run until the timeout limit is reached.

In the example you show, the student has pasted in the test code as well as the function they were asked to write, so will get the same error on every test.

I guess the short answer is: no, testing is always aborted if a runtime error occurs. However, the longer answer is that you can if you want write a template grader for a question or question type; it can catch the runtime error and simply return a 'wrong answer' result (i.e. a mark of 0). Combinator template graders give you total control of the entire grading process and the feedback delivered to the student.

In reply to Richard Lobb

Re: Error during check cases

by miki dream -
Thank you very much Richard,

We are aware that the given example was a bad one since the code of the student was all broken.
Our intention were more on few test cases we created in order to check edge cases of a question, while the student's code was good they failed in few edge cases and the coderunner did not continue checking the other edge cases...

We would try to write a template grader in order to catch the errors of a specific edge cases...

Very much appreciate it!

In reply to miki dream

Re: Error during check cases

by Richard Lobb -

OK, I can see there are situations where you might want to continue when testing edge cases.

There's a related discussion in this forum thread. There's an alternative suggestion there, which you could consider if you're testing edge cases. You might be able to write your tests like:

try:
     answer = call_to_student_func(tricky_edge_case_params)
     print(answer)
except Exception as e:
     print(f"*** Exception ({e}) ***")

CodeRunner will then not see the runtime error, as it's caught by the test itself. Testing will then continue with the next test case.

However, that strategy doesn't work with write-a-whole-program questions. For those you'll need to use the template grader approach.