CodeRunner: "Mark must be greater than zero"

Greetings,

I am an assistant to an Introductory Programming course in the University of Porto (Portugal).

We are determined to use CodeRunner!

In the past, we have rolled out our own web application whereby:

We made 4 public tests available that they could see when submitting code (no punishment)
In the end of the exam, we had 4 private tests that were used to evaluate their code (each worth 25%).

We would like to keep this configuration. We have already seen that we can disable "all-or-nothing" and use "show" for the public tests and "hide" for the private tests.

Everything is working great, except that we would like to define public tests with mark=0 and private tests with mark=25.

I wonder why you disallow mark=0?

edit_coderunner_form.php
} else if (floatval($mark) <= 0) {
    $errors["testcode[$i]"] = get_string('negativeorzeromark', 'qtype_coderunner');
}

Why floatval($mark) <= 0 instead of floatval($mark) < 0 ?

Or, is there any other way to configure our setup? Maybe using prechecks? We are currently using precheck=empty so that we could allow them to run pylint without punishment, but we want to punish submissions using the penalties.

Thank you very much for the great software!

-- Ricardo Cruz

Re: "Mark must be greater than zero"

por Richard Lobb - martes, 12 de marzo de 2019, 22:05

Hi Ricardo

Fair comment - I guess there's no compelling reason why you shouldn't have a test case worth zero. I've just never wanted to do it, myself.

Allowing zero marks for a test case isn't totally trivial because each test's result is displayed as a green tick, an orange tick or a cross depending on the fraction of the marks the test case earns (custom graders can award fractional marks for each test case). The fraction is undefined (divide by zero) if a test case is worth 0 marks.

I'll think about whether it's worth changing the code but in the meantime I think if you use a mark of 0.001 for the testcase the students won't see the difference from zero. Or if that seems too ugly, set those test cases to 1 mark and the other tests to some large number, again essentially giving the public test cases negligible marks. Students only see their total mark for the question as a fraction of the possible mark.

Another option is to set Precheck to Selected and set the public test cases to Precheck only - they then won't be run during the main Check.

Lastly - the ultimate fallback - a combinator grader gives you the freedom to implement any grading scheme you like, including arbitrarily complex penalty computation.

Hope that helps

Richard

Re: "Mark must be greater than zero"

por Ricardo Cruz - miércoles, 13 de marzo de 2019, 05:09

Dear Richard.

It's funny because our own web application (which we have been using), we implemented colors in a similar way. The color was green if student mark=max mark, red if student mark=0, and orange otherwise (e.g. there was a penalization due to wrong letter-cases or spaces). However, with our tool this wasn't a problem for us because we had a public and a private switch.

There is no problem, it's okay. The course starts in September, and we will be using the plugin as such. Maybe in the future we can propose a patch. :)

The regent wants to use 1 for each public test, 24 for each private test with a penalization of 10.

The problem with using Pre-check is that we want to penalize wrong submissions. We want them to be proficient in programming and debugging the program in the computer first, and think twice before insert it in Moodle and "checking" the code.

We will also explore combinator-template grading. Thank you!