Pretests

Pretests

by Vladimir Ilyin -
Number of replies: 13

Is it possible to use pretests in CodeRunner?
To run some first tests before final testing.

F.e. students do not pay attantion for the output format,
use tabs as delimeters instead of spaces and so
get zero points for almost right solution.

May be it is possible to check the solution
before finishing test
with one or two first test to avoid that mistakes?

In reply to Vladimir Ilyin

Re: Pretests

by Ismael Figueroa -

I think you could simulate the desired behavior with a hidden test with the "Hide rest if fail". Put this kind of test at the beginning. I'm not sure whether the rest of the tests will actually be executed, but at least the output should only show up until the first failing "pretest"

In reply to Vladimir Ilyin

Re: Pretests

by Richard Lobb -

I'm not quite sure how you want the "pre-tests" to work, but here are some ideas to explore. If they don't help, please feel free to ask again, with a specific example and stating how you would like CodeRunner to behave.

  1. Use a penalty regime like 0, 0, 10, 20, ...  That gives the student a couple of free attempts, allowing them to sort out their minor formatting errors etc. [Our standard penalty regime nowadays is 0, 10, ..., giving one free attempt.]
  2. Set up the quiz with two versions of your question. The first one, worth zero marks, includes only one or two tests (probably the ones exposed in the "For example" table). The second one is the "real" question, worth marks. A variant on this is to have a completely separate training quiz, which students do before attempting the real one.
  3. Use a NearExactMatch grader rather than an ExactMatch grader. This tolerates white-space variations. Or use a RegularExpression grader that matches only the bits you really care about.
  4. Turn off All-or-Nothing grading and give partial marks for tests that check for answers that are almost right.
Richard


In reply to Richard Lobb

Re: Pretests

by Vladimir Ilyin -

The case 2 is most apropriate, I think.

Thank you very much!

In reply to Vladimir Ilyin

Re: Pretests

by Vladimir Ilyin -

I think that the best is to make CodeRunner's questions adaptive.

But I don't want to make whole quiz adaptive.

Is it possible to make CR questions only with the "Check" button? 

In reply to Vladimir Ilyin

Re: Pretests

by Richard Lobb -

Currently at least, you need to run the entire quiz in Adaptive Mode if you want CodeRunner to behave properly. If you wish to include other questions in the quiz, but you wish them to get marks for them only if their first submission is correct, you can set the penalty to 100%.

Richard

In reply to Richard Lobb

Re: Pretests

by Vladimir Ilyin -

It is not very good to allow students check their answers immidietly
and do not give them ability to correct wrong answers. They will
be worried about this situation all the time...

I don't  understand where adaptive_adapted_for_coderunner behavior is.
I don't see it in any list...
Is it selected automatically when I choose "adaptive"?

In reply to Vladimir Ilyin

Re: Pretests

by Richard Lobb -

Just in case I wasn't clear, I meant that you'd use the 100% penalty only on non-CodeRunner questions. We sometimes do that in tests and exams. It essentially just tells the students not to guess. But you're probably right that it's discouraging for a student to attempt a multichoice question, click Check, discover they're wrong, and know that there's nothing they can do about it. A less brutal approach is to use a 50% penalty. True/False questions, of course, must always have a 100% retry penalty.

adaptive_adapted_for_coderunner is indeed the behaviour that's used by a CodeRunner question when the quiz behaviour is set to adaptive. It's a subclass of the base adaptive behaviour class.

Richard

In reply to Richard Lobb

Re: Pretests

by Vladimir Ilyin -

Is it very difficult to add adapted deferredfeedback behavior for CodeRunner?
For showing Check button for CR questions only.

In reply to Vladimir Ilyin

Re: Pretests

by Richard Lobb -

It wouldn't be particularly difficult to implement. I'm just not sure I want to. My concerns:

  1. I'm not sure the behaviour you describe is what users expect or want. The deferred feedback behaviour is defined to defer the feedback. A CodeRunner question with a check button would break that specification. I agree that I always want the check button in a CodeRunner question, but mightn't there be some teachers who want students to develop and test their code, and then, taking full responsibility for it, submit it without the luxury of being able to check it?
  2. If I did that for deferred feedback behaviour, why not for all other behaviours, too? That then becomes much more work, and in addition makes the issue of (1) even bigger - should CodeRunner questions just ignore all the specified behaviours and do their own thing?
The change would also call for quite a lot of reworking of the documentation and internal messages.

However, I might yet be persuaded that it's a good idea. Comments and suggestions, anyone?

Richard

In reply to Richard Lobb

Re: Pretests

by Tim Hunt -

There is a precident for "just ignore all the specified behaviours and do their own thing".

Moodle 'Essay' questions, irrespective of what the quiz 'How questions behave' option is set to, use the 'Manually graded' behaviour.

So, it is certainly technically possible to have it so that CodeRunner questions always use qbehaviour_adaptive_adapted_for_coderunner, ignoring the quiz setting. I can see arguments for that approach. For the student, writing code where you cannot run it, see if it works, and then fix it if necessary, is a completely inauthentic experience. Being able to vary the penalty strikes me as sufficient control for the teacher. Always using Adaptive mode strikes me as better then allowing teachers to select a different behaviour, then displaying a message on screen that it does not work properly.

As Richard says, the other way to go would be to make CodeRunner work with all Moodle behaviours, but this would be more work. It would be a lot of work to make ..._adapted_for_coderunner versions of all the existing behaviours. The other approach would be to make changes in Moodle core, so that the the set_qt_var bit is no longer needed. This is a long-standing request. See https://tracker.moodle.org/browse/MDL-30442. I quite like Richard's propsed way to handle this, which is backwards compatible for existing question types. That would only leave
adaptive_adapted_for_coderunner doing the enhanced penalty regime, which woudl be less code, and that bit would not be necessary for other behviours. (Of course, any Moodle core change could only go into Moodle 3.3, which would be released next May, so this is a long-term view.)

(While I am talking about Moodle core issues, https://tracker.moodle.org/browse/MDL-54959 looks like it also have the potential to simplify adaptive_adapted_for_coderunner if it ever gets done.)

In reply to Tim Hunt

Re: Pretests

by Richard Lobb -

Thanks Tim. Interesting. Always forcing the use of adaptive_adapted_for_coderunner isn't something I'd ever considered. I had just assumed that if a quiz were running using behaviour X, whatever behaviour the question type chose had to be a subclass of X

I just tried replacing the qtype_coderunner_question::make_behaviour method with

public function make_behaviour(question_attempt $qa, $preferredbehaviour) {
    // Regardless of the preferred behaviour, always use an adaptive
    // behaviour.
    return  new qbehaviour_adaptive_adapted_for_coderunner($qa, $preferredbehaviour);
}
and it seemed to work. I assume you're telling me that's safe? Wunderbar!


Richard

In reply to Richard Lobb

Re: Pretests

by Tim Hunt -

Yes, that code is safe if that is the behaviour you want.

There probably also needs to be a matching change in the documentation.

In reply to Tim Hunt

Re: Pretests

by Richard Lobb -

Thanks Tim. I'm not sure if it's exactly the behaviour I want but, as you said above, it is undoubtedly an improvement on what I was doing before, and at near-zero cost.

Documentation already updated :)

Richard