CodeRunner: Random tests

Hi,

I am trying to author some questions that use random data in a file - the students program then reads the data and outputs results based on that data (it's actually stage one of an assembler). I am familiar with the discussions elsewhere regarding David Bowes and his fork of CodeRunner. However, I am looking for something a little more simplistic and lightweight.

I got very, very close to getting this working just using a customized template. The template generates the file containing the random data and a variable containing the expected output from the program. Where I am stuck is getting the grader to use the contents of the variable in place of the TEST.expected or TESTCASES[n].expected attribute. I have tried updating the TESTCASES[n].expected attribute inside of the template (not trivial, but possible; n seems almost random. I have two tests for which n=2 and n=7 ???), however the grader ignores the change.

I could create a custom grader, but that seems a lot more work. Where can I access the default implementation of the exact match grader, to use as a basis? Or is there a way of changing the expected output inside of the custom template?

I've posted the question below. The first test is with known static data. The second is the random based one.

questions-CA-GCE-COMPSCI-1.2.4.c Assembly language.-20180220-1703.xml

Re: Random tests

de către Richard Lobb- miercuri, 21 februarie 2018, 22:24

Sorry, Stephen, but there's no way to inject an 'expected' field computed by the template code (which is running on Jobe, in Java) back into the standard graders (which are running on the Moodle server in PHP). The architecture diagram here might help to clarify that. You really have no choice but to use a template grader in this situation.

I'm personally not a huge fan of randomly generated test data, but your problem did get me a little interested. So I wrote my own version of your question, using a per-test template grader in Python. I simplified the problem a bit, requiring the student to write a program that tokenises standard input rather than a method. Here's a screenshot and I attach the exported question (as well as your question).

Result table

It's had only a bit of very superficial testing but it might be useful as a starting point, if you're a Python programmer.

Richard

assemblylanguagequestions.xml

Re: Random tests

de către Stephen Parry- joi, 22 februarie 2018, 11:01

Hi Richard,

Thanks for your solution.

I meant from within the twig part of the template (which is still server PHP).

I have managed to engineer a partial solution, using my language of choice, Java. The one bit I have not achieved is the two stage compile / execute part so that syntax errors do not bomb my whole test - hopefully deciphering your python code I can find the right command line to compile for Jobe. I have successfully generated source files and redirected output within the one stage so far.

The problem with fixed problems is that I teach 13-18 year olds how to program, and they cheat A LOT. Alos, some need a lot of attempts to get there solution right. On the one hand, they deliberately avoid functions, loops and conditionals wherever they think they can get away with it and on the other they are forever looking over their neighbour's shoulder looking to see what the question is so they can pre-engineer a solution. Being able to parameterize a couple of elements of the problem tends to kill the cheating dead and reinforces the need for the constructs.

Re: Random tests

de către Richard Lobb- joi, 22 februarie 2018, 14:12

I meant from within the twig part of the template (which is still server PHP).

Ah, sorry. Now I see what you mean. But still I'm afraid there's no path from the Twig expansion of the template into the testcases, which aren't Twigged.

Certainly cheating is a plague. I'm not sure that randomising test cases help much, however, as students who copy their friend's working code will still pass all the (different) randomised tests. David Bowes' system presents each student with a different randomised question, which ensures that a student's code doesn't work for their friends. My personal preference however is to teach students that copying code is downright stupid, as the purpose of the exercises is to teach them programming. We run invigilated tests and final exams, which students must pass in order to pass the course. Students who have simply copied coursework crash and burn in such tests. A good lesson in life? We also run programs that detect identical submissions on larger exercises, and penalise both students. However, I haven't tried teaching pre-tertiary students, and I can see that it's probably harder to get the message across to them. But then again, don't they need to get the message that you can't through life by plagiarising everything?!

Another useful strategy is to minimise the marks they get for the CodeRunner exercises, which you provide as practice material for actual summative assessment tests (again using CodeRunner).

By the way, I recommend that you always have at least one hidden test case so that students can't simply write a program that works only for the visible tests.

Richard

Re: Random tests

de către Stephen Parry- vineri, 23 februarie 2018, 05:22

Thanks for the suggestion on the hidden test case.

I can control the cheating during tests, it's formative sessions that are more of a problem - I don't mind if the students share answers (the summatives will shake out ones who have not actually learned from copying), providing the solutions they share are actually correct. Sharing questions, however, leads to ugly shortcuts and the protest "Well it works so why should I change it?" Many of the students I have taught in recent years could not even see the point of being in a Comp Sci lesson, let alone actually doing anything properly. They were so far outside their comfort zone, they would do anything to avoid engaging brain and learning. Fortunately my current batch are all level 3 (16-18) and a lot more switched on.

My question is now fully working with compilation and runtime checking. However, I think I have found a more generally useful, generally applicable and easily adaptable way of doing things IF we had template_from_string available in CR, without the same level of re-engineering seen in David's fork. I have raised an issue on GitHub to that effect.

Re: Random tests

de către Stephen Parry- vineri, 23 februarie 2018, 07:40

Am I going mad? I accidentally stumbled on some strange behaviour from {{ include template }} - it seems to do what I wanted from template_from_string. Instead of load the template from the filesystem as described in the Twig docs, it seems to render the string itself. Weird and undocumented, but very useful! I am making progress with some reusable templates....

Re: Random tests

de către Richard Lobb- sâmbătă, 28 aprilie 2018, 21:28

Hi Stephen

If you're still after a lightweight question randomisation capability via Twig, you might want to check out the latest release of CodeRunner (v3.4). The randomisation capabilities are explained here.

Richard