Matlab function test framework

Matlab function test framework

von Angus Wallace -
Anzahl Antworten: 4

Hi all,

I've made a lot of progress in automatic question generation using Matlab (where I generate about 30 variations of each question), and am now getting into the more complex questions where the students write functions.

I have a Matlab file that generates the XML which I then import to CodeRunner/Moodle. This sets up the random question varients. I also write the function that serves as the correct answer. What I've been doing is, in the 'extras' section I use my function to generate the correct output and then write that into the XML so that I can use Octave's 'assert' function to compare the students' outputs.

In XML, this looks like:

<extra>
                <text>try inData = struct; inData.adult=2;inData.children=2;inData.pensioner=2;inData.infant=2;[a{1:2}] = zooEntryCalculator(inData);studentCost = cost; stuBreakdown = a{2}'; breakdown= [10.6 0 24.14 23.64 6.68] assert(studentCost, cost); assert(breakDown, stuBreakDown); disp('OK') catch disp('WRONG'); end</text>

where the numerical matrix is the output of my program, inputted to CodeRunner/Moodle as a string.

This is very kludgy, and I feel like there must be a better way. I really want to avoid text-matching outputs, as I find that it doesn't work reliably, and I like being able to use Octave's unit-testing framework, but as the function outputs become more complex, it's harder to compare them. For example, if a function outputs a struct, are there any better options?

What would be ideal would be to somehow define my own function in the extra code section, then do head-to-head tests (my function vs the students') -- but I can't see how to do that elegantly.

Any suggestions welcome!

Cheers, Angus

Als Antwort auf Angus Wallace

Re: Matlab function test framework

von Richard Lobb -

Hi Angus

I script all my Matlab questions in Python. The script runs to many hundreds of lines and checks lots of style things like maximum function length, code layout and use of comments. Template parameters allow me to configure individual questions to require or ban certain constructs or the use of certain functions. This is all so different from your approach that I'm not sure I have much to offer. I don't generate pools of questions in the way you are doing. I've never used Octave's unit testing framework, so I can't help with that either.

However, a colleague in Maths, Jenny Harlow, does one thing which might be of interest to you. Since she is teaching numerical maths, it's primarily numbers she's interested in. Her question types are also scripted in Python. When grading a student's submission she first runs her sample answer and extracts the integers and floating point numbers from the output. Then she runs the student answer, extracts the numeric output from that, and compares the sets of numbers to a specified tolerance. The sample answer can be expanded directly into the template as {{ QUESTION.answer }} or, if you're scripting the process, can be stored in a string with a line like (for Python)

sample_answer = """{{ QUESTION.answer | e('py') }}"""

It sounds to me like you could do something similar - compare the output from your sample answer with the output from the student's answer.

I don't have any particularly nice way of comparing structs either. I usually compare them field-by-field. I've also used the fieldnames function to ensure the right names are present. That function can also be used to implement a custom struct printer. The function orderfields might also be useful to produce a standardised output for structs.

I'm not sure if that helps. But I suspect that the sorts of things you're trying to do would be a lot easier if you scripted the process in another language, running the student code and, probably, the sample answer, in a subprocess. I and all my colleagues now use Python for scripting but for some years I used to use Octave for the job, using the system call to run subprocesses. But, really, scripting string manipulation and OS calls in Octave is an exercise in masochism.

Richard

Als Antwort auf Richard Lobb

Re: Matlab function test framework

von Angus Wallace -
Thanks for your detailed reply, Richard,

I'm not quite clear how you script your Matlab questions in python -- what does that mean? Do you mean that python starts a non-interactive version of Matlab or Octave as a subprocess?

Would you be happy to show me a bit of your python code?

Thanks again, Angus
Als Antwort auf Angus Wallace

Re: Matlab function test framework

von Richard Lobb -

Hi Angus

The general method of scripting questions in Python is documented here.

I attach a simple prototype file for a Python script that runs Octave and collects the stdout and stderr from the run, which you can then postprocess in Python in any way you please. I include a simple demo question. I just wrote this and it hasn't had much testing: it probably has some rough edges but it should give you the idea.

It's a per-test template so it runs once for each test case which is inefficient for questions with lots of testcases, but combinator templates are much harder to write. Note that if the script generates any output to stderr this is taken as a run-time error by CodeRunner and testing aborts at that point.

If you want to fix the out-by-one line number errors you need to postprocess the stderr output from the run.

Richard

Edit 15 March: updated xml file, which was using wrong escapers - should be using e('py') not e('matlab').

Als Antwort auf Richard Lobb

Re: Matlab function test framework

von Richard Lobb -

Oops, I used the wrong escaper in the template - should be e('py') not e('matlab'). I've replaced the .xml file in the above answer with the corrected one.