Question Authors' Forum

Matlab

 
Picture of Richard Lobb
Re: Matlab
by Richard Lobb - Wednesday, 22 June 2016, 10:02 AM
 

Hi Kristy

Good to hear that the modified octave_function template has been working well for you. I've used a similar template in the first week of my Matlab course to set questions in which I define a few variables and then ask the student to write some code using those variables to print a derived value. Presumably that's how you're using it too?

However, as soon as possible I switch to some variant of the (unmodified) octave_function template, i.e.

{{ STUDENT_ANSWER }}
format free
{{ TEST.testcode }};

With this template, I ask students to write a function to some spec and my test code then calls their function and prints the results. You can see an example of this type of question in my response to your first question in this thread - the sqr function. The exported xml for that question was attached to that posting if you want to look at it more closely. [If you're used to Matlab, you may be surprised that this template works, as in Matlab you can't define a function in a file then write global code that calls it. But Octave does not suffer from this restriction.]

Doing things this way round has several advantages:

  1. It gets students writing functions as soon as possible. While they don't initially really understand functions properly, they don't have any major problems writing ones if you specify exactly what they have to do. They just have to know the syntax for a function declaration. This sets them up much better for later in the course when they're expected to decompose larger scripts or functions into smaller ones by themselves.
  2. If you explicitly require that the student's function does not produce any output, your test code has complete control over the output from each test. This eliminates the frustration that occurs when students produce output that isn't exactly in the same format as the expected output. 
  3. Your tests can use fprintf to control numeric precision to reduce (but not eliminate) problems with rounding errors. Alternatively, your test code can compute the absolute difference between the expected and the actual results and print 'ok' or 'fail' accordingly.
  4. The environment in which the student's code runs is now completely explicit - the only variables they have available to them are the function parameters and the only result they have to produce is the function's return value. This improves the clarity of the question and also means you don't have to specify in great detail the expected output format. [You can still ask students to write functions that produce output if you like, but then of course you do need to specify the expected output carefully.]
You ask why the two Matlab prototypes are so much more complicated than the octave_function prototype. Mostly just because they include a custom disp function, written in Octave, that produces output in the same format as Matlab's disp function. My students are using Matlab for their coding but I am testing their code with Octave, so I try to minimise differences. I often test their functions by calling disp to display their answers and if they do the same tests in Matlab before submitting, and see noticeable differences between their output and the expected output, they get worried!

Richard

Picture of Angus Wallace
Re: Matlab
by Angus Wallace - Friday, 9 February 2018, 8:25 PM
 

Hi Richard,

Just joining this thread, I've had a look at your matlab_PROTOTYPES.xml file, and I can't really follow how the two examples are supposed to work. In particular: they don't include any test cases (right?). where do the test cases go, and how are they evaluated?

Do you have a simple example that uses coderunner with testcases that you can share via xml? Something as simple as 'extract the 4th row of Matrix x' would be great.

Many thanks, Angus

Picture of Angus Wallace
Re: Matlab
by Angus Wallace - Friday, 9 February 2018, 10:55 PM
 

I thought it would make sense to attach what I'm working on, maybe someone can point out where I'm going wrong. There are two main problems with this (of which I'm aware!)

  1. it evaluates to 1 or 0, so when I 'show' the first example to the student, they see only a 1 and not the row or column that I want them to extract. I can't see how to evaluate whether their answer is right as a matlab test and also have something nice to print. (I want to avoid string comparisons, and would rather use isequal, as it seems more robust)
  2. it doesn't work in coderunner. When I submit an answer, the page seems  to click and it goes back to the answer without grading it. I presume the process that's evaluating the answer is crashing?

Any suggestions would be gratefully received!

Cheers, Angus

Picture of Richard Lobb
Re: Matlab
by Richard Lobb - Friday, 9 February 2018, 11:56 PM
 

Hi Angus

A prototype isn't an example of a question - rather, it defines a question type. See section 11 of the documentation. But I suggest you don't worry about prototypes for now. Wait until you've written a few questions yourself and have a better idea of how prototypes work.

Your example question is on the right track (though FYI the file you attached is lacking the <quiz> tag at the start). Testing scripts is always a bit messy compared to testing functions, because there are three phases to each test instead of two:

  1. Set up a context for the student. In your case this involves defining the value of x.
  2. Running the student answer in that context.
  3. Checking if the student answer did the right thing, i.e. that it left the workspace in the right state. In your case that means a new variable y must exist and must contain the correct value for the given x.

I see you've correctly used the testcode for step 1 and used the extra code for step 3. However, your template code doesn't actually make use of the extra code, so you're not actually doing anything after the student code has been run.

A better template for this question would be:

{{ TEST.testcode }};
format free
{{ STUDENT_ANSWER }}
{{ TEST.extra }}

With that change, using as an answer

y = x(5, :);

and tweaking the expected output a bit, the question sort of works. However, as you point out, it doesn't give very friendly feedback to the student. All they see are lines telling them the expected output was 1 and the actual output was 0, say.

A simpler approach would be to use disp(y) for the extra and then set the expected field to whatever the displayed slice value should be. However, you say that string comparisons aren't sufficiently robust, though I'm not quite sure why in this particular case, as I don't see any scope for rounding errors. But certainly you have to be careful of rounding errors in general, so then you have to write more elaborate tests. For example you might do a test like the following (in pseudocode):

if y not defined
    print "You haven't defined y"
else if y != expected value within given tolerance
    print "Your value of y is wrong."
    print "Expected ... "
    print "Got ..."
else
    print "OK"

Sure, this gets tiresome very quickly, which is when you start writing your own question prototypes to abstract all the detail into the prototype.

There's also a problem with this approach if the student's code generates output, say because the student omitted the ';' from the end of the assignment statement. You should at least warn the student in the question that they mustn't print anything and must terminate their assignment statement with a semicolon. More elaborate approaches involve catching any student output yourself and giving an appropriate error message, but that is way too complicated for now.

Another thing that you're possibly doing wrong, if you're not getting any output at all, is running the question in the wrong mode. You should be running it in adaptive mode, and you must also make sure that the review options are set to display feedback during the attempt. Otherwise you or the student won't see the test results until the quiz is closed.

Hope that's enough to get you going again

Richard

Picture of Richard Lobb
Re: Matlab
by Richard Lobb - Saturday, 10 February 2018, 12:23 PM
 

Just to clarify my previous answer: the "test code" for which I gave pseudo code has to be run after the student's code. So you either have to put in the extra field (which can have as much code as you like in it) or in the template after the {{ STUDENT_ANSWER }} line. Putting it in the template, and using the 'expected' field just to contain the expected answer (with no newline on the end) gives you something like:

{{ TEST.testcode }};
format free
{{ STUDENT_ANSWER }}

if ~exist('y')
    disp('You haven''t assigned a value to y');
else
    if ~isequal(y, {{ TEST.expected}})
        disp(y)  % Will appear in the 'Got' column
    else
        disp('{{ TEST.expected }}')  % Will be marked correct
        % or disp(y) if there are no rounding or formatting issues
    end
end

Richard

Picture of Angus Wallace
Re: Matlab
by Angus Wallace - Saturday, 10 February 2018, 2:17 PM
 
That’s really helpful, thanks Richard. I’ll have a play with this and see what I can work out.

Cheers, Angus
Picture of Angus Wallace
Re: Matlab
by Angus Wallace - Monday, 12 February 2018, 3:15 PM
 

Hi Richard,

Thanks for your help so far -- your comments about the extra code in particular is very useful!

I have tried to go to string matching and I'm having a lot of trouble. I think it may be due to the formatting of the strings being different. I'm trying to generate the XML from within Matlab so that I can have many variants of the questions to present to the students, so I'm trying to calculate/capture output into a string in Matlab and then write that into the XML. I've tried a few ways of doing this, and what I've got now is:

evalc('disp((0:10:90)+sliceNum)');

where the resulting string is written into the XML as the expected answer, viz.

<expected>
                <text>    96    86    76    66    56    46    36    26    16     6

</text>
      </expected>


but, as you can see, there is an additional line being written.. I think that might be preventing the answer from being properly graded. What I'm not clear about is the exact formatting of the XML that I need to provide so that the answers will match. Is this documented anywhere? (ie. spaces before/after each number, tabs, newline characters, etc, for horizontal arrays, vertical arrays and matrices)


One other thing -- I was unable to see 'adaptive mode' anywhere -- where can I find this?

Thanks again!

-Angus

Picture of Angus Wallace
Re: Matlab
by Angus Wallace - Monday, 12 February 2018, 6:24 PM
 

Hi Richard,

I've worked it out.

I found the 'adaptive mode' under Question behaviour > How questions behave in the quiz configuration.

I was right -- it was the formatting. I needed to print the results in Matlab to better-match what was generated by Octave. In Matlab, I printed thus:

sprintf(' %d', (0:10:90)+sliceNum)

which puts exactly one space before every number.

I have one other question ;-) I'm generating many variants of each question, but I only want to present each student with one variant. Is there a way to do this? (I thought there was, but I'm now having difficult making it work)

Cheers, Angus

Picture of Richard Lobb
Re: Matlab
by Richard Lobb - Monday, 12 February 2018, 9:30 PM
 

Hi Angus

It sounds like you're making good progress.

If white-space is an issue you might want to consider changing the base question type you're working with to use "Near exact match" grading instead of the default "Exact match". The meanings of the different grading options, which are available only when you customise a question and of course in the prototype, are documented in line - click the '?' icon beside the 'Grading' field. 

If you want to present a student with a randomly selected variant from the questions you're generating, you can use Moodle's Random question type. Put all your generated CodeRunner questions questions in the same category and insert into the quiz a Random question that draws from that category.

Cheers

RIchard

Picture of Angus Wallace
Re: Matlab
by Angus Wallace - Monday, 12 February 2018, 10:08 PM
 

One of my colleagues very kindly showed me how to do random subsets of the questions. Ok.. Onward!


Cheers, Angus