Question Authors' Forum

Matlab

 
Picture of Kristy Hansen
Matlab
by Kristy Hansen - Friday, 20 May 2016, 4:58 PM
 

Hi,

I'm new to CodeRunner and am having trouble finding documentation on setting up Matlab/Octave questions. Is it possible that you could send me some examples so that I can try to understand how it works? Is a template required? I am having trouble working out what I should put in the empty fields without knowing what the template is.

Regards,

Kristy

Picture of Richard Lobb
Re: Matlab
by Richard Lobb - Friday, 20 May 2016, 10:31 PM
 

Hi Kirsty

Out of the box, there's only one Octave/Matlab question type, called Octave function

You can find out a bit more about any selected CodeRunner question type by opening the Question Type Details section in the question authoring form and you can see the template that each question type is using by clicking the Customise checkbox. In the case of Octave function, you'll be told that the question type is experimental and has had very little use, but that's not true any longer. I'll remove that on the next push.

The template for the Octave function type is simply

{{ STUDENT_ANSWER }}

format free
        
{{ TEST.testcode }};

which means that the script passed to Octave for execution just consists of the student's code (whatever they pasted into the answer box), a command to use Octave's 'free' mode for output formatting (see the Octave documentation) and then whatever code you've chosen to paste into each of the Test case <n> boxes.

Here's what a trivial "Write an Octave function" would look like to a student:


I attach a Moodle XML export of this question; if you import this and open it you'll see how it's set up.

There are two other Octave question types provided if you import the file uoc_prototypes.xml from the db folder in the source into the Moodle Question bank. They are called Matlab function and Matlab script, though the names are slight misnomers as they actually use Octave. [We have several other Matlab question types that we use in-house, but they're way more complicated and not appropriate if you're just starting to use CodeRunner.] The Matlab function question type is more-or-less the same as the Octave function type except that the template starts with a re-declaration of the disp function that more-closely mimics Matlab's disp function so that the output the students see in a CodeRunner question is the same as they get (usually) when they run their code in Matlab. It doesn't use Octave's free output format mode.

Matlab script is a somewhat different type of question. The question type details for it say "Used for Matlab scripts (so-called). Runs the test code first (which probably sets up a context) and then runs the student's code, which may or may not generate output dependent on the context. Finally the extra code is run (if any)." I rarely use this. It allows you to ask questions of the form Given variables a and b, write code to compute or print some derived value c. The test code defines a and b, the student's code computes or prints c. If they just compute c, you can put code in the Extra field of the question authoring form to display c yourself, or check it in some other way. This question type is useful only for introductory questions when students are just learning to write simple assignment statements or expressions. Getting them to write functions early in their course provides a much safer context for feeding input parameters to them and getting printed output or computed return values back again.

If your students are developing in Matlab and submitting code to be graded with Octave, you should be warned that there are a few gotchas, particularly when students submit bad code - Matlab accepts all sorts of rubbish that Octave warns them about. There are also subtle differences in how some functions work and of course most of the toolboxes are missing from Octave. But if your students are using Octave too, none of these issues matter.

You should also be warned that regardless of whether you're using Octave or Matlab, you must be very careful with floating point tests, as the student's full-precision numeric output will only rarely be the same as yours. At very least you need to use fprintf to limit the precision to which numeric data is displayed and even then things can go wrong. One of my colleagues has developed an Octave question type that compares all floating point numbers to a specified tolerance, but this is a very complex question type that does lots of other clever stuff as well (e.g. lots of style checking) so ask us about it in a year or so :-)  In the meantime, if you run into this problem, you can always write more sophisticated tests that, rather than just printing the student's answer, compare it with the expected output and print a message like YES or NO (so the expected output is then YES).

Hope that's enough to get you going,

Enjoy,

Richard

Picture of Kristy Hansen
Re: Matlab
by Kristy Hansen - Wednesday, 25 May 2016, 11:57 AM
 

Dear Richard,

Thank you very much for the comprehensive reply :) I have been making great progress since I last wrote to you and have managed to code up several successful questions! I managed to import the .xml example that you attached to your message but I cannot find the db folder in the source for the other two examples. Should I be looking for a folder called "source" ?

Thanks again for your help.

Regards,

Kristy

Picture of Richard Lobb
Re: Matlab
by Richard Lobb - Wednesday, 25 May 2016, 8:41 PM
 

Hi Kirsty

Sorry for the confusion. I meant the db folder within the CodeRunner source. You'll find it in the github repository, here.

Richard

Picture of Kristy Hansen
Re: Matlab
by Kristy Hansen - Sunday, 29 May 2016, 4:34 PM
 

Hi Richard,

Thank you for the link and I'm sorry to bother you again but I am not sure how to save the .xml file from that location in order to be able to import it. I am not at all familiar with .xml format so that probably doesn't help :(. Can you please tell me how I can do this?

Regards,

Kristy

Picture of Richard Lobb
Re: Matlab
by Richard Lobb - Sunday, 29 May 2016, 7:33 PM
 

Hi Kristy

Steps are as follows:

  1. In your browser, navigate to https://github.com/trampgeek/moodle-qtype_coderunner/tree/master/db
  2. Left-click the link to the xml file you want e.g. uoc_prototypes.xml
  3. Left-click the "Raw" button at the top of the text window. That should get you a browser window containing just the xml file itself, without other stuff like line numbers.
  4. Save that file (probably by typing CTRL/S though it's browser/OS dependent) somewhere suitable, say as "questions.xml".
  5. Use Moodle's Course Administration menu to find Question Bank > Import. Click that link.
  6. You should now be looking at a Import questions from file form. Select Moodle XML format as the file type and use the Choose file dialogue box to select the saved questions.xml. Click Upload this file
  7. Click the blue Import button. You should get presented with a list of all the questions you just imported.
Hope it works this time. If not I'll just email you the files :)

Richard

Picture of Kristy Hansen
Re: Matlab
by Kristy Hansen - Monday, 6 June 2016, 1:34 PM
 

Hi Richard,

Sorry to make this seem so complicated! But I can't save the file as .xml. It just isn't a listed option for me. I can only save as .txt and then that format is not accepted using the import option. I would really appreciate it if you could email me the file :)

Thanks,

Kristy

Picture of Richard Lobb
Re: Matlab
by Richard Lobb - Thursday, 9 June 2016, 9:54 PM
 

Hi Kristy

OK, I attach an XML file containing the two prototypes: matlab_function and matlab_script. However, please please don't use these unless you understand question prototypes properly (see the CodeRunner manual, section 11). Normally, prototypes are installed by a system administrator at the system level, where they can't be altered by users. Experienced users manage their own question prototypes, but it is important that you know what you're doing. Several users have crashed CodeRunner by replicating prototypes, thinking they were sample questions for them to experiment with.

A pertinent question at this stage might be: what is the problem that you're trying to solve? If you're trying to ask a particular question that isn't supported by the existing question types, the first step is usually to edit the template on a one-off basis to do what you want. Only when you find yourself using the same template over and over again do you normally start using prototypes. If you want help designing a template for a particular question you have in mind, feel free to ask.

-- Richard

Picture of Kristy Hansen
Re: Matlab
by Kristy Hansen - Friday, 17 June 2016, 4:17 PM
 

Hi Richard,

Thank you very much for the attachment. I can appreciate what you mean when you say that you need to be careful with prototypes in order to avoid crashing the system.

I was actually hoping to create a question where students design a user defined function. Could you help me with that? I guess the prototypes that you sent are not really designed for this are they? Actually, I am wondering why the octave_function prototype is so much simpler than these latest versions that you sent? Are the latest ones more robust?

So far I have just been using the following template (slightly modified from octave_function) and it has been working well:

{{ TEST.testcode }};

format free

{{ STUDENT_ANSWER }}

Sorry for my delayed replies - it is a busy time of the semester for me!

Regards,

Kristy


Picture of Richard Lobb
Re: Matlab
by Richard Lobb - Wednesday, 22 June 2016, 9:54 AM
 

Hi Kristy

Good to hear that the modified octave_function template has been working well for you. I've used a similar template in the first week of my Matlab course to set questions in which I define a few variables and then ask the student to write some code using those variables to print a derived value. Presumably that's how you're using it too?

However, as soon as possible I switch to some variant of the (unmodified) octave_function template, i.e.

{{ STUDENT_ANSWER }}
format free
{{ TEST.testcode }};

With this template, I ask students to write a function to some spec and my test code then calls their function and prints the results. You can see an example of this type of question in my response to your first question in this thread - the sqr function. The exported xml for that question was attached to that posting if you want to look at it more closely. [If you're used to Matlab, you may be surprised that this template works, as in Matlab you can't define a function in a file then write global code that calls it. But Octave does not suffer from this restriction.]

Doing things this way round has several advantages:

  1. It gets students writing functions as soon as possible. While they don't initially really understand functions properly, they don't have any major problems writing ones if you specify exactly what they have to do. They just have to know the syntax for a function declaration. This sets them up much better for later in the course when they're expected to decompose larger scripts or functions into smaller ones by themselves.
  2. If you explicitly require that the student's function does not produce any output, your test code has complete control over the output from each test. This eliminates the frustration that occurs when students produce output that isn't exactly in the same format as the expected output. 
  3. Your tests can use fprintf to control numeric precision to reduce (but not eliminate) problems with rounding errors. Alternatively, your test code can compute the absolute difference between the expected and the actual results and print 'ok' or 'fail' accordingly.
  4. The environment in which the student's code runs is now completely explicit - the only variables they have available to them are the function parameters and the only result they have to produce is the function's return value. This improves the clarity of the question and also means you don't have to specify in great detail the expected output format. [You can still ask students to write functions that produce output if you like, but then of course you do need to specify the expected output carefully.]
You ask why the two Matlab prototypes are so much more complicated than the octave_function prototype. Mostly just because they include a custom disp function, written in Octave, that produces output in the same format as Matlab's disp function. My students are using Matlab for their coding but I am testing their code with Octave, so I try to minimise differences. I often test their functions by calling disp to display their answers and if they do the same tests in Matlab before submitting, and see noticeable differences between their output and the expected output, they get worried!

Richard

Picture of Angus Wallace
Re: Matlab
by Angus Wallace - Friday, 9 February 2018, 8:25 PM
 

Hi Richard,

Just joining this thread, I've had a look at your matlab_PROTOTYPES.xml file, and I can't really follow how the two examples are supposed to work. In particular: they don't include any test cases (right?). where do the test cases go, and how are they evaluated?

Do you have a simple example that uses coderunner with testcases that you can share via xml? Something as simple as 'extract the 4th row of Matrix x' would be great.

Many thanks, Angus

Picture of Angus Wallace
Re: Matlab
by Angus Wallace - Friday, 9 February 2018, 10:55 PM
 

I thought it would make sense to attach what I'm working on, maybe someone can point out where I'm going wrong. There are two main problems with this (of which I'm aware!)

  1. it evaluates to 1 or 0, so when I 'show' the first example to the student, they see only a 1 and not the row or column that I want them to extract. I can't see how to evaluate whether their answer is right as a matlab test and also have something nice to print. (I want to avoid string comparisons, and would rather use isequal, as it seems more robust)
  2. it doesn't work in coderunner. When I submit an answer, the page seems  to click and it goes back to the answer without grading it. I presume the process that's evaluating the answer is crashing?

Any suggestions would be gratefully received!

Cheers, Angus

Picture of Richard Lobb
Re: Matlab
by Richard Lobb - Friday, 9 February 2018, 11:48 PM
 

Hi Angus

A prototype isn't an example of a question - rather, it defines a question type. See section 11 of the documentation. But I suggest you don't worry about prototypes for now. Wait until you've written a few questions yourself and have a better idea of how prototypes work.

Your example question is on the right track (though FYI the file you attached is lacking the <quiz> tag at the start). Testing scripts is always a bit messy compared to testing functions, because there are three phases to each test instead of two:

  1. Set up a context for the student. In your case this involves defining the value of x.
  2. Running the student answer in that context.
  3. Checking if the student answer did the right thing, i.e. that it left the workspace in the right state. In your case that means a new variable y must exist and must contain the correct value for the given x.

I see you've correctly used the testcode for step 1 and used the extra code for step 3. However, your template code doesn't actually make use of the extra code, so you're not actually doing anything after the student code has been run.

A better template for this question would be:

{{ TEST.testcode }};
format free
{{ STUDENT_ANSWER }}
{{ TEST.extra }}

With that change, using as an answer

y = x(5, :);

and tweaking the expected output a bit, the question sort of works. However, as you point out, it doesn't give very friendly feedback to the student. All they see are lines telling them the expected output was 1 and the actual output was 0, say.

A simpler approach would be to use disp(y) for the extra and then set the expected field to whatever the displayed slice value should be. However, you say that string comparisons aren't sufficiently robust, though I'm not quite sure why in this particular case, as I don't see any scope for rounding errors. But certainly you have to be careful of rounding errors in general, so then you have to write more elaborate tests. For example you might do a test like the following (in pseudocode):

if y not defined
    print "You haven't defined y"
else if y != expected value within given tolerance
    print "Your value of y is wrong."
    print "Expected ... "
    print "Got ..."
else
    print "OK"

Sure, this gets tiresome very quickly, which is when you start writing your own question prototypes to abstract all the detail into the prototype.

There's also a problem with this approach if the student's code generates output, say because the student omitted the ';' from the end of the assignment statement. You should at least warn the student in the question that they mustn't print anything and must terminate their assignment statement with a semicolon. More elaborate approaches involve catching any student output yourself and giving an appropriate error message, but that is way too complicated for now.

Another thing that you're possibly doing wrong, if you're not getting any output at all, is running the question in the wrong mode. You should be running it in adaptive mode, and you must also make sure that the review options are set to display feedback during the attempt. Otherwise you or the student won't see the test results until the quiz is closed.

Hope that's enough to get you going again

Richard

Picture of Richard Lobb
Re: Matlab
by Richard Lobb - Saturday, 10 February 2018, 12:02 PM
 

Just to clarify my previous answer: the "test code" for which I gave pseudo code has to be run after the student's code. So you either have to put in the extra field (which can have as much code as you like in it) or in the template after the {{ STUDENT_ANSWER }} line. Putting it in the template, and using the 'expected' field just to contain the expected answer (with no newline on the end) gives you something like:

{{ TEST.testcode }};
format free
{{ STUDENT_ANSWER }}

if ~exist('y')
    disp('You haven''t assigned a value to y');
else
    if ~isequal(y, {{ TEST.expected}})
        disp(y)  % Will appear in the 'Got' column
    else
        disp('{{ TEST.expected }}')  % Will be marked correct
        % or disp(y) if there are no rounding or formatting issues
    end
end

Richard

Picture of Angus Wallace
Re: Matlab
by Angus Wallace - Saturday, 10 February 2018, 2:17 PM
 
That’s really helpful, thanks Richard. I’ll have a play with this and see what I can work out.

Cheers, Angus
Picture of Angus Wallace
Re: Matlab
by Angus Wallace - Monday, 12 February 2018, 3:15 PM
 

Hi Richard,

Thanks for your help so far -- your comments about the extra code in particular is very useful!

I have tried to go to string matching and I'm having a lot of trouble. I think it may be due to the formatting of the strings being different. I'm trying to generate the XML from within Matlab so that I can have many variants of the questions to present to the students, so I'm trying to calculate/capture output into a string in Matlab and then write that into the XML. I've tried a few ways of doing this, and what I've got now is:

evalc('disp((0:10:90)+sliceNum)');

where the resulting string is written into the XML as the expected answer, viz.

<expected>
                <text>    96    86    76    66    56    46    36    26    16     6

</text>
      </expected>


but, as you can see, there is an additional line being written.. I think that might be preventing the answer from being properly graded. What I'm not clear about is the exact formatting of the XML that I need to provide so that the answers will match. Is this documented anywhere? (ie. spaces before/after each number, tabs, newline characters, etc, for horizontal arrays, vertical arrays and matrices)


One other thing -- I was unable to see 'adaptive mode' anywhere -- where can I find this?

Thanks again!

-Angus

Picture of Angus Wallace
Re: Matlab
by Angus Wallace - Monday, 12 February 2018, 6:24 PM
 

Hi Richard,

I've worked it out.

I found the 'adaptive mode' under Question behaviour > How questions behave in the quiz configuration.

I was right -- it was the formatting. I needed to print the results in Matlab to better-match what was generated by Octave. In Matlab, I printed thus:

sprintf(' %d', (0:10:90)+sliceNum)

which puts exactly one space before every number.

I have one other question ;-) I'm generating many variants of each question, but I only want to present each student with one variant. Is there a way to do this? (I thought there was, but I'm now having difficult making it work)

Cheers, Angus

Picture of Richard Lobb
Re: Matlab
by Richard Lobb - Monday, 12 February 2018, 9:30 PM
 

Hi Angus

It sounds like you're making good progress.

If white-space is an issue you might want to consider changing the base question type you're working with to use "Near exact match" grading instead of the default "Exact match". The meanings of the different grading options, which are available only when you customise a question and of course in the prototype, are documented in line - click the '?' icon beside the 'Grading' field. 

If you want to present a student with a randomly selected variant from the questions you're generating, you can use Moodle's Random question type. Put all your generated CodeRunner questions questions in the same category and insert into the quiz a Random question that draws from that category.

Cheers

RIchard

Picture of Angus Wallace
Re: Matlab
by Angus Wallace - Monday, 12 February 2018, 10:08 PM
 

One of my colleagues very kindly showed me how to do random subsets of the questions. Ok.. Onward!


Cheers, Angus