Issues with the "Check" Button

Issues with the "Check" Button

por James Bryan -
Número de respuestas: 10

Hi Everyone,

I have been setting up a quiz with coderunner questions to familiarize myself with CodeRunner, however, I have been unable to get the Check box to work correctly.

After creating the question and going into preview mode, with 1 test case, completing the question and pressing "Check" gives the intended result each time. However, whenever I add a 2nd test case, I get the error message:

Expected 2 test results, got 1. Perhaps excessive output or error in question?

I have attached images of each section of the question set up below, as I simply have no idea as to what I am doing incorrectly.

Question Type:


General:


Test Case 1:


Test Case 2:


Question Preview Options:


If you require any more information to solve the issue, please ask.


Thank you

Regards

James

En respuesta a James Bryan

Re: Issues with the "Check" Button

por Jenny Harlow -

Richard is off tramping until the end of the week.  I've not seen that error before.  I tried creating the same question using the settings as you showed in your attachments (including having 'all or nothing grading' unchecked).  I got no problems but could be using a different version of CodeRunner to you, and/or have some other crucial difference.  Follow-up questions/suggestions:

(1) You have not edited the provided python3 prototype in any way? 
(2) What version of CodeRunner are you using? (sub-questions - is it a new install?)
(3) What answer you are using for that question?  You could show it in another screen dump.
(4) Try checking the 'Template debugging' check box (top section of the question editing form) and then saving and previewing and clicking 'Check' again - it shows what is happening one layer under the hood.  Again, screendump it.

Jenny


En respuesta a Jenny Harlow

Re: Issues with the "Check" Button

por James Bryan -
Hi Jenny, 


1) I have not edited the provided python3 prototype.

2) I pulled and installed the latest version from github on the 23rd of December

3)  Answer below:


4) This is the output when I have "template debugging" selected


Thank you

Regards

James

En respuesta a James Bryan

Re: Issues with the "Check" Button

por Jenny Harlow -

Okay. I think I have been able to replicate the problem and if I am on the right track then there is a problem with the python3 prototype BUT that has been around for over a year and although I think I can see why the issue might not have been picked up, it does seem odd... Pending Richard's return, I don't want to suggest any messing with the prototype because it still seems more likely that I am wrong than that the prototype has a glitch. 

I have two possible workarounds for you, one of which (if it works) will also add some evidence for my diagnosis.

Workaround (1).  Check the 'customise' checkbox (top section again).  The customise section will open up.  The only thing to change is the entry in the test splitter box.  Change this to

|#<ab@17943918#@>#\n|ms

(copy and paste from here might be best although the only change is actually replacing the space before |ms at the end with \n )

Save and preview and check again. If that works then I think my diagnosis is right, but you'll have to make the same tweak on all experimental questions you try which is not ideal.

Workaround(2)

Don't do any of the customise stuff above, just put a single space into the standard input box for your first test case.  This makes the way the question is graded behave differently, so each test is run separately.  On my testing that dealt with the problem I think may be occuring.  Again you may have to do that for each experimental question you try ... out of interest, do you recall removing any spaces from the test case standard input from the default new question form?  When I create a new question it automatically has a space in which may be masking the problem behaviour, but my version of CodeRunner is slightly different to yours.

Of course I may just be chasing shadows...

Jenny

En respuesta a Jenny Harlow

Re: Issues with the "Check" Button

por James Bryan -

Hi Jenny,

When I went to implement Workaround 1, I found that the "Test Splitter (regex)" textbox already contained the entry you suggested, however, the "SEPERATOR" variable in the template did not contain the preceding '|' or the succeeding '\n|ms', as displayed below.


Workaround 2 fixed the issue. When I checked test case 1 and test case 2, I found that neither of the "Standard Input" sections contained a space. Once I inputted a space, the issue was resolved. When I added new test cases, the space was not already inputted.

Thank you.


Am I correct in saying that, when a student presses the "Check" box while attempting a CodeRunner quiz question, the results box with test cases should be displayed informing them as to whether their answer is correct or not?

While the check box now works as intended in preview mode, when I log in as a student user, answer the question and press the "Check" box, the answers section is not being displayed, even though both test cases have been set to "Show", leaving the student user with no indication as to whether their answer is correct or incorrect. The code is being sent to the Jobe server, compiled and ran, as when the answers are submitted, the correct marks are being awarded on both correct and incorrect answers.

Once the "Finish Attempt" button has been selected and the quiz review has begun, the "results" box and the test cases are all displayed as expected.


If the answers box should be displayed for a student when they select the "Check" box during the quiz, is there any screenshots or information I can provide to aid in solving the issue?

Thank you for your help. It is greatly appreciated.

Regards

James

En respuesta a James Bryan

Re: Issues with the "Check" Button

por Jenny Harlow -
If workaround 2 has helped for the preview Check that is good but my guess about what might be wrong was clearly incorrect so we'll have to leave Richard to try to help with what's going wrong with the basic prototype...

For the behaviour of the question from a student point of view, I assume that you are viewing the question as a question in a quiz, and in that case it sounds as though you may not have the actual Quiz set to Adaptive mode.  Have a look at the Quiz settings, Question Behaviour section, and specifically the 'How questions behave' box.  Make sure that is set to Adaptive Mode (just plain Adaptive Mode to be able to see how the penalty system works).  Save the settings and try again. 

As I understand it, for all Moodle question types, the quiz settings will determine the question behaviour (adaptive, etc) when the questions are used in a quiz.  The CodeRunner questions show in adaptive mode when the individual question is previewed (eg, by the author), but it is the quiz settings that matter for the behaviour from the student point of view.

Let us know if that fixes things.

Jenny
En respuesta a Jenny Harlow

Re: Issues with the "Check" Button

por James Bryan -

Hi Jenny,

I changed the behaviour of my original quiz to adaptive mode, however the results box still did not show when I pressed "Check" as a student user, so I did a couple of tests.

Test 1)

I created a duplicate of the quiz, and set the quiz mode to adaptive mode during the quiz creation process. I then added the question, tested the quiz and the results box was displayed when I pressed "Check" as a student user.

Test 2)

After this I created another duplicate of the quiz. I left the quiz mode as "deferred" during the creation process and added my question. Before any user answered the question, I changed the quiz mode to adaptive mode. The results box was not displayed when I pressed "Check" as a student user.

Test 3)

I created another quiz duplicate, left the quiz mode as "deferred" during the creation process. Immediately after creating the quiz I changed the quiz mode to "adaptive". I then added my question. the results box was not displayed when I pressed "Check" as a student user.

From this testing, changing the "Quiz Mode" from "Deferred" to "Adaptive Mode" outside of the Quiz set up process does not appear to actually affect the quiz, resulting in a student user being unable to see the results box when pressing the "Check" button.

Thank you for your help. It is greatly appreciated.

Regards

James

En respuesta a James Bryan

Re: Issues with the "Check" Button

por Jenny Harlow -

Hi Jason

I have investigated and discovered that there is another aspect of the quiz settings that I should have also told you to change on your original quiz, and that is under the 'Review options' - the 'During the Attempt' list.  As well as having the 'Question behaviour' as Adaptive you also needed to have (at least) these two boxes checked under 'Review Options' - 'During the attempt':

- Marks
- Specific Feedback

Here are the relevant bits of the settings I use for my CodeRunner Quizzes:


Image of some quiz settings

(The 'Whether correct' checkbox is not necessary for getting the results table for a CodeRunner question in particular, BUT if you have other question types mixed in, I think it may matter.  I always have it checked for adaptive quizzes.)

The experience you had trying to change a quiz to adaptive (failing to get adaptive feedback) and then setting up a new quiz as adaptive and getting the appropriate feedback is (I think) due to Moodle trying to be helpful.  When you set up the new quiz and set the question behaviour to adaptive from the start, Moodle automatically gave some appropriate settings to the 'Review Options' including those two crucial 'During the Attempt' options.  However, when you originally set up a quiz with some other question behaviour (deferred?) Moodle also provided some suitable 'Review Options' for that and these do not automatically get changed if you subsequently change the Question Behaviour setting (eg from deferred to adaptive).  I have just replicated that sequence - start with deferred, save, then change to adaptive, and indeed I had to change those review options to get what I expect when I click the Check button on a quiz question. 

If you still have version 1 of the quiz you should be able to edit the settings to include those two checkboxes.

As far as I know the default settings for a new quiz are controlled at the site admin level so if it is your own install I guess you could alter them so that any new quiz automatically has the appropriate core settings.  It is very well worth working out what you need for different types of quizzes - I also work on a course that uses non-CodeRunner quizzes and the quiz settings are completely different for that course.

Jenny

En respuesta a Jenny Harlow

Re: Issues with the "Check" Button

por Richard Lobb -

Hi Jason

I'm back from my holiday. Jenny and I have discussed this baffling problem of yours at length, and we may at last have come up with a theory that explains it. Are you perhaps using a Moodle system running on a Windows server rather than a Linux server?

Regardless of whether or not your Moodle is Windows-based, please try the following experiment. Take out those extra spaces in the standard input field for the test case(s) (Jenny's cunning workaround #2), and make sure the question now fails again as it did originally. Then click the "Customise" checkbox, and change the value of the "Test splitter (regex)" field to

|#<ab@17943918#@>#\R|ms

i.e., replace the \n with \R. Save the changed question. Note that if you make changes to the question, you need to open a new preview window to test the changes or you need to edit the answer in some trivial way such as adding an extra space to the end of a line, in order to defeat the automatic caching of run results.

If that solves the problem, please then try changing it back to

|#<ab@17943918#@>#\n|ms
and see if the problem returns. [My guess is it won't.]

If the first change doesn't solve the problem, please instead try changing the field to

|#.*#\R|ms
and let us know what happens.

Sorry about the problems and thank you for your patience.

Richard

En respuesta a Richard Lobb

Re: Issues with the "Check" Button

por James Bryan -
Hi Richard and Jenny,


Happy New Year!


I am running Moodle on a Windows server.

I implemented the change you suggested Richard, and that worked. Before, I got the 

Expected 2 test results, got 1. Perhaps excessive output or error in question?

error message, then after I changed the "Test splitter (regex)" field to 

|#<ab@17943918#@>#\R|ms

and back, the error did not appear and the answers were being marked correctly.


For the changes Jenny suggested, I made a new quiz, initially set it to "Deferred" feedback, added my question, changed the Question Behaviour to "Adaptive Mode" and then, as you suggested, selected the "Marks" and "Specific Feedback" options in the Review Options. I removed the whitespace in the "standard input box" that was previously suggested. The quiz question worked just as intended, no errors were being produced and the tests and marks section was displayed perfectly.

Thank you both, so much, for helping me resolve this.

Kindest Regards

James

En respuesta a James Bryan

Re: Issues with the "Check" Button

por Richard Lobb -

Hi James

Excellent news and Happy New Year to you too!

It looks like the problem that Jenny picked up with the python3 prototype only affects Windows-based Moodle servers. These are fairly rare which is why we haven't seen the issue before.

You should now implement the fix (changing the regex to use \R rather than \n - changing it back again is strictly optional) in the prototype for the python3 question type, which is the question named BUILT_IN_PROTOTYPE_python3 in the System > CR_PROTOTYPES category. You need to be a Moodle admin to do this. Once that's done, python3 questions should work out of the box, without the need to customise each one. The same fix will be needed for the python2,  java_method and octave_function question types if you use any of them. I don't think it will be needed for any of the other built-in question types, but please let me know if I'm wrong.

I'll fix the issue for the next release. Many thanks for your patience and help in finding the problem. Fingers crossed that there are no other bugs affecting Windows-based Moodle servers.

Richard