Question Authors' Forum

Invalid questions not saved

Picture of Michel Wermelinger
Invalid questions not saved
by Michel Wermelinger - Friday, 23 December 2016, 4:07 AM

I was writing a question + answer and kept saving it as I worked. The answer was not passing the pylint test but I still saved it, expecting to be able to come back later to fix it. However, when I returned to the list of questions, my question wasn't there. A colleague of mine also got the same behaviour (but without using pylint). 

It therefore seems that a question with an invalid answer is not saved, in spite of what the button says.  Is it possible to give some warning to the author? Maybe the save button should be disabled in such cases, or a popup should appear saying that the question was not saved because the answer doesn't pass the tests.

Picture of Richard Lobb
Re: Invalid questions not saved
by Richard Lobb - Friday, 23 December 2016, 10:51 AM

Hi Michel

Welcome to CodeRunner; sorry about the frustrations. [I should explain to anyone else reading this that Michel is at UK Open University, where they're running the latest development version of CodeRunner, a.k.a. the version 3.1 Release Candidate. This version runs the sample answer, if given, as part of the form validation process. Earlier versions didn't do that, requiring authors to manually check that their sample answer passed the tests.]

When Moodle questions are saved, the question type plug-in's form validation method is called. It returns a list of errors associated with different parts of the form. Any errors abort the save and instead the form is re-displayed, with errors messages shown in red at the appropriate places in the form. So what you're seeing is the standard Moodle question-save behaviour. However, a local user has also complained about it, so I think it's clear that with a complex question type like CodeRunner the standard behaviour is no longer appropriate. The question is: what to do about it?

I'm quite sure that having the ability to check the sample answer while saving is a step forward (it was done at the request of the community), but it looks like the way it's handed needs to be improved. I'd like to involve Tim Hunt in the discussion, assuming he's tuned in, as some of the approaches require changes to the Moodle question engine rather than to CodeRunner.

Here are some possibilities:

  1. There's a "Validate on Save" checkbox associated with the sample answer. It currently defaults to "Yes". A trivial change would be to set the default to "No". Perhaps we should do this anyway, Tim, until we have a better UI solution?
  2. If a save is aborted due to a validation failure, the re-displayed form should be scrolled so that the first error message, in red, is in the window. AFAIK, this isn't something I can control from within CodeRunner, however.
  3. As Michel suggests, I could pop up a dialogue box whenever a Save is aborted, warning the user. [I think I've seen something in the Moodle API that allows PHP code to trigger an Alert box, though I've never used it. I'll look into that.]
  4. The recently-added Moodle standard "Save changes and continue editing" button is useful but is also, I think, part of the problem. When there was just a Save button, either the Save worked, taking you back into the question bank, or it failed, taking you back into the form. This gave a fairly clear indication of success or failure. But the new button takes you back into the form regardless, so you don't actually know if the Save succeeded. The previous 2 suggestions would help a lot here, but I wonder if it's even appropriate to be doing form validation when the author has clearly signalled that the question is incomplete, anyway? Should the framework be changed to bypass form validation in this case?
As a workaround until we've come up with a better solution, Michel, I suggest that you hold off entering a sample answer until you've got the rest of the question written or that you uncheck the Validate On Save box prior to every non-final Save.

Thanks for the feedback. 


Tim at Lone Pine Koala Sanctuary
Re: Invalid questions not saved
by Tim Hunt - Tuesday, 10 January 2017, 7:22 AM

Sorry for the slow reply. I had a nice long break over Christmas.

1. In the long term, I think this would be a step in the wrong direction. May be a necessary short-term measure.

2. would require a Moodle core change, so would not be a quick win. Might be a good idea thought. Should probably apply to all Moodle forms.

3. alert pop-ups are just annoying.

I have an alternate suggestio, which is: as well as adding a specific error to the various places in the form. If any error occurs, we should also add an error message to the first form field (that seems to be Question type - I don't think you can add error messages above headings) saying "Question not saved. There are problem with the values in fields: Answer"

Picture of Richard Lobb
Re: Invalid questions not saved
by Richard Lobb - Tuesday, 10 January 2017, 10:04 AM

Thanks Tim. You'll see that for now I've run with (1), but the status of "ValidateOnSave" is recorded in the database. This seems to work well in practice. A question author can use "Save and continue editing" repeatedly until the question is in its final state. Then they can turn on validation, save the question, and validation is on thereafter. The worst thing that can happen is that an author forgets to turn on validation and finishes with an invalid question or answer, for which they've only themselves to blame. With validation on by default, authors could lose entire questions that they'd been working on, which was much more annoying and was perceived as my fault. I'll think about changing the default when the current system has bedded in.

I do like your alternate suggestion, though. I'll look into that.