Warnings result in a run-error, and run-errors cause the answer to be incorrect

Warnings result in a run-error, and run-errors cause the answer to be incorrect

by Michael Fairbank -
Number of replies: 1

Hi, I noticed something I consider strange on CodeRunner.

We had some Python packages slightly out-of-sync on the Jobe server, and got this message appearing in Moodle, when submitting the student answer through code-runner:

"***Run error*** /usr/local/lib/python3.10/dist-packages/gym/utils/passive_env_checker.py:233: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`. (Deprecated NumPy 1.24)  if not isinstance(terminated, (bool, np.bool8))"

This question was marked through a regular expression check, searching for the word "Success".  The test script still managed to print the word "Success" despite the above warning- so the I think question should have been marked right.  But the above "run error", which only seemed to be a warning, was enough for the question to be marked wrong.

We fixed the problem by upgrading the packages on Jobe.  But as synchronising packages in Python can be tricky sometimes, with all their dependencies.  Importing libraries such as TensorFlow always seem to bring up a million warnings.  With this in mind, should the above warning ever have resulted in a "run error", and should the "run error" have stopped the regular expression matching from being run and marking the question as correct?

In reply to Michael Fairbank

Re: Warnings result in a run-error, and run-errors cause the answer to be incorrect

by Richard Lobb -
When a job is run on Jobe, any output to the stderror stream is deemed to be a run time error. This is universal rule that works pretty well across all languages and question types. There's no generic way to determine if some particular stderr output is something that can reasonably be ignored or not. So the behaviour you describe is fundamental to all the basic built-in question types.

However, you can write your own question types that run the student code in a subprocess, capturing both stdout and stderr output and selectively discard stderr output if it is, say, just warnings.