Python unittest

Python unittest

by Peter Sander -
Number of replies: 7

Hello,

In my course I'm trying to brainwash the students into integrating unit testing into their code development toolkit. CodeRunner is a fantastic tool for testing student code, and now I'm trying to use it to test code that tests student code.

Here's a ridiculously minimal example to illustrate an issue that I'm having:

import unittest

class Person:
    pass

class PersonTest(unittest.TestCase):
    def testWhatever(self):
        pass

if __name__ == '__main__':
    unittest.main()

The above code (the student answer) gives the following result:

Testing unittest

The test runs, but where does the ***Error*** come from? (I can't see anything helpful when running with Template Debugging.)

Thanks in advance for helping.

The idea behind all this is to give some of the testing code in the Answer Box Preload, with the students then submitting their code + the unittest-ing code into CR. CR runs the unittest code which tests the student code.

In reply to Peter Sander

Re: Python unittest

by Richard Lobb -

Weird things like this can arise if Python runs out of memory. When you import numpy, for example, you need to raise the memory limit for the job. The limit on the number of processes can also be an issue when running extra library code.

Try customising the question, open the Advanced Customisation panel and set the memory limit to 0 (meaning no limit). If that fixes it, experiment to find a realistic memory limit. If that doesn't fix it, try increasing the process-limit via the sandbox parameters field - see here.

If that doesn't work either, post back and I'll look into it more closely.

In reply to Richard Lobb

Re: Python unittest

by Peter Sander -

Hi Richard,

Thanks for your speedy (as always) reply!

I've set the MemLimit to 0 and the Parameters to {"numprocs": 50}, however I'm still getting that pesky ***Error*** notification...

Curiously, setting the Template Debugging indicates that it's making two seemingly identical Runs. There must be something specific in the way unittest executes.


In reply to Peter Sander

Re: Python unittest

by Peter Sander -

This may (or may not) be relevant. Instead of using the built-in Python3 prototype, I've hacked together a LOCAL_PROTOTYPE_python_checkr to run Python code (just repurposing my LOCAL_PROTOTYPE_java_checkr) to see what would happen. It's almost the same as with the built-in prototype - the unittest runs ok, the ***Error*** warning still appears. The only difference is that there's now just one Run indicated by Template Debugging.

That would seem to point to some unittest strangeness.

Although I don't know where the ***Error*** is coming from - CR or unittest. Can't find any reference to such a warning in unittest.


In reply to Peter Sander

Re: Python unittest

by Peter Sander -

Another piece of evidence: replace the whole template with just the Python code from my initial question above. Test runs correctly, ***Error*** still present with two Runs.

Can't get a simpler example!

In reply to Peter Sander

Re: Python unittest

by Richard Lobb -

Thanks for the extra testing. All is now clear. The unittest module writes all its output to stderr, even when the run is clean. CodeRunner interprets any output to stderr as an error (which it should be, under normal circumstances). I assume the unittest was written that way in order to make it possible to separate its output from that of the code being tested but it's a bit of an abuse of the 'stderr' concept.

To fix it, you'll have to catch stderr output and redirect it to stdout. Depending on what sort of template you want to use, you might be able to do this by setting sys.stderr to a StringIO object and printing its contents at the end of the run, or you might have to run the entire task using the subprocess module, redirecting stderr to stdout. Or maybe the unittest module provides a solution itself?

Post back if you need more help. Note that there's a 12 hour time difference between NZ and Germany, giving us a rather long communication lag time.

Richard


In reply to Richard Lobb

Re: Python unittest

by Peter Sander -

That sounds about right. Will check it out tomorrow, getting a bit late now.


In reply to Peter Sander

Re: Python unittest

by Peter Sander -

Bingo! Intercepting the output from unittest does the trick. And I have to diddle that output anyway to neutralize the displayed time since the test run times would vary between users. Here's my quick hack:

if __name__ == '__main__':
    # find all tests in this module
    import __main__
    suite = unittest.TestLoader().loadTestsFromModule(__main__)
    with io.StringIO() as buf:
        # run the tests
        with contextlib.redirect_stdout(buf):
            unittest.TextTestRunner(stream=buf).run(suite)
        # process the results - neutralise the time taken
        # since this can vary depending on the phases of the moon
        result = buf.getvalue()
        start = result.find(' in ') + len(' in ')
        end = result.find('s', start, len(result))
        result = result.replace(result[start:end], 'X.XXX')
        print(result)
It needs re-packaging into a more useful form. What's a bit curious is that it redirects stdout (also works with stderr), but it works.

Thanks a bunch for your quick and insightful help!