CodeRunner: Testing raising of Exceptions in Python

I want to test if a function raises a specific exception. For example, this silly example:

def myfun(a):
    if a == 1:
        raise TypeError("wrong")
    if a == 2:
        raise ValueError("wrong too")

Now I want to test if the correct Exception Type and message was thrown for different inputs.

My idea was to change the template to this:

{{ STUDENT_ANSWER }}

__student_answer__ = """{{ STUDENT_ANSWER | e('py') }}"""

SEPARATOR = "#<ab@17943918#@>#"

{% for TEST in TESTCASES %}
try:
    # ??? How to indent the code here correctly ???
    {{ TEST.testcode }}
except Exception as e:
    n = type(e).__name__
    # Put here all Exception Names we expect to handle
    if n not in ['TypeError', 'ValueError']:
        raise e from None
    else:
        print(repr(e))
{% if not loop.last %}
print(SEPARATOR)
{% endif %}
{% endfor %}

I.e., to check if a exception is thrown, which we want and then simply print repr of the exception and compare that.
I wondered, however, if this is a sane idea or would produce any problems? Is there any better way to do it?

One Problem I already found is, that the TEST.testcode must not have more than one line, because the indentation will be wrong... (Which is for this type of question not a big problem).
Another problem could be, that if the student uses the wrong type of exception, it will crash instead of showing the test as failed. I could change the code to always print repr of the exception, but that would mean that for example, certain exceptions where I want a detailed error log, would just print the exception name and message... But I guess that cannot be avoided?

Re: Testing raising of Exceptions in Python

por Paul McKeown - miércoles, 6 de agosto de 2025, 11:29

Hi,

It looks like you are trying to put your test cases into the template rather than writing them as actual test cases.

For example, using the standard Python3 template, you should could simply use the following test cases. (See attached example question - here).

Test1:
try:
    myfun(1)
except TypeError as e:
    print('Yay you raised a TypeError with message', e)
except Exception as e:
    print('Whoops you raised the wrong type of exception')
print('Carrying on...')

Expected1:
Yay you raised a TypeError with message wrong
Carrying on...


Test2:
try:
    myfun(2)
except ValueError as e:
    print('Yay you raised a ValueError with message', e)
print('Carrying on...')

Expected2:
Yay you raised a ValueError with message - wrong too
Carrying on...


Test3:
# This checks that student isn't just printing
# the expected result by printing something
# different when catching the exception for a=1.
# Do a similar thing for a=2
try:
    myfun(1)
except TypeError as e:
    print('TypeError raised and nothing printed by function')
except Exception as e:
    print('Whoops you raised the wrong type of exception')
print('Carrying on...')

Expected3:
TypeError raised and nothing printed by function
Carrying on...


Test4:
try:
    myfun(100)
except TypeError as e:
    print('TypeError raised and nothing printed by function')
except Exception as e:
    print('Whoops you raised the wrong type of exception')
print('Carrying on...')

Expected4:
Yaya, 100 is not exceptional!
Carrying on...

Note, students might try writing answers that just print the expected output and avoid raising any exceptions. Hence the need for parallel test cases such as Test3 above. If you didn't have cases such as this then students could write code like the following and pass the tests.

def myfun(a):
    """ Exceptionally poor exception raising! """
    if a == 1:
        print("Yay you raised a TypeError with message wrong")
    elif a == 2:
        print("Yay you raised a ValueError with message wrong too")
    else:
        print(f'Yaya, {a} is not exceptional!')

It's also possible to have multiple catch clauses on your tries so you can do different things with different exceptions. In the above examples I put a catch-all clause after the more specific clause so that I can tell if the wrong type of exception was raised.

Hopefully this is enough to get your rolling again :)

Cheers, Paul

Re: Testing raising of Exceptions in Python

por Sebastian Bachmann - miércoles, 6 de agosto de 2025, 17:54

Thanks - I had it like that before, but I notice that I did not put enough context in my question ;)
The problem with this approach is, that I would spoil the answer to the next question in the test cases... I.e., this question is about raising exceptions, while there is another question about catching them.
So my initial thought was to simply call the function and put the traceback and error message in the expected field, but that does obviously not work, as raised exceptions make all tests fail.

So I got some ideas: I could write my test template in such a way, that it replicates the traceback but does not crash the test. In that case, I hope it is clear what they should expect from calling the function and I can write my checks like this:

try:
   {{ TEST.testcode | replace({'\n': '\n    '}) }}
except Exception as e:
   # print out the exception in a deterministic way for testing...
   pass
else:
   raise YouCheatedTheTestException("function did not raise an exception!")

I think also the inspect module may help me to check the traceback. with {{ TEST.extra }} I also should be able to pin it to a specific exception that should be handled.
Btw, I found out that the replace filter can be used to add the required indentation.

My other idea was to combine raising and catching in one question. For example, by specifying that two functions should be programmed, where one raises the exceptions and the other has to call this function and handle them. However, testing this is even a bit more trickier and would probably involve some monkey patching of the student's answer. And I cannot show separate tests for each function, as it would spoil the answer again.

Re: Testing raising of Exceptions in Python

por Sebastian Bachmann - miércoles, 6 de agosto de 2025, 19:30

I came up now with this template:

{{ STUDENT_ANSWER }}

__student_answer__ = """{{ STUDENT_ANSWER | e('py') }}"""

SEPARATOR = "#<ab@17943918#@>#"

import traceback
import sys

{% for TEST in TESTCASES %}
{% set assert_exception = TEST.extra starts with 'assert_exception' %}
try:
    {{ TEST.testcode | replace({'\n': '\n    '}) }}
except Exception as e:
    exc_type, exc_value, exc_traceback = sys.exc_info()
{% if assert_exception %}
{% set EXC_ARGS = TEST.extra|split(' ') %}
    if exc_type.__name__ != '{{ EXC_ARGS[1] }}':
        raise e from None
{% endif %}
    print("***Laufzeitfehler***")
    print("Traceback (most recent call last):")
    print("  ...")
    print(f"{exc_type.__name__}: {exc_value}")
else:
{% if assert_exception %}
    print("Keine Exception geworfen, aber erwartet!")
{% else %}
    pass
{% endif %}
{% if not loop.last %}
print(SEPARATOR)
{% endif %}
{% endfor %}

The test then looks like this:

It is still a bit sketchy though...

Re: Testing raising of Exceptions in Python

por Paul McKeown - miércoles, 6 de agosto de 2025, 20:02

Hi,

Aha, that makes it a bit trickier...

If you want to hide the exception handling then you could put the actual test calls inside the Extra template data for each test and add this after the testcode. The template would look like the following:

{{ STUDENT_ANSWER }}
__student_answer__ = """{{ STUDENT_ANSWER | e('py') }}"""
SEPARATOR = "##"
{% for TEST in TESTCASES %}
{{ TEST.testcode }}
{{ TEST.extra }}
{% if not loop.last %}
print(SEPARATOR)
{% endif %}
{% endfor %}

And a first test case like the following:

I've attached an example question with such a setup, here.

Hopefully this makes it a bit easier and flexible to write these sorts of questions.

Cheers,

Paul

Re: Testing raising of Exceptions in Python

por Sebastian Bachmann - miércoles, 6 de agosto de 2025, 20:31

Oh well, yes, that makes it much easier! I think I got a bit carried away with the implementation of a good exception checker :D
Thanks!

Best,
Sebastian