Test cases which allow multiple types of Numpy arrays

Test cases which allow multiple types of Numpy arrays

by Peter Bratby -
Number of replies: 4

I am authoring the following question related to Numpy arrays.

"Write a function even_array(n) which returns a Numpy array containing even numbers 0 to 2n inclusive."

My specimen answer is

import numpy as np

def even_array(n):
    return np.arange(0, n*2 + 2, 2)

which returns an array of floats. However, the student might choose to answer the question in a different way such as

import numpy as np

def even_array(n):
    return np.array([2*i for i in range(n+1)])

This answer is also correct, but returns an array of integers.

A test case print(even_array(2)) with expected output [0. 2. 4] would pass the first answer but fail the second.

I could change the test case to the following:

print(even_array(2).astype(float))

but I think this could be confusing to the students, who won't have been introduced to different types of Numpy array at this point.

My proposed solution is to customise the template as follows, which also prevents the function returning something of type other than Numpy array (for example a Python list):

{{ STUDENT_ANSWER }}

__student_answer__ = """{{ STUDENT_ANSWER | e('py') }}"""

SEPARATOR = "#<ab@17943918#@>#"

import numpy as np
{% for TEST in TESTCASES %}
result = {{ TEST.testcode }}
if isinstance(result, np.ndarray):
    print(result.astype(float))
else:
    print("The function does not return a Numpy array.")
{% if not loop.last %}
print(SEPARATOR)
{% endif %}
{% endfor %}

I'd welcome any opinions on whether this is a sound solution, or whether there is a simpler or better way to make this work. I'm a novice Coderunner user so it is quite possible I've missed something obvious!


In reply to Peter Bratby

Re: Test cases which allow multiple types of Numpy arrays

by Richard Lobb -

Our approach is a bit different. Our students have already done several weeks of Python, so are expected to know the distinction between ints and floats. So we hammer home the difference between np.array([1, 2, 3]) and np.array([1.0, 2.0, 3.0]) from the very start. And we tell them about dtype=float and dtype=int very early as well.

Thus we would ask "Write a function even_array(n) which returns a Numpy array containing even integers 0 to 2n inclusive."  I would personally prefer to disallow floats as the answer, as technically a float can't be even or odd. And in fact, I think np.arange(0, 2 * n + 2, 2) will return an array of ints unless n is a float, won't it? 

However, sometimes you do have to accept both float and int answers, in which case some sort of ploy like you've used is necessary. In that case your solution is fine, although hiding code from students can cause confusion, as the output they get when they run their code doesn't exactly match what CodeRunner appears to be producing.

If we do want to add hidden test code, our in-house question types allow us to prefix (or suffix) TEST.testcode with TEST.extra. We can then put any hidden code in TEST.extra which is more flexible and saves us having to customise questions. 

In reply to Richard Lobb

Re: Test cases which allow multiple types of Numpy arrays

by Peter Bratby -
Thanks Richard, this is really useful. I suspect that the optimum solution can only be determined after testing this on real students, and fortunately I will have a formative opportunity (on a different module) in Term 1 before setting a proper exam during Term 2.

My students are scientists, not computer scientists or engineers, and so my philosophy is to get the students programming 'real' applications as quickly as possible, if necessary skipping non-essential details (such as the types of numpy arrays). Perhaps this is something I can revisit.

I can't immediately see how i would adapt my solution to using TEST.extra. Is it possible to view the in-house question types somehow?
In reply to Peter Bratby

Re: Test cases which allow multiple types of Numpy arrays

by Richard Lobb -
All our in-house question types are available on github here. However, I doubt they're what you want right now, as they're extremely complex with minimal documentation.

I attach an xml export of an implemention of your even_array question. It looks like this (displaying the integers as floats):

Screenshot of even_array question.

Behind the scenes what's happening here is:

  1. The template has been changed to a useful general-purpose form that runs the contents of the globalextra field before running each test. Further each test runs the contents of its TEST.extra field before each test. 
    {{ STUDENT_ANSWER }}

        SEPARATOR = "#<ab@17943918#@>#"
        {{ QUESTION.globalextra }}
        {% for TEST in TESTCASES %}
        {{ TEST.extra }}
        {{ TEST.testcode }}
        {% if not loop.last %}
        print(SEPARATOR)
        {% endif %}
        {% endfor %}
  2. I've set the globalextra field to Python code that redefines the print function to convert any parameters that are numpy arrays to floats. It's best to use globalextra for this rather than TEST.extra to avoid repeating it for every test. 
    import numpy as __np__
__saved_print__ = print
def print(*args, **kwargs):
    newargs = [arg.astype(float) if isinstance(arg, __np__.ndarray) else arg for arg in args]
    __saved_print__(*newargs, **kwargs)

However, with this approach, be prepared to field questions from students asking why the coderunner output displays all the numbers with a decimal point after them, whereas when they test their own code there are no decimal points.

In reply to Richard Lobb

Re: Test cases which allow multiple types of Numpy arrays

by Peter Bratby -
Thanks, Richard. This makes perfect sense now.

Based on this discussion, and using the 'keep it simple for now' principle, I think I will go with your first suggestion (i.e. only accepting a specific array type). I'll understand better once real students start using it!