Validating test code

Validating test code

by Anton Dil -
Number of replies: 2

Greetings Richard,

We ran into an interesting problem with a Java question:

Our solution used a wildcard import, and this meant that some test code we had compiled okay when validating. Minus the wildcard import, the test code didn't compile, so some students who didn't use a wildcard in their answer had to add one for a test case, which felt odd.

I did a simple test:

- Create a java_class type question (which has no imports)
- Add a test case that requires an import, like ArrayList<String> x = new ArrayList<>();
- Don't enter an Answer, but tick the validate box
- It saves fine without complaining. 

Once I add compiling code to the Answer, even commented out (//class X {}) then the issue with the test case is raised. But if you again remove all text from the Answer box and validate it saves fine.

I guess that a template ought to include required imports for the test cases, as we wouldn't really expect the Answer to contain imports for arbitrary tests.  But it was odd that we didn't spot the problem earlier, in spite of validating our solution.

A side question is whether an empty Answer with validate ticked ought to compile in this case - an empty Java source code file is actually compilable, and validating an empty file would give a way of checking whether test cases require imports.

For the time being I've just made a note to self - take care with imports!


In reply to Anton Dil

Re: Validating test code

by Richard Lobb -
Hi Anton

There are really two things going on here.

Firstly, if there is no sample answer, there is no validation run at all. I accept that the validation checkbox should be greyed out or invisible in this case, but it hadn't occurred to me until now that someone might have expected some sort of validation when there was no answer to validate. But I can see that you might see it as somehow validating the template and the test cases.

The other point is more subtle. There certainly can be interactions between author's code and testcases. If test cases need extra code that you don't wish to display to the students, it can often be put into the test-case extra fields or the global-extra field. The question-type template needs to be suitably adjusted to incorporate these, of course. Or you can hard-code the extra imports into the template itself.
In reply to Richard Lobb

Re: Validating test code

by Anton Dil -

Hi Richard, thank you for explaining further. I see what you mean about using the 'extra' fields and how that could be incorporated in a template. That looks like a possible solution, if we can remember to check when it's needed! (Perhaps one should develop tests in another environment first.)