Question Authors' Forum

Hi Anton

Thanks for the interesting post.

Java is the only language with built-in question types that I haven't personally used in teaching, because we changed from Java to Python for first-year teaching about the time I started developing CodeRunner. The three built-in question types (write-a-method, write-a-class and write-a-method) were just ones that I thought might be useful. Had I tried using them in teaching I would have undoubtedly come up with extra and/or different ones.

Now that I look more closely, I think your comment that the 'write-a-method' test could be misleading is valid. The template for that question type is

public class Main {

    {{ STUDENT_ANSWER }}

    public static void main(String[] args) {
        Main main = new Main();
        main.runTests();
    }

    public void runTests() {
        {{ TEST.testcode }};
    }
}

but if the student doesn't understand the context, the test does indeed look anomalous, as it appears to be calling a global function, something that's not possible in Java.

You could instead modify the runTests method in the template to:

public class Main {

    {{ STUDENT_ANSWER }}

    public static void main(String[] args) {
        Main main = new Main();
        main.runTests();
    }

    public void runTests() {
        System.out.println({{ TEST.testcode }});
    }
}

Does that suit you better? 

I'll consider adding that into the distribution as a new alternative Java write-a-method question type. However, although it looks more logical in the above test, it is a bit more restrictive in that the test case now must be a single expression rather than a sequence of one or more Java statements. With the existing question type, you could instead write the tests in multiple lines, e.g.

int[] data = {10, 17, -3};
int total = sumSquares(data);
System.out.println(total);

which might perhaps be simpler for learners, with the caveat that they still need to understand that the test code is wrapped as a method in the same class as their method declaration. [Edit: I belatedly realise that writing the tests like that with the built-in Java method question type doesn't quite work, because its combinator template combines all the tests into a single method. That gives syntax errors due to the variables being redeclared. So to make that form of test work with Java you either have to customise the question and turn off the combinator template or make a new question type like the built-in one with the combinator disabled. Edit to Edit: actually, the combinator template can be easily changed to allow tests like this, just by wrapping each test case in braces, so it runs in a separate independent block. ]

A further restriction on that new format of question is that you can't test void functions using it, whereas you can test them with the existing write-a-method question type just by dropping the call to System.out.println from each test.

You're right of course that there is an degree of fuzziness in the conversion of a method return value to a string. However, you have to assume some conversion in order to give the student an example of the value that their method should return, as in "Your function should return the value 23". Passing the return value as a parameter to println is the easiest conversion but it's over to the question author to choose whatever conversion suits them. Occasionally we do add extra test cases (usually hidden ones, or at least ones that display only if they fail) that check the type of the value returned by the method/function.

You raise the question of checking parameter names. This is part of the thorny question of how you check style. In Python we use the pylint program as a pre-processor - if a student submission fails pylint checks it's not even tested. However, no program can check if variable names are "good" or not, so ultimately you still have to have humans check code at some stage in a course.

I've added to this site an article that a colleague, Jenny Harlow, and I have written on CodeRunner, to appear in ACM Inroads. See here.  It reports on our experiences teaching with CodeRunner and discusses some of the issues with style checking.

-- Richard