Support files overriden by student input?

Support files overriden by student input?

by Anton Dil -
Number of replies: 8

My question is what happens in this scenario:

I add a support file, say X.java, which is used in testing Y.java

Student provides both X.java and Y.java in their answer.

My impression is that the support file is ignored, or perhaps a .class file is overwritten, but I would appreciate confirmation and understanding of how this works.

Thanks!

In reply to Anton Dil

Re: Support files overriden by student input?

by Richard Lobb -
Author-supplied support files are all write-locked, and cannot be overridden by student-supplied files. Or even by your own template code when the job is run. If the student attempts to use a filename that conflicts with an author's filename, they get a "Disallowed file name" message when the click Check.

We use filenames starting with double underscores for author-supplied support files to minimise any risk of file-name conflicts. However, in Java, where class names are locked to filenames you might instead wish to prescribe a regular expression for the filenames that students can use for their uploaded files to avoid such conflicts.

Richard
In reply to Richard Lobb

Re: Support files overriden by student input?

by Anton Dil -
Thank you for the information Richard, however I am talking about answers (particularly Java classes) posted to the answer box rather than uploaded as files.  Can you elaborate on how that works please?
In reply to Anton Dil

Re: Support files overriden by student input?

by Richard Lobb -
Sorry - I misunderstood.

Java filenames are rather messy. Assuming you're using the java_program question type and have not set an explicit filename (see below) CodeRunner attempts to infer the main class name using a regular expression. [As your posting to the developers' forum points out, this doesn't match all possibilities, and can be fooled, for example, by a commented-out main class. Indeed it's not actually possible to infer the class name with 100% reliability without a full parse.]

The inferred class name, with .java appended, is used as the name for the file to be loaded into the temporary working directory on the Jobe server. Support files are then loaded into the working directory. If any of these has the same name as the already loaded main program, the latter gets silently overwritten, so that when the program is run it is your support file that runs rather than the student-supplied file. I admit that that's not nice and should instead give an explicit error; I've made a note of this in our TODO list.

Or at least the above is what I would expect after looking through the code and seems to be confirmed by a simple test I just did. Do you have any counter-examples?

You can override the default filename inferencing above by explicitly setting a sourcefilename in the Advanced Customisation panel:
Advanced customisation image
However, this is a bit cumbersome and it's generally best to avoid customisation where possible. But it might be a workaround if you're actually encountering problems in practice. Are you? [I don't teach Java myself, but I would have thought such problems were unlikely.]
In reply to Richard Lobb

Re: Support files overriden by student input?

by Anton Dil -
Hi Richard, Sorry for the long pause....

I'm using the java_class question type.

Here's an example of the behavior:

Question:
Write the class Ocean that has a toString() method returning new Fish().toString();

Expected Answer:

class Ocean
{
public String toString()
{
return new Fish().toString();
}
}

Support file: Fish.java looks like this:

class Fish
{
public String toString()
{
return "fish";
}
}

So the expected output from printing toString is 'fish'.

However, the Student's answer is:

class Ocean
{
    public String toString()
    {
        return new Fish().toString();
    }
}
//thinking they also needed to supply Fish!
class Fish
{
    public String toString()
    {
        return "walrus"; //and coming up with their own behaviour
    }
}

The test
Ocean o = new Ocean();
System.out.println(o.toString());
fish
fails, as the output is 'walrus' instead of 'fish'.

I suppose it depends on the template to some extent, which generally looks like this:

{{ STUDENT_ANSWER | replace({'public class ': 'class '}) }}

public class __tester__ {

    public static void main(String[] args) {
        __tester__ main = new __tester__();
        main.runTests();
    }

    public void runTests() {
        {{ TEST.testcode }};
    }
}

So it may be to do with the order in which the files are compiled. I'm suspecting:

1  javac  Fish.java -> Fish.class
2. javac  __tester__.java  -> __tester__.class, Ocean.class, Fish.class, i.e. overwriting the original Fish.class

I don't actually know what order javac compiles files in, but maybe javac is invoked more than once? Or could it be down to the naming of the template file, or dumb luck...

Anyway, this is my experience - the student's .class file overwrites the support file's .class file. (I may have confused things by suggesting that .java files were being overwritten.) Maybe that's a less surprising thing than the student's answer isn't used and a file they can't see (the Support file) is used.

However, it would be good to confirm if the compiler is invoked twice and in the order above, or if something else could happen, because right now I'm suspecting it is just how it worked out on this occasion and in another example the support file could be compiled second.

And to be clear: they're not uploading files, they're pasting text into the answer box.

Thanks!

In reply to Anton Dil

Re: Support files overriden by student input?

by Richard Lobb -
Thanks for the great little example, Anton. The situation is pretty much exactly as you surmise except that there's no explicit 'javac Fish.java' command executed at any stage, because there's no assumption about the nature of support files (are they source files or data files?). So the only point at which Fish.class get created is when __tester__.java gets compiled, making the existence of Fish.java redundant. If the student doesn't provide an inner class Fish, then of course the Java compiler will compile Fish.java in the process of compiling __tester__.java.

If you were to provide Fish.class as a support file instead of Fish.java, and the student were to have an inner class called Fish, then the compilation of the __tester__.java gives a not-very-helpful error message like the following (which occurs because support files are write-protected):

Error compiling Fish

I don't teach Java, and my Java is very rusty, so I'm not sure what to suggest as a solution if this issue is causing you problems. A couple of possible approaches might be:
  1. Include a test case that uses the reflection API to check if the Fish class is an inner class. Something like https://www.geeksforgeeks.org/java-program-to-check-if-a-given-class-is-an-inner-class/
  2. Write a question type that lets you inspect the student's answer, say in Python, before compiling it and running it. There was a long forum discussion many years ago in which Peter Sander developed a much more general question type for Java, though I'm regret to say I haven't really followed up on it as I intended. See https://coderunner.org.nz/mod/forum/discuss.php?d=22
In reply to Richard Lobb

Re: Support files overriden by student input?

by Anton Dil -
Thank you Richard, this makes sense and seems reasonable.

BTW it's not an inner class in this case, as then the .class file would have a different name, like __Tester__$1Fish.class and presumably you could search for that file's existence in "." but I am not worried about this. It makes sense that the test would fail as the student has changed the expected behaviour of a supplied class.

The different outcome of supplying Fish.class is interesting and somewhat tempting.
In reply to Anton Dil

Re: Support files overriden by student input?

by Richard Lobb -
Ah yes - of course. Thanks for clarifying. My Java has 20 years rust on it, and was never all that great to begin with.

However, you could probably still use the reflection API to detect if the correct Fish class were being used, if you added your own "signature" method to your Fish support class (i.e. some weird method present only as an identifier). If that method isn't present you could print a nice clear message to the student. They could probably spoof this if they wanted but it avoid accidental inclusion of Fish. And you could probably prevent spoofing with a bit more ingenuity too. Hiding the actual test case code from public view would be a good start.
In reply to Richard Lobb

Re: Support files overriden by student input?

by Anton Dil -

Yes, thank you for the suggestions.