Hi Carl
If you're setting "write a function" type questions you can put code in the template to dump or inspect the output file after the student's code has been run. But it sounds like you're asking "write a program" questions, and that's a bit harder.
My normal approach in trickier cases like this is to use a Python question in which the Ace editor is set for C++ but the execution language is python3. The template is then a Python3 script that compiles and executes the student code and then (in your case) either dumps the student output file to stdout or runs a file comparator of some sort. It might even just run Linux diff. Something like the following:
import subprocess
COMPILE = "g++ -Wall -Werror -o program program.cpp".split()
RUN = "./program"
f = open("program.cpp", "w")
f.write(__student_answer__)
f.close()
try:
subprocess.check_output(COMPILE, stderr=subprocess.STDOUT, universal_newlines=True)
except subprocess.CalledProcessError as e:
print("Error compiling program", file=sys.stderr)
print(e.output, file=sys.stderr)
else:
try:
subprocess.check_call(RUN, stderr=subprocess.STDOUT, universal_newlines=True)
# Code to check the output file goes here. The simplest version might
# be something like:
print("After running program, output file blah.txt was:")
print(open("blah.txt").read())
except subprocess.CalledProcessError as e:
print("Error running program", file=sys.stderr)
print(e.output, file=sys.stderr)
A variant on this might be to change the RUN command to something like "./program; diff expected.txt blah.txt" and run the subprocess in the shell (with the shell=True argument to check_call). [Caveat: I've not actually tried doing this myself.]
You could make this into a more generic question type by having a template parameter like POST_RUN, which defines what happens after the student's code has been run.
One slight disadvantage of this approach is that compile errors show up as Runtime Errors, because any output to stderr within a run is interpreted as a run error. But in practice this doesn't seem to cause much confusion as the compile errors are obvious (and, in the mode in which we run, shouldn't occur anyway, as students are expected to compile and test their code before submitting it).
Richard