Wow, you've had a busy day while I was asleep!
That's exactly the sort of thing I would have recommended if you hadn't beaten me to it. Well done!
The files initially loaded into the working directory are all write protected for security, so that student code can't corrupt question-author-supplied files. So indeed you needed to create yourself a separate directory first.
My Java's pretty rusty, but it all seems OK to me. The documentation for -d says the directory must already exists but you haven't created the 'dist' directory explicitly. I assume the documentation is out of date?
The code assumes the main class is called Main. Another option might be to assume that if the student provides a class in the answer box, then that is the main class?
If you haven't already done so, I suggest making that into a new question type. Then, over time, you add template parameters to add functionality, such as checking for required constructs in student code, style checks, etc.
I assume you've discovered that you can set the Ace language separately from the template language, so students get Java syntax colouring?
Something to consider for the future is whether you could profit from a combinator template. You've written a per-test template, and that's how the JavaProgram question type usually behaves anyway; by default, even with combinator templates, CodeRunner drops back to one-test-at-a-time mode when each test has its own standard input. But that means you edit and recompile the student code for each test. If you instead used a combinator template with the Allow multiple studins box checked, too, you could get by with just a single compile. Assume the compile works, you would then loop to run each test, setting up standard input from the TEST.stdin field. You need to insert the SEPARATOR string between outputs.
However, that's probably more complexity than you want right now, and all it does is speed up testing a bit.