I think your best approach here is to switch to using a question type scripted in Python, in which you take charge of the whole compile-and-run process. The documentation section on
Implementing or supporting new languages points the way forward, using C as an example.
The template code in the documentation includes the lines
return_code = subprocess.call("gcc {0} -o prog prog.c".format(cflags).split())
if return_code != 0:
print("** Compilation failed. Testing aborted **", file=sys.stderr)
You would need to replace those lines with code that compiles the Java program and analyses the result to check if it's just a preview warning; if so, accept the compilation as valid. This means you'll want to use the more-general subprocess.run rather than subprocess.call, capturing the stdout and stderr for analysis.
Another example of this approach to compiling and running other languages is the built-in multilanguage question type, which runs C, C++, Java or Python. You could mostly use the template code for that question type, stripping out all the non-Java stuff and tweaking the few lines of code that do the compilation. To see the template, start to create a new CodeRunner question, choose Multilanuage as the CodeRunner question type and click the Customise checkbox to see the template code. Use that instead of the C compile-and-run code in the above-linked documentation section.
One caveat: both the documentation and the multilanguage question type use a per-test-case template grader. However, if you're going to be making extensive use of this approach with Java (which has big compile overheads) you'd probably want a combinator template to save recompiling on each testcase. But I suggest using the per-test-case approach for a start, as it's easier.