CodeRunner: Queuing up long running questions

First off: thank you for this tool - it looks great for many of the use cases we are seeing with our courses. I have just started working with it, but so far the installation and the local docker of the Jobe server went off perfectly.

I am trying to understand if there is some way to handle very long running submissions. In particular, what if the code submitted by a student takes 5-10 minutes (or more) to compile and evaluate? The specific example I am looking at is for hardware synthesis - users submit code in Verilog which gets put through a synthesis tool and is evaluated to see if it meets timing constraints. I can script all of this, but the compilation itself can easily take several minutes.

As of now, (provided I increase the timeout) it looks as though once I submit my answer, the page basically hangs there waiting for the result. Is it possible to have the page say "submission accepted, check back later for the result" or something like that?

In other words, that would probably make it more like an assignment evaluator rather than a quiz. Is this feasible with coderunner?

Apologies if this has been answered before - I have looked through the past discussions on this and the developer forum but couldn't find anything that fits.

Thanks,

Nitin

Re: Queuing up long running questions

por Richard Lobb - sábado, 25 de enero de 2020, 20:40

Interesting question. I've never had to deal with such long run-times myself.

CodeRunner was originally designed for use in laboratories with large classes. It has since moved into many other application areas including assignments, tests and final exams. But the goal has always been to provide a fast turn-around, where students submit their code and are rewarded (or not) with nice green ticks within a few seconds. Your use case doesn't fit that model.

You can increase the timeout within the question, although the default upper-limit, set by the Jobe server, is only 50 seconds. You can in turn raise that by editing the file

<JOBE_HOME>/application/config/config.php

and raising the constant cputime_upper_limit_secs. But you're right that the page where the student submits code will still hang around, waiting for the response to the RESTful POST request. The browser might time out and, additionally, long turn-around times would have major loading implications for the Jobe server with even moderate-sized classes.

While the original Jobe REST API allows for the possibility of a deferred execution and subsequent polling for the result, I've never actually implemented it. And if I had, I don't know how I could make use of it within the Moodle quiz-question architecture.

I don't have an easy solution for you. However, some users have written CodeRunner questions where the student is required to submit their code to a completely separate repository, such as gitlab, where their code is automatically compiled and run. When that is complete, students are then required to submit the URL of their repo submission to CodeRunner which then grades the state of the code in the remote repo. I haven't done this myself, but there's some discussion of that idea and other related possibilities in this thread.

Note that if you want the Jobe server to be able to access remote repos you'll need to open holes in its firewall.

Re: Queuing up long running questions

por Nitin Chandrachoodan - domingo, 26 de enero de 2020, 01:58

Thanks, I sort of figured that this may not be the default use case.

The thread you linked to has some nice pointers - will follow up on those as well.

One other low overhead possibility I am looking at is to just allow students to submit code to an assignment, and then use the Jobe sandboxing to run the code and get results - of course with timeouts changed.

In any case, coderunner has opened up the possibility of having many "micro" assignments that I had not thought of before. Thanks!

Nitin