I couldnt find this info anywhere - what kind of hardware would you recommend for the jobe server in terms of disk space and cores?
The answer depends on what languages you're using, the tasks that you're setting the students and the likely submission rate (e.g. how many students are using it under exam-type pressure at any one time). So I can only tell you about our own servers; perhaps other users can report on their own experience?
We use CodeRunner with lots of languages; Matlab is the biggest resource hog, with Java a distant second. Nowadays we use Octave rather than Matlab, so I don't have recent data on Matlab.
Our main production server is a VM on a Blade server; it's configured for 8 CPUs and 16GB RAM. Even running a Java exam with 250 students this server is only lightly loaded - the load factor rarely exceeds 1, and since it has 8 CPUs I wouldn't even be getting interested until it exceeded 10. I like it to be over-configured rather than under-configured, but I'd still be quite comfortable if it only had 4 CPUs and 8 GB RAM. Alternatively, I'd be happy to run exams with at least 500 students on the current system and I suspect even 1000 would be fine if we had that many seats.
The amount of disk space isn't very relevant as nothing much is saved on Jobe during grading except the resource files that question authors add (usually negligible). So you just need enough to install all the languages you want to run plus a bit. 20 GB should be ample, unless you want to run Matlab.
There's a load-tester, loadtesting2.py, which I use for testing the server before major tests and exams. It simulates a set of students repeatedly logging into Moodle and submitting an answer to a trivial question. With 20 simulated students throwing Python submissions at the server as fast as they can (waiting for a response to each before resubmitting), our Moodle + Jobe system can handle a steady rate of over 10 submissions per second, but the bottleneck is Moodle not Jobe. At 12 submissions per second, the Moodle server load factor gets up to around 22 but Jobe only gets to around 3.
The Moodle server has the same configuration as Jobe, so a rough rule of thumb might be that the Jobe server needs at most half the CPUs and half the RAM of the Moodle server.
One more data point for you: jobe2.canterbury.ac.nz is the outward-facing Jobe server used by: coderunner.org.nz, people installing coderunner, several users who have requested a key without limits and by a popular on-line Java textbook. It has only 2 CPUs and 4 GB RAM. AFAIK it never gets overloaded, because it doesn't run intensive tests/exams. [But having said that, I don't monitor it closely, so I wouldn't really know.]
Richard
Thanks for the elaborate answer, that helps a lot :)
We are just starting out with coderunners this semester, so I'm not sure yet how many teachers will end up actually using it, but we will probably not have more than around 20 students in an exam at once, so I'm going to go with similar specs to your outward-facing Jobe server.