Ah, with only 2 cores the situation becomes a bit clearer. I'm not seeing a memory size in the above specs - are you perhaps light on memory as well? My guess at what has been happening in your testsubmit runs is that 10 jsimultaneous jobs (each with a 2 second wait inserted to ensure they don't finish quickly) has overloaded your 2-core server. I've just read that the "resource temporarily unavailable" can be issued when a fork fails due to lack of memory as well as due to hitting the 'numprocs' limit.
The first thing you should do is edit <jobehome>/application/config/config.php. Change the value of 'jobe_max_users' from 8 to 2, i.e. edit the line to
$config['jobe_max_users'] = 2
Re-run the installer (<jobehome>/install).
Now rerun the testsubmit program. What happens?
I've just changed the Jobe code to set the default value of the 'jobe_max_users' config parameter to the number of CPUs rather than the current default of 10. The change will go out on the next push.
You say your course of 170 students had sporadic error messages. What were the error messages? If a free Jobe user can't be allocated to run the job within 10 seconds, you should get a quite explicit message that the Jobe server has overloaded. However, if the server overloads prior to the jobe_max_users limit being reached - which seems to be what is happening with your testsubmit runs - you'll probably get more inexplicable errors.
You ask how many students a 2-core Jobe server can handle. I really can't answer that question as it depends on how much memory and CPU time a typical student run takes and the rate at which jobs are being submitted to the server. You can do a back-of-the-envelope calculation like the following (after measuring how long a typical student job takes):
total_cpu_time_required = num_students * time_per_run * num_coderunner_questions_in_quiz * num_tries_per_question
Compare that with:
total_cpu_time_available = quiz_duration * num_cores
The latter should be many times larger than the former in order to allow for the bursty nature of student submissions (usually the start of a timed quiz is the busiest) and for the Apache and communication overheads.
But that's still a very crude calculation and can be complicated by things like excess regradings at the end of a timed quiz (if all students finish together). You really need to measure what's happing. I usually run top on our jobe servers during a test or exam: the instantaneous load factor should stay below about half the number of CPUs for comfort.
I'd recommend you upgrade to an 8-core Jobe server as soon as possible, regardless of the above calculations.