CodeRunner: Inquiry about Jobe Server Configuration for Optimal CPU Utilization

We have recently deployed a standalone Jobe server with the following specifications:

CPU: Dual Intel Xeon E5-2680v4 processors (28 cores, 56 threads total)
RAM: 64GB DDR4
Storage: SSD storage
Network: Gigabit Ethernet

The Jobe server is integrated with our Moodle learning management system, where we've configured the Jobe server address in the CodeRunner settings. Basic code submission and grading are working correctly.

However, we've encountered a performance issue during bulk re-grading of submissions. When we initiate re-grading for multiple assignments, the process takes significantly longer than expected. During these operations, we've observed that:

The Jobe server's CPU utilization remains relatively low (many of them are 0%)
Not all CPU cores are actively engaged in processing tasks
The re-grading process appears to be processing tasks sequentially rather than in parallel

Our current configuration includes:

Moodle correctly configured to communicate with the Jobe server
Basic testing with testsubmit.py shows good performance for individual submissions

We would greatly appreciate your guidance on:

What specific Jobe configuration parameters should we adjust to maximize parallel processing capabilities?
Are there any server-level optimizations (Apache, PHP, system settings) that would improve concurrent task handling?
What monitoring approaches do you recommend to identify bottlenecks in our setup?
Are there known limitations in the current Jobe architecture that might explain this behavior?

The server resources appear to be underutilized, and we believe there's potential for significantly better performance given our hardware capabilities.

root@jobe:/var/www/html/jobe# python3 testsubmit.py
Supported languages:
c: 13.3.0
cpp: 13.3.0
java: 21.0.5
nodejs: 18.19.1
octave: 8.4.0
pascal: 3.2.2
php: 8.3.6
python3: 3.12.3

Valid Python3 OK
Python3 with stdin OK
Syntactically invalid Python3 OK
Python3 runtime error OK
Python3 file I/O OK
Testing use of interpreter args with Python3 OK
Testing use of runargs args with Python3 OK
Python3 program with customised timeout OK
Python3 program with support files OK
Valid Python3/pylint program OK
Invalid Python3/pylint program OK
UTF-8 output from Python3 (will fail unless Jobe set up for UTF-8) OK
Test good C hello world OK
Test compile error C hello world OK
Test use of compileargs with C OK
Test runtime error C hello world OK
Test timelimit on C OK
Test outputlimit on C OK
Memory limit exceeded in C (seg faults) OK
Infinite recursion (stack error) on C OK
C program controlled forking OK
A C program with ASCII non-UTF-8-compatible output OK
Valid Octave OK
octave with stdin OK
Syntactically invalid Octave (treated as runtime error) OK
Syntactically valid Nodejs hello world OK
Syntactically invalid Nodejs OK
Correct Php program OK
Syntactically incorrect Php program OK
Correct Java program OK
Correct Java program without supplied sourcefilename OK
Syntactically incorrect Java program OK
Java program with a support class (.java) OK
Java program with Unicode output (will fail unless Jobe set up for UTF-8) OK
Test good C++ hello world OK
Test compile error C++ hello world OK
Good Hello world Pascal test OK
Fail Hello world Pascal test OK

38 tests, 38 passed, 0 failed, 0 exceptions

Checking parallel submissions in C
Test good C hello world OK
Test good C hello world OK
Test good C hello world OK
Test good C hello world OK
Test good C hello world OK
Test good C hello world OK
Test good C hello world OK
Test good C hello world OK
Test good C hello world OK
Test good C hello world OK
All done

Testing a submission with an excessive cputime parameter
OK
root@jobe:/var/www/html/jobe#
root@jobe:/var/www/html/jobe# python3 testsubmit.py --perf c
Measuring performance in c
1 parallel submits: OK. 4 jobs/sec
2 parallel submits: OK. 9 jobs/sec
4 parallel submits: OK. 20 jobs/sec
8 parallel submits: OK. 41 jobs/sec
16 parallel submits: OK. 44 jobs/sec
32 parallel submits: OK. 58 jobs/sec
64 parallel submits: OK. 61 jobs/sec
128 parallel submits: OK. 90 jobs/sec
256 parallel submits: OK. 144 jobs/sec
512 parallel submits: OK. 115 jobs/sec
1024 parallel submits: OK. 139 jobs/sec
2048 parallel submits: FAIL.
1536 parallel submits: OK. 146 jobs/sec
1792 parallel submits: OK. 155 jobs/sec
1920 parallel submits: FAIL.
1856 parallel submits: OK. 148 jobs/sec
1888 parallel submits: FAIL.
1872 parallel submits: OK. 149 jobs/sec
1880 parallel submits: FAIL.
1876 parallel submits: OK. 150 jobs/sec

Re: Inquiry about Jobe Server Configuration for Optimal CPU Utilization

by Richard Lobb - Tuesday, 25 November 2025, 8:47 PM

There are two separate issues here:

How to tune Jobe to make best use of your high-performance Jobe server?
How to improve the performance of bulk-regrades?

With 28 cores you should be able to improve the performance significantly by raising the value of $jobe_max_users in jobe/app/Config/Jobe.php. It defaults to 16 so that at most 16 jobs can be running at once. I suggest trying 28 then 56, running testsubmit with --perf and monitoring the load factor and memory usage as well as the maximum throughput. Typical job size may be relevant here - you don't want to run yourself out of memory through having too many jobs running, though that shouldn't be a problem in C. You could try with different languages as well. Java is problematic, since the JavaVM tries to gobble up all available resources when it starts. The community has added extra tuning parameters for Java but I've never explored them myself. Except for Java, the $jobe_max_users parameter is pretty much the only thing you can usefully tweak, so just experiment.

Bulk regrading can certainly be a problem. However, CodeRunner doesn't have any say in how regrading runs - it's just a question type and the Moodle core handles regrades, calling CodeRunner's grade_response function for each submission to be graded. As you observe, regrading is done with a single process, doing questions sequentially. There's no direct way to run multiple threads, unless there's new functionality I haven't noticed yet.

But some ways to mitigate the problem:

Turn on the Grade Cache in the CodeRunner settings. This caches the results of Jobe runs, so if you're regarding a question that hasn't been changed, the job never even gets sent to Jobe. This gives you a dramatic speedup in such cases. Management of the cache (i.e. selective purging if it consumes too much disk) is via moodle/question/type/coderunner/cachepurgeindex.php.
If regrading multiple assignments, do each one as a separate request, so they run in parallel.
When initiating a regrade of a single assessment item, make sure to select only the questions that have been changed.
Initiate multiple regradings of selected subsets of the students, using the "Regrade selected" option. You will have to initiate each regrade from a different incognito browser window. This essentially runs all your regrades in parallel. Caveat: I haven't tried this with a high level of parallelism but it should work.

As an aside: your profile says you're in Christchurch, which is where I live. So I assume that's not correct. Where are you, in fact?

Re: Inquiry about Jobe Server Configuration for Optimal CPU Utilization

by bi hex - Wednesday, 26 November 2025, 6:24 AM

Performance Comparison Table (All Configurations)
Configuration Max Burst Handling Sustained Throughput Peak Throughput Rate
56 users 243 jobs 22 jobs/sec 19 jobs/sec
50 users 256 jobs 21 jobs/sec 20 jobs/sec
48 users 256 jobs 22 jobs/sec 18-19 jobs/sec
45 users 256 jobs 22 jobs/sec 19 jobs/sec
After several rounds of performance testing, we have ultimately selected the maximum users setting of 45 as the final parameter.
Thank you for your valuable assistance throughout this tuning process.

Re: Inquiry about Jobe Server Configuration for Optimal CPU Utilization

by Tim Hunt - Tuesday, 25 November 2025, 10:24 PM

Acutally, the problem you describe is not a Jobe tuning issue. As you say "The re-grading process appears to be processing tasks sequentially rather than in parallel". It appears that way becuase that is how the Moodle code is written. When you choose to regrade some quiz attempts, it does a simple loop over those attempts, processing them one at a time.

In your case, where you want the entire regrade to complete as fast as possible, that is not the best. (A more modern approach, in terms of Moodle code, might be to queue a whole bunch of ad-hoc tasks, one per attempt, and then they can be processed in parallel.

However, there are up-sides in the way Moodle works. To give just one example.

Yesterday, some other teacher, runs their big CodeRunner quiz, which consumes a significant proportion of the Jobe server capacity.
Today, you are running your big CodeRunner quiz, which requires a significant proportion of the Jobe server capacity to run reliably.
But, unknown to you, the other teacher is reviewing his exam results today, and spots the need for a regrade.

If that regrade tries to run as fast as possible, using as much Jobe capacity as possible, that could go very badly. At least with Moodle's current implementation, the regrade can only take up one bit of the Jobe capacity.

I realise that is not a perfect answer, but I am just saying, that there are plusses and minuses to any change.

Re: Inquiry about Jobe Server Configuration for Optimal CPU Utilization

by bi hex - Wednesday, 26 November 2025, 6:27 AM

Thank you again for sharing your expertise and providing the broader context around these system design decisions. It's helpful to understand both the technical considerations and the practical realities of managing shared educational resources.

Best regards,