A user emailed me the following question, which I thought I should share here.
I recently ran a test with 270 students using coderunner. I saw somewhere on the coderunner manual page that the specs used at UC some time ago was 4 core Moodle, dual core jobe that handled 300-400 students, so I set that on Google Cloud (where I am running the servers on) but during the test the jobe server got overloaded (100% for the full hour) with many students complaining that the jobe server was not responsive xD (I scheduled resit test for the bunch to resolve this). Do you have different setup for tests?
Anyway, that was my first experience doing a large sized test that didn’t go as well as I hoped. One aspect could be the latency between VM in the google cloud that didn’t help, but maybe I underestimated the workload taken by the jobe server. Straight after the test, the cpu usage dropped to 2~3%. Do you have any recommendations for more fluent running of tests that are large sized? I am planning to use it for the exam as well, and currently I am going to do a stress test with students a couple of days before the exam to find a suitable setting on the google cloud. But if you have any other suggestions, that would be great and I can explore as well J
Also I will be doing some matplotlib stuff so I guess I need higher spec then too!
Thought I’d ask in case you or others you know had similar issues with me.
My response (slightly edited) was:
Thank for for sharing that, and I sorry to hear you had such a bad
experience. Certainly I would like to understand what happened and help
avoid a recurrence.
Firstly, I'm slightly alarmed at what you say the coderunner manual says (or said) about loading. The manual currently states:
A single 4-core Moodle server can handle an average quiz question submission rate of about 60 quiz questions per minute while maintaining a response time of less than about 3 - 4 seconds, assuming the student code itself runs in a fraction of a second. We have run CodeRunner-based exams with nearly 300 students and experienced only light to moderate load factors on an 8-core Moodle server. The Jobe server, which runs student submissions (see below), is even more lightly loaded during such an exam.
I don't state there what the Jobe server specs are: it too is an 8-core machine. Both servers have 16 GB RAM.
It's also no longer true in our environment that the jobe server is more lightly loaded than the moodle server during a Python exam though it's still true for a C exam. Our in-house Python question type includes a pylint run plus a heap of other style checks including a parser run. As a consequence of all that overhead, the cost of a single Python run is much more than if you use a default python3 question type. With our tests now, the Jobe server and the Moodle server are equivalently loaded.
There is more extensive info on Jobe configuration here: https://coderunner.org.nz/mod/forum/discuss.php?d=64
To help me get a handle on the problems you had, can you fill me in on some details, please:
- What language were you testing?
- What question type were you using?
- How many jobe runs on average would you expect per question? [This will usually be 1 for a write-a-function question using a combinator template but will be 1-run per test case for a question that reads from standard input using the default python3 question type.]
- How much memory does your Jobe server have?
- Do you have questions that require significant CPU time, e.g. that do some sort of numerical calculations like root finding, d.e. solving etc?
Certainly you should do some load testing before your next test/exam. The unsupported undocumented load-tester that comes with CodeRunner, loadtesting2.py, might be a starting point but it's a bit tricky to set up. You need to set up a special 'Testing' course on the moodle server; a backup of that course is here: https://github.com/trampgeek/moodle-qtype_coderunner/blob/master/tests/fixtures/loadtesting_pseudocourse_backup_V3.0.0.mbz. Then you will need to change various constants in the source code to match your set up (username, password, path to course, etc). It will probably take you at least a full day to get going. My own load testing results from 2 years back are here.
Follow up
The user advised that his Jobe server was 2 vCPUs with 7.5 GB RAM, running on Google cloud services and he had possibly mis-remembered or mis-read old information such as https://transformingassessment.com/sites/default/files/files/TA_webinar_3_Sep_2014_CodeRunner.pdf. He will do some stress testing and report any interesting results here.