Developers' Forum

Error while executing java based questions: timelimit exceeded

 
Picture of KiranKumar Rama
Error while executing java based questions: timelimit exceeded
by KiranKumar Rama - Thursday, 26 July 2018, 7:51 AM
 

Hi,

We are facing an issue only when executing java questions as "timelimit exceeded" and below it "command terminated....". Attached screenshot for reference. Can you please suggest a way forward for this issue resolution.

Also are there any options or configurations that can be changed so that jobe server can support 2000 students at a time? we are using 8 core 24 GB ubuntu 16.04 64 bit server installed with Jobe server only. Moodle is deployed in another server with similar configuration. We tried once with 2000 students writing an exam at once, but unfortunately it was not good and hence checking the possibilities to tweak any options for handling such scenario.



Picture of Richard Lobb
Re: Error while executing java based questions: timelimit exceeded
by Richard Lobb - Thursday, 26 July 2018, 11:01 PM
 

You can change the timeout for the entire question type by editing the prototype. Open the Advanced Customisation panel and set the value you want in the TimeLimit (secs) entry field. You're limited to a maximum of 30 seconds by the Jobe server. It's not recommended that you do this with the built-in prototypes because they get completely replaced on every version upgrade; you should create your own version of the prototype first - a "user-defined prototype".

If you want to increase the timelimit for just one question, you will have to click the "Customise" checkbox for that question in order to access the Advanced Customisation panel. Please be aware that questions customised in this way are largely "cut off" from the parent prototype, so that if you change the prototype's template, say, the customised child still continues to use the old template.

Running an exam with 2000 students at once sounds like a tall order. I'm not sure that an 8-core Moodle server could handle that many students, even if the Jobe server were infinitely fast. If your exam is in Java I'd been even more skeptical about how do-able it is.

Factors influencing the load on Jobe are:

  1. The rate at which students are submitting answers. This is the most important factor, of course.
  2. The computational cost of running a typical answer - solving numerical equations, for example, could be arbitrarily expensive. You may also need to consider the rate of submission of bad answers with endless loops. Each such submission runs until the task timeout throws it off the server, chewing up significant Jobe CPU time.
  3. Whether you're using a question type that uses a combinator template to combine all test cases into a single run. If not, the cost of testing an answer is proportional to the number of test cases, with each one requiring a separate compile-and-execute task submission to Jobe.
  4. Whether the question uses standard input; if so, even with a (simple) combinator template, you'll get a separate compile-and-execute task submission for each test case.
  5. Whether students are developing their code directly within CodeRunner or whether they have separate development environments and submit only when they believe their code is working.
  6. Whether you've turned on the Precheck option. Since Prechecks are free, students tend to throw code at the server with gay abandon, and each click on Precheck involves a Jobe submission. I've seen students Precheck a single question 20 or more times.

If you wish to estimate the server requirements I suggest you first estimate the rate of task submissions. For example, 2000 students, a 2 hour exam with 20 questions and an expectation that each student will submit every question exactly once gives you an average submission rate of 6 submissions per second. The peak load will be several times that.

Then login to the Jobe server and use the time command to measure the cost of running a typical task, keeping in mind points 3 and 4 above. A minimal Java compile-and-run on my laptop (Core I7) takes 1.5 seconds of CPU time. If you assume 8 CPUs could run 8 such tasks in parallel (wildly optimistic) you'd have a maximum throughput of 5 jobs/second. This assumes that you have plenty of RAM and I/O bandwidth. Python and C are much cheaper. Running Hello world in Python takes 0.02 secs CPU time on my laptop, compared to 1.5 secs in Java. A C compile-and-run is even faster.  My experience is that with Python and C, it's the Moodle server that's the bottleneck, not the Jobe server. With Java it's almost certainly the Jobe server.

I'll leave the rest of the calculations to you, but those rough estimates suggest that you'd need a minimum of three or four 8-core Jobe servers to run a Java exam. And that's assuming optimal answers to all the above load factors. Also, I'm guessing you'd need around double the capability on your Moodle server. With C or Python, your present Jobe server should be OK but the Moodle server would still need more capacity.

Many sites run multiple Moodle servers and the Open University runs two Jobe servers back to back with a router (or some network device) that distributes jobs to the two servesr. Tim Hunt may be able to give you more details on that and also advise on the maximum question submission rate that a single Moodle server like yours can handle, disregarding Jobe.

Richard

Tim at Lone Pine Koala Sanctuary
Re: Error while executing java based questions: timelimit exceeded
by Tim Hunt - Monday, 30 July 2018, 7:19 PM
 

I can't give much information about running large simultaneous quizzes, because most of our quizzes are open for several days or weeks for students to do at a time that suits them. Therefore, the load is pretty smooth, although there are small peaks just before any deadline. In total, currently our students are answering on average about 2,000 CodeRunner questions per week in total, which is hardly taxing our servers. (It is probably 100,000 to 200,000 across all question types. This is still a small fraction of our sever load, which is mostly students accessing content or interacting in forums.)

We do have multiple Jobe servers behind a load-balancer, but this is mostly for resiliency (so we can upgrade one server at a time without affecting students). Given the above load, both servers average about 1% CPU use. Therefore, we don't really worry about performance testing. We could clearly handle a 10x increase in CodeRunner use with no problems, and any actual growth is likely to be quite gradual. Anyway, scaling-out the Jobe servers is likely to get you a long way.

As Richard says, scaling up Moodle is likely to be the harder part. It is worth getting set up to do load-testing. We use JMeter for that.  It takes a bit of work to get JMeter to attempt a Moodle quiz. You need to grab some values out of the page after starting an attempt and then use them to submit the subsequent pages, but it is perfectly doable. I would start by seeing how far you can scale a quiz comprising multiple choice or true/false questions first, to get a baseline, and only then move to trying with CodeRunner questions.

Picture of KiranKumar Rama
Re: Error while executing java based questions: timelimit exceeded
by KiranKumar Rama - Thursday, 9 August 2018, 5:00 AM
 
Thanks Tim Hunt. I will work on the load testing and will see the behaviour.
On the other hand, as part of short tweaks, based on some forums to increase the performance, i changed the configuration of apache and php to have bit more values for memory, however still the server is not supporting more than 400 users at a time.
Earlier i had similar environment without coderunner based quizes and with same server configuration, but the operating system is Centos6 and moodle (couldnt remember exactly but almost 8 versions back from current one) had a good performance to support almost 600 users on an average.

Does CentOS provides good support to moodle executions than Ubuntu? - just a high level query to consider.
Any thoughts / experience on this front please..
Tim at Lone Pine Koala Sanctuary
Re: Error while executing java based questions: timelimit exceeded
by Tim Hunt - Thursday, 9 August 2018, 7:24 AM
 

Sorry, I don't have any real thoughts.

It can be helpful to use profiing to see what is acutally happening when pages load. See https://docs.moodle.org/dev/Profiling_PHP and https://tjhunt.blogspot.ca/2013/05/performance-testing-moodle.html.