Question Authors' Forum

CodeRunner V3.3.0 released

Picture of Richard Lobb
Re: CodeRunner V3.3.0 released
by Richard Lobb - Thursday, 1 March 2018, 9:01 PM

Thanks Martin. Good to hear.


Picture of Martin Zwerschke
Re: CodeRunner V3.3.0 released
by Martin Zwerschke - Thursday, 10 May 2018, 6:19 AM
Announcement says:
  • A question can be customised to use a different Jobe sandbox from normal, allowing for load balancing and specialist Jobe servers e.g. for a language not supported by the production Jobe server.

I wonder if this is the right way for load balancing. If one fixes a question to one jobe server you can not switch over coderunner to another jobe like before.

I had some outings of or jobe server  recently because of unsuccessful dynDNS updates (my fault, but anyway).

I managed to switch back to my experimental home jobe server but I wonder if it woul be possible to enter more than one jobe server into the coderunner plugin settings ?

If one fails, another can do the job(e)s instead. Perhaps, if more than one of them is running, the load coud be balanced automatically.

In this context  I would like to feature-request a kind of "coderunner availabilty announcement "plugin for Moodle. 

I already had something like this realized with a small javascript inside a normal Moodle announcement.

So my students were able to see if Coderunner is clear and ready to check their answers at all.

But after I set up the firewall on jobe server this can not work any more, for traffic to and from jobe server is blocked save the traffic to the Moodle server.

So client's java script can not do it's test call to ("REST API get_languages()" ) like before.

So the jobe-status has to be checked by the Moodle server instead (inside coderunner php).

Picture of Richard Lobb
Re: CodeRunner V3.3.0 released
by Richard Lobb - Thursday, 10 May 2018, 9:35 AM

Hi Martin

Thanks for the feedback.

You're certainly right that setting a per-question or per-question-type jobe server is not the right way to go about load balancing. I shouldn't have even mentioned it, as it could only ever be useful as an emergency measure and even that's questionable. Being able to set a different Jobe server for a particular question was once requested by one of our staff members and, as it turned out to trivial, I added it.

I think the right way to do load balancing, or at any rate a much better way, is to use some form of DNS load balancing. That's scalable and requires no changes to CodeRunner. Open University in the UK does that. I understand that Round-robin DNS load balancing is trivial (just provide the DNS server with a list of Jobe IP numbers?) but that more elaborate load balancers, which poll the servers to determine their current load and availability, are possible.

Since I'm employed as a teacher not a software developer, I tend to focus my development efforts on things that are of relevance to me and my department. I can't recall an occasion when our production Jobe server has gone down unexpectedly, so providing a "coderunner availabilty announcement" feature doesn't seem like a priority to me. If the Jobe server does go down, students submitting code should get a message saying that the Jobe server appears to be down and to try again later. That's good enough for my purposes. That said, I appreciate that different sites have different needs so if there enough other users out there who would like such a feature, I'm prepared to reconsider. Squeak now, you other users!


Tim at Lone Pine Koala Sanctuary
Re: CodeRunner V3.3.0 released
by Tim Hunt - Thursday, 10 May 2018, 8:33 PM

Yes, at the Open University, our Jobe 'server' is actually two servers sitting behind a load-balancer.

However, I don't think round-robin DNS will work. The problem is files, where one evaluation of a student response may require several Jobe API calls. So, we set our load-balancer to use a bit of session persistence. I don't remember the exact config now, but I could dig it out.

Picture of Richard Lobb
Re: CodeRunner V3.3.0 released
by Richard Lobb - Thursday, 10 May 2018, 9:35 PM

Thanks Tim.

We've not actually used a round-robin DNS here and I'm interested to hear that you think it might not work. I just looked at my code and I would have expected it to work OK when different tests run on different Jobe servers. I don't currently allow students to submit files, only question authors. Files are cached on the Jobe server forever (even with several thousand CodeRunner questions, we only have around 700 files cached on Jobe), so only ever get uploaded once. If Jobe is asked to run a test case for which it doesn't have the required file(s), it returns an error code and Moodle must then upload the missing file(s) and try again.

So if a round robin DNS doesn't work I might have a bug somewhere - please let me know.

This discussion is helpful, because I now realise another downside of allowing students to upload files with their submissions - I'll need to implement a file cache expiry mechanism in Jobe. 


Picture of Martin Zwerschke
Re: CodeRunner V3.3.0 released
by Martin Zwerschke - Thursday, 10 May 2018, 10:31 PM

Thank you for this diskussion. Tim's approach sounds interesting but I don't think it will really work out for me.

The DNS names of my two jobe servers are dynamic because in Germany we usually get a different IP from our providers each time a system connects via DSL. Even if it holds it's connection, our providers usually interrupt and reconnect at midnight giving a new IP.

So we have to update DNS adresses via a service like dynDNS automatically with help of daemons in VDSL router or on the server directly exposed to VDSL connection. If for some reason the dynamic DNS update fails, I can not reach the jobe server any more. It is not the jobe service itself to get stuck. (Though I also run a MaximaPool docker on the same server, that could crash the server some time).

The DNS server is runnig on the service providers side - no way to let it do a round robin.

Our students connect from home as well as from our school's LAN. So twisting DNS on our LAN's DNS server would not solve the problem for home use.

I thought of two our three jobe entries in the coderunner plugin settings. If a connection to primary jobe server fails, coderunner connects to the secondary. This would realize failover instead of  load balancing in the first place. But it could also be used for load balancing, if both servers are up.

For the "Coderunner is working" announcement, I will try to proof in Javascript, if the SSH port of the jobe machine can be reached at all. If no it can show the outage, if yes the chance that coderunner is up is quite high. But of course doing a periodic test call to REST-API from inside Moodle would be a better proof of a working jobe machine.

But being a teacher myself I can understand, that Richard's time for development is limited...

Picture of Richard Lobb
Re: CodeRunner V3.3.0 released
by Richard Lobb - Friday, 11 May 2018, 11:09 PM

I really don't think you should be trying to run Jobe on a server with a dynamic IP. It's fundamentally unreliable.

Have you considered running Jobe on something like an Amazon EC2 instance or a Digital Ocean server? I just brought up Jobe on a T2.micro Amazon EC2 instance (1 CPU, 1 GB RAM) and also on a Digital Ocean 1 CPU 1 GB RAM droplet. In both cases I used Ubuntu 16.04 base systems. I had to use jdk8 not jdk9 but otherwise all went very smoothly. Both run and I also did a bulk test of 1200 Python questions on the Digital Ocean droplet.

The smallest DO droplet (1 CPU 1 GB RAM) costs only $5 per month, which is probably a lot less than the cost of the electricity to run your home server! Such a small server may or may not be sufficient, depending on what load you need to place on it, but surely your institution can afford to support you on a larger Amazon server, DO droplet or equivalent if necessary?


Picture of Martin Zwerschke
Re: CodeRunner V3.3.0 released
by Martin Zwerschke - Saturday, 12 May 2018, 1:35 AM

> surely your institution can afford to support you on a larger Amazon server, DO droplet or equivalent

It's not a matter of costs, I Think. But you don't know German communal burocracy  :-)  

But if i plug in a server more or less to the mains, no one cares. 

My home server is mainly for failover and experimenting. I have another one running in the cellar of our school next to the main electrical power supply connected to a otherwise not used and free VDSL line. This server is really isolated from any other network, so it would do no bigger harm if a hacker break into. This is why I risked to use it several weeks without firewall. Outages occured only the first  time I really switched on the firewall, because I forgot to let pass the dynamic DNS-Updater service.

Until then it was running like a charme. Same for my home server, that was protected by a firewall from the beginning. But it is behind a DSL-Router, that does the dynamic DNS-Update itself and when i copied the ufw settings to the school's server I forgot about the difference.

Of course it would be better to rent a dedicated server with fixed IP. But I think, I leave this as an option if bad realiability really becomes a problem.

Thanks for your hints anyway.