CodeRunner: "Connection Reset" error from outside network

Hi,

I'm running into a weird, and really problematic issue.

I installed Moodle + CodeRunner on a VM on the internal network of the CS Dept. When the students access the quizzes in the lab, within the internal network, everything works fine.

However, when students or anyone tries to make the quiz from outside, we get "Connection reset" errors, either when pressing "Check", or when trying to go to the next questions in the quiz...

Could this be related to the fact that I haven't configured a mail account for Moodle?

Checking the server logs I see errors like:

Database transaction aborted automatically in /var/www/html/moodle/mod/quiz/processattempt.php, referer: http://lambda.inf.ucv.cl/moodle/mod/quiz/attempt.php?attempt=253&page=

Re: "Connection Reset" error from outside network

de Ismael Figueroa - miércoles, 7 de septiembre de 2016, 04:04

I'm still checking the logs, in access.log I see:

01.214.121.123 - - [06/Sep/2016:12:58:15 -0300] "POST /moodle/mod/quiz/processattempt.php HTTP/1.1" 404 36831 "http://lambda.inf.ucv.cl/moodle/mod/quiz/attempt.php?attempt=314&page=8" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"
::1 - - [06/Sep/2016:12:58:18 -0300] "GET /jobe/index.php/restapi/languages HTTP/1.1" 200 648 "-" "MoodleBot/1.0"
::1 - - [06/Sep/2016:12:58:18 -0300] "GET /jobe/index.php/restapi/languages HTTP/1.1" 200 648 "-" "MoodleBot/1.0"
::1 - - [06/Sep/2016:12:58:18 -0300] "POST /jobe/index.php/restapi/runs HTTP/1.1" 200 726 "-" "MoodleBot/1.0"

While in error.log I see:

[Tue Sep 06 12:58:15.070331 2016] [:error] [pid 21623] [client 201.214.121.123:57304] Default exception handler: Ha introducido datos fuera de la secuencia normal. Esto puede ocurrir si utiliza los botones Atr\xc3\xa1s o Adelante de su navegador; por favor no los utilice durante la prueba. Tambi\xc3\xa9n puede ocurrir si hace clic sobre algo mientras se carga una p\xc3\xa1gina. Haga clic en <strong>Continuar</strong> para <strong>seguir.</strong> Debug: \nError code: submissionoutofsequencefriendlymessage\n* line 2165 of /mod/quiz/attemptlib.php: moodle_exception thrown\n* line 88 of /mod/quiz/processattempt.php: call to quiz_attempt->process_attempt()\n, referer: http://lambda.inf.ucv.cl/moodle/mod/quiz/attempt.php?attempt=314&page=8

Re: "Connection Reset" error from outside network

de Richard Lobb - miércoles, 7 de septiembre de 2016, 10:01

Hi Ismael

I haven't seen this problem myself. Connection resets from out-of-sequence browser requests are common enough when you use the browser Back button or if you have multiple sessions open to the server from within the same browser, but I've never seen them in normal quiz operations.

Are you sure they're CodeRunner related? What happens if you have a quiz containing only standard Moodle question types (multichoice etc) - do you still get this problem?

Richard

Re: "Connection Reset" error from outside network

de Ismael Figueroa - miércoles, 7 de septiembre de 2016, 12:08

I'm not quite sure this has to do with CodeRunner itself, but what is weird is the different behavior between internal university network vs external network.

I'm suspecting it may have to do with the communication to the JOBE server, which is in the same host (I know, bad practice). Why? Because the connection reset error also appears when pressing "Check". But maybe I'm wrong as perhaps that button makes a call to "processattempt" too...

So far I isolated the issue to the following error message:

Default exception handler: A required parameter (attempt) was missing Debug: \nError code: missingparam\n* line 495 of /lib/setuplib.php: moodle_exception thrown\n* line 552 of /lib/moodlelib.php: call to print_error()\n* line 39 of /mod/quiz/processattempt.php: call to required_param()\n

I'm sadly disappointed from this experience, as I really want CodeRunner to succeed in our school, so any help would be appreciated!

Re: "Connection Reset" error from outside network

de Richard Lobb - miércoles, 7 de septiembre de 2016, 13:44

I'm pretty sure this problem can be solved, so don't panic (yet)!

Certainly a click on 'Check' does a full Moodle quiz attempt submission. I'm inclined to doubt this is CodeRunner related; please do try the test with non-CodeRunner questions. If it turns out to happen without CodeRunner as well, then we can start looking elsewhere.

Richard

Re: "Connection Reset" error from outside network

de Tim Hunt - miércoles, 7 de septiembre de 2016, 18:22

Like Richard, I think an important diagnostic step is to see what happens with a simple Moodle question type (e.g. multiple choice). That will really help work out where the problem is.

However, if you think about Moodle as a software application running on the server, it really should not care how the bits travel over the network to reach it. I mean, there should be no way that in can even tell whether the request originated from within your internal network on not. The network must be doing something weird with requests coming in from outside before the data gets to Moodle, and you need to work out what it is.

The fact it works from your internal network means that the Moodle + CodeRunner install is working fine, if the network traffic can get to it.

Most of the errors you have found are really symptoms of the problem, not the cause. Problems like this are often linked to how users are logging in, and are most common if you are using some form of single-sign-on system. What happens when a user presses check should be like this:

POST to .../mod/quiz/processattempt.php. What this does is
1. Verify this is a legitimate request by a logged-in user.
2. If so, process the submitted data, to update the state of the question in the database.
3. Issue a redirect to...
GET of .../mod/quiz/attempt.php?attempt=... to display the updated attempt.

What happens if there is a problem with the user's session is:

POST to .../mod/quiz/processattempt.php
1. This incorrectly fails to detect that the user is already logged in with a current session, so issues a redirect to... [Note, this is the moment when you get the "Database transaction aborted automatically" error in the logs.]
GET of .../login/index.php so that the user can log in again.
1. If you are using a single-sign-on system, then page may be able to automatically create a new Moodle session for the user, based on other cookies the user has. If so, there is another redirect to ...
possibly another GET of something like .../login/index.php?testsession=... - this is part of re-establishing the Moodle session. This also redirects to ...
GET of .../mod/quiz/processattempt.php. Once the user is logged back in, Moodle tries to re-direct them back to where they where. However, the original request was a POST, and the redirect can only be a GET, which means that all the post data has been lost. This is what leads to the "Error code: submissionoutofseq" error.

So, as you can see, the real problem is something with your single-sign-on system, and it is not implausible that could behave differently depending on where requests are originating.

Re: "Connection Reset" error from outside network

de Ismael Figueroa - jueves, 8 de septiembre de 2016, 10:27

Thank you Richard and Tim for your feedback.

Regarding Tim's post, I've extensively googled for this issue and came across several of his posts that explain the issue. Thank you for that.

After enabling mod_dumpio in the Apache server, I've managed to, I hope, distill the issue.

This cases consider the same user, quiz and question. What I did was to access the quiz, inside a CodeRunner question, and I pressed the "Check" button. Then I got the "network reset" error, or the successful checking. Regarding your previous questions, the error also appears in non-Code Runner questions.

In the good case (error_ok.log):

Line 1: POST to lambda.inf.ucv.cl/moodle/mod/quiz/processattempt.php?attempt=324&page=2
Then CodeRunner does its job correctly, querying the JOBE server
Line 175: The response to the POST is a 303 See Other with Location: lambda.inf.ucv.cl/moodle/mod/quiz/attempt.php?attempt=324&page2
Line 184: GET lambda.inf.ucv.cl/moodle/mod/quiz/attempt.php?attempt=324&page2
The redirect is followed and everything works OK.

In the bad case (error_bad.log):

Line 1: POST to lambda.inf.ucv.cl/moodle/mod/quiz/processattempt.php?attempt=324&page=2
Then CodeRunner does its job correctly, querying the JOBE server
Line 173: The response to the POST is a 303 See Other with Location: lambda.inf.ucv.cl/moodle/mod/quiz/attempt.php?attempt=324&page2
Then the redirect never happens!! There is no corresponding GET

I attach 2 trimmed logs that pinpoint the situation.

Overall the MoodleSession cookie is kept around, and is not deleted. Also, we are not using a single-sign-on, only manual accounts.

Considering all things together, I think the main difference between the internal and external network is DNS resolution. After some more googling I found that there exist some: "Redirection on CORS with preflight" think, meaning that 303 redirects are not followed.

Now, I have no idea about how to fix this issue... so any further help is greatly appreciated!

Best regards

moodle_logs_connection_reset.zip

Re: "Connection Reset" error from outside network

de Ismael Figueroa - viernes, 9 de septiembre de 2016, 06:40

Ok this is weird :(

I don't know if the error solved itself or what is going on.

First:

I replaced the $CFG->wwwroot with the hardcoded IP address of the server, which is a public IP by the way
Then I tried to replicate the issue, and everything worked, but the URL showed the IP

At this point I was more than happy, I can live with an ugly URL with a hardcoded IP address, but:

Then I followed this advice: https://moodle.org/mod/forum/discuss.php?d=201308 and things were still working
Then I removed the "ServerName lambda.inf.ucv.cl" directive from Apache's 000-default.conf

Things were still working, now with the nice URL.

But then, I added again the ServerName directive, I put $CFG->wwwroot back to 'http://lambda.inf.ucv.cl/moodle' I restarted the apache server, and now the quizzes are still working from the outside

I'm leaving the settings just as they are supposed to be, that is:

$CFG->wwwroot = "http:lambda.inf.ucv.cl/moodle"
But without the ServerName directive.

So I don't know what is going on, perhaps something changed from the university networking side, perhaps I just removed something that was weird and wrong in some of the config files... Anyways, tonight I and several students will try from the external networks of our homes, to see whether the problem persists or not...

Re: "Connection Reset" error from outside network

de Ismael Figueroa - viernes, 9 de septiembre de 2016, 09:20

Forget what I said in the above post.

The issue seems to be related to the OS/Browser combination.

I'm accessing the quiz from a Windows 10 Asus Transformer, and I get the connection reset error, but from the external network only! In this case, changing the $CFG->wwwroot to the ip address did nothing... I get the error both from Google Chrome and from Edge. When I try to do the same with the internal university network, everything works fine.

When I use my Macbook Air, it works in both the external and internal networks. It works in Chrome and Safari too...

Yesterday when I was testing internal vs external network I was also using only the Macbook for the Internal network and the Asus for the external network :(

Still, I don't know how to fix the error.

Cheers

Re: "Connection Reset" error from outside network

de Richard Lobb - viernes, 9 de septiembre de 2016, 09:46

Well, it's good (from my point of view) to know that it's not a CodeRunner-specific problem, anyway. And possibly good that you're apparently getting a better understanding of the problem.

I did look through the logs and it seemed to me that some of the checks/submissions succeeded. [I looked for 303 responses from Apache and then looked to see what happened subsequently. I thought I saw several that worked, though I haven't time to go back and check right now.] If that's the case, then you have somewhat random behaviour and it's highly unlikely that server configuration is the issue. The most likely explanation is then just that you have an unreliable network, with packets being dropped by the firewall or the ISP.

Richard

Re: "Connection Reset" error from outside network

de Ismael Figueroa - domingo, 18 de septiembre de 2016, 10:22

Sadly, the issue still persists. I have several students complaining they can't make the quizzes from their homes... If you have any pointers, I'll be thankful!

Cheers