memory exhausted when compiling regular expressions

Re: memory exhausted when compiling regular expressions

de către Richard Lobb-
Număr de răspunsuri: 9
Your program compiles and runs fine on a couple of my servers.

The default memory allocation for a C++ compile is 500 MB and for the subsequent run is 200 MB. I experimented a bit and I can compile that program with virtual memory reduced to 10 MB. Something is broken badly somewhere.

What's the output if you run the following C++ program from within CodeRunner?

#include <iostream>

int main() {
    system("bash -c 'g++ --version; ulimit -a'");
    return 0;
}

 

On my system the output is

g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) 20000
pending signals                 (-i) 63673
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) 6
max user processes              (-u) 21
virtual memory          (kbytes, -v) 200000
file locks                      (-x) unlimited

 

Also ... have you had any other issues running C++ programs? Does the problem only occur when compiling? Does it compile and run OK if the regex is a local variable of main?

Ca răspuns la Richard Lobb

Re: memory exhausted when compiling regular expressions

de către Henning Koehler-
My setup generates the following output:

g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) 20000
pending signals (-i) 31475
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 8192
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) 6
max user processes (-u) 21
virtual memory (kbytes, -v) 200000
file locks (-x) unlimited

Compiler and OS seem a bit older, but otherwise seems the same (except for pending signals - not sure what that is). Other C++ programs run just fine, and any use of regex variables (whether local or global) causes the compiler error above.

Your suggestion did give me an idea how to do further testing though with the following code:

#include <cstdlib>
int main() {
    system("bash -c 'echo \"#include<regex>\nint main() { std::regex foo(\\\"bar\\\"); }\"'");
    system("bash -c 'echo \"#include<regex>\nint main() { std::regex foo(\\\"bar\\\"); }\" | g++ -x c++ -'");
}


and it seems that it's an issue with the g++ compiler itself rather than CodeRunner:



Forwarded this to our IT department to sort out now, thanks for the help.
Ca răspuns la Henning Koehler

Re: memory exhausted when compiling regular expressions

de către Richard Lobb-
Interesting, but I fear that your IT department will come back at you, pointing out that the code runs fine if you don't use CodeRunner :-(

There's a related discussion here: https://coderunner.org.nz/mod/forum/discuss.php?d=492. Again the code they were using ran fine on our University of Canterbury outward-facing Jobe server. I don't think we ever figured out why their Jobe server was giving such errors. Note that their problem went away as soon as the memory limit for the job was set to 0, turning off memory limits altogether. You might want to try that on your system too, though I certainly don't recommend it for production use, as there's then no protection against run-away memory gobbling tasks.
 
This link might be worth trying too - it changes the g++ garbage collector's behaviour: https://stackoverflow.com/questions/75391757/compiling-using-g-error-virtual-memory-exhausted-cannot-allocate-memory.
 
It's odd that I can run your code fine on our systems. There is clearly some difference in how your Jobe server is set up, but what? Can you ask your IT department how many cores and how much memory your Jobe server has, please? Strange things have happened occasionally on Jobe servers with only 1 or 2 cores and also on Jobe servers with lots of cores. The latter case (lots of cores) can create issues with parallelised software that attempts to allocate worker threads proportional to the number of cores; an attempt to create an excessive number of workers can cause a fork failure which some software will mis-diagnose as a memory limit having been reached.
 
When/if you do figure out the problem, please post back here for the benefit of anyone else who hits it.
 
Richard
 
 
Ca răspuns la Richard Lobb

Re: memory exhausted when compiling regular expressions

de către Henning Koehler-
Changing the g++ garbage collector behaviour seems to do the trick, i.e. the following compiles fine:

#include
int main() {
system("bash -c 'echo \"#include\nint main() { std::regex foo(\\\"bar\\\"); }\"'");
system("bash -c 'echo \"#include\nint main() { std::regex foo(\\\"bar\\\"); }\" | g++ --param ggc-min-expand=10 -x c++ -'");
}

Will see what the IT department comes back with. I assume the Jobe server is using a sandbox environment with its own compilers?
Ca răspuns la Henning Koehler

Re: memory exhausted when compiling regular expressions

de către Richard Lobb-
The Jobe server uses a very old version of a restrictive environment called runguard, from the domjudge programming contest server. It uses all the standard Linux compilers and other applications, but restricts the environment by a combination of ulimit settings, environment variable clensing and a watchdog timer. So when you run g++ from the terminal you're running the same version as when the jobs arrive via CodeRunner, except for the additional restrictions.

If the ggc-min-expand parameter solves your problem and you need a workaround, you can add that parameter to any question (including to the prototype) via Customise > Advanced customisation > Sandbox > Parameters . It will be something like

{"compileargs": ["--param ggc-min-expand=10"]}

However, I'd really like to know what the fundamental problem is.
Ca răspuns la Richard Lobb

Re: memory exhausted when compiling regular expressions

de către Henning Koehler-
It seems another options that gets it to work is to set MemLimit to 50 MB - which is interesting as based on the previous output I would have thought it to be unlimited by default. IT is now investigating how to set this globally (rather than on a per-question basis).


Ca răspuns la Henning Koehler

Re: memory exhausted when compiling regular expressions

de către Richard Lobb-
That's very interesting, thanks. Also very strange. That value is used to set the ulimit for the task. The default memory limit during compile should be 500 MB, dropping to 200 MB during run. And the test you did earlier, showing the ulimit output for all parameters, indeed showed it was set to 200 MB. It looks like making it smaller is solving the problem. Let me know if you can cast any further light on this.
 
BTW: if you apply that setting to the prototype for you C++ questions, it will apply to all questions of that type. You don't have to set it on each question.
Ca răspuns la Richard Lobb

Re: memory exhausted when compiling regular expressions

de către Henning Koehler-
Mostly speculation, but it sounds like the C++ compiler (when dealing with regular expressions) tries to use a lot of memory, up to the amount it thinks is available (set by MemLimit). This can then exceed the virtual memory used by the sandbox, causing the error. With MemLimit set to something smaller than 200MB the compiler will restrain itself and the error goes away. That would match up with gcc-min-expand fixing the issue. The compiler generates a lot of garbage-collectable memory, but it thinks there's 500MB available in total so garbage collection doesn't kick in early enough to prevent the out-of-memory error.
Ca răspuns la Henning Koehler

Re: memory exhausted when compiling regular expressions

de către Henning Koehler-
Actually, it seems like the IT department might have made some changes since I tested: works now without any customization, with the following settings:

g++ (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) 20000
pending signals (-i) 31468
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) 6
max user processes (-u) 31
virtual memory (kbytes, -v) 500000
file locks (-x) unlimited

I've tried with even bigger virtual memory (realized that this is what corresponds to MemLimit) up to 2GB and it still works. Best guess they implemented the gcc-min-expand solution - I'll let you know if I find out.
Ca răspuns la Henning Koehler

Re: memory exhausted when compiling regular expressions

de către Richard Lobb-
It's sounding to me like the Jobe server originally was simply light on physical memory. A quick web search suggests that if a Linux task tries to use more than 50% physical RAM + swap space it will get a memory error. I'm guessing your IT guys just upped the amount of physical memory.