CodeRunner: memory exhausted when compiling regular expressions

I found that even trivial regular expressions used in a C++ program solution cause an out-of-memory error during compilation (local compilation works fine). Is that an issue with CodeRunner setup, the compiler used on the CodeRunner server (set up by my institution, I don't have direct access), or something else? Trying to get a handle on how to address this.

#include <regex> static std::regex foo("bar"); int main() { return 0; } Syntax Error(s) virtual memory exhausted

#include <regex> static std::regex foo("bar"); int main() { return 0; } Syntax Error(s) virtual memory exhausted

Re: memory exhausted when compiling regular expressions

de către Richard Lobb- vineri, 3 martie 2023, 19:51

Your program compiles and runs fine on a couple of my servers.

The default memory allocation for a C++ compile is 500 MB and for the subsequent run is 200 MB. I experimented a bit and I can compile that program with virtual memory reduced to 10 MB. Something is broken badly somewhere.

What's the output if you run the following C++ program from within CodeRunner?

#include <iostream>

int main() {
    system("bash -c 'g++ --version; ulimit -a'");
    return 0;
}

On my system the output is

g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) 20000
pending signals                 (-i) 63673
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 8192
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) 6
max user processes              (-u) 21
virtual memory          (kbytes, -v) 200000
file locks                      (-x) unlimited

Also ... have you had any other issues running C++ programs? Does the problem only occur when compiling? Does it compile and run OK if the regex is a local variable of main?

Re: memory exhausted when compiling regular expressions

de către Henning Koehler- luni, 6 martie 2023, 20:03

My setup generates the following output:

g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) 20000
pending signals (-i) 31475
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 8192
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) 6
max user processes (-u) 21
virtual memory (kbytes, -v) 200000
file locks (-x) unlimited

Compiler and OS seem a bit older, but otherwise seems the same (except for pending signals - not sure what that is). Other C++ programs run just fine, and any use of regex variables (whether local or global) causes the compiler error above.

Your suggestion did give me an idea how to do further testing though with the following code:

#include <cstdlib>

int main() {

system("bash -c 'echo \"#include<regex>\nint main() { std::regex foo(\\\"bar\\\"); }\"'");

system("bash -c 'echo \"#include<regex>\nint main() { std::regex foo(\\\"bar\\\"); }\" | g++ -x c++ -'");

}

and it seems that it's an issue with the g++ compiler itself rather than CodeRunner:

Forwarded this to our IT department to sort out now, thanks for the help.

Re: memory exhausted when compiling regular expressions

de către Richard Lobb- luni, 6 martie 2023, 21:06

Interesting, but I fear that your IT department will come back at you, pointing out that the code runs fine if you don't use CodeRunner :-(

There's a related discussion here: https://coderunner.org.nz/mod/forum/discuss.php?d=492. Again the code they were using ran fine on our University of Canterbury outward-facing Jobe server. I don't think we ever figured out why their Jobe server was giving such errors. Note that their problem went away as soon as the memory limit for the job was set to 0, turning off memory limits altogether. You might want to try that on your system too, though I certainly don't recommend it for production use, as there's then no protection against run-away memory gobbling tasks.

This link might be worth trying too - it changes the g++ garbage collector's behaviour: https://stackoverflow.com/questions/75391757/compiling-using-g-error-virtual-memory-exhausted-cannot-allocate-memory.

It's odd that I can run your code fine on our systems. There is clearly some difference in how your Jobe server is set up, but what? Can you ask your IT department how many cores and how much memory your Jobe server has, please? Strange things have happened occasionally on Jobe servers with only 1 or 2 cores and also on Jobe servers with lots of cores. The latter case (lots of cores) can create issues with parallelised software that attempts to allocate worker threads proportional to the number of cores; an attempt to create an excessive number of workers can cause a fork failure which some software will mis-diagnose as a memory limit having been reached.

When/if you do figure out the problem, please post back here for the benefit of anyone else who hits it.

Richard

Re: memory exhausted when compiling regular expressions

de către Henning Koehler- marți, 7 martie 2023, 00:44

Changing the g++ garbage collector behaviour seems to do the trick, i.e. the following compiles fine:

#include
int main() {
system("bash -c 'echo \"#include\nint main() { std::regex foo(\\\"bar\\\"); }\"'");
system("bash -c 'echo \"#include\nint main() { std::regex foo(\\\"bar\\\"); }\" | g++ --param ggc-min-expand=10 -x c++ -'");
}

Will see what the IT department comes back with. I assume the Jobe server is using a sandbox environment with its own compilers?

Re: memory exhausted when compiling regular expressions

de către Richard Lobb- marți, 7 martie 2023, 09:58

The Jobe server uses a very old version of a restrictive environment called runguard, from the domjudge programming contest server. It uses all the standard Linux compilers and other applications, but restricts the environment by a combination of ulimit settings, environment variable clensing and a watchdog timer. So when you run g++ from the terminal you're running the same version as when the jobs arrive via CodeRunner, except for the additional restrictions.

If the ggc-min-expand parameter solves your problem and you need a workaround, you can add that parameter to any question (including to the prototype) via Customise > Advanced customisation > Sandbox > Parameters . It will be something like

{"compileargs": ["--param ggc-min-expand=10"]}

However, I'd really like to know what the fundamental problem is.

Re: memory exhausted when compiling regular expressions

de către Henning Koehler- miercuri, 8 martie 2023, 12:50

It seems another options that gets it to work is to set MemLimit to 50 MB - which is interesting as based on the previous output I would have thought it to be unlimited by default. IT is now investigating how to set this globally (rather than on a per-question basis).

Re: memory exhausted when compiling regular expressions

de către Richard Lobb- miercuri, 8 martie 2023, 13:39

That's very interesting, thanks. Also very strange. That value is used to set the ulimit for the task. The default memory limit during compile should be 500 MB, dropping to 200 MB during run. And the test you did earlier, showing the ulimit output for all parameters, indeed showed it was set to 200 MB. It looks like making it smaller is solving the problem. Let me know if you can cast any further light on this.

BTW: if you apply that setting to the prototype for you C++ questions, it will apply to all questions of that type. You don't have to set it on each question.

Re: memory exhausted when compiling regular expressions

de către Henning Koehler- miercuri, 8 martie 2023, 14:24

Mostly speculation, but it sounds like the C++ compiler (when dealing with regular expressions) tries to use a lot of memory, up to the amount it thinks is available (set by MemLimit). This can then exceed the virtual memory used by the sandbox, causing the error. With MemLimit set to something smaller than 200MB the compiler will restrain itself and the error goes away. That would match up with gcc-min-expand fixing the issue. The compiler generates a lot of garbage-collectable memory, but it thinks there's 500MB available in total so garbage collection doesn't kick in early enough to prevent the out-of-memory error.

Re: memory exhausted when compiling regular expressions

de către Henning Koehler- miercuri, 8 martie 2023, 15:31

Actually, it seems like the IT department might have made some changes since I tested: works now without any customization, with the following settings:

g++ (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) 20000
pending signals (-i) 31468
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) 6
max user processes (-u) 31
virtual memory (kbytes, -v) 500000
file locks (-x) unlimited

I've tried with even bigger virtual memory (realized that this is what corresponds to MemLimit) up to 2GB and it still works. Best guess they implemented the gcc-min-expand solution - I'll let you know if I find out.

Re: memory exhausted when compiling regular expressions

de către Richard Lobb- miercuri, 8 martie 2023, 17:59

It's sounding to me like the Jobe server originally was simply light on physical memory. A quick web search suggests that if a Linux task tries to use more than 50% physical RAM + swap space it will get a memory error. I'm guessing your IT guys just upped the amount of physical memory.