Continue testing after a test case runtime error

Continue testing after a test case runtime error

by Richard Lobb -
Number of replies: 0

This posting is just to share a github posting by Eduardo Marques so it's more accessible to the community. The author had observed a strange behaviour of the fork() system call in CodeRunner while trying to write a question type that continues testing after a runtime error. I suggested an approach using a per-test template but his answer using a combinator template is better. Here's his last posting:

Thanks, your solution had a small but important ingredient that paved the way for my solution to work: the dup2(1,2) call in the child process. The difference is that I use a combinator template. This is more efficient as we save time on compiling just one C/C++ program as I understand, rather than one compiled program per test case. My template is as shown next.

#include <iostream>
#include <fstream>
#include <string>
#include <cmath>
#include <vector>
#include <algorithm>
#include <unistd.h>
#include <sys/wait.h>

using namespace std;
#define SEPARATOR "#<ab@17943918#@>#"


int main() {
{% for TEST in TESTCASES %} 
       pid_t pid = fork();
       if (pid == 0) {
            dup2(1, 2);
            {{ TEST.extra }};
            {{ TEST.testcode }};
       } else if (pid > 0) {
            waitpid(pid, 0, 0);
            {% if not loop.last %}cout << SEPARATOR << endl;{% endif %}

{% endfor %}
    return 0;

A bit of context that perhaps you may find interesting: we are compiling programs with AddressSanitizer and UndefinedBehaviorSanitizer enabled (gcc and clang support both), hence crashes in student submissions are more frequent (and deterministic!), and we need to accommodate for those cases more often. An additional minor note is that AddressSanitizer uses 20 terabytes (!) of virtual memory hence MemLimit needs to be set to 0. This line in your code thus applies to other cases beyond Matlab.