Signal 25 with c# question

Signal 25 with c# question

by Mark Stern -
Number of replies: 22

I am trying to create a C# question using dot net (my code is in Python).

When I run (from a Python subprocess):

dotnet new console

I get signal 25. As far as I can tell, that is SIGXFSZ

I presume I need to increase the file size allowed, but I cannot see any settings in CodeRunner for this.

The command works (as a shell command) on the JOBE server.

How do I fix this?

Thanks,

Mark Stern

In reply to Mark Stern

Re: Signal 25 with c# question

by Richard Lobb -
Jobe server parameters are documented here. The one you probably want is disklimit - the maximum file size in MB. Default 20.You can set this value in the Advanced Customisation > Sandbox > Parameters field as, e.g., {disklimit: 100}.

Caveats: 
  1. I haven't often used this parameter so am not totally confident that it works as expected.
  2. Jobe uses an old version of the domjudge runguard.c package which limits resource usage, including filesize, using setrlimit. If the limits are exceeded you normally get a custom Jobe error message. I've never seen SIGXFSZ, but from what I'm seeing on the web, this should only occur if the executable code is linked with unix95.o. I'm not sure how compatible runguard is with unix95.o.
In reply to Richard Lobb

Re: Signal 25 with c# question

by Mark Stern -
That did not help. The files it is creating do not seem that big anyway. So I tried uploading them (as a tar file) and untarring them in the code. That worked. However, when I build the project using dotnet, I am back to signal 25.
In reply to Mark Stern

Re: Signal 25 with c# question

by Richard Lobb -
I assume you've tried with a stupidly large version of disklimit, like 2000?

Have you also set the memlimit to 0, since the default value is almost certain to fail if you're running dotnet with python? That disables the memory limit altogether which might not be a good idea in a production server but is best for testing. Also try increasing numprocs with the sandbox parameter setting, e.g. {"disklimit": 2000,"numprocs":1000} [I see I missed the quotes in the command earlier - hopefully you fixed that error?]
In reply to Richard Lobb

Re: Signal 25 with c# question

by Mark Stern -
OK, I tried all that, but I am still getting signal 25.

I found
https://coderunner.org.nz/mod/forum/discuss.php?d=323

It seems that others have tried to do this, and failed. For now, I think the only solution is to use Mono.
Mono is not a good solution, because it is no longer being maintained (as far as I can tell) and is already
6 versions of C# behind. However, until somebody fixes CodeRunner to work with dotnet, it is probably
the best I can do.
In reply to Mark Stern

Re: Signal 25 with c# question

by Richard Lobb -
OK, I've just looked into this. Indeed C# dotnet is a pig to get running in Jobe. However, I attach a question prototype and an example that seems to work. 

Warnings:
  1. (Of course) dotnet needs to be installed on Jobe first. The command I used was just

    sudo apt-get install dotnet-sdk-8.0

  2. In order to get it going I've had to turn off the resource limits on memory and filesize; I'm hoping that the dotnet environment manages these sufficiently well to prevent the Jobe server thrashing.
  3. I'm unhappy about the performance; even a trivial run takes 2 - 3 seconds on my development laptop. That would make the question type very problematic with large classes in tests or exams, unless you have a cluster of jobe servers.
  4. I'm not a C# programmer and have no personal need for this question type. So I've done only minimal testing, using just the one example question that's attached. There will undoubtedly be issues that you will need to fix yourself.
Caveat emptor!

Please would you report back on the results of any testing you do, particularly in a production environment. If it proves satisfactory (perhaps with changes you might wish to suggest), I could consider adding it to the built-in prototoypes for CodeRunner.

In case you're interested, here are the hacks and tweaks I had to make to get it working:

  • The filesize ulimit was, I think, the cause of your 25 errors. I couldn't find any value that would allow dotnet to run, and Jobe does not directly support disabling this ulimit. However, I was able to exploit a loophole: the runguard package that Jobe uses multiplies a given file size in kilobytes (not to be confused with the disklimit that Jobe is given, which is in megabytes!) by 1000 to convert it to bytes. If that gives an integer overflow runguard sets the filesize ulimit value to unlimited. So I used a value of disklimit that triggered that integer overflow. [I've since updated my development copy of Jobe to use a value of -1 for disklimit to disable the filesize limit, but I haven't pushed that update yet.]
  • I also couldn't find a ulimit value on memsize that would dotnet would run with, so I set the memory limit in the question type to 0 to disable that ulimit as well.
  • dotnet seems to need a ridiculous number of processes. So I set a ulimit (numprocs) of 1000.
  • dotnet expects a home directory, but for security reasons Jobe tasks run without one. dotnet also wants a directory (usually within the home directory) to store temporary config files. So I had to create temporary directories for these and add environment variables to reference them.
  • When a build fails (i.e. a compilation error) dotnet spews a vast heap of rubbish to stdout (not to stderr - why not?!). I used what command line flags I could find to minimise the rubbish and then added a post-filter to strip out only the meaningful lines for display.

** EDIT ** A more advanced version of this question type that uses a combinator template grader is given later in this thread.

In reply to Richard Lobb

Re: Signal 25 with c# question

by Mark Stern -
Thanks. Do I need to be an admin to install this?
In reply to Mark Stern

Re: Signal 25 with c# question

by Richard Lobb -
You need to have admin rights on the jobe server to install dotnet. But you should be able to import the xml files into your Moodle course as a normal teacher. I strongly recommend you have a separate category for prototypes.

However,  your other postings suggest you're used to dealing with prototypes so I'm wondering ... did something go wrong when you tried importing?
In reply to Richard Lobb

Re: Signal 25 with c# question

by Mark Stern -
OK, I figured out how to import those XML files into Moodle, and I have converted all my "write a program in C#" questions to use your prototype, without any problem (all I had to do was change the question type).

I am also going to need a "write a C# method" prototype, but I should be able to take what you have done and adapt it.

I am perplexed by your hack for disklimit. My understanding is that PHP (we are talking about PHP, right?) does not have integer overflow - it automatically switches to float.

Thanks again for all your help.
In reply to Mark Stern

Re: Signal 25 with c# question

by Richard Lobb -
Good to hear. If you get a "write a C# method" question type working, would you mind posting it back here, please?

Are you going to be able to live with the 2 - 3 sec run times? Hopefully, you don't have large classes running CodeRunner based tests or exams.

To clarify the disklimit hack: the bit of code that checks for integer overflow is in the runguard component of Jobe. Runguard is from the open-source DOMjudge program contest system, and is written in C. I'm using a very old version.
In reply to Richard Lobb

Re: Signal 25 with c# question

by Mark Stern -
Ahhh. It is C, not PHP. That explains it.

I am working on a  "write a C# method" question type. At the moment I have a "write a C# method" question (attached). However, it is too slow to be useful, and should have 16 test cases instead of 6. However if I add the other 10 test cases I get a 404 error. The reason that it is so slow is that it seems to be ignoring the 'Is Combinator' setting - it is doing a complete run for each test. I do not know why. The documentation for "Allow multiple stdins" is too confusing, so I tried both settings. However, I do not think it is relevant, because I am not (intentionally) using stdin. 

In reply to Mark Stern

Re: Signal 25 with c# question

by Richard Lobb -
I see you have turned on template debugging, which was the right thing to do. But did you notice that the first run includes all the testcases (the true combinator run) whereas the subsequent ones are running each test case separately?

When using a combinator (and assuming no stdins), CodeRunner first tries running all the tests as a single run. If that results in a runtime error or any output to stderr, CodeRunner drops back to running each test separately until another runtime error occurs at which point it aborts the testing.

The problem with this question is that your way of combining the testcases into a single run is flawed - each testcase declares a variable daysInMonth so in the combinator run you get a "redeclared variable" compile error from C# which results in stderr output. CodeRunner then attempts to re-run each test separately, which now works fine because the same template is used but with the TESTCASES variable being an array with just one testcase on each run. Note that you get a different behaviour when there's a syntax error in the student code: in that case the first run fails as before but the second run also fails with the same syntax error and testing is aborted.

The fix is just to wrap each testcase in braces to give it a local context, i.e.

{% for TEST in TESTCASES %}
{
        {{ TEST.testcode }};
}
{% if not loop.last %}
        Console.WriteLine("##");   // Testcase separator
{% endif %}
{% endfor %}

I also saw that there's a problem with the error messages given to the student - the line numbers are all wrong because of the extra hidden text at the start. You might want to consider post-processing the compile error messages to subtract the appropriate number of lines from each error line number.

In reply to Richard Lobb

Re: Signal 25 with c# question

by Mark Stern -
Thank you for the explanation. I did not notice that the first run includes all the test cases, but I should have done. I did notice that there were 7 runs, even though there were only 6 tests.

Is there any way to turn off the fall back to one run per test? Alternatively, is there a way of passing information from one run to the next? I tried writing to a file, but the file seems to be gone at the start of the next run (probably in a different directory).
In reply to Mark Stern

Re: Signal 25 with c# question

by Richard Lobb -
You can't turn off the fall-back as it's an error-recovery mechanism. But you can take complete control of the grading process by using a combinator template grader (https://github.com/trampgeek/moodle-qtype_coderunner#combinator-template-grading), which - if written properly - should ensure the fall back never occurs. However, they're way harder to write, as you have to ensure that you handle all possible error conditions including timeouts in order to always return a valid grading response. And you'd have to turn on the "multiple stdins" options to prevent the default behaviour of running tests singly when each test has a separate stdin.

Most of our University of Canterbury question types use combinator template graders as they give you maximum flexibility. Some of our question types are visible at https://github.com/trampgeek/ucquestiontypes. However, this is intended as an inhouse repo so don't expect any documentation, and in fact some of the question types probably don't even work. Also, our question types are particularly complicated because we have our own OO framework for running multiple languages, and we have lots of template parameters, graphical output, ...

No, there's no simply way to pass information between testcase runs. You could create a writeable directory on Jobe and store stuff there, but that would open a huge security hole. There is a QUESTION.graderstate attribute (see https://github.com/trampgeek/moodle-qtype_coderunner#the-twig-question-variable) but this is only for use with combinator template graders and is for passing info across multiple student submissions, not between individual tests. If you're using a combinator template grader such a mechanism isn't required, anyway.

What problem are you trying to solve? If you're trying to write questions where testcases have standard inputs, then given the poor performance of dotnet, you'll almost certainly need to turn on the "multiple stdins" option and handle the stdin setup for each test case yourself (probably in a template grader). [** EDIT ** I see I already turned on multiple stdins in the prototype I provided, only not in the context of a template grader.]
In reply to Richard Lobb

Re: Signal 25 with c# question

by Richard Lobb -

OK, given that there may be other users needed a C# dotnet question type, and the difficulties of writing combinator template grader questions, I decided to throw one together. 

I attach the C# dotnet prototype rewritten to use a combinator template grader. I also attach the demo question with Precheck enabled and some demonstrations of the different test case settings.

Note that the question type has two global constants that set the maximum total run time for the question (the time budget) and the maximum time allowed for any particular test. The first of these should be 1 second less than the Jobe Time limit to ensure that the job times itself out rather than getting killed by Jobe. Normally we use template parameters for these, but I thought things were probably complicated enough already.

Caveat: this question type has had minimal testing, so please give it a thorough workout and let me know of any issues with it.

In reply to Richard Lobb

Re: Signal 25 with c# question

by Mark Stern -
Thanks. I tried the demo using the new prototype, and I noticed that all the tests had ticks, but only the last test showed as green. Is that expected behavior? I am running it as a teacher.
In reply to Richard Lobb

Re: Signal 25 with c# question

by Mark Stern -
I am trying to solve 2 problems:
1. If the combinator fails, it runs all the tests as separate runs, which has terrible performance.
2. If the combinator fails, it does not tell me why.

The way I was trying to solve these problems was:
Whenever I get an error, in addition to writing it to STDERR and terminating, write it to an error file.
At the start of a run, if the error file (from a previous run) exists, output its contents and then terminate.

However, it does not work because the error file is not preserved between runs.

I have some other ideas to solve these problems, but I will give your new combinator template grader prototype a try first.
In reply to Mark Stern

Re: Signal 25 with c# question

by Mark Stern -
I now have a working C# dot net prototype for "Write a method" (attached). It does not support input from STDIN (all input must be in the form of parameters to the method).

I noticed that the template PROTOTYPE_DOTNET_CSHARP.xml has an unterminated DIV tag on line 89.
In reply to Mark Stern

Re: Signal 25 with c# question

by Richard Lobb -

Great, thanks Mark.

Since I last posted, I have been working on the latest release of CodeRunner and decided to include a folder of unsupported question types. I included an updated version of the dotnet C# file I posted to this thread earlier. It includes some functionality refinements and three template parameters; its documentation is as below.

I'd like to include your write-a-function version in the folder too, if that's ok? I'd add attribution at the start. But I wonder if you'd first like to extend it to include the extra functionality I've since added to the write-a-program question?

One other thing: did you notice that the question type can be sped up by building a template project on the Jobe server? This is documented within the prototype itself as:

    Performance is also significantly improved (~ 1 sec less build
    time) if a template project is built at location /home/jobe/dotnettemplate
    by commands such as
    
    sudo bash
    mkdir /home/jobe/dotnettemplate
    cd /home/jobe/dotnettemplate
    dotnet new console --use-program-main

-- Richard

========================================

CodeRunner question type: csharpdotnet

A prototype for a C# question type that tests a C# "write-a-program" question using the dotnet framework. It runs all tests in a single Jobe run with a single compilation using a Combinator Template Grader.

Template parameters:

total_time_budget: the maximum time allowed (seconds) for the entire testing of the question (the compilation time plus all tests). Should be at least 1 second less than the maximum time limit set by the Jobe server, which is usually 50 secs. Default: 10. 

per_test_timeout: the maximum time allowed (seconds) to run any one of the tests. It does not include the compile time, as compilation is done before the various tests are run. Default 3.

warnings_are_errors: true to treat any warning messages as errors. Otherwise, warnings are not displayed to the user unless there are also errors present. Default: true.

Example template parameter: {"total_time_budget": 20, "per_test_timeout": 5}

WARNINGS: 

  1. You will need to have installed the dotnet package on your jobe server (sudo apt-get install dotnet-sdk-8.0).
    • For improved efficiency, you should also install an empty console project at /home/jobe/dotnettemplate; see the template for the required terminal commands.
  2. This question type has had hardly any testing and has never been used in a production environment.
  3. Performance is very poor, because it takes around 2 seconds to compile a C# program. Hence, use of this question type in a test or exam is likely to overload the jobe server except with very small classes or when you have a large pool of Jobe servers.
  4. Dotnet does not play well with the usual Jobe 'ulimit' resource limitations, so the memory limit and disklimit (amount of disk i/o) have both been disabled. It is potentially possible for a rogue task to disable the Jobe server by exceeding these limits, although the watchdog timer should kill the job within around 10 seconds and the server should then recover. This theory has not been tested in practice.

In reply to Richard Lobb

Re: Signal 25 with c# question

by Mark Stern -
You are welcome to include my template. I will try to add the suggested improvements, but currently I cannot import your latest template. I get:

Error importing question Error parsing XML: Mismatched tag at line 279, char 25

I did not see anything in the documentation about pre-building the project on the JOBE server, although I did see the code that was obviously intended to achieve that. I will have a play with that.
In reply to Mark Stern

Re: Signal 25 with c# question

by Richard Lobb -
The file imports fine for me. Please try again:
  1. In Chrome, visit https://github.com/trampgeek/moodle-qtype_coderunner/blob/master/unsupportedquestiontypes/prototypes/PROTOTYPE_dotnet_c%23.xml
  2. Click the Download raw file link at the top right of the image: 
  3. Save it somewhere with a name like PROTOTYPE_dotnet_c#.xml
  4. Import that file, making sure to select the file type as Moodle XML
In reply to Richard Lobb

Re: Signal 25 with c# question

by Mark Stern -
Thanks, that worked. I have updated the prototype to include the extra functionality, apart from per_test_timeout which does not make sense for this prototype.
In reply to Mark Stern

Re: Signal 25 with c# question

by Richard Lobb -
Many thanks Mark. I've added that to the "unsupported question types" repo on github, adding the line "Contributor: Mark Stern". I also took the liberty of adding 'rstrip()' method calls on both the expected and got fields when testing for correctness. I thought it was a bit odd to require newlines on all the testcases except the last one. Most (all?) the built-in question types perform rstripping in this way, and in fact also rstrip all lines as well since whitespace on the end of a line is invisible and confusing if it causes errors.

Also, I added an example question - static int sqr(int n) - since I'm trying to have at leat one example of each question type.