Signal 25 with c# question

Re: Signal 25 with c# question

by Mark Stern -
Number of replies: 4
Thank you for the explanation. I did not notice that the first run includes all the test cases, but I should have done. I did notice that there were 7 runs, even though there were only 6 tests.

Is there any way to turn off the fall back to one run per test? Alternatively, is there a way of passing information from one run to the next? I tried writing to a file, but the file seems to be gone at the start of the next run (probably in a different directory).
In reply to Mark Stern

Re: Signal 25 with c# question

by Richard Lobb -
You can't turn off the fall-back as it's an error-recovery mechanism. But you can take complete control of the grading process by using a combinator template grader (https://github.com/trampgeek/moodle-qtype_coderunner#combinator-template-grading), which - if written properly - should ensure the fall back never occurs. However, they're way harder to write, as you have to ensure that you handle all possible error conditions including timeouts in order to always return a valid grading response. And you'd have to turn on the "multiple stdins" options to prevent the default behaviour of running tests singly when each test has a separate stdin.

Most of our University of Canterbury question types use combinator template graders as they give you maximum flexibility. Some of our question types are visible at https://github.com/trampgeek/ucquestiontypes. However, this is intended as an inhouse repo so don't expect any documentation, and in fact some of the question types probably don't even work. Also, our question types are particularly complicated because we have our own OO framework for running multiple languages, and we have lots of template parameters, graphical output, ...

No, there's no simply way to pass information between testcase runs. You could create a writeable directory on Jobe and store stuff there, but that would open a huge security hole. There is a QUESTION.graderstate attribute (see https://github.com/trampgeek/moodle-qtype_coderunner#the-twig-question-variable) but this is only for use with combinator template graders and is for passing info across multiple student submissions, not between individual tests. If you're using a combinator template grader such a mechanism isn't required, anyway.

What problem are you trying to solve? If you're trying to write questions where testcases have standard inputs, then given the poor performance of dotnet, you'll almost certainly need to turn on the "multiple stdins" option and handle the stdin setup for each test case yourself (probably in a template grader). [** EDIT ** I see I already turned on multiple stdins in the prototype I provided, only not in the context of a template grader.]
In reply to Richard Lobb

Re: Signal 25 with c# question

by Richard Lobb -

OK, given that there may be other users needed a C# dotnet question type, and the difficulties of writing combinator template grader questions, I decided to throw one together. 

I attach the C# dotnet prototype rewritten to use a combinator template grader. I also attach the demo question with Precheck enabled and some demonstrations of the different test case settings.

Note that the question type has two global constants that set the maximum total run time for the question (the time budget) and the maximum time allowed for any particular test. The first of these should be 1 second less than the Jobe Time limit to ensure that the job times itself out rather than getting killed by Jobe. Normally we use template parameters for these, but I thought things were probably complicated enough already.

Caveat: this question type has had minimal testing, so please give it a thorough workout and let me know of any issues with it.

In reply to Richard Lobb

Re: Signal 25 with c# question

by Mark Stern -
Thanks. I tried the demo using the new prototype, and I noticed that all the tests had ticks, but only the last test showed as green. Is that expected behavior? I am running it as a teacher.
In reply to Richard Lobb

Re: Signal 25 with c# question

by Mark Stern -
I am trying to solve 2 problems:
1. If the combinator fails, it runs all the tests as separate runs, which has terrible performance.
2. If the combinator fails, it does not tell me why.

The way I was trying to solve these problems was:
Whenever I get an error, in addition to writing it to STDERR and terminating, write it to an error file.
At the start of a run, if the error file (from a previous run) exists, output its contents and then terminate.

However, it does not work because the error file is not preserved between runs.

I have some other ideas to solve these problems, but I will give your new combinator template grader prototype a try first.