Question Authors' Forum

How to make running time and memory usage two more requirements for the program to be correct in CodeRunner?

How to make running time and memory usage two more requirements for the program to be correct in CodeRunner?

by Zhang Jeremy -
Number of replies: 3

Hello there, I have met a problem when I am adding additional features to CodeRunner. I have noticed that it is possible to set the limit of running time and memory usage in the advanced settings of CodeRunner quiz, but I am just so confused about how to change the codes in order to make the program to be shown correct only when all the test cases have passed and it satisfies the requirements of running time and memory usage.

To be specific, the teacher can set the largest value for the running time and memory usage and then the program that the student has submitted will only be shown in green correct when all the test cases are right while the running time and memory usage all has not exceeded the limit number. If the program has exceeded the limt of running time or memory usage, even if all the test cases have passed, the feedback should still be red incorrect.

I just wonder how I can change the codes in some files like "render.php" in order to realize this extensions to CodeRunner? I guess some files in CodeRunner itself shoud be changed and probably some files in the Jobe server should also be editted.  I really need some help. Thank you so so much. 

In reply to Zhang Jeremy

Re: How to make running time and memory usage two more requirements for the program to be correct in CodeRunner?

by Richard Lobb -
I don't fully understand what you're trying to achieve here that isn't already built in. Regardless, I recommend against changing the CodeRunner code base. If you use a combinator template grader the template code gives you complete control of the compiling, execution and feedback. I usually use Python but any language can be used.

It sounds like you're trying to track CPU usage and memory usage while still permitting tests to exceed those limits. That's not really possible - consider a task that's stuck in an endless loop, for example. Alternatively, maybe you wish to continue testing even after a particular test has crashed due to exceeding the limits? That requirement is discussed in this forum thread: https://coderunner.org.nz/mod/forum/discuss.php?d=526



In reply to Richard Lobb

Re: How to make running time and memory usage two more requirements for the program to be correct in CodeRunner?

by Zhang Jeremy -
Hello there, thank you for replying! I think I haven't explained clearly what I want to achieve haha. I mean if all the test cases can pass, but the running time or memory usage is larger than the limit number, when we check the feedback of the program, I hope it should show several lines telling us that every test case is correct as usual, but also tell us that the running time or memory usage doesn't satify the requirements, so the final judge result should still be incorrect although all the test cases are correct. And the final judge result should only be shown totally correct when every test case is correct and the running time and the memory usage doesn't exceed the value we define.
In reply to Zhang Jeremy

Re: How to make running time and memory usage two more requirements for the program to be correct in CodeRunner?

by Richard Lobb -
How do you propose to deal with tasks that take really excessive time and/or memory? Are you going to let them run for 10 minutes (consuming valuable Jobe server CPU time)? And what if they demand 20 GB RAM (bringing the server to its knees)? What if the task runs some test cases in a reasonable time but demands far too much time on some others? What if the time taken on each test case is within some specified limits but the total time over all test cases is excessive?

How are you going to specify all these various time and memory constraints within a CodeRunner question?

Also, please realise that tracking memory usage by a task is very difficult and requires the use of cgroups on the Jobe server, which is not currently set up to use them. This is hard.