!! DANGER ZONE !!

]]> A_PROTOTYPE_python3_MAIN Last Updated: Oct 12, 2024

How Results Are Displayed to Students

  • Input - Got - Expected - Points
    • What inputs were given for each test
    • What THEIR code printed/outputted (Got)
    • What was expected by the test auto-grader
    • The number of points ( 2 / 2) for that specific test.
  • Scratchpad code always ignored during testing
  • If a test fails, best efforts are attempted to continue on to other tests
    • Helps prevent a student failing future tests 3,4,5 as a result of failing test 2
  • You can customize what is displayed by clicking Customize and changing the list of dictionary values in "Result columns"

 

Order Of Events For Test Cases

Review "Test Variables and Functions" as well as "Template Parameters" for more info on the embedded variables named below.

  1. Global Extra, followed by Test Extra (useful for individual tests) are injected PRIOR to the student code. 
  2. Student Code is appended, and also saved in the variable student_code
    • Given the sequence, you can't inspect student code in your global/test extra
  3. Prohibited code snippets are checked for in student code.
    • If any are found and PROHIBIT_SPECIFIC_CODE_PHRASES is set to True, the test fails
  4. Substitutions to be made in student_code is completed
    • If ENABLE_STUDENT_CODE_SUBSTITUTIONS is set to True, all changes are made prior to running any code.
    • An important one set by default and replaces except: with except Exception as e:
  5. Syntax errors are checked for in the student code.
    • The code is linted using ast
    • If any errors are found, they are printed if OUTPUT_STUDENT_ERRORS_TO_USER is set to True
  6. Runtime errors are checked for by executing the student code, not including any test case code.
    • This runs to check for simple run time errors a student may have typed. Similar to pressing "Run Code", but via the test cases.
  7. Errors: if any errors do occur, best efforts are made to match up the error line numbers to the student code
    • Scratchpad code numbers are not an issue as code here is ignored and not tested
    • You can access TEST_ERROR_DESCRIPTION, TEST_ERROR_TYPE, and TEST_ERROR_LINE_NUMBER in the event of any errors in your Test Case code.
  8. Finally, all code is run together
    • Any pre code (Global or Test Extra) + Student Code + Test Case Code

 

Test Case Functions & Variables

Student Code

Student's raw code can be accessed in a multi-line string in your test cases.

  • student_code
    • Use Cases:
      • if "import" in student_code
      • student_code.splitlines() then iterate thru each line looking for something

Removing/Capturing Print Statements

Test code can told to silence print statements using functions built into template:

  • print_off()
    • No args needed and returns nothing. Just call it and any output (print/input/etc)
  • print_on()
    • Reenables any stdout back to the output screen
      • You might turn print off before running their code, then back on after/during your tests.
      • For example, off in test extra then on in your test case code.
    • Returns any output since print_on was engaged.
      • You could search this for a particular output if you wanted
    • Returns in a form of a list (each line is a string element)
 
Check For Beginning Comments
  • check_first_three_lines()
    • Checks for 3 comments in the first 5 lines of student code
    • Will print one of 2 results:
      • Passed comment test
      • Failed comment test - add 3# at start of your code

Template Parameters

  • There are a number of built-in options that are run during each test case that you may want turned on/off on a particular question.
  • These have default values already preset in the prototype question.
  • To modify these for a single question, you simply copy and paste the JSON formatted record below into the Template params and adjust as needed 

{
    "SCAN_FOR_SYNTAX_ERRORS": true,
    "SCAN_FOR_RUNTIME_ERRORS": true,
    "OUTPUT_STUDENT_ERRORS_TO_USER": true,
    "OUTPUT_TEST_ERRORS_TO_USER": true,
    "ALLOW_TRACEBACK_DISPLAYED_ON_TEST_ERROR": true,
    "PROHIBIT_SPECIFIC_CODE_PHRASES": false,
    "ENABLE_STUDENT_CODE_SUBSTITUTIONS": false
}

What They Do
  • SCAN_FOR_SYNTAX_ERRORS
    • Check student code for syntax errors prior to running test code
  • SCAN_FOR_RUNTIME_ERRORS
    • Check student code for runtime errors prior to running test code    
  • OUTPUT_STUDENT_ERRORS_TO_USER
    • Output any errors in the student code itself (run on its own)
  • OUTPUT_TEST_ERRORS_TO_USER
    • Output any errors generated during the test case to the student
  • ALLOW_TRACEBACK_DISPLAYED_ON_TEST_ERROR
    • Display full traceback error (Note: this exposes your test code to the student)
  • ENABLE_STUDENT_CODE_SUBSTITUTIONS
    • Replace specific code phrases in student code as listed
  • PROHIBIT_SPECIFIC_CODE_PHRASES
    • Restrict any code phrases (import, etc)

If you want to further customize PROHIBIT_SPECIFIC_CODE_PHRASES or ENABLE_STUDENT_CODE_SUBSTITUTIONS (they are both dictionary constants in the test template code), then at the top of the question choose "Customise", then under the new Customisation section edit "Template" and mod those variables.

Randomization

In Test Cases
  • Anywhere in your test case code you'll need to enter random.seed(some_number) so the test is predictable
  • Random library is already imported in the test template

Scratchpad "Run Code" Option
  • When not running tests, if students run "Run Code" from the scratchpad and is trying to use both random and multiple inputs, we have an issue.
    • With each input entered, full code is run again (the previous input is then injected)
    • This gives the illusion of a full operational program, but really it's just re-run every time there is another input().
    • This means if we are randoming anything, it's re-randomized with every input!
      • For example "guess a number" games results in the original number being change with every input.
    • Therefore, if there are any programs with more than 1 input, it needs to be seeded.
  • This is prebuilt for you in the scratchpad template (Prototype extra)
    • Be default, it will not seed the randomizer with there are <= 1 "input(" detected in the code
    • Anything more than 2, the number of characters in the students code is used as the seed value
      • If a student notices it's always the same, tell them to press space or make a new line. This changes the seed
    • If you want to edit this, customize the question and change the max_allowed_inputs in Prototype Extra

 

]]> 5.0000000 0.0000000 0 A_PROTOTYPE_python3_MAIN 2 0 0, 0, 0, 10, 20, ... 0 0 0 18 100 1 1 0 0 #\n|ms]]> python3 NearEqualityGrader 5 1 1 twig 0 1 scratchpad Your work is autosaved every 20 seconds.

Run Code Button

You can press 'Run Code' as often as you'd like for no penality. This will execute your code and show your output.

If the 'Run Answer Code?' box is:
Checked = Run code in Answer Box
Unchecked = Run code in Scratchpad


Check Button

Pressing 'Check' will run prebuilt 'tests' on code in your Answer Box.

If penalty regime (shown just above the answer box) is not zero 0 only, then you could be docked marks if you press it. Ask if you're not sure!


What Is Graded

Only code in the answer box is graded. The scratchpad is basically 'scrap paper' and not part of your answer.


Using The Scratchpad

If you have the 'run answer code?' box checked it will actually run code from both boxes. It will run the answer code 1st, then whatever is in the scratchpad.

So if the box is checked, you'll want to remove/comment out anything in your scratchpad so code in both boxes doesn't run together.

" }]]> 0 0 10240 1 0 2 * MAX_OUTPUT_CHARS: __saved_print__("\\\\n*** Excessive output. Job aborted ***", file=sys.stderr) sys.exit(1) else: __saved_print__(*params, **keyparams) """ # Add input precode and apply Random Seed based on character number if inputs are found if input_count > max_allowed_inputs: subproc_code = seed_code + ini_subproc_code else: subproc_code = ini_subproc_code #### FULL Precode (does not include student code) subproc_code += PRE_CODE ##### STUDENT Code Only og_student_code = """{| ANSWER_CODE |} {| SCRATCHPAD_CODE |} """ replaced_student_code = og_student_code.replace("except:", "except Exception as e:") ## NOTE: Precode and Student Code Mixes as Needed Below # Normalizes code lines numbers by removing precode injected line numbers from student code def adjust_error_message(stderr, line_offset): # Adjust the line numbers in the traceback using a more precise regex def adjust_line_number(match): # Extract the current line number from the traceback line_number = int(match.group(1)) # Adjust the line number by subtracting the offset return f'line {line_number - line_offset}' # Try to remove precode line numbers try: adjusted_stderr = re.sub(r'line (\d+)', adjust_line_number, stderr) return adjusted_stderr except: # If precode adjustment failed, return original numbers return "NOTE: Line numbers below are not correctly adjusted\n" + stderr # Cut short any stdout or stderr output def truncated(s): return s if len(s) < MAX_OUTPUT_CHARS else s[:MAX_OUTPUT_CHARS] + " ... (truncated)" # Checks the syntax BEFORE executing. def check_syntax(code_to_test): try: ast.parse(code_to_test) return "" except SyntaxError: catcher = io.StringIO() traceback.print_exc(limit=0, file=catcher) return catcher.getvalue() # Prepare variables for analysis stdout = "" stderr = check_syntax(replaced_student_code) # Check that student code has no syntax errors if stderr == "": # No syntax errors, run code program_code = subproc_code + replaced_student_code with open("prog.py", "w", encoding="utf-8") as outfile: outfile.write(program_code) # Calculate the number of precode lines line_offset = subproc_code.count('\n') proc = subprocess.Popen( [sys.executable, "prog.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, ) try: stdout, stderr = proc.communicate(timeout=3) returncode = proc.returncode except subprocess.TimeoutExpired: # Run timed out. Maybe infinate loop? proc.kill() stdout, stderr = proc.communicate() returncode = 13 # If an error was detected, try to adjust line numbers. If not, return original error if stderr: stderr = adjust_error_message(stderr, line_offset) else: returncode = 1 # Syntax errors # CodeRunner requires a JSON dump. Assemble it here. output = { "returncode": returncode, "stdout": truncated(stdout), "stderr": truncated(stderr), "files": [], } # Dump for CodeRunner print(json.dumps(output)) ]]> A_PROTOTYPE_python3_SIMPLE Template Layout
  • Global Extra and Test Extra (extra for specific tests) is injected PRIOR to the question. 
  • Before running student code, it's evaluated for presence of:
    • quit()
    • exit()
    • syntax errors
    • runtime errors
    • attempts to ensure error line numbers match up for the student
    • Note: You can add more to this if statement via customizing so you can edit it
  • Student code is then run as part of the test.
  • Test code for each test runs AFTER the student code.

 

Test Code

  • Removing/Capturing Print Statements
    • Test code can told to silence print statements using functions built into template:
      • print_on()
        • No args needed and returns nothing. Just call it
      • print_off()
        • Returns any output since print_on was engaged. 
        • Returns in a form of a list (each line is a string element)

 

  • Seeding Randomization
    • Anywhere in your test code enter random.seed(some_number)
    • Random library is already imported in the test template

 

Scratchpad "Run Code" Option & Randomization

  • When not running tests, if students run "Run Code" from the scratchpad and is trying to use both random and multiple inputs, we have an issue.
    • With each input entered, full code is run again (the previous input is then injected)
    • This gives the illusion of a full operational program, but really it's just re-run every time there is another input().
    • This means if we are randoming anything, it's re-randomized with every input!
      • For example "guess a number" games results in the original number being change with every input.
    • Therefore, if there are any programs with more than 1 input, it needs to be seeded.
  • This is prebuilt for you in the scratchpad template (Prototype extra)
    • Be default, it will not seed the randomizer with there are <= 1 "input(" detected in the code
    • Anything more than 2, the number of characters in the students code is used as the seed value
      • If a student notices it's always the same, tell them to press space or make a new line. This changes the seed
    • If you want to edit this, customize the question and change the max_allowed_inputs in Prototype Extra

 

How Results Are Outputed to Students

  • What was Inputed
  • What THEIR code gave (Got)
  • What was expected by the auto-grader
]]> 5.0000000 0.0000000 0 A_PROTOTYPE_python3_SIMPLE 2 1 0 0 0 0 18 100 1 1 0 1 #\n|ms]]> python3 NearEqualityGrader 1 1 None 0 {} 0 scratchpad Your work is autosaved every 20 seconds.

Run Code Button

You can press 'Run Code' as often as you'd like for no penality. This will execute your code and show your output.

If the 'Run Answer Code?' box is:
Checked = Run code in Answer Box
Unchecked = Run code in Scratchpad


Check Button

Pressing 'Check' will run prebuilt 'tests' on code in your Answer Box.

If penalty regime (shown just above the answer box) is not zero 0 only, then you could be docked marks if you press it. Ask if you're not sure!


What Is Graded

Only code in the answer box is graded. The scratchpad is basically 'scrap paper' and not part of your answer.


Using The Scratchpad

If you have the 'run answer code?' box checked it will actually run code from both boxes. It will run the answer code 1st, then whatever is in the scratchpad.

So if the box is checked, you'll want to remove/comment out anything in your scratchpad so code in both boxes doesn't run together.

" }]]> 0 0 10240 1 0 2 * MAX_OUTPUT_CHARS: __saved_print__("\\\\n*** Excessive output. Job aborted ***", file=sys.stderr) sys.exit(1) else: __saved_print__(*params, **keyparams) """ # Add input precode and apply Random Seed based on character number if inputs are found if input_count > max_allowed_inputs: subproc_code = seed_code + ini_subproc_code else: subproc_code = ini_subproc_code #### FULL Precode (does not include student code) subproc_code += PRE_CODE ##### STUDENT Code Only og_student_code = """{| ANSWER_CODE |} {| SCRATCHPAD_CODE |} """ ## NOTE: Precode and Student Code Mixes as Needed Below # Normalizes code lines numbers by removing precode injected line numbers from student code def adjust_error_message(stderr, line_offset): # Adjust the line numbers in the traceback using a more precise regex def adjust_line_number(match): # Extract the current line number from the traceback line_number = int(match.group(1)) # Adjust the line number by subtracting the offset return f'line {line_number - line_offset}' # Try to remove precode line numbers try: adjusted_stderr = re.sub(r'line (\d+)', adjust_line_number, stderr) return adjusted_stderr except: # If precode adjustment failed, return original numbers return "NOTE: Line numbers below are not correctly adjusted\n" + stderr # Cut short any stdout or stderr output def truncated(s): return s if len(s) < MAX_OUTPUT_CHARS else s[:MAX_OUTPUT_CHARS] + " ... (truncated)" # Checks the syntax BEFORE executing. def check_syntax(code_to_test): try: ast.parse(code_to_test) return "" except SyntaxError: catcher = io.StringIO() traceback.print_exc(limit=0, file=catcher) return catcher.getvalue() # Prepare variables for analysis stdout = "" stderr = check_syntax(og_student_code) # Check that og student code has no syntax errors if stderr == "": # No syntax errors, run code program_code = subproc_code + og_student_code with open("prog.py", "w", encoding="utf-8") as outfile: outfile.write(program_code) # Calculate the number of precode lines line_offset = subproc_code.count('\n') proc = subprocess.Popen( [sys.executable, "prog.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, ) try: stdout, stderr = proc.communicate(timeout=3) returncode = proc.returncode except subprocess.TimeoutExpired: # Run timed out. Maybe infinate loop? proc.kill() stdout, stderr = proc.communicate() returncode = 13 # If an error was detected, try to adjust line numbers. If not, return original error if stderr: stderr = adjust_error_message(stderr, line_offset) else: returncode = 1 # Syntax errors # CodeRunner requires a JSON dump. Assemble it here. output = { "returncode": returncode, "stdout": truncated(stdout), "stderr": truncated(stderr), "files": [], } # Dump for CodeRunner print(json.dumps(output)) ]]>