CodeRunner: Weird problem with the Unit tests

I've got a weird problem with the unit tests when I run them on my computer, which don't happen when I run them elsewhere (e.g. Github Actions)

This is using Moodle 3.11 (or 3.10), PHP 7.4 and CodeRunner 4.1.0+ (or 4.0.1).

The failures I get are all like:

$ vendor/bin/phpunit --testsuite qtype_coderunner_testsuite --filter qtype_coderunner_cpp_questions_test::test_good_sqr_function
Moodle 3.10.4 (Build: 20210510), e7514424c252a524f29f7e6e8e5f8c887d7cd17f
Php: 7.4.23, ou_pgsql: 11.4, OS: Windows NT 10.0 AMD64
PHPUnit 8.5.14 by Sebastian Bergmann and contributors.
E                                                                   1 / 1 (100%)
Time: 12.17 seconds, Memory: 130.00 MB
There was 1 error:
1) qtype_coderunner_cpp_questions_test::test_good_sqr_function
assert(): assert(!isset($this->prototype)) failed
C:\Users\tjh238\workspace\moodle_ou_live\question\type\coderunner\question.php:876
C:\Users\tjh238\workspace\moodle_ou_live\question\type\coderunner\question.php:846
C:\Users\tjh238\workspace\moodle_ou_live\question\type\coderunner\classes\jobrunner.php:64
C:\Users\tjh238\workspace\moodle_ou_live\question\type\coderunner\question.php:547
C:\Users\tjh238\workspace\moodle_ou_live\question\type\coderunner\tests\cppquestions_test.php:43
C:\Users\tjh238\workspace\moodle_ou_live\lib\phpunit\classes\advanced_testcase.php:80
ERRORS!
Tests: 1, Assertions: 0, Errors: 1.

The assert is in get_prototype on the question class. The problem is that not only is this bing called withing the question (via jobrunner and -> get files), but it is also called when the question is initialised by tests/helper.php.

Aha! found it. Please could you, where you run the unit tests yourself, check the output of:

php -r "echo ini_get('zend.assertions');"

php -r "echo ini_get('assert.active');"

If I turn off assertions, then the tests pass for me locally.

So, I think

This assertion is currently having no effect for almost all CodeRunner developers because they have assertions disabled.
Currently, either the assertion is wrong, or the code is wrong. (If it was me, would change the code to remove the assertion, and instead make get_prototype do nothing the second and subsequent time it is called on a particular question.

Thanks.

Re: Weird problem with the Unit tests

por Richard Lobb - jueves, 14 de octubre de 2021, 11:46

Yes, thanks Tim. Matthew Toohey pointed this out to me a couple of weeks ago. Indeed the assertion is wrong, but I wasn't seeing it either because assertion checking was off, apparently by default.

I fixed it in my development version but haven't pushed yet. I don't remember when/why I put that assertion in. Checking through the code I see that there do appear to be redundant calls to get_prototype but it's difficult to prove they are redundant given the many different paths through the code. So I've left the calls in place and now just return immediately if the prototype has already been defined, exactly as you suggest. Alternatively you can just delete the assertion, which is essentially how the code has been running for months if not years. It's just a bit inefficient to search for the prototype a second time when it has already been found.

Re: Weird problem with the Unit tests

por Tim Hunt - jueves, 14 de octubre de 2021, 20:12

Great! Thanks.

(I am speculating that this might be coming up now becuase PHP 7.4 changed the default for what to do with assertions when you install). Anyway, good to know that the fix is on the way.)