I am linking the new(-ish) bulk-test output for testing upgrades, but I have a couple of suggestions:
- Where it re-prints a summary of all the failures at the end, it woudl be useful if that list gave some sort of contextual information about where the failing questions are from.
- I personally would like green/red styling of passes and fails.
- We had some weird ones, where a question failed in the bulk run, but then passed when run separately. However, I now cannot reproduce that.
- For most failures, it is OK to follow the link to questiontestrun.php, and from there to "See this question in the question bank". However, when the failure is something like "Failed to load question Unknown question: qtype_coderunner/Prototype python3_cosc121 is unavailable in this context, or does not exist. ****", then that path does not work, because questiontestrun.php page give a fatal error.
The bulk tester certainly could do with some polishing. It was mostly code copied from the Stack plugin, but I thought of it as primarily something for administrators and never put much effort into making it look nice. However, I find it's getting used increasingly by question authors, too, so it would be worth investing some effort.
I like the sound of all your suggestions. However, I'm in the middle of a busy teaching spell at present so wouldn't be able to put any time into CodeRunner development for a while - possibly not until the end of the teaching year in November. I'd certainly be grateful for any pull requests.
In the past I've seen cases where a question appeared to fail in the bulk run but was OK when I went to look at it, too. But I haven't had any cases recently. It was never repeatable so debugging was problematic. The questions that failed were always fine in production use so whatever caused the problem was specific to the bulk tester. Let me know if it reoccurs for you.