Dear Martin,

I'm not clear on what is your specific problem with the question template. I have checked on my server, and I'm using the default nodejs question type, that is, using applications/library/nodejs_task.php  

Can you provide a more specific error message or screenshot?

Regarding the installation of nodejs and related libraries, I'm using nvm to install node:

> nvm ls
->      v6.10.0
         system
default -> v6.10.0

And the executable at /usr/bin/nodejs, as specified in nodejs_task.php is:

> /usr/bin/nodejs --version
v4.8.0

And the installed packages are:

> npm ls
/var/www
└─┬ jsdom@9.2.1
  ├── abab@1.0.3
  ├── acorn@2.7.0
  ├── acorn-globals@1.0.9
  ├── array-equal@1.0.0
  ├── cssom@0.3.1
  ├── cssstyle@0.2.36
  ├─┬ escodegen@1.8.0
  │ ├── esprima@2.7.2
  │ ├── estraverse@1.9.3
  │ ├── esutils@2.0.2
  │ ├─┬ optionator@0.8.1
  │ │ ├── deep-is@0.1.3
  │ │ ├── fast-levenshtein@1.1.3
  │ │ ├── levn@0.3.0
  │ │ ├── prelude-ls@1.1.2
  │ │ ├── type-check@0.3.2
  │ │ └── wordwrap@1.0.0
  │ └─┬ source-map@0.2.0
  │   └── amdefine@1.0.0
  ├── iconv-lite@0.4.13
  ├── nwmatcher@1.3.8
  ├── parse5@1.5.1
  ├─┬ request@2.72.0
  │ ├── aws-sign2@0.6.0
  │ ├── aws4@1.4.1
  │ ├─┬ bl@1.1.2
  │ │ └─┬ readable-stream@2.0.6
  │ │   ├── core-util-is@1.0.2
  │ │   ├── inherits@2.0.1
  │ │   ├── isarray@1.0.0
  │ │   ├── process-nextick-args@1.0.7
  │ │   ├── string_decoder@0.10.31
  │ │   └── util-deprecate@1.0.2
  │ ├── caseless@0.11.0
  │ ├─┬ combined-stream@1.0.5
  │ │ └── delayed-stream@1.0.0
  │ ├── extend@3.0.0
  │ ├── forever-agent@0.6.1
  │ ├─┬ form-data@1.0.0-rc4
  │ │ └── async@1.5.2
  │ ├─┬ har-validator@2.0.6
  │ │ ├─┬ chalk@1.1.3
  │ │ │ ├── ansi-styles@2.2.1
  │ │ │ ├── escape-string-regexp@1.0.5
  │ │ │ ├─┬ has-ansi@2.0.0
  │ │ │ │ └── ansi-regex@2.0.0
  │ │ │ ├─┬ strip-ansi@3.0.1
  │ │ │ │ └── ansi-regex@2.0.0
  │ │ │ └── supports-color@2.0.0
  │ │ ├─┬ commander@2.9.0
  │ │ │ └── graceful-readlink@1.0.1
  │ │ ├─┬ is-my-json-valid@2.13.1
  │ │ │ ├── generate-function@2.0.0
  │ │ │ ├─┬ generate-object-property@1.2.0
  │ │ │ │ └── is-property@1.0.2
  │ │ │ ├── jsonpointer@2.0.0
  │ │ │ └── xtend@4.0.1
  │ │ └─┬ pinkie-promise@2.0.1
  │ │   └── pinkie@2.0.4
  │ ├─┬ hawk@3.1.3
  │ │ ├── boom@2.10.1
  │ │ ├── cryptiles@2.0.5
  │ │ ├── hoek@2.16.3
  │ │ └── sntp@1.0.9
  │ ├─┬ http-signature@1.1.1
  │ │ ├── assert-plus@0.2.0
  │ │ ├─┬ jsprim@1.2.2
  │ │ │ ├── extsprintf@1.0.2
  │ │ │ ├── json-schema@0.2.2
  │ │ │ └── verror@1.3.6
  │ │ └─┬ sshpk@1.8.3
  │ │   ├── asn1@0.2.3
  │ │   ├── assert-plus@1.0.0
  │ │   ├── dashdash@1.14.0
  │ │   ├── ecc-jsbn@0.1.1
  │ │   ├── getpass@0.1.6
  │ │   ├── jodid25519@1.0.2
  │ │   ├── jsbn@0.1.0
  │ │   └── tweetnacl@0.13.3
  │ ├── is-typedarray@1.0.0
  │ ├── isstream@0.1.2
  │ ├── json-stringify-safe@5.0.1
  │ ├─┬ mime-types@2.1.11
  │ │ └── mime-db@1.23.0
  │ ├── node-uuid@1.4.7
  │ ├── oauth-sign@0.8.2
  │ ├── qs@6.1.0
  │ ├── stringstream@0.0.5
  │ └── tunnel-agent@0.4.3
  ├── sax@1.2.1
  ├── symbol-tree@3.1.4
  ├── tough-cookie@2.2.2
  ├── webidl-conversions@3.0.1
  ├─┬ whatwg-url@3.0.0
  │ └── tr46@0.0.3
  └── xml-name-validator@2.0.1

I'm not quite sure why there are different version numbers between the "node" and "nodejs" executables...

Please let me know if this is helpful to you,

Best regards

Hi Ismael,

Your TEMPLATE looks really nice and I'm eager to try it out myself. 
Though I'm running into problems with the Jobe server that does not recognize the required node packages (e.g. jsdom). 

Could you maybe share some information on how you setup the jobe server such that a certain set of nodejs packages are available for the scripts?

Thanks for posting, Peter. That looks like it should be very useful to other Java users.

Richard

Here's a slightly more finished version of what I've been cobbling up for Java code questions. The java_code_checkr.py is downloadble from here: https://github.com/sanderator/CodeRunner-prototypes.git.

Prototype template for static and / or dynamic checking of student answer java code specifications.

The file java_code_checkr.py should be included as a CodeRunner Support file in a prototype, eg, LOCAL_PROTOTYPE_java_checkr. Java code questions then use this prototype as their Question type.

The support file java_code_checkr.py has the functions documented below:

compile_and_run

compile_and_run(student_answer, testcode, import_static=None, xception=None, ncoding='utf-8')

Assembles code (student answer, support files, tester class.
Then compiles and (hopefully) runs the tester code.

Example of use in template:

from java_code_checkr import compile_and_run
compile_and_run("""{{STUDENT_ANSWER | e('py')}}""", '''{{TEST.testcode}}''')

 When the student answer is expected to throw a given exception there is an additional argument:

from java_code_checkr import compile_and_run
compile_and_run("""{{STUDENT_ANSWER | e('py')}}""", '''{{TEST.testcode}}''',
    xception='''{{QUESTION.parameters.exception}}''')

where the CodeRunner question type / Template params look like:

{"exception": "IllegalArgumentException"}

compile_and_findbugs

compile_and_findbugs(student_answer, testcode, import_static=None, xception=None, ncoding='utf-8')

Compiles the tester code and runs FindBugs on the bytecode.

Example of use in template:

from java_code import compile_and_findbugs
compile_and_findbugs("""{{ STUDENT_ANSWER | e('py') }}""", '''{{TEST.testcode}}''')

check_for_author

check_for_author(student_answer, existing_author=None)

Checks for an author tag in the javadoc comments. With an existing
author argument, checks that another author has been added.
This is in case the student was to modify existing code with an existing author.
If there's no additional author, then raises an error and stops further testing.

Example of use in template:

from java_code_checkr import check_for_author
check_for_author("""{{ STUDENT_ANSWER | e('py') }}""", '''{{QUESTION.parameters.existing_author}}''')

where the CodeRunner question type / Template params look like:

{"existing_author": "J. Random Author"}

check_for_enum

check_for_enum(student_answer, enum)

Verifies that the appropriate enum is declared.
If that's not the case, then raises an error and stops further testing.

Example of use in template:

from java_code_checkr import check_for_enum
check_for_enum("""{{ STUDENT_ANSWER | e('py') }}""", '''{{QUESTION.parameters.enum}}''')

where the CodeRunner question type / Template params look like:

{"enum": "DayOfWE"}

check_for_enum_in_switch

check_for_enum_in_switch(student_answer, enum_const)

Verifies that the appropriate enum constant is used in a switch case statement.
If that's not the case, then raises an error and stops further testing.

Example of use in template:

from java_code_checkr import check_for_enum_in_switch
check_for_author("""{{ STUDENT_ANSWER | e('py') }}""", '''{{QUESTION.parameters.enum_const}}''')

where the CodeRunner question type / Template params look like:

{{"enum": "DayOfWE", "enum_const": "SATURDAY"}}

check_for_extends

check_for_extends(student_answer, subclass, superclass)

Checks for 'class subclass extends superclass'.
If that's not the case, then raises an error and,stops further testing.

Example of use in template:

from java_code_checkr import check_for_extends
check_for_extends("""{{ STUDENT_ANSWER | e('py') }}""", '''{{QUESTION.parameters.subclass}}''', '''{{QUESTION.parameters.superclass}}''')

where the CodeRunner question type / Template params look like:

{"subclass": "Sub", "superclass":"Souper"}

Sorry this is coming in in bits-and-pieces but it's very much a work-in-progress.

I've now got FindBugs working in a CR template. This does pretty comprehensive static code analysis on the student answers. For instance, 

where FindBugs complains that the case statements are missing breaks, and that there's no default: case. When the code is cleaned up:

Still got some issues to work out, such as the line numbers of problems being off.

Thanks for those two postings, Peter. I've just built the first of your java_code_test.py versions into a java_general question type to be included in a file 3rd_party_PROTOYPES.xml, which will be part of the next release. The question text includes all the explanatory comment at the start of your support file. I assume you're happy with this (it includes your Copyright note)?

I also assume that this new question type is intended to replace the earlier one you submitted, so I won't be including that.

That's an interesting approach, to move most of the template code into the support file. The disadvantage is that it's harder to make use of template parameters but certainly it makes it easier to edit and manage the code. 

It seems many teachers take the path of including code to do some sort of static checking of a student's submission before running it. In our first year course we run all code through the pylint Python style checker, with tight constraints on things like maximum function length, maximum line length, use of global variables, etc. We also have our own extra code to disallow or to require the use of certain constructs, as you've done in your enum example.

In my TODO list is a plan to include pre-run checking as a separate phase of the execution sequence so that you don't get the pre-run check failures coming up as Runtime Errors. That will however add a bit of overhead, as it will yet another Jobe submission at the start.

Thanks for contributing

Richard

Moving right along...

I also noticing from letting students loose on CodeRunner led to occasional unfortunate behaviour. For example, having given them code specs where they were to use enums in a switch, they might be tempted to just go with if-then-else-if-then-else... To CodeRunner's tester the output looks like what's wanted and the test case passes. Not what I wanted.

So I've cobbled up a few code-checker modules to actually do some static code analysis (extremely basic, it's not FindBugs but the students don't know that  :^)  ) to check whether their code is vaguely in-spec. If the code is out-of-spec CodeRunner pops an informative exception and they get to try again.

This works along the same principles as my last post - there's a module file java_code_checker.py included in the LOCAL_PROTOTYPE_java_checker question prototype which does the work of code-analysis.

For the enum checker, the template looks like:

from java_code_checker import *
from java_code_tester import *

student_answer = """{{ STUDENT_ANSWER | e('py') }}"""

student_answer = assemble_student_answer(student_answer, '{{QUESTION.parameters.enum}}')

# checks if enum used
check_for_enum(student_answer, '{{QUESTION.parameters.enum}}')

# loads and filters support files
support_files = assemble_support_files()

# assembles and saves test file from student answer and support files
assemble_tester(student_answer, support_files, '''{{TEST.testcode}}''')

# take it for a spin
compile_and_run()

These checks need a bit more configuration via the template parameters. For example, for the enums example I need to configure the name of the enum via

{"enum": "DayOfWeekend"}

because I need the name of the enum in order to include the possibility of a static import of its identifiers. One day I'll regex that bit of cruft away. One day...

This is nowhere near serious static code analysis, but it seems to keep the students a bit more honest.

Right, now that I've been using CodeRunner in anger in a Java-based course, I've had another go at making it workable. Instead of customizing templates, which quickly became unmanageable, I've put the code into modules which are then loaded as support files. This reduces the amount of customization of templates to an import statement and a couple of function invocations. For example, to test student answer functionality the customized template becomes:

from java_code_tester import *

student_answer = """{{ STUDENT_ANSWER | e('py') }}"""

# manipulates student answer
student_answer = assemble_student_answer(student_answer)

# loads and filters support files
support_files = assemble_support_files()

# assembles and saves test file from student answer and support files
assemble_tester(student_answer, support_files, '''{{TEST.testcode}}''')

# take it for a spin
compile_and_run()

I've worked this into a LOCAL_PROTOTYPE_java_tester to use as a question prototype and to be customize further as needed. Pre-loading the module file java_code_tester.py into the prototype as a support file means not having to remember to load it into every question; it's already available.

The java_code_tester.py file is attached below.

CodeRunner has parameter Sandbox-Option
So we can run java programs with  -Djava.security.manager But it has not work for me.

But we can use SecurityManager this way 

public class __Test__ {
    static{
        System.setSecurityManager(new SecurityManager());
    }
    //...
}
{{ STUDENT_ANSWER | replace({'public class ': 'class '}) }}

I think this code is safe enough.

Аnd it is a very weak protection disallow imports, but we cal filter any using of standart packages by twig replace function.

Hope this information will be useful for somebody.

Certainly not the right way, but by far the simplest (until the students catch on)...

In the question prototypes I've set up (in another forum thread), the only imported packages are java.util and java.util.stream. OTOH, nothing to stop them using java.net or java.nio or whatever classes by their complete name :^( . I suppose one could filter their answers for given package name occurrences in class names and raise a fuss.