File upload/submission question type

File upload/submission question type

por Trent Lewis -
Número de respuestas: 13

Hi All,

I'd like to float the idea of a "file upload/submission" question type for CodeRunner.  CodeRunner is excellent for testing solutions to questions that small (a single/flle/page of code) and targeted and I commended the creators for their excellent work. 

However, students are appear to more comfortable working in an IDE when developing code solutions and this is how one would develop code in the workplace.  CodeRunner is an excellent tool for small quiz items, however, students feel a disconnect between the code in the IDE and the snippets in the CodeRunner quiz item.  This disconnect appears to lead to an added anxiety during a test.   This might be overcome with more practice with CodeRunner quiz items out of test conditions and more adept students do not raise concerns for their own work, but novice students tend to end up submitting code that compiles/runs in the IDE, but does not “work” in the quiz item.  

A potential solution might be to allow the student to develop in the IDE and submit to the quiz a single file or a package file (for Java this might be a .jar file) that is produced from the IDE.  If the quiz item had an upload file option this could allow the student to upload the package file that could be tested in the CodeRunner Quiz framework on the jobeserver still using the output text matching.”

It could also be interesting to entertain the idea of allowing code to be retrieved from a version control repository.

Thanks in advance,

Trent

En respuesta a Trent Lewis

Re: File upload/submission question type

por Richard Lobb -

Thanks for raising this Trent.

“Students are appear to more comfortable working in an IDE when developing code solutions and this is how one would develop code in the workplace. 
Totally agree. That has always been our model here, too. Although having said that, I think there are quite a few other users who don't operate that way. For example, the University of Auckland's Moodle site says "Because the system is on-line students do not need to have programming tools installed on their lab or private computer."

This might be overcome with more practice with CodeRunner quiz items out of test conditions
Yes, I think this is an important point. On-line tests and exams can be very stressful for students. In the past, when we had a special on-line exam environment that students were thrown into only for exams, a significant percentage of students crashed badly. But we now use CodeRunner for all labs, assignments, tests and exams. Students work with it both at home and in our labs from day 1 in the course. As a result, I think our on-line exams are no more stressful than written ones and possibly even significantly less stressful. It is still very important, however, to start every test or exam with some really trivial questions (as with written exams) to help students relax a bit.

Now, I’m not even sure that it is possible to have a “file upload” option for a quiz item in Moodle, so the request might not be technically possible anyway!
I certainly could have an option to upload data from a file rather than cutting and pasting. However, would this really  reduce the stress for your students? My feeling is that the stress arises because of the uncertainty and unfamiliarity with CodeRunner rather than the particular sequence of mouse and keyboard events required to transfer the code across.

If the quiz item had an upload file option this could allow the student to upload the package file that could be tested in the CodeRunner Quiz framework on the jobeserver still using the output text matching.
Yes, this is a reasonably compelling use case. However, presentation of the results is now less clear as students can no longer see on screen exactly what they submitted. They would have to be able to download again the jar file they submitted to see what the errors were. [I'm thinking of cases where they somehow submitted an incorrect jar or one containing the wrong version of a file.] 


It could also be interesting to entertain the idea of allowing code to be retrieved from a version control repository.
Uploading directly from a repository into Moodle is semi-doable even now. The "student answer" could be just the student's repository URL, and the template could begin with something equivalent to

git clone {{ STUDENT_ANSWER }}
The rest of the template could then cd into the directory and test the student's code. I've done something very similar to that in a web programming course, where students developed a website to certain specs and just pasted its url into the student answer.

However, this approach has a few problems, such as:

  • As with jar file upload, it's harder to understand the table of results, as the submitted code isn't clearly visible.
  • Grading is not automatically triggered by a commit to the repo; students have to independently request it.
  • The state of the repo changes over time so it would be difficult to go back through a student's submissions to see what happened unless the template were able (somehow) to record the state of the repo as the time it was being graded on each submission.
  • jobe needs to have git installed and the firewall configured to allow access to the repo(s).
I'd be really interested to hear from anyone else on the subject of file upload. Who else wants it? Why?

Richard

En respuesta a Richard Lobb

Re: File upload/submission question type

por Trent Lewis -

Any further thoughts on a file upload.

I've investigated the version control (git) option, but my institution is not particularly okay with opening up the firewall on the jobeserver :(

I'd be happy to trial and test a file upload question type :)

Trent

En respuesta a Trent Lewis

Re: File upload/submission question type

por Tim Hunt -

A file upload option could be implemented. The standard Moodle 'Essay' question type shows that the Moodle quiz system is capable of handling files as part of a question response.

However, one of the things that gives CodeRunner its flexibility is the use of a template to combine all the bits of code from the teacher and student. One would need to think through how that should be extended to cope with responses involving files.

For a response that is a single file, it is likely to be easier for the student to just copy and paste the content of their IDE into the question.

En respuesta a Tim Hunt

Re: File upload/submission question type

por Richard Lobb -

OK, I'll look into copying appropriate bits of the essay question type into CodeRunner to allow file upload, but I'll do this only in the development branch for a start, as I'm still not sure where it's going.

Tim's question of how to deal with the files is a good one. My assumption is that any files attached to the student submission should just be copied into the working directory before the template-generated code is run. But then what?

The easiest approach from a developer's standpoint, as it doesn't require any extended functionality, is to use the template code as a control script, probably written in Python. It could just list the working directory to see what was uploaded and then take whatever action was necessary, e.g. unzipping a .zip attachment or compiling a Java program with a -jar <upload.jar> command line parameter. 

For extra flexibility I could add a template parameter FILE_NAMES, which contains the names of all uploaded files. One could then deal with uploaded files at the Twig level rather than at the template_scripting level. However you're limited in what you can do there, e.g., you can't add command line options for compiling. So I'm not sure that having the FILE_NAMES variable really helps much and it might just confuse inexperienced authors. Mostly you're going to have to fire the job off to Jobe for execution anyway, so dealing with the files there, in a much more powerful language, would usually be my preferred approach. Nowadays virtually all our advanced question types are written as Python scripts.

Thoughts, anyone? Particularly, Trent - what did you have in mind?

BTW: don't expect any lightning action on this - our second semester is about to start and I don't usually feel like playing with CodeRunner development while I'm teaching.

Richard


En respuesta a Richard Lobb

Re: File upload/submission question type

por Trent Lewis -

Getting back into my CodeRunner head space (many months later)!

Did you progress any further with this?  I like your thinking above.  I agree that if a submission is a single file (that is text only) that it should be just a cut'n'paste into an answer box.  But larger, well-formed code might require multiple files (e.g. each class in Java should be in a separate file).

I do also see that this file upload for complex, multi-file code is probably not what CodeRunner is conceptually designed for as a quiz is more about testing parts of code, rather than large assignment submission.  However, CodeRunner works so well that it'd be a great extension ;)

I could see it as the student produces the smaller parts in the first few questions of the quiz and then needs to bring it all together with a multiple file submission (or a jar/binary) to show understanding of the bigger picture would be good.  This allows students to get marks for the "simpler" components and those who can conceptually bring it together gain extra marks at the end.

So, I am happy with having a file upload and then processing in python.  A FILE_NAMES would be a great idea.

Happy to test out anything.

Trent

PS. just about to starting investigating the CodeRunner and version control idea...

En respuesta a Trent Lewis

Re: File upload/submission question type

por Richard Lobb -

This is very timely, Trent, as I'm just switching back into CodeRunner development mode, now that our teaching term has (almost) finished. I've got a list of things I want to do; file upload is on the list, near the top, so it should happen.

I think I'll just treat files uploaded by the students the same as support files included by the author. They just get copied into the working directory. Also, they're read only so they can't get changed when testing, which could introduce unwanted side-effects between tests.

I'll post back when it's done.

Richard


En respuesta a Richard Lobb

Re: File upload/submission question type

por dirk grunwald -

You may also want to look at http://www.inginious.org -- this is a autograding system that can interface with Moodle using LTI. It doesn't fit into a 'quiz context' but provides more capabilities on the backend grading (...because you have to cons it up yourself using a shell script..). It uses docker containers during evaluation so is slower / less responsive.

There are other similar options we've explored - we use coderunner extensively in our labs, but are looking for "assignment grading" solutions that need to integrate with version control but don't need to be as responsive.

For browser based development, there's the Moodle "VPL" ( http://vpl.dis.ulpgc.es/ ). We've also been using Cloud9 (c9.io) but that's now merging into Amazon and harder to use. Other web IDE's like codenvy are an option, but we haven't explored them yet.

We've considered inginious.org, autolabproject.org and okpy.org as replacements for an in-house solution. Our current thinking is to go with inginious in the short term and then adopt a solution using a university hosted gitlab and standard CI tools like jenkins. To handle the "file-up load", we were going to create an LTI application that uses a token to manipulate the gitlab instance and e.g. check in a student's solution for them. This would mean a simple interface for students who have yet to master git, but full git functionality for later classes.

En respuesta a dirk grunwald

Re: File upload/submission question type

por Trent Lewis -

Thanks for the hints and pointers.

I have just the other day successfully created a question type in CodeRunner that takes as an ‘answer’ an SVN revision number and then the jobeserver checkouts from a locally hosted subversion server, compiles and runs (using ant) and then compares against the tests given in the CodeRunner question setup.  The student just uses a standard IDE to develop the code and commits to the repository.  Allowing the student to specify the revision number as the ‘answer’ gives them control over what they submit and allows multiple submissions.  They could even, in theory, complete all the work, committing each stage/checkpoint  to the revision control system and then go to Moodle/CodeRunner and submit each of the revision numbers that satisfy the different questions.

It still needs some fine tuning but happy to share once I have made it a bit more generic and sort out some quirks.  One of the trickiest parts was getting the University IT to allow holes in the various firewalls!

I’m sure Richard will give me a slap over the wrist for this as I’m sure that is not the intention of CodeRunner :)

Trent

En respuesta a Trent Lewis

Re: File upload/submission question type

por Richard Lobb -

Certainly no slap over the wrist from me - it sounds great, Trent! Please share. 

I did something vaguely similar a few years ago when teaching a web programming course - the student's "answer" was an URL of a website they'd implemented, and the template code then explored and graded their site. And like you, I had to punch holes in the firewalls to make it happen.

I'm only a control freak with regard to CodeRunner itself - the more uses you can find for it the better, as far as I'm concerned :)

Richard

En respuesta a Richard Lobb

Re: File upload/submission question type

por Trent Lewis -

I’ll keep you up to date and share it when it’s done. 


Trent

En respuesta a Trent Lewis

Re: File upload/submission question type

por Trent Lewis -

So...I just finished a semester using a CodeRunner+SVNRepo question type...and...it went really well!

It allow students to commit their complete project and any support files from their IDE (NetBeans and Java in this case) which also gave their revision number and then head over to Moodle/CodeRunner question and submit the revision number.  The question then checks out the code (based on the revision where the root of repo was based on the students Moodle username so no cross contamination or code stealing) and used Ant to build and run the code which was then subjected to the standard CodeRunner test cases.  We had to set up a dummy SVN user that had ip restricted access to the repository rather than hard coding a text password (students managed to find that quickly!)

Students seemed to easily get the hang of it and it allowed me to automatically provided feedback on more complex programming problems (something bigger than a cut and paste allows).  It also gave students more practice at using version control which is always good.  For major assignments I could also easily checkout the final code revision and review it for myself (and use coderunner as a self-review tool for the students).

I'll sanitize the question type a bit more once my teaching settles and maybe even make some notes on it if anyone is interested.

Trent

En respuesta a Trent Lewis

Re: File upload/submission question type

por Richard Lobb -

Great posting - thanks Trent. I forwarded it to all our Software Engineering staff to show them how things should be done :)  I'd certainly be interested in hearing more when/if you have time.

Richard

En respuesta a Richard Lobb

Re: File upload/submission question type

por Jim Newton -
Was feature implemented?

Having students copy/paste large amounts of code into the text box which gets magically spliced into template code is very brittle.  If the user's code is not parsable, then the error messages are completely useless.  For example if the user code is missing a parenthesis or quotation mark, the output does not indicate such.
If the user's code was available as a file, then it would be easy to tell whether that file is parsable.