Requirements

TestFest requires Racket 5.1.3 or higher:

http://racket-lang.org/download/

TestFest requires sendmail to send account notification messages to students.

We've used TestFest on a Debian/Xen virtual machine with 256 MB RAM. Running students' code usually requires more resources than running the Web service itself.

Installation

From the command-line, create a sample assignment:

$ cd src $ racket sample.rkt

If there are no errors, you'll see the following message:

Created sample database at testfest.db. Username and password are [email protected]

From the same directory, start the TestFest server:

$ racket server.rkt

Visit http://localhost:8080 and login with the credentials above.

Try TestFest

Login as a TA, select the "ae" assignment, and then select "Test Suites" on the left. TAs can submit tests and solutions in exactly the same manner as students. Submit the following test:

(test (calc (parse '{+ 3 7})) 10)

As a TA, select "Pending Tests" and approve the test. You'll need to click Refresh to update the list.

Return to "Test Suites" and Refresh to see that the test is accepted.

Select "Solutions" and submit the following correct solution to the AE assignment:

(define-type AE [num (n integer?)] [add (lhs AE?) (rhs AE?)] [sub (lhs AE?) (rhs AE?)])

(define (parse sexp) (cond [(number? sexp) (num sexp)] [(and (list? sexp) (= 3 (length sexp))) (case (first sexp) [(+) (add (parse (second sexp)) (parse (third sexp)))] [(-) (sub (parse (second sexp)) (parse (third sexp)))] [else (error 'parse "unknown operator")])] [else (error 'parse "unknown term")]))

(define (calc an-ae) (type-case AE an-ae [num (n) n] [add (lhs rhs) (+ (calc lhs) (calc rhs))] [sub (lhs rhs) (- (calc lhs) (calc rhs))]))

Refresh a few times to see the solution pass the test suite.

Workflow

After logging in, you'll see "ae" on the top-left toolbar. This is a sample assignment to write an interpreter for (A)rithmetic (E)xpressions. Click on "ae" to reveal the assignment toolbar.

The assignment toolbar has "Test Suites" and "Solutions" on the left; these are visible to students and TAs. The items on the right-hand side are only visible to TAs.

Students submit test suites from the "Test Suite" pane. Submitted tests run against the TA-written solution, which is visible to TAs under the "Solution" pane. If the test suite passes the TA-written solution, it appears under "Pending Tests". If the test suite signals an error, TAs are not notified. However, students frequently blame TAs for writing faulty solutions or specifications. Sometimes, the TA solution is faulty, so TestFest allows the solution to be changed and allows previously accepted to be rejected.

While the assignment is out, TAs must periodically check and accept/reject submitted tests under "Pending Tests". Once a test is accepted by the TA, it is added to a global pool of tests. Submitted solutions are tested against all tests in the pool.

Creating New Users

You can invite new users from the command-line:

$ cd src $ racket users.rkt [email protected] [email protected]

This command uses email addresses as usernames, generates new passwords, and notifies users. A shorter variant:

$ racket users.rkt --suffix "@cs.brown.edu" arjun sk

Creating New Assignments

The TestFest Web application allows TAs to create new PLAI and PLAI-GC based assignments. New kinds of assignment require some programming. You'll need to add a case for checking test suites:

https://github.com/brownplt/testfest/blob/master/src/background.rkt#L120

and for checking solutions:

https://github.com/brownplt/testfest/blob/master/src/background.rkt#L48

TestFest represents students' submissions as "solution" and "test-suite" records, both defined in types.rkt:

https://github.com/brownplt/testfest/blob/master/src/types.rkt#L41

For example, the run-test function sandboxes and runs a single test suite and solution:

https://github.com/brownplt/testfest/blob/master/src/background.rkt#L221

Run-test produces an s-expression that represents test results. This s-expression is printed to a string by result->string/port:

https://github.com/brownplt/testfest/blob/master/src/background.rkt#L13

The result is stored in the solution-status field of the solution:

https://github.com/brownplt/testfest/blob/master/src/background.rkt#L67

This result is displayed to students, so it must not print confidential information, such as other students' tests and solutions.