This page provides answers to frequently asked questions about Web-CAT. If you have a question that does not appear here, add it to the end of the list and check back for an answer later.

Our FAQ is just getting started, so feel free to contribute questions (or answers!).

1. What Is Web-CAT?

For a quick tour, watch the SubmissionWalkthrough movie (24Mb, 5min).

The name Web-CAT stands for Web-based Center for Automated Testing. It was originally conceived as a portal for student-focused services to help teach software testing, but the first (and most successful) plug-in was one for automated grading (the Grader Subsystem). Web-CAT has since become well-known because of its automated grading capabilities, which seem to have a greater impact on student learning than any of the other services platform for providing a number of student support services to help students learn software testing. Although the Grader Subsystem supports traditional models of automated program grading--by comparing the output produced by student code against some reference--it also supports grading of assignments where students do their own testing and helps encourage test-driven development.

1.1. What is Test-driven Development?

Test-driven development (also called test-first coding) is a development technique where one writes unit tests and new code features in alternation. For each small feature or addition to be made, the programmer first writes a corresponding set of unit tests that express the expectations and intent for the feature's behavior, and then writes new code. The tests are used to check that the code meets the expressed expectations. Ideally, one constantly bounces back and forth between writing a test or two, and writing a few more lines of code. See the TestDrivenDevelopment page for more information and references. The Web-CAT Grader allows a student to submit his or her test cases along with the solution, and grades on test validity and test completeness as well as code correctness.

1.2. Does Web-CAT Support Assignments That Don't Require TDD?

In short, yes.

The actions used to compile, execute, and evaluate student code can be tailored by the instructor, who can upload one or more "grading plug-ins" to be used. Further, the feedback generated for students is completely controlled by these plug-ins. As a result, virtually any assessment and evaluation scheme desired can be used with Web-CAT, including traditional "output comparison"-style grading.

Since the author of Web-CAT advocates the use of TDD, most existing grading plug-ins follow that approach, but any other approach can be used as well. In fact, a fairly sophisticated "output comparison" back-end written has been written in Java for use with Web-CAT (but is currently still in development status).

In addition, the JavaTddPlugin allows instructors to set up assignments where student-written tests are purely optional, and scoring is based only on reference tests provided by the instructor (although the default configuration is to require student-written tests). When an assignment is configured this way, students will get extra feedback if they write their own tests (including full coverage analysis, for example), but whether or not they include tests, and how well they cover their own code will not affect scoring.

There also is a grading plug-in for C++ that provides for more traditional stdin/stdout execution of student programs. Students can write their own input file(s), and Web-CAT will show them the results. Instructors can provide "reference" input file(s) as well, with results that are only visible to course staff. The CppStdioPlugin does not include student testing performance in its scoring (instead, the CppTddPlugin provides for grading C++ programmers on how well they test their own code).

At the same time, however, remember that the learning advantages that have been experimentally documented for students using Web-CAT are based on assignments where students did write their own tests using TDD. If you don't require students to write their own tests, or grade them on how well they do this, your students may not get as much value out of their assignments. See OurPublications for more details.

As more instructors use Web-CAT, we anticipate that a wider variety of grading plug-ins will become available.

1.3. Why Should Students Use TDD?

When students write tests for their own code, they initially see it as just extra work. However, after using it for a while, students quickly begin to see benefits they can appreciate themselves. Students are less likely to turn work in late, have greater confidence that their own code works the way they intended, have an easier time finding errors as they are introduced, and avoid "big bang" integration. See OurPublications for articles describing empirical evidence for these claims (particularly our article in JERIC).

1.4. What Programming Languages Does Web-CAT Support?

Web-CAT provides flexible, extensible programming language support for any programming language that can be installed on the server. The actions used to compile, execute, and evaluate student code can be tailored by the instructor, who can upload one or more "grading plug-ins" to be used. As a result, to use a new language, all the instructor needs to do is ensure the language is installed on the server, and then provide the necessary grading and feedback plug-in(sor use existing ones where possible. Plug-ins have already been written for Java (1.4 and 1.5), Scheme (both MIT Scheme and DrScheme), Prolog (SWI), Pascal (FreePascal), Standard ML (SML/NJ), and c++ (g++ 3.4.x), and new plug-ins can be adapted from existing ones.

1.5. Do I Need to Use Eclipse?

No. Anyone can submit a zip/jar/tar/tgz file directly using a web browser, as shown in the SubmissionWalkthrough movie (24Mb, 5min).

For students who do wish to use Eclipse, we have a submission plug-in they can download and install, but it is purely for convenience.

The plug-in will add a "Submit..." link to the Project menu. When clicked, this action will query the Web-CAT server to find out what assignments are open and let the student pick one from the list. It will then jar up their eclipse project, submit it to Web-CAT, and pop up the results within Eclipse (or in your web browser, depending on your OS).

You can see a screen shot of the menu addition as well as a screen shot of the submission wizard.

The plug-in installs the normal way (unzip it in your eclipse/plugins directory). Configuration is via Window->Preferences->Electronic Submission, and all you have to do is set the URL to the Web-CAT server.

The submission plug-in can be downloaded from our sourceforge site. Read the recipe on how to ConfigureEclipseSubmitter.

1.6. Can I Use BlueJ's Submitter?

Yes, it is also possible to use BlueJ's submitter extension to submit directly to Web-CAT. That extension ships with stock BlueJ distributions. All you need is a one-line config file to point it to the Web-CAT server in order to use it. Read the recipe on how to ConfigureBlueJsSubmitter.

1.7. Can I Supply My Own Code or Data Files for Student Assignments?

Yes. Instructors who are setting up assignments have their own personal "file space" where they can upload resources that they wish to use for one or many assignments (data files, supplementary jar files, configuration files for static analysis tools, etc.). You can manage this file space through Web-CAT's web interface, including uploading files, creating subdirectories, downloading/deleting items or entire folders, and even direct editing of text files on-line.

The Java grading plug-in allows instructors to provide a supplementary jar or an (arbitrarily complex) subdirectory of class files that you would like to have added to the classpath for grading assignments. You can set up a common set of options that you'd like to reuse across several assignments where you describe these kinds of parameters just once (and can have as many of these as you'd like). For any individual assignment, you can also specify an additional supplementary jar or classpath dir that applies to that assignment only.

The C++ grading plug-in provides a similar mechanism for instructors to supply a directory of header files to place on the include path, compiled library files to include at link time, or extra data files to be placed in the current working directory while student code is evaluated.

On the VT server, we preserve the contents of this "personal file space" from semester to semester, so that once you take the time to upload your favorite resources, they are always around for you to use on your assignments (until you decide to change). This is also the same place where instructors can upload reference tests to use in automatic grading.

1.8. How Do Instructors Specify Their Tests?

For the Java grading plug-in, standard JUnit test classes are used (as *.java source files to be compiled against student code). You can have as many as you want for a given assignment. There are no limitations except for some basic naming conventions. Students use the same format for the test cases they write. JUnit is widespread enough that there are many useful resources for teaching and learning how it works.

In the future, we are imagining optionally expanding this to provide our own custom subclass of junit.framework.TestCase that transparently adds some useful features that current users have requested. Instructors who want to take advantage of these features will only have to change the name of the baseclass they use, but this will be optional.

The C++ grading plug-in uses CxxTest-style tests. Other plug-ins use different formats, depending on the programming language used for submissions.

1.9. As an Instructor, Can I Run My Own ANT Script on Student Code?

The current Java grading plug-in actually uses a carefully designed ANT script (see http:/ant.apache.org/) to control the build and analysis process for student code. It is pretty general, in that it can handle virtually any project structure that folks are likely to use. However, it does not support the use of any code generation tools to generate Java files from other non-Java sources.

However, instructors can actually upload their own plug-ins and use them for assignment grading all by themselves, using just the web interface. If you want to use a custom ANT build script, the best thing to do might be to take the current Java plug-in, add to or enhance its build.xml file, and upload it as a custom plug-in for your own use. You can edit/modify/re-upload a new build.xml file for your custom plug-in at any time if you need to change its behavior.

The C++ grading plug-in also uses ANT to control the build and analysis process for student code. We use the various ANT tasks from the ant-contrib project at http://ant-contrib.sourceforge.net/ to compile and link C++, regardless of the directory hierarchy or project structure used by students.

2. Using Web-CAT

2.1. What's It Like to Submit an Assignment?

For a quick tour, watch the SubmissionWalkthrough movie (24Mb, 5min).

Students can upload their submissions via a web browser, and no other software is required to use Web-CAT. Multi-file submissions are accepted in the form of zip or jar archives, and are automatically expanded as needed. All submissions and their results are archived independently. Further, automated submission support is also provided for users of BlueJ and Eclipse. Extending submission support to other environments is also possible.

2.2. What's It Like to Set Up an Assignment?

For a quick tour from the instructor's side, watch the CreateAnAssignmentWalkthrough (34Mb, 9min).

2.3. What Options Can Instructors Control?

Because assignment processing and feedback generation are completely plug-in-based, the exact options available to an instructor depends on the plug-ins that are being used for an assignment. However, for an example, you can browse through the JavaTddPlugin/ParameterDescriptions, which lay out the controls available for our most-widely used plug-in (for Java).

2.4. How Do Students Receive Feedback?

Students receive their feedback via their web browser. The form of the feedback is completely customizable by the grading plug-ins used by the instructor, but typically takes the form of an integrated web-viewable feedback report that includes a scoring summary and a color-coded, syntax-highlighted view of each source file. For languages where static or dynamic analysis tools are provided, this report may also include an execution count for each line, with under-tested lines highlighted, as well as individual line-by-line errors or diagnostics. Instructors, TAs, or grading staff can directly add comments and point deductions to the color-highlighted source code view using only a browser. For a visual example of typical feedback produced by the JavaTddPlugin, watch the SubmissionWalkthrough movie (24Mb, 5min).

2.5. What Happens When Hundreds of Students Submit Simultaneously?

We typically run 500-600 users per semester on our main server at Virginia Tech. Of course, these are spread over multiple courses, and usually no more than 100-150 have simultaneous deadlines. Load runs fine most of the time.

The only load problems students see are when an assignment deadline is approaching and many students are submitting, especially if multiple courses have accidentally assigned work due at the same time.

Processing a single Java assignment takes between 15-45 seconds, on average. Of course, this depends on many factors, including the likelihood that students will write code with infinite loops or other non-terminating behaviors. The server only grades one submission at a time for a number of technical reasons. Handling extremely large number of student sessions and submissions is not a problem, but submissions queue up and are processed on a first-come-first-served basis.

So the problem is that if 100 students all submit at the same time, it will take about 1500-4500 seconds (or 25-75 minutes) for them to all receive feedback. Students can continue to access the server, look up results (say, on earlier submissions), and make new submissions, but none of that will speed up when results come back on work they've already submitted.

Web-CAT uses several techniques to ensure single individuals don't monopolize the queue, and also to make sure that administrative requests (say, to regrade assignments) don't affect new submissions from students.

The bottom line is that students learn very quickly (and we tell them early) about the load as an assignment deadline approaches. I tell my students explicitly that if they're making a submission within 1 hour of the deadline, they should expect to wait--possibly a long time--before they see their feedback. Instead, we encourage them to start early and submit early if they want to take advantage of the feedback. In other words, the delays near deadlines are very predictable, and are the price for waiting too late. If you want to avoid them, it is simple--don't wait until the last minute.

In practice, this actually works quite well. Students are very good at moderating their own behavior once they know what is going on. The key is to make sure they have reasonable expectations about what will happen, so that they don't get surprised when they can least afford it.

2.6. Can Server Response Times Be Improved by Configuration?

A faster server that is dedicated just to Web-CAT use is best. A multi-processor server is even better. But there's not much you can do beyond that, I don't think. It is possible to move the database off to a separate server, use a web server (like apache) to serve static content instead of forcing Tomcat to do that also, and even move the web server off to a separate server. However, there will still be queue problems if hundreds of students are shooting for the same deadline.

Overall, though, this hasn't been a problem in practice. While our class sizes have gone down, we used to regularly use Web-CAT in courses of up to 300 students, and one of the longest-running non-VT servers regularly has classes that are even larger. Once students learn to expect long queues when a deadline approaches, they can deal with it.

If you're proactive about communicating expectations and explaining how to avoid the long waits, that is w-a-y more effective than server tweaking. I personally haven't seen queue sizes > 10 in any of my classes in the past year, even with classes well over 100 students in size. But I attribute that to students who have some experience with Web-CAT and who know to anticipate (and avoid) the delays.

We are experimenting with cluster computing this year so that we can put a pool of CPUs on the grading queue, but at best this gives just a linear speedup.

3. Using Web-CAT in Your Courses

3.1. Can I Try It to See What It is Like?

You can TryOutWebCat from our server to see how it works.

3.2. How Can I Use It?

After you TryOutWebCat, you can request to UseWebCatRemotely from our server for your course. We offer remote hosting for any institution that wishes to use it rather than maintain their own server. We currently host courses at elevent different universities from our main server, which has processed over a quarter million submissions (approximately 25 thousand each semester).

If you prefer, you can also install your own server. See the "Can I Install Web-CAT at My Institution?" FAQ earlier on this page.

3.3. Why Does Web-CAT Generate Warnings on Most of My Source Lines?

Some of the "tryout" assignments you can use to take Web-CAT for a test drive are configured with Checkstyle and PMD settings we use in our freshman classes, and which may differ in some minor ways from your preferred coding style. In particular, the Checkstyle options expect code to be indented every 4 columns using spaces instead of tabs (tabstops are interpreted as being every 8 columns). Checkstyle is also configured to enforce the use of braces in all control structures. Further, opening braces are expected to begin on a new line immediately beneath the linguistic structure they are a part of, instead of being "cuddled" on the right edge of the previous line.

Feel free to use the alternate versions of the "tryout" assignments that are not labelled as "tighter formatting" or "full static checks" (and which have more liberal settings), or adjust your coding style for the "tighter" tryout assignments.

If you decide to use Web-CAT in a class, you are free to configure the Checkstyle and PMD settings to values that suite your institution's preferred style conventions.

4. Getting a Copy of Web-CAT

4.1. Is Web-CAT Open Source?

Yes, Web-CAT is licensed under the terms of the GNU General Public License. Source code and binary releases for both Web-CAT and our EclipsePlugins are available via our SourceForgeProject.

When we originally started the project, there were a number of obstacles to creating an OpenSourceWebCatProject, but we have overcome all of them in the intervening time.

4.2. Can I Install Web-CAT at My Institution?

Yes ... or we offer remote hosting on our professionally maintained production servers. The choice is yours. Currently, there are more than 15 separate institutions running their own Web-CAT servers. See Setting Up Your Own Web-CAT Server later in this FAQ for more details.

5. Setting Up Your Own Web-CAT Server

5.1. How Do I Install Web-CAT?

Web-CAT is distributed as a WAR file, and requires Java (1.5 or greater), MySQL, and an appropriate servlet container (such as Tomcat). Installing Apache as well can help speed response by delivering static html resources (additional configuration required). If you have a server equipped with this software to run web applications, or if you are reasonably familiar with installing and configuring web applications, then you can install Web-CAT. There are live servers running Web-CAT under linux (a couple of different flavors), Windows, and OS X.

The servlet has a built-in self-installation wizard that runs the first time you visit the application. The self-installer walks you through the process of setting up database connectivity, creating database tables, picking your authentication mechanism, creating an administrator account, and populating the database with sample data.

Before trying to install Web-CAT, be sure to read the SystemRequirements. Then you can watch the InstallationWalkthrough and read how to InstallWebCat.

While we strive to make installation as easy as possible, we're still working the kinks out of some installation issues on some server platforms, so please be patient with us. Our development team will be happy to help you get off the ground if you're willing to invest the required effort. In return, we hope that you will contribute to the Web-CAT project by helping us improve our installation instructions--in the form of recipes in our WebCatCookbook or FAQ answers on this page. For more information, read HowToContribute.

5.2. How Do I Create New Courses?

Most first-time users set up a course or two during Web-CAT installation as part of the self-installation wizard. Be sure you do not want to just CreateaCourseOffering--check on the difference between CoursesAndCourseOfferings if you are not sure. To add a new course, follow the cookbook recipe describing how to CreateaNewCourse.

5.3. How Do I Create New User Accounts?

If you are an instructor and you want to create an account for a student, the easiest way to create their account is to create a CSV file containing their information and then UploadYourCourseRoll. This will create their account if it does not already exist, and will also enroll them in your course. You can do this for your entire class, or do it for just one or two students you have to add later.

If you are an administrator and want to add a new account for an instructor or anyone else, follow the recipe to CreateaNewUser.

5.4. How Do I Download Scores?

If you are an instructor using Web-CAT and wish to export your assignment scores for use in Excel or for uploading to an on-line gradebook, use the Grading tab and select the Download Scores subtab. Scores can be exported in Blackboard-compatible or Moodle-compatible CSV formats, or in a more detailed CSV format that you can load into a spreadsheet and manipulate yourself.

If you'd like to be able to import your downloaded scores into another on-line tool that requires a different CSV format, contact our development team. If you can give us the details of the CSV format you need and the external tool is common enough that others will use it, we'll definitely consider adding additional download format options.

6. Common Installation Issues

6.1. NoClassDefFoundError on NSTimeZone

Some people have reported an unhandled error when trying to install Web-CAT. The installation wizard does not appear and instead a diagnostic message indicating the servlet failed to initialize correctly appears, citing the following Java exception:

java.lang.NoClassDefFoundError: com.webobjects.foundation.NSTimeZone

This error appears on linux systems that are configured to use gcj as the default Java installation. The best solution is to switch to using Java's SDK to run Tomcat--see SwitchToSunJdk. If you also UpdateBootstrapJar, you'll get better diagnostics for this problem instead of an unhandled exception (but the solution is still the same).

6.2. NoClassDefFoundError: Could Not Initialize JFreeChart

On linux servers, some people have reported a problem after a successful install, once their server is in use. They report that Web-CAT produces unexpected exceptions when you visit any pages that contain charts, such as those graphing student results. The Web-CAT exception page cites the following Java exception:

Error: java.lang.NoClassDefFoundError
Reason: Could not initialize class org.jfree.chart.JFreeChart

Or possibly:

Error: java.lang.NoClassDefFoundError
Reason: Could not initialize class sun.awt.X11GraphicsEnvironment

If this occurs, check the environment variables for the pseudo-user under which Tomcat is running. Check for any definition for the DISPLAY environment variable that is visible to Tomcat, an remove any such definitions.

This problem appears to be caused internally in Sun's AWT implementation, where Java is trying to connect to the X11 display identified by the DISPLAY environment variable, but the user id under which Tomcat is running does not have permission to connect (or there is no active DISPLAY). Removing the environment variable setting for the Tomcat user will fix this problem. You may also need to add -Djava.awt.headless=true to Tomcat's startup arguments as well.

6.3. Unexpected Exceptions During Installation

If you are using Tomcat and deployed Web-CAT simply by adding the WAR file to Tomcat's webapps directory, you may inadvertently end up with a corrupted install. Because of the size of the Web-CAT WAR file, race conditions during a live auto-deploy under Tomcat can cause problems in correctly unpacking the WAR. To remedy this, shut down Tomcat entirely and completely remove any Web-CAT deployment directory inside the webapps directory. Then manually unzip the WAR file yourself, say, into the directory webapps/Web-CAT (there is no need to keep the version number in the app directory). After unpacking, remove the WAR file from the webapps directory entirely (you can move it to another location if you like). Then restart Tomcat.

7. Contributing to Web-CAT

7.1. Can I Help Develop the Code?

Yes! See a discussion of HowToContribute. You can also contribute through our SourceforgeProject.

Right now, the best place to contribute is to work on new plug-ins. We are currently working on plug-ins for python and ruby. If you're interested, contact our development team.