Supressing Compilation Errors

Greetings everyone,

  So, part of the description of this discussion area is, "...how to write better tests for various classroom purposes." That's exactly what I'm trying to do. I'm trying to supress compiler errors and replace them with "gentler" reminders to check the assigment description. Students often make submissions that contain an incorrect class name, method declaration, or Java package. I'd like to be able to catch the cryptic (at least to them) "cannot find symbol" messages and replace them with my own. Is this possible?

Thanks,

~Jim

Groups:

Comments

Stephen Edwards

Funny you should ask

We've done a fair amount of work in this area, and regularly do this for most of our Java assignments.

First, let me describe the approach we've been using for a long time, and then tell you where it is going.  Most of our Java JUnit reference tests are written using Java reflection so there are zero compile-time references to or dependencies on the student's work.  This means there is never such a thing as a "compilation error" in the reference tests.  Instead, all references to student code and/or features are resolved at run-time.  We don't use Java's reflection API directly, though, since it is clunky.  We have a small helper library that provides all the needed features to write code that is about the same size as the original (virtually line for line), but that uses reflection instead of directly referencing student classes.  Further, any errors in reflective lookups turn into exceptions with nice, student-friendly messages, which then turn into test case failures where those messages become the Web-CAT "hints".  As a quick example, instead of writing this:

public void testSomething()
{
    Person p = new Person("First Last");
    assertEquals("First", p.getFirstName());
    p.transmorgrify(17);
    assertEquals(23, p.getAgeMinus(5));
}

We would instead write something like this:

public void testSomething()
{
    Object p = create("Person", "First Last");
    assertEquals("First", invoke(p, String.class, "getFirstName"));
    invoke(p, "transmorgrify", 17);
    assertEquals(23, invoke(p, int.class, "getAgeMinus", 5).intValue());
}

It's not perfect, but it is manageable. The error messages cover all the failure possibilities (misnamed classes or methods, forgetting to declare something public, incorrect parameter signatures, you name it), and also handle real Java-like method lookups (inheritance search, overloading, parameter type matching, auto-boxing/unboxing, etc.). One of our students co-wrote a journal paper about the library, which serves as its main documentation.

So yes, it is definitely possible. As for where it is going, we've also developed a bytecode rewriting system that allows plain, normally written reference tests that have been successfully compiled (say, against your reference solution when you tested the assignment before publishing it :-)) to be transformed into a purely reflection-based equivalent using this library. This would allow instructors to write "plain old" reference tests just like always, but transparently get the advantages of the pure reflection-based approach without having to write any of it by hand. Unfortunately, there are a couple of technical issues that are keeping us from going production with this yet.

If this approach is something you're interested in, let me know and I can point you to the paper describing the library, as well as to the library's download location. And, of course, if/when you have suggestions for improvement, they would be very welcome!

earlyjp

Thanks!

Hello Stephen!

Once again, thanks for your prompt attention and pointer to the Reflection API. I experimented a bit with this last night, and I developed the following (not a complete test):

  @Test
    public void testInstance() {
        // Check that the class can be instantiated
        Object o = null;
        Class classDefinition = null;
        try {
            classDefinition = Class.forName("csc212hw3.Calculator");
            o = classDefinition.newInstance();
        } catch (Exception e) {
            fail("Could not create an instance of the Calculator class. Check the class name and Java package for correctness.");
        }
    
        // instance created -- check method declarations
        assumeNotNull(classDefinition);
        Method[] methods = classDefinition.getDeclaredMethods();

        // Check correct number of methods
        assertTrue("Incorrect number of methods in the Calculator class. Expected: 6, but found " + methods.length, methods.length == 6);

        // check for integer version of max()
        boolean correct = false;
        for (Method m : methods) {
            //System.out.println("\nDEBUG: Generic - " + m.toGenericString());
            if (m.toGenericString().equals("public int csc212hw3.Calculator.max(int,int)")) {
                correct = true;
            }
        }
        assertTrue("The integer version of the max() method is not declared correctly", correct);

and so forth. I can produce error messages in this fashion that are more instructive, and that's great.

There is a lot written in the JUnit documentation about how tests should be independent of one another, and therefore the order of test execution is arbitrary. However, I would clearly want to run the above test before running something like this:

 @Test
    public void TestMin2() {
        Calculator calc = new Calculator();
        // Check when a < b < c
        assertEquals("Double version of min() produces incorrect value", 20.1, calc.min(20.1, 20.2, 20.3), 0);
        // Check when a > b > c
        assertEquals("Double version of min() produces incorrect value", 20.1, calc.min(20.3, 20.2, 20.1), 0);
        // Check when a = b = c
        assertEquals("Double version of min() produces incorrect value", 20.2, calc.min(20.2, 20.2, 20.2), 0);
    }

I did some experiments a while back with the JUint @BeforeClass directive and placing Reflection statements in such a test, but that led to unexpected reporting behavior in Web-CAT (although, I can't remember exactly what I noticed). Do you concern yourselves with test dependency/ordering, or do all of your tests contain the Reflection examples you noted above?

Thanks again,

~Jim

P.S. I will read the paper you mentioned.

P.S.S. Web-CAT is beyond AWESOME!

Stephen Edwards

> P.S. I will read the paper

> P.S. I will read the paper you mentioned.

You can find it here:

http://www.bentham-open.com/tosej/articles/V007/TOSEJ130422001.pdf

Stephen Edwards

The real problem is that the

The real problem is that the second test your wrote (testMin2()) directly references the class, its constructor, and the method(s).  If there are any mistakes in the student's code, this test method won't even compile. In fact, any test class it is contained in won't compile. So the problem is worse than wanting to run testInstance() before testMin2()--if testInstance() fails, it may indicate that testMin2() won't even compile.

If you remove testInstance() entirely, and translate testMin2() to the format I outlined above, then three things happen. First, testMin2() always compiles, so that's never a problem. Second, if any methods used in testMin2() are missing or misdeclared (the same goes for the class, its constructor, etc.), then testMin2() will fail with an appropriate diagnostic indicating what the problem is/was (which is basically what testInstance() is trying to do). Third, your tests can remain order-independent, since testInstance() has been completely eliminated and there is no more need to run a "pre-check" before testMin2(). Basically, translating testMin2() as I outlined above causes it to act as both its own pre-filter that checks for the proper declarations of exactly the methods/features it uses, and as a test case that also checks the behaviors of those methods. In fact, every test case will have exactly these properties, so there's no need for test case ordering to ensure features are available.