Adjusting the Adaptive Timeout

I have some GUI tests that take a rather long time to run and they are timing out too early at 10 seconds. Is there a nice way to lengthen the timeout for those individual tests or the entire file if needed?  I've tried adding a new AdaptiveTimeout as a Rule but that seems to just add an additional timeout to the tests and doesn't replace the original.

Groups:

Comments

Stephen Edwards

Hmmm ... adding a new

Hmmm ... adding a new AdaptiveTimeout as a Rule within your test case should override the original as well ... UNLESS your test case inherits from student.TestCase.  Does it?

Right now, subclasses of student.TestCase inherit the static rule included in that base class, so yes, adding a new rule does not replace/override the existing rule.  However, our custom test runner (which is used on Web-CAT, and typically also used for student.jar users who include that jar on the classpath before the JUnit jar) will silently inject an AdaptiveTimeout rule in any test case that isn't a subclass of student.TestCase as well, as long as that test class doesn't already have an adaptive timeout rule.  Curiously, this means that it should be possible to override the default adaptive timeout by including your own in your test class--as long as your test class doesn't inherit from an ancestor (such as student.TestCase) that already provides such a rule.

Unfortunately, because of the way the AdaptiveTimeout rule works, specifying per-test-method timeouts is not directly mappable onto the rule.  Also, the rule doesn't easily have access to the information it would need to make per-test-method settings easy to specify.  It would be possible to add this machinery, but it is a fair amount of work (compared to the limited number of cases where it would be used).

Scott Turner

It inherits from GUITestCase,

It inherits from GUITestCase, so, yes, it is inheriting from student.TestCase.  I'll play around with the timeouts like you mentioned below.

Stephen Edwards

The timeout limits are adjustable

Look in your plug-in's reusable option set for these two options:

Default Test Case Time Limit (in ms)

Default Test Case Maximum Time (in ms)

Set them to the values you prefer (for example: 30000ms + 60000ms).  The defaults are 10s and 20s, respectively.  That's all that is needed.

Scott Turner

Time Out Not Behaving as expected

So, I played around with those opition and set the timeout to 120s and the max to 180.  Now, instead of one test failing due to the timeout, all of the tests are failing because only the failed test is being run now and it times out. Is there a way to change the timeouts programmatically? It would be nice the the longer time out to apply to just one test or just one file.

Stephen Edwards

Do you also have an overall plug-in timeout set?

Do you also have a timeout set for the entire plug-in on the assignment's properties page?  Is it possible that this liimit is cutting off processing of test methods prematurely?

Scott Turner

Nuts.  I originally had that

Nuts.  I originally had that set much higher but it seems that I forgot to change it when I set it up this semester. I'll go back and try it with both of those.

Stephen Edwards

We need to remove that setting

The idea was to eliminate that timeout setting for the JavaTddPlugin altogether and rely on the adaptive timeout behavior exclusively, but we wanted to make sure the adaptive timeout was working correctly first.  So at the moment, both settings are still present--and both are enforced.  Maybe it is time to remove the overall timeout completely--what do you think?

Scott Turner

I agree that the overall

I agree that the overall timeout setting is redundant if the adaptive one is functioning correctly.  I'm a little torn over how visible the adaptive timeout should be.  Where it is now is sort of hard to find but, at the same time, I can count on one hand the number of times I've had to change the timeout to something besides the default of 45 seconds. Maybe improving the tooltip for the options so that it lists the types of options available (style, testing, timeout, etc.) would be useful in this case.