How to Re-run failed JUnit tests immediately?
You can do this with a TestRule. This will give you the flexibility you need. A TestRule allows you to insert logic around the test, so you would implement the retry loop:
public class RetryTest {
public class Retry implements TestRule {
private int retryCount;
public Retry(int retryCount) {
this.retryCount = retryCount;
}
public Statement apply(Statement base, Description description) {
return statement(base, description);
}
private Statement statement(final Statement base, final Description description) {
return new Statement() {
@Override
public void evaluate() throws Throwable {
Throwable caughtThrowable = null;
// implement retry logic here
for (int i = 0; i < retryCount; i++) {
try {
base.evaluate();
return;
} catch (Throwable t) {
caughtThrowable = t;
System.err.println(description.getDisplayName() + ": run " + (i+1) + " failed");
}
}
System.err.println(description.getDisplayName() + ": giving up after " + retryCount + " failures");
throw caughtThrowable;
}
};
}
}
@Rule
public Retry retry = new Retry(3);
@Test
public void test1() {
}
@Test
public void test2() {
Object o = null;
o.equals("foo");
}
}
The heart of a TestRule
is the base.evaluate()
, which calls your test method. So around this call you put a retry loop. If an exception is thrown in your test method (an assertion failure is actually an AssertionError
), then the test has failed, and you'll retry.
There is one other thing that may be of use. You may only want to apply this retry logic to a set of tests, in which case you can add into the Retry class above a test for a particular annotation on the method. Description
contains a list of annotations for the method. For more information about this, see my answer to How to run some code before each JUnit @Test method individually, without using @RunWith nor AOP?.
Using a custom TestRunner
This is the suggestion of CKuck, you can define your own Runner. You need to extend BlockJUnit4ClassRunner and override runChild(). For more information see my answer to How to define JUnit method rule in a suite?. This answer details how to define how to run code for every method in a Suite, for which you have to define your own Runner.
Re-run failed Test Script 'completely' using JUnit
JanakiL,
It is a very good question. I tried to find some solution but i didn't manage to find clean solution for this task.
I can only propose to do some workaround that eventually will work.
So, in order to re-run suite you need to do following steps:
You need to create @ClassRule in order to execute whole suite.
All Suite you can retry using following code:public class Retrier implements TestRule{
private int retryCount;
private int failedAttempt = 0;
@Override
public Statement apply(final Statement base,
final Description description) {
return new Statement() {
@Override
public void evaluate() throws Throwable {
base.evaluate();
while (retryNeeded()){
log.error( description.getDisplayName() + " failed");
failedAttempt++;
}
}}
retryNeeded() – method that determines whether you need to do retry or not
This will retry all tests in your suite. Very important thing that retry will go after @AfterClass method.
In case you need to have “green build” after successful retry you need to write a bunch of another gloomy code.
- You need to create @Rule that will not allow “publish” failed result. As example:
public class FailedRule extends TestWatcher {
@Override
public Statement apply(final Statement base, final Description description) {
return new Statement() {
@Override
public void evaluate() throws Throwable {
List<Throwable> errors = new ArrayList<Throwable>();
try {
base.evaluate();
} catch (AssumptionViolatedException e) {
log.error("", e.getMessage());
if (isLastRun()) {
throw e;
}
} catch (Throwable t) {
log.error("", t.getMessage());
if (isLastRun()) {
throw t;
}
}
};
};
}
}
isLastRun() – methods to verify whether it is last run and fail test only in case ir is last run.
Only last retry needs to be published to mark you test failed and build “red”.
3. And finally in your test class you need do register two rules:
@Rule
public FailedRule ruleExample = new FailedRule ();
@ClassRule
public static Retrier retrier = new Retrier (3);
In the Retrier you can pass count of attempts to retry.
I hope that somebody could suggest better solution !
Re-run failed Junit 5 tests without reporting them as failed in test results
Finally I used https://github.com/artsok/rerunner-jupiter library. It throws TestAbortedException before repetition of failed test and such failure is marked as ignored test. The result of build is marked as successful in this case. It was needed to replace @Test annotation to my custom annotation and add @TestTemplate to it, also I extended from Extension class of the library and applied my implementation of extension to class of my basic test using @ExtendWith. Looks like the only way to use the library without using of @ReplaceIfExceptions... annotation on each test method.
Rerun flaky JUnit test in case they failed
I would suggest to fix your tests or rewrite them so they will only fail if something is broken. Maybe you can mock away the things that tend to fail. If you are depnending on a database connection, maybe you could use a sqlite
or smething which is local.
But there is also a plugin which can retry a build:
https://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin
Simply install the plugin, and then check the Post-Build action "Retry build after failure" on your project's configuration page.
If you want to rerun tests in JUnit-context, take a look here: SO: How to Re-run failed JUnit tests immediately?
Related Topics
How Come Invoking a (Static) Method on a Null Reference Doesn't Throw Nullpointerexception
Simpledateformat Parsing Date with 'Z' Literal
Java Generating Non-Repeating Random Numbers
Deploying Spring 5.X on Tomcat 10.X
String.Replaceall(Regex) Makes the Same Replacement Twice
How to Change UI Depending on Combo Box Selection
Get Width and Height of JPAnel Outside of the Class
Get Source Jars from Maven Repository
Getting Random Numbers in Java
Java.Lang.Outofmemoryerror: Gc Overhead Limit Exceeded
Xml Configuration Versus Annotation Based Configuration
Initial Size for the Arraylist
What's the Best Way to Distribute Java Applications
How to Use Annotations to Define Different Types of Relationships in Hibernate 4 and Spring
Java Reverse an Int Value Without Using Array
Similarity String Comparison in Java