continuous integration - How to fix failed tests count in teamcity for re-executed tests? -
we using teamcity selenium tests execution specflow+specrun, problem teamcity count re-executed tests.
for example if test failed first time re-executed 2 times more, in teamcity see 3 tests failed, 1 test.
also if first re-execution fail, other 2 success, reported in teamcity 2 failed, 1 passed, need reported 1 test has been passed.
it possible configure in teamcity using service messages or else?
updated:
based on answer can gather logs using powershell script , change build status using teamcity service messages:
$path = get-childitem %teamcity.build.checkoutdir%\projectfolder\bin\remote\testresults\specrun.log $file = get-content $path $total = $file | select-string "total:" $passed = $file | select-string "succeeded:" $failed = $file | select-string "failed:" write-host $( "##teamcity[buildstatus text='tests {0}, {1}, {2}']" -f $total, $passed, $failed )
teamcity behaving expected. teamcity reports testing framework tells him. process is:
- execute step run tests.
- gather output logs.
- extract stats: number of test executed, passed , failing.
- fail build if tests fail if configured that.
you have configure testing framework (specflow+specrun in case) change way reports re-execution. you should configure specflow log re-executions in different way... if can.
in general should avoid re-execution of tests. tests should work @ first attempt. should prepare state if needed, test, , clean state.
the need of re-execution indicates dependency other tests. like:
- test 001: search user foo1. (fails)
- test 002: adds user foo1. (ok)
- re-execution of test 001: search user foo1. (ok)
the correct way test001 not have dependency of test002, adds user foo1, test search , clean user foo1.
Comments
Post a Comment