Yesterday’s build of a commercial application we are working on was reported as “with a bug”. An annoying dialog appeared here and there after a relatively long time of uninterruptible work of the application’s console. Need to say that reproduction of such bugs (“the console got tired and a dialog appears after an hour or more…”) is not a piece of cake. At least, no one wants to await the dialog in the debugger for an hour or two.
Now, the serious part of the product is being tested with PowerShell. Whereas the system part is being tested with C# (the ‘system team’ prefers writing code what I call ‘writing own bugs’), the UI including simple interactions like selecting users from a domain is done in PowerShell. The majority of the code exploits UIAutomation and TMX modules.
Below is the yesterday’s report (a TMX results report):
As can be seen, the average time of a test case is 8005/(2682-42)=3,03 seconds.
The fail/pass rate is 161/2682=6%.
Data is nothing if you have none to compare. Below is a report done a week ago (test cases number varies from run to run as the autotest developer and maintainer switches test cases on or off due to changes in the UI or for analysis and making amendments to the test suite):
The ‘happy’ run has 0,5% fail/pass rate (0.0% fail/pass rate is hardly achievable during the time the product’s UI is being changed).
The average time is better than in a normal case and counted as 4623/(2251-35)=2,08 seconds per test case, and there is still room for further improvements. Preferably, that fail-run increments were bound to 20%, not 50%.
Anyway, the red peak of 6% of the height of a percent chart seems appropriate. Isn’t it a success of using PowerShell in software testing?