Quality report: a great success of PowerShell in testing


Yesterday’s build of a commercial application we are working on was reported as “with a bug”. An annoying dialog appeared here and there after a relatively long time of uninterruptible work of the application’s console. Need to say that reproduction of such bugs (“the console got tired and a dialog appears after an hour or more…”) is not a piece of cake. At least, no one wants to await the dialog in the debugger for an hour or two.

Now, the serious part of the product is being tested with PowerShell. Whereas the system part is being tested with C# (the ‘system team’ prefers writing code what I call ‘writing own bugs’), the UI including simple interactions like selecting users from a domain is done in PowerShell. The majority of the code exploits UIAutomation and TMX modules.

Below is the yesterday’s report (a TMX results report):

Suites:31 Passed:19 Failed:9 Not tested:3
Scenarios:227 Passed:206 Failed:21 Not tested:0
Test cases:2682 Passed:2479 Failed:161 Not tested:42 Time spent:8005 seconds

As can be seen, the average time of a test case is 8005/(2682-42)=3,03 seconds.

The fail/pass rate is 161/2682=6%.

Data is nothing if you have none to compare. Below is a report done a week ago (test cases number varies from run to run as the autotest developer and maintainer switches test cases on or off due to changes in the UI or for analysis and making amendments to the test suite):

Suites:31 Passed:19 Failed:8 Not tested:4
Scenarios:197 Passed:188 Failed:9 Not tested:0
Test cases:2251 Passed:2204 Failed:12 Not tested:35 Time spent:4623 seconds

The ‘happy’ run has 0,5% fail/pass rate (0.0% fail/pass rate is hardly achievable during the time the product’s UI is being changed).

The average time is better than in a normal case and counted as 4623/(2251-35)=2,08 seconds per test case, and there is still room for further improvements. Preferably, that fail-run increments were bound to 20%, not 50%.

Anyway, the red peak of 6% of the height of a percent chart seems appropriate. Isn’t it a success of using PowerShell in software testing?

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: