Daily automation: eliminating the fragility of tests


Why do test fail? It’s the question every automated tester investigated into. Today we’ll discuss unexpected changes in the software under tests. This topic is hardly relevant to developers or even to testers in projects where no inexplicable changes allowed (have you seen any?).

When a new build is baked, and all nobody-knows-what changes are baked in, your tests might not find something vital and omit a great amount of the product functionality or stop completely.

One of solutions we accepted is to enable the search using wildcards. To our surprise, the wildcard search within a not great number of controls works with the same speed as the exact search. Wildcard search within a great number of controls is slower than exact search, possibly the latter uses caching or some hidden indices that we can’t use. There is a place for further investigation. In case of the fruitless search, both searches use almost the same time, the time the timeout told.

Task: use UIAutomation to get a control by a fragment of its name.

Requirements: compare times that needed for exact-match search and wildcard search.

Solution: below is the script that runs a process, gets its window, gets a control by exact match or using wildcard and reports the time used (part of the samples package) :

Set-StrictMode -Version Latest;
ipmo [path]\UIAutomation.dll;
[UIAutomation.Preferences]::OnSuccessDelay = 0;

# regular search
[UIAutomation.Preferences]::DisableExactSearch = $false
Write-host "exact matching (one of few controls of the MenuItem type):";
(Measure-Command {Start-Process services.msc -PassThru | Get-UIAWindow | Get-UIAMenuItem -Name File}).TotalSeconds
Stop-Process -Name mmc;

# search with wildcards
[UIAutomation.Preferences]::DisableExactSearch = $true
Write-host "wildcard matching (one of few controls of the MenuItem type):";
(Measure-Command {Start-Process services.msc -PassThru | Get-UIAWindow | Get-UIAMenuItem -Name Fil*}).TotalSeconds
Stop-Process -Name mmc;

# regular search
[UIAutomation.Preferences]::DisableExactSearch = $false
Write-host "exact matching (one of a heap of controls of the Edit):";
(Measure-Command {Start-Process services.msc -PassThru | Get-UIAWindow | Get-UIAEdit -Name BranchCache}).TotalSeconds
Stop-Process -Name mmc;

# search with wildcards
[UIAutomation.Preferences]::DisableExactSearch = $true
Write-host "wildcard matching (one of a heap of controls of the Edit):";
(Measure-Command {Start-Process services.msc -PassThru | Get-UIAWindow | Get-UIAEdit -Name Branch*}).TotalSeconds
Stop-Process -Name mmc;

# regular search
[UIAutomation.Preferences]::DisableExactSearch = $false
Write-host "exact matching (one of few controls of the MenuItem type):";
(Measure-Command {Start-Process services.msc -PassThru | Get-UIAWindow | Get-UIAMenuItem -Name File}).TotalSeconds
Stop-Process -Name mmc;

# search with wildcards
[UIAutomation.Preferences]::DisableExactSearch = $true
Write-host "wildcard matching (one of few controls of the MenuItem type):";
(Measure-Command {Start-Process services.msc -PassThru | Get-UIAWindow | Get-UIAMenuItem -Name *il?}).TotalSeconds
Stop-Process -Name mmc;

# regular search
[UIAutomation.Preferences]::DisableExactSearch = $false
Write-host "exact matching (one of a heap of controls of the Edit):";
(Measure-Command {Start-Process services.msc -PassThru | Get-UIAWindow | Get-UIAEdit -Name Workstation}).TotalSeconds
Stop-Process -Name mmc;

# search with wildcards
[UIAutomation.Preferences]::DisableExactSearch = $true
Write-host "wildcard matching (one of a heap of controls of the Edit):";
(Measure-Command {Start-Process services.msc -PassThru | Get-UIAWindow | Get-UIAEdit -Name ?orksta*i*n*}).TotalSeconds
Stop-Process -Name mmc;

# regular search
[UIAutomation.Preferences]::DisableExactSearch = $false
Write-host "exact matching (one of few controls of the MenuItem type - it takes the full time timeout is set to):";
(Measure-Command {Start-Process services.msc -PassThru | Get-UIAWindow | Get-UIAMenuItem -Name nothing}).TotalSeconds
Stop-Process -Name mmc;

# search with wildcards
[UIAutomation.Preferences]::DisableExactSearch = $true
Write-host "wildcard matching (one of few controls of the MenuItem type - it takes the full time timeout is set to):";
(Measure-Command {Start-Process services.msc -PassThru | Get-UIAWindow | Get-UIAMenuItem -Name not?ing}).TotalSeconds
Stop-Process -Name mmc;

# regular search
[UIAutomation.Preferences]::DisableExactSearch = $false
Write-host "exact matching:";
(Measure-Command {Start-Process services.msc -PassThru | Get-UIAWindow | Get-UIAEdit -Name nothing}).TotalSeconds
Stop-Process -Name mmc;

# search with wildcards
[UIAutomation.Preferences]::DisableExactSearch = $true
Write-host "wildcard matching:";
(Measure-Command {Start-Process services.msc -PassThru | Get-UIAWindow | Get-UIAEdit -Name ?othing}).TotalSeconds
Stop-Process -Name mmc;

As can be seen, the search among controls that are not huge in number is so fast as exact matching. However, when it works among a significant number of control of the type chosen, search is slowing down. Why? Possibly, there caching is needed. No answer at the moment. the third part of the test script clearly shows that in an unhappy case both types of search fail at the same speed, after timeout expires.

Thus, for small applications or separate parts of a big one, there is the reason to use wildcard wherever it’s possible. Just to eliminate surprises your developers might easily involve you in.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: