UIAutomation performance test
I conducted a test that is pretty interesting. As many might know, the more objects on a form, the slower your searches down the Automation tree.
It was measured that search on a form containing an Infragistics grid with 250 rows and 6 columns (i.e., 1500 cells) may take up to fifteen minutes. It depends on many conditions, for example, if a host is relatively ‘fresh’, it takes 383 seconds, i.e. just more than six minutes.
To demystify the topic, I ran the following test:
a simple .NET form with a DataGrid:
The ‘button1’ button creates two columns, the ‘button2’ button adds ten rows per click.
This is the test script:
Get-UIAWindow -n Form1 | Get-UIAButton -n button1 | Invoke-UIAButtonClick; for ($i = 0; $i -lt 20; $i++) { Get-UIAWindow -n Form1 | Get-UIAButton -n button2 | Invoke-UIAButtonClick; Write-Host "$(($i + 1) * 10) rows"; "$(($i + 1) * 10) rows" >> "C:\1\grid_report.txt"; Measure-Command -Expr { Get-UIAWindow -n Form1 | Get-UIATable | Get-UIAControlDescendants | %{Write-Host "$($_.Current.Name)`t$($_.Current.AutomationId)`t$($_.Current.ClassName)`t$($_.Current.ControlType.ProgrammaticName)`t$(($_.GetCurrentPattern([System.Windows.Automation.ValuePattern]::Pattern) -as [System.Windows.Automation.ValuePattern]).Current.Value)";} } >> "C:\1\grid_report.txt"; }
The script initializes the form by pressing the ‘button1’ button and performs twenty cycles. Each cycle, the script increases the number of rows in the grid and measures the search time.
In other words, our test measures the speed of search below the grid for the increasing number of rows with the increment equaling to 10.
The results are in the table (the test was performed on a Intel965 box with Windows 8 RP, 6GB RAM and the lowest Crucial M4 SSD):
Surprisingly, even though time the search took is increasing with the increase of the rows number, the objects per second time gradually decreases.
Here are two charts to make it more perceptible:
Both charts tell us two things:
- the objects per second time is growing not to fast as the number of objects in the grid does, it grows slower
- somewhere after 150 objects, time of the test begins increasing faster and faster.
The latter fact will be the reason for other tests in the near future (I’ll be informing the community about results).
Finally, the bare speed is not so valuable without other measurements. How much memory has been consumed? After the test, the Task Manager displays that:
The amount of memory is not so shocking if we compare it with the amount SharpDevelop consumes. One instance of SharpDevelop is the instance which the test app has been run from. The second instance did no more than was open with the UIAutomation solution loaded.
The normal amount of memory PowerShell consumes on this host is between 32 and 34 Megabytes, varying from run to run:
The test app, the script and the results are in the Box at the right.
Daily automaiton: how to deal with windows?
On the project forum, questions about windows have been posted today. I think, the community is interesting in he answers.
Question:
Please, give some examples how to wait for new window with unpredictable Name value(it can be various error or notify messages)?; how to enumerate all opened windows? I want to remove windows to clean desktop after unsuccessful run of application.
Answer:
1) how to wait for a new window with random Name?
first of all, there are ways to take a window by its ProcessId/pid, ProcessName/pn and its process object. If you issue the process, all the following would work:
# you've set a name for process somewhere in the code $processName = "calc"; # get the window by its process id Get-UIAWindow -pid (Start-Process $processName -PassThru).Id # get the window by its process name Get-UIAWindow -pn (Start-Process $processName -PassThru).ProcessName
second, you also can use a part of its name (supposing that some part of a name is uncahngeble from run to run);
Start-Process calc; Get-UIAWindow -n *lato*
Note that rare windows that are not seen by MS Automation and could be found by using FindWindow, still don’t support wild cards (it will be done relatively soon). I know only one such an example: on a 64-bit OS, a 32-bit MMC snap-in that is not shown in the TaskBar, runs its own window instead of the main MMC window (It’s really rare case).
2) Which opened windows do you want to enumerate?
2.1) window with known to you a part of name:
Start-Process calc; Start-Process calc; Start-Process calc; Get-UIADesktop | Get-UIAControlChildren -ControlType Window | ?{$_.Current.Name.Contains("alcu")}
2.2) if you need to collect all the dialog of the application under test, there is search down the Automaiton tree
# you should run PowerShell as Asministrator to reproduce this test Start-Process services.msc -PassThru | Get-UIAWindow | Get-UIAMenuItem -Name File | Invoke-UIAMenuItemClick | Get-UIAMenuItem -Name Opt* | Invoke-UIAMenuItemClick; # now we have several windows in the process mmc: the main window, the Options window # and one or more internal frames of mmc itself # as mmc has windows called snap-ins (or possibly even more windows I'm not aware of) # in the code below, you don't need to use the Get-UIAWindow cmdlet # because the last window you've successfully gotten # stored in the [UIAutomation.CurrentData]::CurrentWindow variable Get-UIAControlDescendants -ControlType Window | %{$_.Current.Name;} # in this case, we have gotten only two windows, both with names # in more complicated cases, you need to use several properties to select a window Get-UIAControlDescendants -ControlType Window | %{Write-Host "$($_.Current.Name)`t$($_.Current.AutomaitonId)`t$($_.Current.ClassName)";}
there also are Search- cmdlets, but these are considered legacy as they use TreeWalker what means more cross-process callc during the test:
# the search for a specified window Search-UIAControl -Name Options -ControlType Window # the code assumes that you already have the main window # and the same as the code: [UIAutomation.CurrentData]::CurrentWindow | Search-UIAControl -Name Options -ControlType Window
As can be seen, the Search-UIAControl cmdlet is more specific (like every good search), whereas Get-UIAControlDescendants returns all that is of type specified.
2.3) if you need to catch inexplicable windows like a dialog in the process of the application under test, use the Register-UIAWindowsOpenedEvent cmdlet (it’s recommended to use also the Unregister-UIAEvent cmdlet from time to time during the test suite):
Start-Process services.msc -PassThru | Get-UIAWindow | Register-UIAWindowOpenedEvent -EventAction {param($src, $e) [System.Windows.Forms.MessageBox]::Show("A new window is open: $($src.Cached.Name)");}; Get-UIAMenuItem -Name File | Invoke-UIAMenuItemClick | Get-UIAMenuItem -Name Opt* | Invoke-UIAMenuItemClick;
2.4) finally, if you need to catch a windows of another process that appears during the test, use the standard Register-WMIObjectEvent cmdlet (there are a lot of ways to write the code as parameters of the cmdlet can be used in various combinaitons):
the smaple from Richard Siddaway’s Blog:Â http://msmvps.com/blogs/richardsiddaway/archive/2009/11/07/powershell-wmi-events.aspx
3) you can eliminate processes, it’s the best choise in many situations
Start-Process calc; Start-Process calc; Start-Process calc; Stop-Process -Name calc
However, some applications should save their data or close connections before exiting, so that you need to use something like in the item 2.1 (Get-UIADesktop | Get-UIAControlChildren: main windows are always children of the desktop) to get all of them to perform exit via main menu or hitting the Close button…
Release: version 0.7.12 that does work with Metro UI from UIAutomationSpy a bit more comfortable is out
Visit our Metro UI automation page.
Today, three hosts (Windows 8 RP x64, Windows 8 RP x86, Windows Server 2012 RC) agreed on that the version 0.7.12 is the version we can start with.
What we now can do for testing Metro UI applications?
- we can start UIAutomationSpy manually and type or paste script code into it, or type a path to the script file. Automated run with preloaded code or script file is not done yet
- we can show the Start screen (the Show-UIAMetroStartScreen cmdlet) from UIAutomationSpy
- we can call charm (the Show-UIAMetroCharm cmdlet)
- we also can call the app/ the Start screen menu (the Show-UIAMetroMenu cmdlet)
- we can select an app at the Start screen (the Get-UIAListItem cmdlet issuing somewhat after the Get-UIADesktop cmdlet, as the Get-UIA[ControlType] cmdlets require input object)
- we can start an app by sending an InvokePattern-click to it (the Invoke-UIAListItemClick cmdlet)
All that is mentioned above can be done with the version 0.7.12 and using existing blog posts.
Note: the UIASendKeys utility is deprecated. the Invoke-UIAHotkey is not recommended for use and will be rewritten soon.
Metro automation: a nightmare of code generation
Visit our Metro UI automation page.
I’ve been struggling with code generation in UIAutomationSpy for a couple of hours. What’s the problem?
Code that UIAutomationSpy generates for Windows applications is what you can run immediately after recording (okay, some polishing or additions may be needed). At least, from today, when I removed the first line of code (i.e., RootElement) and UIAutomaitonSpy is now so close to the Start-UIARecorder-generated code as it might be.
For example, this is code for Notepad (an emtpy document, I click File, the dialog opens and I close it):
Get-UIAWindow -Class 'Notepad' -Name 'Untitled - Notepad' | ` Get-UIAMenuBar -AutomationId 'MenuBar' -Name 'Application' | ` Get-UIAMenuItem -AutomationId 'Item 1' -Name 'File' Get-UIAWindow -Class 'Notepad' -Name 'Untitled - Notepad' | ` Get-UIAMenuBar -AutomationId 'MenuBar' -Name 'Application' | ` Get-UIAMenuItem -AutomationId 'Item 1' -Name 'File' | ` Get-UIAMenu -Class '#32768' -Name 'File' | ` Get-UIAMenuItem -AutomationId 'Item 1' -Name 'New' Get-UIAWindow -Class 'Notepad' -Name 'Untitled - Notepad' | ` Get-UIADocument -AutomationId '15' -Class 'Edit' Get-UIAWindow -Class 'Notepad' -Name 'Untitled - Notepad' | ` Get-UIAWindow -Class '#32770' -Name 'Open' | ` Get-UIATitleBar -AutomationId 'TitleBar' -Name 'Open'
The code is readable (to some extent), though even such code should be prepared to run well.
In fact, the code is raw and the Start-UIARecorder cmdlet still works better:
Get-UIAWindow -n *notepad* | Get-UIAMenuItem -Name File | Invoke-UIAMenuItemExpand | Get-UIAMenuItem -Name Open* | Invoke-UIAMenuItemClick
Anyway, a Windows app is not a mythical beast. Metro UI is more complicated; below is a typical piece of generated code:
Get-UIAList -AutomationId 'GridRoot' -Class 'GridRoot' -Name 'Start Apps' | ` Get-UIAPane -AutomationId 'ScrollViewer' -Class 'TouchScrollViewer' | ` Get-UIAPane -AutomationId 'content' -Class 'GridContent' | ` Get-UIAGroup -AutomationId 'Group 2305843009213693952' -Class 'GridGroup' -Name 'Mail' | ` Get-UIAListItem -AutomationId 'microsoft.windowscommunicationsapps_8wekyb3d8bbwe!Microsoft.WindowsLive.Chat' -Class 'GridTileElement' -Name 'Messaging'
This code simpy finds the Messaging app. This is a dead code. To make the code alive, we need:
- add Get-UIADesktop before the first instruction
- add Show-UIAMetroStartScreen before the first instruction
- remove the line 2:Â Get-UIAPane -AutomationId ‘ScrollViewer’ -Class ‘TouchScrollViewer’ | `
- remove the line 3:Â Get-UIAPane -AutomationId ‘content’ -Class ‘GridContent’ | `
- remove the part of line 5:Â -AutomationId ‘microsoft.windowscommunicationsapps_8wekyb3d8bbwe!Microsoft.WindowsLive.Chat’ -Class ‘GridTileElement’
After editing, code looks like:
Get-UIADesktop | Get-UIAList -AutomationId 'GridRoot' -Class 'GridRoot' -Name 'Start Apps' | ` Get-UIAGroup -AutomationId 'Group 2305843009213693952' -Class 'GridGroup' -Name 'Mail' | ` Get-UIAListItem -Name 'Messaging'
Finally, the right code would be (for Windows 8 RP):
Get-UIADesktop; Show-UIAMetroStartScreen; Get-UIAListItem -Name *Internet*Explorer* | Invoke-UIAListItemClick; Show-UIAMetroStartScreen; Get-UIAListItem -Name *people* | Invoke-UIAListItemClick; Show-UIAMetroStartScreen; Get-UIAListItem -Name *messag* | Invoke-UIAListItemClick; Show-UIAMetroStartScreen; Get-UIAListItem -Name *phot* | Invoke-UIAListItemClick; Show-UIAMetroStartScreen; Get-UIAListItem -Name *calen* | Invoke-UIAListItemClick;
and for Windows Server 2012:
Get-UIADesktop; Show-UIAMetroStartScreen; Get-UIAListItem -Name *Internet*Explorer* | Invoke-UIAListItemClick; Show-UIAMetroStartScreen; Get-UIAListItem -Name *control* | Invoke-UIAListItemClick; Show-UIAMetroStartScreen; Get-UIAListItem -Name *computer* | Invoke-UIAListItemClick; Show-UIAMetroStartScreen; Get-UIAListItem -Name *task*man* | Invoke-UIAListItemClick; Show-UIAMetroStartScreen; Get-UIAListItem -Name *admi*tool* | Invoke-UIAListItemClick;
As can be seen, the problem is that code UIAutomationSpy generates is far away of what is working…
The worst of all, the code for Windows applications can be useful from the top control to the current, whereas the code for Metro UI apps should be filtered, edited and some commands should be added to it.
This requires two branches of code generation, doesn’t it?
There is a video based on code samples (HD mode is recommended if you want to read the code).
function Start-MetroApp { param( [string]$appName ) Show-UIAMetroStartScreen; Get-UIAListItem -Name $appName | Invoke-UIAListItemClick; } Get-UIADesktop; Start-MetroApp Music; Start-MetroApp Finan*; Start-MetroApp '*internet*explorer*'; Start-MetroApp people;
Metro automation: UIAutomationSpy displays even more
Visit our Metro UI automation page. This post is deprecated.
The version 0.7.11 is out specially for Metro UI testers and developers. There are a couple of improvements to UIAutomationSpy:
- now you can see the full path to a control in terms of the UIAutomation module.
- you can see the same graphically. Okay, just as a tree.
- you do not need UISendKeys anymore. It simply does not ship. Use the Invoke-UIAHotKey cmdlet as shown on the picture
- access the charm? Not a problem!Â
- finally, how to pull the application menu? Send Win+Z to the Start screen  or to an application:
Happy testing!
P.S. All the faces on tiles belong to their respective owners.
Quality report: a great success of PowerShell in testing
Yesterday’s build of a commercial application we are working on was reported as “with a bug”. An annoying dialog appeared here and there after a relatively long time of uninterruptible work of the application’s console. Need to say that reproduction of such bugs (“the console got tired and a dialog appears after an hour or more…”) is not a piece of cake. At least, no one wants to await the dialog in the debugger for an hour or two.
Now, the serious part of the product is being tested with PowerShell. Whereas the system part is being tested with C# (the ‘system team’ prefers writing code what I call ‘writing own bugs’), the UI including simple interactions like selecting users from a domain is done in PowerShell. The majority of the code exploits UIAutomation and TMX modules.
Below is the yesterday’s report (a TMX results report):
As can be seen, the average time of a test case is 8005/(2682-42)=3,03Â seconds.
The fail/pass rate is 161/2682=6%.
Data is nothing if you have none to compare. Below is a report done a week ago (test cases number varies from run to run as the autotest developer and maintainer switches test cases on or off due to changes in the UI or for analysis and making amendments to the test suite):
The ‘happy’ run has 0,5% fail/pass rate (0.0% fail/pass rate is hardly achievable during the time the product’s UI is being changed).
The average time is better than in a normal case and counted as 4623/(2251-35)=2,08Â seconds per test case, and there is still room for further improvements. Preferably, that fail-run increments were bound to 20%, not 50%.
Anyway, the red peak of 6% of the height of a percent chart seems appropriate. Isn’t it a success of using PowerShell in software testing?
Metro automation: now, UIAutomationSpy displays more
Visit our Metro UI automation page.
I’ve been working on a material for upcoming posts about Metro UI automation and found that it’s often inevident from where a control grows. I mean that any information about the immediate parent and the ancestry at all is a necessity for writing scripts.
With that said, a new version of UIAutomationSpy (sooner a half-version as it’s not the time to publish such small changes as a version) would be accepted by the public well.
Right now there is a version that displays the immediate parent’s code along with the code of a control you are observing.
Here we are hovering over the Start screen. The full path to a tile is shown in the lower rectangle:
The picture on the lower-left tile is a property of http://echo.msk.ru/. It shows that the president, the prime minister and the main speaker of the upper part of Duma are from the city I live in.  Above we’ve hovering over People, now we are hovering over Travel:
The upper-left tile informs us that I have a report on Family Safety. This is a report for an account I’m not related to in any respect. How to spy on somebody you even don’t know with the help from Microsoft, read here.
What is the main part of the Travel application made of?
After we opened the charm (Win+C):
Settings:
Settings -> Feedback:
Dissecting a combo box:
Settings -> Share:
Settings -> Share -> Login to Evernote:
Finally, Settings -> Share -> Mail:
Don’t procrastinate the chance to test your Metro UI application, download it now.
Daily automation: peeling the onion
Sometimes, hit happens. It’s a shame to confess, but rarely even I recommend simple Win32 clicks. Today was the story about such clicks. A member of the test team, not a newcomer to UIAutomation and with thousands of test cases in the suite behind, the guy complained that Invoke-UIAControlClick (i.e., PostMessage) did not work!
I approached, and it didn’t work! The panic was spreading out among the testers, suggestions like ‘use mouse_event!’ or ‘revise the code!’ was hovering around the room.
The answer was simple. The Invoke-UIAControlClick cmdlet as well as the PostMessage/SendMessage Win32 call hits by control’s handle. We attempted to do the following:
Get-UIAWindow -n window_name | Set-UIAFocus | Get-UIATab -n tab_name | Invoke-UIAControlClick -X 500 -Y 125;
What the world was awaiting was:
Get-UIAWindow -n window_name | Set-UIAFocus | Get-UIATab -n tab_name | Get-UIAPane | Get-UIAPane | Get-UIAPane | Invoke-UIAControlClick -X 500 -Y 125;
Each control in this pipeline had a handle and there was no point to hit the tab if the link was on the lowest pane.
Layers are of almost the same size and put one over one. Even spy++ couldn’t help as nobody knew which layers was pointed to by the spy++ target cursor in order to collect messages.
Task:Â help to find the right level.
Requirements:Â write out the script that hit-tests every layer of controls and prints their handles.
Solution:Â below is the script that walks down the hierarchy and prints out handles of controls:
function hittest { param( [System.Windows.Automation.AutomationElement]$element, [int]$X, [int]$Y ) if ($element | Invoke-UIAControlClick -X $X -Y $Y) { Write-Host "The control has handle $($element | Read-UIAControlNativeWindowHandle)"; } } Get-UIAWindow -Name *window*name* | Set-UIAFocus | ` Get-UIATab -Name *tab*name* | ` Get-UIAControlDescendants | ` %{hittest $_ 500 125;}
The script tries to click on each control and outputs the handle value. The Invoke-UIAControlClick cmdlet couldn’t decide was or wasn’t a click successful and returns $true if no crash were during the script execution.
Anyhow, you can use handles to get controls and click by handle in a cycle to find out the control you need.