Tag Archives: UIAutomation

The new elements’ object model is out. A short intro

Many claim that it’s not so easy to decide which pattern cmdlet to use with a particular control. For long time we were tied to what Microsoft’s UI Automation outputs to us. However, there is a way to increase productivity: extension methods. Yes, they are not visible in PowerShell by default, nonetheless I’d like to offer an object model that grew from nothing else but extension methods.

To start, let’s page through the presentation, with binaries of 0.8.7 alpha 3 or higher in your hands. Enjoy!

Metro automation: a nightmare of code generation

Visit our Metro UI automation page.


I’ve been struggling with code generation in UIAutomationSpy for a couple of hours. What’s the problem?

Code that UIAutomationSpy generates for Windows applications is what you can run immediately after recording (okay, some polishing or additions may be needed). At least, from today, when I removed the first line of code (i.e., RootElement) and UIAutomaitonSpy is now so close to the Start-UIARecorder-generated code as it might be.

For example, this is code for Notepad (an emtpy document, I click File, the dialog opens and I close it):

Get-UIAWindow -Class 'Notepad' -Name 'Untitled - Notepad' | `

Get-UIAMenuBar -AutomationId 'MenuBar' -Name 'Application' | `

Get-UIAMenuItem -AutomationId 'Item 1' -Name 'File'
Get-UIAWindow -Class 'Notepad' -Name 'Untitled - Notepad' | `

Get-UIAMenuBar -AutomationId 'MenuBar' -Name 'Application' | `

Get-UIAMenuItem -AutomationId 'Item 1' -Name 'File' | `

Get-UIAMenu -Class '#32768' -Name 'File' | `

Get-UIAMenuItem -AutomationId 'Item 1' -Name 'New'
Get-UIAWindow -Class 'Notepad' -Name 'Untitled - Notepad' | `

Get-UIADocument -AutomationId '15' -Class 'Edit'
Get-UIAWindow -Class 'Notepad' -Name 'Untitled - Notepad' | `

Get-UIAWindow -Class '#32770' -Name 'Open' | `

Get-UIATitleBar -AutomationId 'TitleBar' -Name 'Open'

The code is readable (to some extent), though even such code should be prepared to run well.

In fact, the code is raw and the Start-UIARecorder cmdlet still works better:

Get-UIAWindow -n *notepad* | Get-UIAMenuItem -Name File | Invoke-UIAMenuItemExpand | Get-UIAMenuItem -Name Open* | Invoke-UIAMenuItemClick

Anyway, a Windows app is not a mythical beast. Metro UI is more complicated; below is a typical piece of generated code:

Get-UIAList -AutomationId 'GridRoot' -Class 'GridRoot' -Name 'Start Apps' | `
Get-UIAPane -AutomationId 'ScrollViewer' -Class 'TouchScrollViewer' | `
Get-UIAPane -AutomationId 'content' -Class 'GridContent' | `
Get-UIAGroup -AutomationId 'Group 2305843009213693952' -Class 'GridGroup' -Name 'Mail' | `
Get-UIAListItem -AutomationId 'microsoft.windowscommunicationsapps_8wekyb3d8bbwe!Microsoft.WindowsLive.Chat' -Class 'GridTileElement' -Name 'Messaging'

This code simpy finds the Messaging app. This is a dead code. To make the code alive, we need:

  1. add Get-UIADesktop before the first instruction
  2. add Show-UIAMetroStartScreen before the first instruction
  3. remove the line 2: Get-UIAPane -AutomationId ‘ScrollViewer’ -Class ‘TouchScrollViewer’ | `
  4. remove the line 3: Get-UIAPane -AutomationId ‘content’ -Class ‘GridContent’ | `
  5. remove the part of line 5: -AutomationId ‘microsoft.windowscommunicationsapps_8wekyb3d8bbwe!Microsoft.WindowsLive.Chat’ -Class ‘GridTileElement’

After editing, code looks like:

Get-UIADesktop | Get-UIAList -AutomationId 'GridRoot' -Class 'GridRoot' -Name 'Start Apps' | `
Get-UIAGroup -AutomationId 'Group 2305843009213693952' -Class 'GridGroup' -Name 'Mail' | `
Get-UIAListItem -Name 'Messaging'

Finally, the right code would be (for Windows 8 RP):

Get-UIAListItem -Name *Internet*Explorer* | Invoke-UIAListItemClick;

Get-UIAListItem -Name *people* | Invoke-UIAListItemClick;

Get-UIAListItem -Name *messag* | Invoke-UIAListItemClick;

Get-UIAListItem -Name *phot* | Invoke-UIAListItemClick;

Get-UIAListItem -Name *calen* | Invoke-UIAListItemClick;

and for Windows Server 2012:

Get-UIAListItem -Name *Internet*Explorer* | Invoke-UIAListItemClick;

Get-UIAListItem -Name *control* | Invoke-UIAListItemClick;

Get-UIAListItem -Name *computer* | Invoke-UIAListItemClick;

Get-UIAListItem -Name *task*man* | Invoke-UIAListItemClick;

Get-UIAListItem -Name *admi*tool* | Invoke-UIAListItemClick;

As can be seen, the problem is that code UIAutomationSpy generates is far away of what is working…

The worst of all, the code for Windows applications can be useful from the top control to the current, whereas the code for Metro UI apps should be filtered, edited and some commands should be added to it.
This requires two branches of code generation, doesn’t it?

There is a video based on code samples (HD mode is recommended if you want to read the code).

function Start-MetroApp
param( [string]$appName )

Get-UIAListItem -Name $appName | Invoke-UIAListItemClick;

Start-MetroApp Music;

Start-MetroApp Finan*;

Start-MetroApp '*internet*explorer*';

Start-MetroApp people;

Metro automation: UIAutomationSpy displays even more

Visit our Metro UI automation page. This post is deprecated.


The version 0.7.11 is out specially for Metro UI testers and developers. There are a couple of improvements to UIAutomationSpy:

  • now you can see the full path to a control in terms of the UIAutomation module.
  • you can see the same graphically. Okay, just as a tree.
  • you do not need UISendKeys anymore. It simply does not ship. Use the Invoke-UIAHotKey cmdlet as shown on the picture
  • access the charm? Not a problem! 
  • finally, how to pull the application menu? Send Win+Z to the Start screen  or to an application:

Happy testing!

P.S. All the faces on tiles belong to their respective owners.

Quality report: a great success of PowerShell in testing

Yesterday’s build of a commercial application we are working on was reported as “with a bug”. An annoying dialog appeared here and there after a relatively long time of uninterruptible work of the application’s console. Need to say that reproduction of such bugs (“the console got tired and a dialog appears after an hour or more…”) is not a piece of cake. At least, no one wants to await the dialog in the debugger for an hour or two.

Now, the serious part of the product is being tested with PowerShell. Whereas the system part is being tested with C# (the ‘system team’ prefers writing code what I call ‘writing own bugs’), the UI including simple interactions like selecting users from a domain is done in PowerShell. The majority of the code exploits UIAutomation and TMX modules.

Below is the yesterday’s report (a TMX results report):

Suites:31 Passed:19 Failed:9 Not tested:3
Scenarios:227 Passed:206 Failed:21 Not tested:0
Test cases:2682 Passed:2479 Failed:161 Not tested:42 Time spent:8005 seconds

As can be seen, the average time of a test case is 8005/(2682-42)=3,03 seconds.

The fail/pass rate is 161/2682=6%.

Data is nothing if you have none to compare. Below is a report done a week ago (test cases number varies from run to run as the autotest developer and maintainer switches test cases on or off due to changes in the UI or for analysis and making amendments to the test suite):

Suites:31 Passed:19 Failed:8 Not tested:4
Scenarios:197 Passed:188 Failed:9 Not tested:0
Test cases:2251 Passed:2204 Failed:12 Not tested:35 Time spent:4623 seconds

The ‘happy’ run has 0,5% fail/pass rate (0.0% fail/pass rate is hardly achievable during the time the product’s UI is being changed).

The average time is better than in a normal case and counted as 4623/(2251-35)=2,08 seconds per test case, and there is still room for further improvements. Preferably, that fail-run increments were bound to 20%, not 50%.

Anyway, the red peak of 6% of the height of a percent chart seems appropriate. Isn’t it a success of using PowerShell in software testing?

Metro automation: now, UIAutomationSpy displays more

Visit our Metro UI automation page.


I’ve been working on a material for upcoming posts about Metro UI automation and found that it’s often inevident from where a control grows. I mean that any information about the immediate parent and the ancestry at all is a necessity for writing scripts.

With that said, a new version of UIAutomationSpy (sooner a half-version as it’s not the time to publish such small changes as a version) would be accepted by the public well.

Right now there is a version that displays the immediate parent’s code along with the code of a control you are observing.

Here we are hovering over the Start screen. The full path to a tile is shown in the lower rectangle:

The picture on the lower-left tile is a property of http://echo.msk.ru/. It shows that the president, the prime minister and the main speaker of the upper part of Duma are from the city I live in.  Above we’ve hovering over People, now we are hovering over Travel:

The upper-left tile informs us that I have a report on Family Safety. This is a report for an account I’m not related to in any respect. How to spy on somebody you even don’t know with the help from Microsoft, read here.

What is the main part of the Travel application made of?

After we opened the charm (Win+C):


Settings -> Feedback:

Dissecting a combo box:

Settings -> Share:

Settings -> Share -> Login to Evernote:

Finally, Settings -> Share -> Mail:

Don’t procrastinate the chance to test your Metro UI application, download it now.

Metro automation: getting the Start screen

Visit our Metro UI automation page. This post is deprecated


It might sound oddly, but the fact is that I’ve spent several hours in attempts to raise the Start screen. I tried moving the cursor over the lower left corner to get the ListItem, a small image of the Start screen. I tried to move cursor over the lower right corner and along the right side to get the desktop charm. I tries to use SendKeys to send Win (^{ESC}) or Win+C key combinations to the desktop. After all, I simply created an app that is running as another, non-privileged process and is able to send Ctrl+Esc. The solution looks overly complicated, nevertheless it works pretty well.

This was, as an example, one of my wrong solutions:

Get-UIADesktop | Get-UIAPane -Class 'Shell_TrayWnd' | Invoke-UIAControlClick -X 10 -Y 10 | Move-UIACursor -X -3 -Y 100;

Get-UIAListItem -Name 'Start screen' | Invoke-UIAListItemClick;

Okay, what is recommended:

1.) download the UIASendKeys.exe, placed separately for this time

2.) unpack it to a folder of your choice and run the application

It is to be run the whole test session.

3.) run UIAutomationSpy

4.) When you need to raise the Start screen (this can be needed frequently as it’s frightening of any movement over the desktop or flickering of applications that got some changes),  go to your working place, the Script tab, and type the following code:

Get-UIAWindow -n UIASendKeys | Get-UIAButton -Name Start* | Invoke-UIAButtonClick;

This raises the Start screen whenever you need it:

The idea behind this solution is the same that I used for testing of several products: run an application under test in another process and connect to it using inter-process communication (UIAutomation, in other words). This method is used in many test systems, NUnit is the first example that has been recalled.

Metro automation: getting started

Visit our Metro UI automation page.


After a long hiatus, yesterday, I suddenly thought that it’s the time to return to Metro automation. To my surprise, rumors have been already circulating that Windows 8 Next Preview is upcoming in hours. Intuition? Maybe. In time is in time.

Yesterday I added to UIAutomaitonSpy the functionality to run PowerShell scripts. Today I polished a bit (here is great volume of work to do).

Let’s start. I tested the following on Windows 8 CP x64 and Windows 8 RP x86 (Release Preview of June, 01st). The binaries was built on a Windows 8 CP x64 box.ImageC

1. download the package. It’s not the default release now.

2. Unpack the package to a certain directory. We need to put in a secure location. One of them is %SystemRoot%, the other is %ProgramFiles%. The latter is more appropriate. In my tests, I creates the “C:\Program Files\1” directory and put the binaries there. You have in the folder UIAutomationSpy.exe, the config, UIAutomation.dll and TMX.dll.

3. One more file you’ve got is a certificate. You need to install it. Alternatively, you can sign UIAutomationSpy.exe with the certificate you possibly have.

If you have no your own certificate, below are screenshots showing how to use the certificate from the package. Run certmgr, for example, from cmd.exe, or from the Start Screen

and follow the pictures:








After you installed the certificate and the application in the secure location (you might set the policy not to require this if you’d like to), you can run the application. If you want to change the policy, run gpedit.msc, for example, from the Start screen

and set the following setting:

After you’ve finished, reload policies:

For the first test, just run the application, agree with UAC and manually run the Start screen by pressing the Win button. Now you should see something like on the picture below:


On the picture, UIAutomationSpy shows the code for the Mail tile. It is bordered with the red rectangle. All that I’d like to offer is to explore tiles:


This is a text box, Edit in terms of UIAutomation.

Tomorrow I’m planning to start discussing how to write and run scripts for Metro UI.

A word of appreciation: Windows 8 accessibility performance improvements

Compared to runs on a Windows 7 x64 box as well as on a Windows 2008 SP2 x64 box, the same script from the post gave the great difference in the work with the FindAll call:

to get the AutomationElement corresponding to the Workstation service (at the bottom of the grid) through the FindFirst call takes the same time as to get this element by using FindAll with iteration through the collection.

If you ran the script on a Window 7 or Windows 2008 box, you probably noticed that the latter way is slower than the former as much  as several times.

On the opposite, on a Windows 8 CP x64 box and on a Windwos Server 8 Beta box, both times are almost equal.

Viva, Microsoft UI Automation team! You embedded something (I think it’s caching always ON) that greatly improved the performance of control collections.

%d bloggers like this: