Metro automation: a nightmare of code generation


Visit our Metro UI automation page.

 

I’ve been struggling with code generation in UIAutomationSpy for a couple of hours. What’s the problem?

Code that UIAutomationSpy generates for Windows applications is what you can run immediately after recording (okay, some polishing or additions may be needed). At least, from today, when I removed the first line of code (i.e., RootElement) and UIAutomaitonSpy is now so close to the Start-UIARecorder-generated code as it might be.

For example, this is code for Notepad (an emtpy document, I click File, the dialog opens and I close it):

Get-UIAWindow -Class 'Notepad' -Name 'Untitled - Notepad' | `

Get-UIAMenuBar -AutomationId 'MenuBar' -Name 'Application' | `

Get-UIAMenuItem -AutomationId 'Item 1' -Name 'File'
Get-UIAWindow -Class 'Notepad' -Name 'Untitled - Notepad' | `

Get-UIAMenuBar -AutomationId 'MenuBar' -Name 'Application' | `

Get-UIAMenuItem -AutomationId 'Item 1' -Name 'File' | `

Get-UIAMenu -Class '#32768' -Name 'File' | `

Get-UIAMenuItem -AutomationId 'Item 1' -Name 'New'
Get-UIAWindow -Class 'Notepad' -Name 'Untitled - Notepad' | `

Get-UIADocument -AutomationId '15' -Class 'Edit'
Get-UIAWindow -Class 'Notepad' -Name 'Untitled - Notepad' | `

Get-UIAWindow -Class '#32770' -Name 'Open' | `

Get-UIATitleBar -AutomationId 'TitleBar' -Name 'Open'

The code is readable (to some extent), though even such code should be prepared to run well.

In fact, the code is raw and the Start-UIARecorder cmdlet still works better:

Get-UIAWindow -n *notepad* | Get-UIAMenuItem -Name File | Invoke-UIAMenuItemExpand | Get-UIAMenuItem -Name Open* | Invoke-UIAMenuItemClick

Anyway, a Windows app is not a mythical beast. Metro UI is more complicated; below is a typical piece of generated code:

Get-UIAList -AutomationId 'GridRoot' -Class 'GridRoot' -Name 'Start Apps' | `
Get-UIAPane -AutomationId 'ScrollViewer' -Class 'TouchScrollViewer' | `
Get-UIAPane -AutomationId 'content' -Class 'GridContent' | `
Get-UIAGroup -AutomationId 'Group 2305843009213693952' -Class 'GridGroup' -Name 'Mail' | `
Get-UIAListItem -AutomationId 'microsoft.windowscommunicationsapps_8wekyb3d8bbwe!Microsoft.WindowsLive.Chat' -Class 'GridTileElement' -Name 'Messaging'

This code simpy finds the Messaging app. This is a dead code. To make the code alive, we need:

  1. add Get-UIADesktop before the first instruction
  2. add Show-UIAMetroStartScreen before the first instruction
  3. remove the line 2: Get-UIAPane -AutomationId ‘ScrollViewer’ -Class ‘TouchScrollViewer’ | `
  4. remove the line 3: Get-UIAPane -AutomationId ‘content’ -Class ‘GridContent’ | `
  5. remove the part of line 5: -AutomationId ‘microsoft.windowscommunicationsapps_8wekyb3d8bbwe!Microsoft.WindowsLive.Chat’ -Class ‘GridTileElement’

After editing, code looks like:

Get-UIADesktop | Get-UIAList -AutomationId 'GridRoot' -Class 'GridRoot' -Name 'Start Apps' | `
Get-UIAGroup -AutomationId 'Group 2305843009213693952' -Class 'GridGroup' -Name 'Mail' | `
Get-UIAListItem -Name 'Messaging'

Finally, the right code would be (for Windows 8 RP):

Get-UIADesktop;
Show-UIAMetroStartScreen;
Get-UIAListItem -Name *Internet*Explorer* | Invoke-UIAListItemClick;

Show-UIAMetroStartScreen;
Get-UIAListItem -Name *people* | Invoke-UIAListItemClick;

Show-UIAMetroStartScreen;
Get-UIAListItem -Name *messag* | Invoke-UIAListItemClick;

Show-UIAMetroStartScreen;
Get-UIAListItem -Name *phot* | Invoke-UIAListItemClick;

Show-UIAMetroStartScreen;
Get-UIAListItem -Name *calen* | Invoke-UIAListItemClick;

and for Windows Server 2012:

Get-UIADesktop;
Show-UIAMetroStartScreen;
Get-UIAListItem -Name *Internet*Explorer* | Invoke-UIAListItemClick;

Show-UIAMetroStartScreen;
Get-UIAListItem -Name *control* | Invoke-UIAListItemClick;

Show-UIAMetroStartScreen;
Get-UIAListItem -Name *computer* | Invoke-UIAListItemClick;

Show-UIAMetroStartScreen;
Get-UIAListItem -Name *task*man* | Invoke-UIAListItemClick;

Show-UIAMetroStartScreen;
Get-UIAListItem -Name *admi*tool* | Invoke-UIAListItemClick;

As can be seen, the problem is that code UIAutomationSpy generates is far away of what is working…

The worst of all, the code for Windows applications can be useful from the top control to the current, whereas the code for Metro UI apps should be filtered, edited and some commands should be added to it.
This requires two branches of code generation, doesn’t it?

There is a video based on code samples (HD mode is recommended if you want to read the code).

function Start-MetroApp
{
param( [string]$appName )

Show-UIAMetroStartScreen;
Get-UIAListItem -Name $appName | Invoke-UIAListItemClick;
}
Get-UIADesktop;

Start-MetroApp Music;

Start-MetroApp Finan*;

Start-MetroApp '*internet*explorer*';

Start-MetroApp people;
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: