Metro automation: the first script recorded!


With a bit of free time, I built a PowerShell host that works over Start Screen and recorded my first Metro script. As a host I used PoshConsole (a flamboyant shell, and I couldn’t use autocomplete for the UIAutomation cmdlets) with the manifest. At first, I tried to compile BGShell, but it falls each run, perhaps as a matter of using Windows Forms Aero for .NET 2.0 or something of this kind.

Okay, I ran PoshConsole, tested first clicks on tiles by commands issued manually and recorded a script: here it is, for your understanding what the structure of controls are (Inspect would show you similar results). In short, script does the following (I did the following):

– press Win to get to Start Screen

– click onto Photos, hovering near links to Photos, Skydrive photos, Facebook, Flickr, and text fields at the top of the screen

– click on Mail, select an account, hover between letters.

Below is the output:

Get-UIAWindow -AutomationId 'Start menu window' -Name 'Start menu' | `
	Get-UIAPane -AutomationId 'Root' -Name 'Start menu' | `
	Get-UIAList -AutomationId 'GridRoot' -Name 'Start Apps' | `
	Get-UIAPane -AutomationId 'ScrollViewer' | `
	Get-UIAPane -AutomationId 'content' | `
	Get-UIAListItem -AutomationId 'microsoft.windowsphotos_8wekyb3d8bbwe!Microsoft.WindowsLive.ModernPhotos' -Name 'Photos LargeTile1 Image';
		#InvokePattern
		#ValuePattern
		#ExpandCollapsePattern
		#GridItemPattern
		#SelectionItemPattern
		#ScrollItemPattern

Get-UIAPane -AutomationId 'Start menu window' -Name 'Start menu';
		# no supported pattterns

Get-UIAWindow -Name 'Photos' | `
	Get-UIAText -Name 'Launching Photos';
		#ValuePattern

Get-UIAWindow -Name 'Photos' | `
	Get-UIAPane | `
	Get-UIAPane -Name 'Windows Photos' | `
	Get-UIAList | `
	Get-UIACustom -Name 'Scrolling Container' | `
	Get-UIAImage;
		#ValuePattern

Get-UIAWindow -Name 'Photos' | `
	Get-UIAPane | `
	Get-UIAPane -Name 'Windows Photos' | `
	Get-UIAList | `
	Get-UIACustom -Name 'Scrolling Container' | `
	Get-UIAText -Name 'SkyDrive photos';
		# no supported pattterns

Get-UIAWindow -Name 'Photos' | `
	Get-UIAPane | `
	Get-UIAPane -Name 'Windows Photos' | `
	Get-UIAList | `
	Get-UIAListItem -Name 'Facebook photos

See your albums';
		#InvokePattern
		#SelectionItemPattern

Get-UIAWindow -Name 'Photos' | `
	Get-UIAPane | `
	Get-UIAPane -Name 'Windows Photos' | `
	Get-UIACustom -Name 'Scrolling Container';
		# no supported pattterns

Get-UIAWindow -Name 'Photos' | `
	Get-UIAPane | `
	Get-UIAPane -Name 'Windows Photos' | `
	Get-UIAList | `
	Get-UIACustom -Name 'Scrolling Container' | `
	Get-UIAText -Name 'Flickr photos' | `
	#TextChangedEvent triggered
#source title:  of the type Document | `
	Invoke-UIAListItemClick | `
	#TextChangedEvent triggered
#source title:  of the type Document;
		# no supported pattterns

Get-UIAWindow -Name 'Photos' | `
	Get-UIAPane | `
	Get-UIAPane -Name 'Windows Photos' | `
	Get-UIAList | `
	Get-UIACustom -Name 'Scrolling Container' | `
	Get-UIAText -Name 'Flickr photos';
		# no supported pattterns

Get-UIAWindow -Name 'Photos' | `
	Get-UIAPane | `
	Get-UIAPane -Name 'Windows Photos' | `
	Get-UIACustom -Name 'Scrolling Container' | `
	#TextChangedEvent triggered
#source title:  of the type Document;
		# no supported pattterns

Get-UIAWindow -Name 'Photos' | `
	Get-UIAPane | `
	Get-UIAPane -Name 'Windows Photos' | `
	Get-UIAList | `
	Get-UIAListItem -Name 'Facebook photos

See your albums';
		#InvokePattern
		#SelectionItemPattern

Get-UIAWindow -Name 'Photos' | `
	Get-UIAPane | `
	Get-UIAPane -Name 'Windows Photos' | `
	Get-UIACustom -Name 'Scrolling Container';
		# no supported pattterns

Get-UIAWindow -Name 'Photos' | `
	Get-UIAPane | `
	Get-UIAPane -Name 'Windows Photos' | `
	Get-UIAList | `
	Get-UIACustom -Name 'Scrolling Container' | `
	Get-UIAImage;
		#ValuePattern

Get-UIAWindow -Name 'Photos' | `
	Get-UIAPane | `
	Get-UIAPane -Name 'Windows Photos' | `
	Get-UIAText -Name 'Photos';
		# no supported pattterns

Get-UIAWindow -Name 'Photos' | `
	Get-UIAPane -Name 'Windows Photos';
		#ValuePattern

Get-UIAWindow -Name 'Photos' | `
	Get-UIAPane | `
	Get-UIAText -Name 'APP PREVIEW';
		# no supported pattterns

Get-UIAWindow -Name 'Photos' | `
	Get-UIAPane | `
	Get-UIAPane -Name 'Windows Photos' | `
	Get-UIAText -Name 'Photos';
		# no supported pattterns

Get-UIAWindow -AutomationId 'window' -Name 'PoSh - C:\1\poshconsole_f9bb2b127402\PoshConsole\bin\AnyCPU\Release (FileSystem)';
		#WindowPattern
		#TransformPattern
		#SynchronizedInputPattern

Get-UIAWindow -AutomationId 'window' -Name 'PoSh - C:\1\poshconsole_f9bb2b127402\PoshConsole\bin\AnyCPU\Release (FileSystem)' | `
	Get-UIADocument | `
	#TextChangedEvent triggered
#source title:  of the type Document;
		#TextPattern
		#SynchronizedInputPattern

Get-UIAWindow -Name 'Photos' | `
	Get-UIAPane -Name 'Windows Photos';
		#ValuePattern

Get-UIAWindow -Name 'Photos' | `
	Get-UIAPane | `
	Get-UIAPane -Name 'Windows Photos' | `
	Get-UIACustom -Name 'Scrolling Container';
		# no supported pattterns

Get-UIAButton -AutomationId 'ImmersiveBackground' -Name 'Launch Start Menu' | `
	#TextChangedEvent triggered
#source title:  of the type Document;
		#InvokePattern

Get-UIAWindow -AutomationId 'Start menu window' -Name 'Start menu' | `
	Get-UIAPane -AutomationId 'Root' -Name 'Start menu' | `
	Get-UIAList -AutomationId 'GridRoot' -Name 'Start Apps' | `
	Get-UIAPane -AutomationId 'ScrollViewer' | `
	Get-UIAPane -AutomationId 'content' | `
	Get-UIAListItem -AutomationId 'Microsoft.XboxLIVEGames_8wekyb3d8bbwe!Microsoft.XboxLIVEGames' -Name 'Xbox LIVE Games' | `
	#TextChangedEvent triggered
#source title:  of the type Document;
		#InvokePattern
		#ValuePattern
		#ExpandCollapsePattern
		#GridItemPattern
		#SelectionItemPattern
		#ScrollItemPattern

Get-UIAWindow -AutomationId 'Start menu window' -Name 'Start menu' | `
	Get-UIAPane -AutomationId 'Root' -Name 'Start menu' | `
	Get-UIAList -AutomationId 'GridRoot' -Name 'Start Apps' | `
	Get-UIAPane -AutomationId 'ScrollViewer' | `
	Get-UIAPane -AutomationId 'content' | `
	Get-UIAListItem -AutomationId 'microsoft.windowscommunicationsapps_8wekyb3d8bbwe!Microsoft.WindowsLive.Mail' -Name 'Mail SwedishPod101.com, SwedishPod101 Word of the Day - 04/27/2012, Friday, April 27th, 2012 Your Swedish word of the day is: väg (Can you guess what it is? Click here';
		#InvokePattern
		#ValuePattern
		#ExpandCollapsePattern
		#GridItemPattern
		#SelectionItemPattern
		#ScrollItemPattern

Get-UIAWindow -Name 'Mail' | `
	Get-UIAPane | `
	Get-UIAPane -Name 'Mail' | `
	Get-UIAList -Name 'Message List' | `
	Get-UIACustom -Name 'Scrolling Container' | `
	Get-UIAListItem -Name 'From Группа разработчиков службы Hotmail, Subject Совет для Hotmail: как избавиться от спама, Received ‎April ‎18, ‎2012, Read' | `
	Get-UIACustom -Name 'Группа разработчиков службы Hotmail

Совет для Hotmail: как избавиться от спама

‎4/‎18/‎12

********************************************************************** Windows Live Hotmail(r) ********************************************************************** Hotmail защищает от спама. Hotmail использует множество инструментов для фил

From Группа разработчиков службы Hotmail, Subject Совет для Hotmail: как избавиться от спама, Received ‎April ‎18, ‎2012, Read' | `
	Get-UIAText -Name 'Совет для Hotmail: как избавиться от спама';
		# no supported pattterns

Get-UIAWindow -Name 'Mail' | `
	Get-UIAPane | `
	Get-UIAPane -Name 'Mail' | `
	Get-UIAList -Name 'Message List' | `
	Get-UIACustom -Name 'Scrolling Container' | `
	Get-UIAListItem -Name 'From Пользователь denisebaker68@hotmail.com, Subject Пользователь denisebaker68@hotmail.com хочет стать вашим другом в службе Windows Live, Received ‎April ‎17, ‎2012, Read' | `
	Get-UIACustom -Name 'Пользователь denisebaker68@hotmail.com

Пользователь denisebaker68@hotmail.com  хочет стать вашим другом в службе Windows Live

‎4/‎17/‎12

Пользователь Пользователь denisebaker68@hotmail.com хочет добавить вас в свой список друзей в службе Windows Live <https://xxxxxxxxx

From Пользователь denisebaker68@hotmail.com, Subject Пользователь denisebaker68@hotmail.com хочет стать вашим другом в службе Windows Live, Received ‎April ‎17, ‎2012, Read' | `
	Get-UIAText -Name 'Пользователь Пользователь denisebaker68@hotmail.com хочет добавить вас в свой список друзей в службе Windows Live <https://xxxxxxxxx' | `
	#TextChangedEvent triggered
#source title:  of the type Document;
		# no supported pattterns

Get-UIAWindow -Name 'Mail' | `
	Get-UIAPane | `
	Get-UIAPane -Name 'Mail' | `
	Get-UIAMenuItem -Name 'Folders' | `
	#TextChangedEvent triggered
#source title:  of the type Document | `
	Invoke-UIAListItemClick | `
	#TextChangedEvent triggered
#source title:  of the type Document | `
	Invoke-UIAButtonClick | `
	#TextChangedEvent triggered
#source title:  of the type Document | `
	Invoke-UIAButtonClick | `
	#TextChangedEvent triggered
#source title:  of the type Document;
		#InvokePattern
Advertisements

3 responses

  1. Hi!
    I am very interested on this.
    Please, how I could contact you for a better explanation?
    I am working on a Win8 project software development. I am a tester and I am very interested to know how you did this.
    Thanks!
    Patricia.

    1. I completely forgot, there IS the place where more or less usable to discuss PowerShell testing. I created a group in LinkedIn called ‘Software Testing Using PowerShell’, it’s free for all to start a discussion, and it can also send me a letter on my corporate e-mail if something new appered there.

  2. Hello Patricia, I’ve been (and will be for the rest of this week) on a vacation without proper Internet access and couldn’t answer.

    Regarding your questions, where (to discuss) is just here or on the Discussions page of the project. (these are public pages, the more people use the technology, the better technology itself)

    About ‘how you did this’ – if I understood right, you’re asking about what to do if you needed to compile an application that is floating over the Metro UI? The answer is quite simple, and to the same extent, hardly usable 🙂 as Microsoft itself didn’t offer yet a proper tool.
    I’ll tomorrow post a post on the topic. Shortly, you need to add a manifest with the uiAccess parameter, do the application topmost, sign it and put it in a secure location (%systemdrive% or %ProgramFiles%, the latter is better).

    The second part, I mean the application that can run scripts, is not written yet (I have a work load and therefore had no time). I’m going to create the application later this week.

    Alexander

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: