Tag Archives: UI Automation

Test Case Management: generating test results on the fly


The truth is that almost nobody writes test cases these days by hands. Okay, for big and serious, and requiring mind efforts things like scenarios, testers write. PowerShell frameworks are all about automation and demand as little manual work as possible. Like other GUI and Web testing tools, PowerShell frameworks generate test results by watching the code execution. Let’s go to the samples:

ipmo [path]\SePSX.dll
[SePSX.Preferences]::EveryCmdletAsTestResult = $true;
Start-SeFirefox | Enter-SeURL "http://google.com" | Get-SeWebElement -Name q | Set-SeWebElementKeys Cheese | Submit-SeWebElement;

What did we expect from this code? We set the EveryCmdletAsTestResult setting on, started an instance of Firefox, navigated to Google and submitted a query. As we saw no errors, we want to get our passed test results. How to do that?

[TMX.TestData]::CurrentTestScenario.TestResults | FL Name,Status

The output is as follows:
Name : Start-SeFirefox
Status : PASSED

Name : Enter-SeURL “http://google.com”
Status : PASSED

Name : Get-SeWebElement -Name q
Status : PASSED

Name : Set-SeWebElementKeys Cheese
Status : PASSED

Name : Submit-SeWebElement;
Status : PASSED

Name :
Status : NOT TESTED
Every cmdlet reported itself as Passed, and five results display this. The sixth result is a pre-generated test result that will be used in the immediately next cmdlet call.

This is too ideal, now we change our code to obtain real-life results. We are seeking for one of Google 2.0 controls, namely ‘q2’ (Marissa Mayer’s gone, hasn’t she?).

ipmo [path]\SePSX.dll
[SePSX.Preferences]::EveryCmdletAsTestResult = $true;
Start-SeFirefox | Enter-SeURL "http://google.com" | Get-SeWebElement -Name q2 | Set-SeWebElementKeys Cheese | Submit-SeWebElement;

The code failed (controls have not been renamed yet), where are our results?

[TMX.TestData]::CurrentTestScenario.TestResults | FL Name,Status

The output is below:
Name : Start-SeFirefox
Status : PASSED

Name : Enter-SeURL “http://google.com”
Status : PASSED

Name : Get-SeWebElement -Name q2
Status : FAILED

Name : Get-SeWebElement -Name q2
Status : FAILED

Name :
Status : NOT TESTED

The third and the fourth results have different exceptions in their Descriptions, though it’s a place where module’s code review needed…

Now, let’s see how time was consumed:

[TMX.TestData]::CurrentTestScenario.TestResults | FL Name,Status,TimeSpent

As can be seen, there is a default 500 milliseconds’ delay after starting the browser, 2 seconds were spent on navigation to the google page, and 5 seconds (the full time of [SePSX.Preferences]::Timeout) were spent on attempts to get the control of our interest.

In practice, testers are often interested in Failed results, much more often that in the list of Passed. ūüôā How to obtain such a list? First of all, we need to import the TMX module. Selenium and UIAutomation modules use TMX indirectly, as a library, but the advanced functionality is available as cmdlets:

ipmo [path]\TMX.dll;
Search-TMXTestResult -FilterFailed | FL Name,LineNumber,Code,Details

If the time of every test result is stored, what benefits do we have? Consider using the following query:

Search-TMXTestResult -OrderByTimeSpent -Descending -FilterFailed | FL Name,TimeSpent,Code

This lists Failed test results and time consumed in the descending mode (for what kind of user the contemporary versions of MS Excel have been written? Maybe, I’m wrong here? Okay, I’ll rewrite). This command lists Failed test results from bigger time spent to smaller.

Test Case Management: filtering and sorting the results


Let’s start from yesterday’s script. I added the code that never would work: getting a non-existing window and finding the “10” button on Calculator. Here is the script:

ipmo [path]\TMX.dll
ipmo [path]\UIAutomation.dll
Start-Process calc
Get-UIAWindow -n calc* | Get-UIAButton -n 1 | Invoke-UIAButtonClick;
Get-UIAWindow -n calc* | Get-UIAButton -n add | Invoke-UIAButtonClick;
Get-UIAWindow -n calc* | Get-UIAButton -n 1 | Invoke-UIAButtonClick;
Get-UIAWindow -n calc* | Get-UIAButton | Set-UIAFocus | Set-UIAControlKeys -Text "1{+}1{=}"

Get-UIAWindow -n "non-existing window" -Seconds 2
Get-UIAWindow -n calc* | Get-UIAButton -n 10 | Invoke-UIAButtonClick;

Stop-Process -Name calc

Search-TMXTestResult
Export-TMXTestResults -As HTML -Path c:\1\calc_test_results_generated.htm
c:\1\calc_test_results_generated.htm

The three lines copy-pasted from yesterday’s post do exactly what they did yesterday: generate thirteen test results. What do the lines I added? The first line tries to get the window that’s rarely seen in the real life (you may compile one, though). The next line searches for the 10 button (that Microsoft declined to add to the tool, with very oblique explanation).

Okay, what should we expect? The instruction ‘Get-UIAWindow -n “non-existing window” -Seconds 2′ fails after two seconds’ time, so does ‘Get-UIAWindow -n calc* | Get-UIAButton -n 10’.

What does it mean for the whole test? These test cases (i.e., commands) are considered as most time-consuming (several queries to the Automaiton tree until the timeout ends this) and, no doubts, failed.

All the test results that failed can be pulled out with the following code:

Search-TMXTestResult -FilterFailed

The worst test result, considering as the worst a test result with the greatest time spent, can be scooped up with the following piece of code;

Search-TMXTestResult -FilterFailed -OrderByTimeSpent -Descending

Test results that are generated automatically have names based on the code. This leads us to the opportunity to filter test results by a code snippet or a keyword:

Search-TMXTestResult -FilterNameContains Button

or

Search-TMXTestResult -FilterNameContains 10

All that described are going to be available today as part of 0.8.0. Preview 1.

Test Case Management: how to generate test cases from the code


Software engineers use test cases for serious testing. Test cases are written rules, written in words or in code, and intended to become test results. Test results reflect which test cases have passed and which have failed.

This means that test cases are theoretical, static part of testing, whereas test results are dynamic, practical part. Life cycle of a test case longs from several builds to several versions of the product the test case belongs to. Life cycle of test results is from run to run. Usually, a run is performed once per build. Sometimes, several runs (i.e., test results) are conducted per build.

Now, let’s take a look at how contemporary test tools publish results. Click’n’play tools generate results for every click. Is this valuable? I think so, because a test engineer often needs to investigate into ‘not clicked’ issue. To sum up, this is an industry-proven practice to write up test results automatically. In fact, nobody would write test cases like ‘click the Next button’ by hands.

Following the industry practices, we also added test results generation from the code. How it works? If the paramater [UIAutomation.Preferences]::EveryCmdletAsTestResult is set to $true, all the results brought to the pipeline will be also stored as test results.

Let’s take the following piece of code:

ipmo [path]\TMX.dll
ipmo [path]\UIAutomation.dll
[UIAutomation.Preferences]::EveryCmdletAsTestResult=$true
Start-Process calc
Get-UIAWindow -n calc* | Get-UIAButton -n 1 | Invoke-UIAButtonClick;
Get-UIAWindow -n calc* | Get-UIAButton -n add | Invoke-UIAButtonClick;
Get-UIAWindow -n calc* | Get-UIAButton -n 1 | Invoke-UIAButtonClick;
Get-UIAWindow -n calc* | Get-UIAButton | Set-UIAFocus | Set-UIAControlKeys -Text "1{+}1{=}"

Search-TMXTestResult
Export-TMXTestResults -As HTML -Path c:\1\calc_test_results_generated.htm
c:\1\calc_test_results_generated.htm

In the code above, we can see thirteen calls of UIA cmdlets. These thirteen calls generate thirteen test results. That’s all, no more manual work.

The Search-TMXTestResult cmdlets outputs all the test results to the pipeline, allowing you to process them in your own way. The Export-TMXTestResults cmdlet creates a simple HTML report (the projects seriously lacks of a HTML/CSS programmer).

When this way to obtain test result would be right:

  • you are playing with a small experimental script and you simply don’t want to spend time for creating test cases to a probe script
  • your test suite is huge, so huge that writing test cases manually makes no sense
  • you are not a professional tester, but test results are valuable for you

Beware: navigation among Metro items


To whom it might frighten: multi-string names. There are no AutomationId here, only complicated names to get a control.

The Start Screen is much better:

What’s inside? The Mail application:

Kobo:

As can be¬†seen from the last two figures, UI Automation is available to some extent. Nevertheless, we are far away from the ultimate happiness: there’s almost nothing to do in Solitaire.

Dictionary.com is also automation-ready:

It’s time to please ourselves with one more fascinating pic from the Store and say to Microsoft: “Think more about automation engineers _before_ the release”:

Happy testing!

(MS) UIAutomation: putting text into a control


Task: demonstrate how to write in text into a text box field.

Requirements: use UI Automation pattern or Win32.

Solution: the typical code is as follows:

Get-UIAWindow -Name $formTitle | `
 Get-UIAEdit -AutomationId UsernameBox | `
 Set-UIAEditText -Text "text";

Despite the name, the Set-UIAEditText cmdlet uses ValuePattern. This is because that Set-UIAEditText as well as any Set-UIA[ControlType]Text cmdlet is an alias of the Invoke-UIAValuePatternSet cmdlet.
the same story with the Set-UIATextBoxText cmdlet, which is the alias for Set-UIAEditText.
Not every time controls that are supposed to support ValuePattern really support it. This may depend on the developer’s code or on a control’s state. For example, a combo box supports ValuePattern if it has a field to write text in.

As promised, accessing such controls programmatically should raise the ModifiedChanged event.

Sometimes, the ValuePattern can be inaccessible. Due to many reasons, it happens, though rarely. Under these circumstances, you may use the Set-UIAControlText cmdlet, the pure Win32 text setter.

Get-UIAWindow -Name $formTitle | `
 Get-UIAEdit -AutomationId UsernameBox | `
 Set-UIAControlText -Text "text";

Notice, however, that it works slightly differently: it puts text before the text that is already in the text field. What means that you need first to clean it up, the control, before putting a new text in.

(MS) UIAutomation: more ways to get a window


There’s been posted a request for more accurate method to get a window or control. Actually, getting a window by process name or window title may be considered by many as totally unreliable.
The application under test may be a page in the browser (several pages and browser instances are not rare these days), an MDI child, a frame. Simply, several instances of a GUI application can be running at the same time.
All these reasons led to adding new ways of getting windows.

Task:¬†demonstrate how to get the window you’ll be using in tests by its process Id, window handle or process object.

Requirements: provide the reader with ready-to-run examples.

Solution: along with Get-UIAWindows -p (ProcessName) and -n (Name), the version 0.6.7 introduces new parameters for the Get-UIAWindow cmdlet.

    • -ProcessId or -pid
Get-UIAWindow -pid (Get-Process -Name mmc).Id
    • -Process or -p or -InputObject
Get-UIAWindow -p (Get-Process -Name mmc)

Earlier, many should do the following in order to start a process and get its handle:

Start-Process C:\Windows\system32\mmc.exe -Wait:$false
Get-UIAWindow -pn mmc -Seconds 10

the code says to start a process and seek for its window for no more than ten seconds.
Now, we are definitely more sure about what process we are going to get. We simply pipeline it to a cmdlet:

Start-Process C:\Windows\System32\mmc.exe -PassThru | Get-UIAWindow -Seconds 10

The sample provided hooks the same window that we ran in it.

One more requested feature is to handle windows and control by its handle. Whence the handle can be taken?
One way is to get the property:

(Start-Process C:\Windows\System32\mmc.exe -PassThru | Get-UIAWindow -Seconds 10).Current.NativeWindowhandle
(Start-Process C:\Windows\System32\mmc.exe -PassThru | Get-UIAWindow -Seconds 10 | Get-UIATree).Current.NativeWindowhandle

the other is to pipeline the handle out from the control:

PS C:\Users\apetrov1> Start-Process C:\Windows\System32\mmc.exe -PassThru | Get-UIAWindow -Seconds 10 | Read-UIAControlNativeWindowHandle
Start-Process C:\Windows\System32\mmc.exe -PassThru | Get-UIAWindow -Seconds 10 | Get-UIATree | Read-UIAControlNativeWindowHandle

After you’ve got the handle, you are free to use it until it’s being disposed by the operating system:

Get-UIAControlFromHandle -Handle 67856 | Get-UIATreeItem -Name 'Console Root'

(MS) UIAutomation: what are profiles (modes) and how to deal with them? Part 1


Task: demonstrate how to change the module settings in a bunch.

Task #2: provide an explanation what settings are.

Requirements: describe profiles and describe individual settings.

Solution: the UIAutomation module has a range of settigns. You can easily print them by running the following cmdlet:

Show-UIAModuleSettings;

The output provides you with a number of variables, some tell you something by their names, some don’t. Below are these settings:

C:\Users\Administrator> Show-UIAModuleSettings

Timeout settings:
[UIAutomation.Preferences]::Timeout = 20000

Error collection settings:
[UIAutomation.Preferences]::MaximumErrorCount = 256

Common actions:
[UIAutomation.Preferences]::OnErrorAction =
[UIAutomation.Preferences]::OnSleepAction =
[UIAutomation.Preferences]::OnSuccessDelay =

Screenshot settings:
[UIAutomation.Preferences]::OnErrorScreenShot = False
[UIAutomation.Preferences]::ScreenShotFolder = C:\Users\Administrator\AppData\Local\Temp

Log settings:
[UIAutomation.Preferences]::Log = True
[UIAutomation.Preferences]::LogPath = C:\Users\Administrator\AppData\Local\Temp\UIAutomation.log

Debugging delays:
[UIAutomation.Preferences]::OnClickDelay = 0
[UIAutomation.Preferences]::OnErrorDelay = 500
[UIAutomation.Preferences]::OnSleepDelay = 0
[UIAutomation.Preferences]::OnSuccessDelay = 500

Transcript settings
[UIAutomation.Preferences]::TranscriptInterval = 200

Highlighter settings:
[UIAutomation.Preferences]::Highlight = True
[UIAutomation.Preferences]::HighlighterBorder = 3
[UIAutomation.Preferences]::HighlighterColor = Color [Red]
C:\Users\Administrator>

What is what here? First of all, there is the need to mention also profiles (modes. Now, there is no decision on how to call them better :)).

Out of the box, you have the Presentation profile being set:

[UIAutomation.Mode]::Profile = [UIAutomation.Modes]::Presentation

This is colorful and not very fast. This is the face of the module: somebody downloads the module, tries to work with and shows to somebody else. The first thing the module should do well is to be accepted by the public. Why? Without the consumers, how should it progress?
Therefore, the first stage of the module life-cycle is being in the hands of newcomers (to the module). A tester, an IT manager or their boss(es), even the tester does through the newcomer stage.
Here comes the presentation mode: highlighting, half a second’s delays on every actions. Just to recline in the chair and watch the anime on the screen. Let’s explain these four settings:
[UIAutomation.Preferences]::OnClickDelay = 0
[UIAutomation.Preferences]::OnErrorDelay = 500
[UIAutomation.Preferences]::OnSleepDelay = 0
[UIAutomation.Preferences]::OnSuccessDelay = 500
If the cmdlet finished successfully (returned a control or called a pattern), it sleep for five hundred milliseconds. The time the spectators can use for exhaling and moving eyes.
If the cmdlet fails, it waits for 500 milliseconds, giving you the chance to stop the presentation or to continue if it’s not critical.
ONClickDelay is used in Win32 click cmdlets like Invoke-UIAControlClick and Invoke-UIAControlContextMenu.
OnSleepDelay is used in specific cases as if your host is overloaded and there is no reason to try to get an AutomationElement without a delay.
When you are debugging the script, the Debug mode is much more useful. It sleep a second after the success and five seconds after a failure.
When you are leaving the office and the script will work unattended, set the Normal mode, where there are no delays at all.
Moreover, all the settings, delays and others, you are allowed to change manually.
Homework: try to play with profiles (modes). Change individual settings. Observe, what are results. Change modes over any individual settings were done. Try change them in any combination.

(MS) UIAutomation: all about a combo box


Task: automate combo box selection.

Task #2: provide a homework helping to learn the control.

Requirements: describe both getting the selection and selecting list items.

Solution: below is the preparation (services.msc is already running, select a service, open the context menu, click Properties):

Get-UIAWindow -Name Services | `
Get-UIADataGrid | `
Get-UIADataItem -Name Alerter | `
Invoke-UIAControlContextMenu | `
Get-UIAMenuItem -Name Properties | `
Invoke-UIAMenuItemClick;

If you are sitting against a Windows 7 box, select for our test another service, for example, BranchCache.

Now the combo box can be seen. It has three or four values (I’m again doing the testing on a Windows XP box as I care about working on older and the newest operating systems): Automatic, Manual, and Disabled. There might or mightn’t be two ‘Automatic…’ items.

How to get what is selected now? Do so:

$((Get-UIAWindow -p mmc | Get-UIAComboBox | Get-UIAComboBoxSelection)[0]).Current

Now, is there a way to set a selection? Who roared ‘SetFocus; SendKeys’? Shut up, it’s not a sport but a torture. If a control is healthy and developers’ hands are sewn directly to shoulders, we can and we must use patterns.

Get-UIAWindow -p mmc | `
Get-UIAComboBox | `
Invoke-UIAComboBoxExpand | `
Get-UIAList | `
Get-UIAListItem -Name Automatic | `
Invoke-UIAListItemSelectItem -ItemName Automatic | `
Invoke-UIAListItemClick;

Too long? Let’s shortenize:

Get-UIAWindow -p mmc | `
Get-UIAComboBox | `
Invoke-UIAComboBoxExpand | `
Get-UIAListItem -Name Automatic | `
Invoke-UIAListItemSelectItem -ItemName Automatic | `
Invoke-UIAListItemClick;

We managed to delete only the Get-UIAList cmdlet. Why?
1) All the Get- cmdlets can perform search from the point they’ve been given with. Given a window, a childwindow or a control, from any of that start, a Get- cmdlet (excluding Get-UIAWindow as it does not have the -InputObject parameter) can continue the search down the Automation tree.
2) All the Invoke- (and pattern-cmdlets Get- and Set- like Get-UIAComboBoxSelection. Recognize them in a way: Get/Set-UIA[ControlType][Pattern name or abbreviation]) cmdlets can’t perform any search. They should be fed with a pipeline from a Get- cmdlet or a cmdlet that passed thru (-PassThru’ed) the object of the AutomationElement type.

To shortenize even more, you may cut out the first call, Get-UIAWindow (as it’s always stored in the [UIAutomation.CurrentData]::CurrentWindow variable), but be very attentively doing so: if you/your test Get-s a window (the main or an active child), closes it and tries to work with subsequent calls of Get-UIAWindow, error definitely will appear.

Homework: practisize working with the combo box, try another values, get the selection (I suppose that you know what is the difference between the selected list item and the selected list item that is braced inside the dotted box), save selection to a file, change selection, read it back from the file you’ve created, set the saved selection to the combo box, and do it until a new post got published. ūüôā

(MS) UIAutomation: how to be informed if a control disappears…


An interesting question has been raised yesterday: how UIAutomation help to deal with a control that is disappearing. As known, MS UIAutomation works with only controls that are visible. It’s either ideology and technical limitation.
How the button I mentioned can disappear? For example, this is a button serving a progress bar of a wizard with the label Stop on its face. The progress bar achieves 100% and the button hides.

Using such an awkward code is not a solution:

while (Get-UIAWindow -p processname | Get-UIAButton -n Stop)

Here comes the help again from MS UI Automation: StructureChangedEvent provides us with six types of EventID, among others is the one we needed – ChildRemoved.

Task: wait for a button to be out of sight.

Requirements: get a signal from the button that can triger the main code. Use StructureChangedEventHandler.

Solution: the following sample represents the use of StructureChangedEvent

ipmo .\UIAutomation.dll
[bool]$global:buttonVisible1 = $true;
[bool]$global:buttonVisible2 = $true;
[bool]$global:buttonAddedAgain = $false;
Get-UIAWindow -n ShowHideButton | Get-UIAButton -n Button | Register-UIAStructureChangedEvent -ChildRemoved -EventAction {"01" >> c:\report.txt; $global:buttonVisible1=$false; "1" >> c:\report.txt;} -Verbose
Get-UIAWindow -n ShowHideButton | Get-UIAButton -n Button | Register-UIAStructureChangedEvent -ChildRemoved -EventAction {"02" >> c:\report.txt; $global:buttonVisible2=$false; "2" >> c:\report.txt;} -Verbose
Get-UIAWindow -n ShowHideButton | Register-UIAStructureChangedEvent -ChildAdded -EventAction {$global:buttonAddedAgain=$true; "3" >> c:\report.txt;} -Verbose
#sleep -Seconds 5
#Write-Host "1" $global:buttonVisible1
#Write-Host "2" $global:buttonVisible2
#Write-Host "3" $global:buttonAddedAgain

while ($global:buttonVisible1 -eq $true -and $global:buttonVisible2 -eq $true){
	sleep -Seconds 1;
	Write-Host "1" $global:buttonVisible1
	Write-Host "2" $global:buttonVisible2
	Write-Host "waiting...";
}
Write-Host "completed";
Write-Host "1" $global:buttonVisible1
Write-Host "2" $global:buttonVisible2
Write-Host "3" $global:buttonAddedAgain

The sample works with a simple program that is a form with two buttons. One button is Show/Hide button, its click event shows/hides the second button.

Just run the program (samples 0.6.2 or newer), after that run the script (it subscribes to events) and click one or several times upon the Show/Hide button.
Soon or later you’ll see both global variables set to $false, and the third to $true.

Restrictions may apply, to read more about the event and which eventId can be used on a purpose, visit StructureChangedEventArgs Constructor

(MS) UIAutomation: administrative rights may be necessary on comtemporary operating systems


Problem: the framework behaves strangely.

Requirements: reclaim the stability.

Solution:¬†run your powershell console or IDE under domain admin credentials or through Run As Administrator. When working with something like Administrative tools (services.msc or compmgmt.msc, or ADUC), or regedit, or your company’s administrative tool, there might be a waterfall of errors, sometimes amazing. The answer is simple: use administrative credentials while working with administrative applications through inter-process¬†communication.

%d bloggers like this: