Tag Archives: software testing

Just for fun: after acquisition, our frameworks are used in a Fortune 50 company…


We’re here:

http://money.cnn.com/magazines/fortune/fortune500/2012/full_list/ , slightly above the fifty line…

Quality report: a great success of PowerShell in testing


Yesterday’s build of a commercial application we are working on was reported as “with a bug”. An annoying dialog appeared here and there after a relatively long time of uninterruptible work of the application’s console. Need to say that reproduction of such bugs (“the console got tired and a dialog appears after an hour or more…”) is not a piece of cake. At least, no one wants to await the dialog in the debugger for an hour or two.

Now, the serious part of the product is being tested with PowerShell. Whereas the system part is being tested with C# (the ‘system team’ prefers writing code what I call ‘writing own bugs’), the UI including simple interactions like selecting users from a domain is done in PowerShell. The majority of the code exploits UIAutomation and TMX modules.

Below is the yesterday’s report (a TMX results report):

Suites:31 Passed:19 Failed:9 Not tested:3
Scenarios:227 Passed:206 Failed:21 Not tested:0
Test cases:2682 Passed:2479 Failed:161 Not tested:42 Time spent:8005 seconds

As can be seen, the average time of a test case is 8005/(2682-42)=3,03 seconds.

The fail/pass rate is 161/2682=6%.

Data is nothing if you have none to compare. Below is a report done a week ago (test cases number varies from run to run as the autotest developer and maintainer switches test cases on or off due to changes in the UI or for analysis and making amendments to the test suite):

Suites:31 Passed:19 Failed:8 Not tested:4
Scenarios:197 Passed:188 Failed:9 Not tested:0
Test cases:2251 Passed:2204 Failed:12 Not tested:35 Time spent:4623 seconds

The ‘happy’ run has 0,5% fail/pass rate (0.0% fail/pass rate is hardly achievable during the time the product’s UI is being changed).

The average time is better than in a normal case and counted as 4623/(2251-35)=2,08 seconds per test case, and there is still room for further improvements. Preferably, that fail-run increments were bound to 20%, not 50%.

Anyway, the red peak of 6% of the height of a percent chart seems appropriate. Isn’t it a success of using PowerShell in software testing?

(MS) UIAutomation: putting text into a control


Task: demonstrate how to write in text into a text box field.

Requirements: use UI Automation pattern or Win32.

Solution: the typical code is as follows:

Get-UIAWindow -Name $formTitle | `
 Get-UIAEdit -AutomationId UsernameBox | `
 Set-UIAEditText -Text "text";

Despite the name, the Set-UIAEditText cmdlet uses ValuePattern. This is because that Set-UIAEditText as well as any Set-UIA[ControlType]Text cmdlet is an alias of the Invoke-UIAValuePatternSet cmdlet.
the same story with the Set-UIATextBoxText cmdlet, which is the alias for Set-UIAEditText.
Not every time controls that are supposed to support ValuePattern really support it. This may depend on the developer’s code or on a control’s state. For example, a combo box supports ValuePattern if it has a field to write text in.

As promised, accessing such controls programmatically should raise the ModifiedChanged event.

Sometimes, the ValuePattern can be inaccessible. Due to many reasons, it happens, though rarely. Under these circumstances, you may use the Set-UIAControlText cmdlet, the pure Win32 text setter.

Get-UIAWindow -Name $formTitle | `
 Get-UIAEdit -AutomationId UsernameBox | `
 Set-UIAControlText -Text "text";

Notice, however, that it works slightly differently: it puts text before the text that is already in the text field. What means that you need first to clean it up, the control, before putting a new text in.

(MS) UIAutomation: more ways to get a window


There’s been posted a request for more accurate method to get a window or control. Actually, getting a window by process name or window title may be considered by many as totally unreliable.
The application under test may be a page in the browser (several pages and browser instances are not rare these days), an MDI child, a frame. Simply, several instances of a GUI application can be running at the same time.
All these reasons led to adding new ways of getting windows.

Task: demonstrate how to get the window you’ll be using in tests by its process Id, window handle or process object.

Requirements: provide the reader with ready-to-run examples.

Solution: along with Get-UIAWindows -p (ProcessName) and -n (Name), the version 0.6.7 introduces new parameters for the Get-UIAWindow cmdlet.

    • -ProcessId or -pid
Get-UIAWindow -pid (Get-Process -Name mmc).Id
    • -Process or -p or -InputObject
Get-UIAWindow -p (Get-Process -Name mmc)

Earlier, many should do the following in order to start a process and get its handle:

Start-Process C:\Windows\system32\mmc.exe -Wait:$false
Get-UIAWindow -pn mmc -Seconds 10

the code says to start a process and seek for its window for no more than ten seconds.
Now, we are definitely more sure about what process we are going to get. We simply pipeline it to a cmdlet:

Start-Process C:\Windows\System32\mmc.exe -PassThru | Get-UIAWindow -Seconds 10

The sample provided hooks the same window that we ran in it.

One more requested feature is to handle windows and control by its handle. Whence the handle can be taken?
One way is to get the property:

(Start-Process C:\Windows\System32\mmc.exe -PassThru | Get-UIAWindow -Seconds 10).Current.NativeWindowhandle
(Start-Process C:\Windows\System32\mmc.exe -PassThru | Get-UIAWindow -Seconds 10 | Get-UIATree).Current.NativeWindowhandle

the other is to pipeline the handle out from the control:

PS C:\Users\apetrov1> Start-Process C:\Windows\System32\mmc.exe -PassThru | Get-UIAWindow -Seconds 10 | Read-UIAControlNativeWindowHandle
Start-Process C:\Windows\System32\mmc.exe -PassThru | Get-UIAWindow -Seconds 10 | Get-UIATree | Read-UIAControlNativeWindowHandle

After you’ve got the handle, you are free to use it until it’s being disposed by the operating system:

Get-UIAControlFromHandle -Handle 67856 | Get-UIATreeItem -Name 'Console Root'

(MS) UIAutomation: what are profiles (modes) and how to deal with them? Part 1


Task: demonstrate how to change the module settings in a bunch.

Task #2: provide an explanation what settings are.

Requirements: describe profiles and describe individual settings.

Solution: the UIAutomation module has a range of settigns. You can easily print them by running the following cmdlet:

Show-UIAModuleSettings;

The output provides you with a number of variables, some tell you something by their names, some don’t. Below are these settings:

C:\Users\Administrator> Show-UIAModuleSettings

Timeout settings:
[UIAutomation.Preferences]::Timeout = 20000

Error collection settings:
[UIAutomation.Preferences]::MaximumErrorCount = 256

Common actions:
[UIAutomation.Preferences]::OnErrorAction =
[UIAutomation.Preferences]::OnSleepAction =
[UIAutomation.Preferences]::OnSuccessDelay =

Screenshot settings:
[UIAutomation.Preferences]::OnErrorScreenShot = False
[UIAutomation.Preferences]::ScreenShotFolder = C:\Users\Administrator\AppData\Local\Temp

Log settings:
[UIAutomation.Preferences]::Log = True
[UIAutomation.Preferences]::LogPath = C:\Users\Administrator\AppData\Local\Temp\UIAutomation.log

Debugging delays:
[UIAutomation.Preferences]::OnClickDelay = 0
[UIAutomation.Preferences]::OnErrorDelay = 500
[UIAutomation.Preferences]::OnSleepDelay = 0
[UIAutomation.Preferences]::OnSuccessDelay = 500

Transcript settings
[UIAutomation.Preferences]::TranscriptInterval = 200

Highlighter settings:
[UIAutomation.Preferences]::Highlight = True
[UIAutomation.Preferences]::HighlighterBorder = 3
[UIAutomation.Preferences]::HighlighterColor = Color [Red]
C:\Users\Administrator>

What is what here? First of all, there is the need to mention also profiles (modes. Now, there is no decision on how to call them better :)).

Out of the box, you have the Presentation profile being set:

[UIAutomation.Mode]::Profile = [UIAutomation.Modes]::Presentation

This is colorful and not very fast. This is the face of the module: somebody downloads the module, tries to work with and shows to somebody else. The first thing the module should do well is to be accepted by the public. Why? Without the consumers, how should it progress?
Therefore, the first stage of the module life-cycle is being in the hands of newcomers (to the module). A tester, an IT manager or their boss(es), even the tester does through the newcomer stage.
Here comes the presentation mode: highlighting, half a second’s delays on every actions. Just to recline in the chair and watch the anime on the screen. Let’s explain these four settings:
[UIAutomation.Preferences]::OnClickDelay = 0
[UIAutomation.Preferences]::OnErrorDelay = 500
[UIAutomation.Preferences]::OnSleepDelay = 0
[UIAutomation.Preferences]::OnSuccessDelay = 500
If the cmdlet finished successfully (returned a control or called a pattern), it sleep for five hundred milliseconds. The time the spectators can use for exhaling and moving eyes.
If the cmdlet fails, it waits for 500 milliseconds, giving you the chance to stop the presentation or to continue if it’s not critical.
ONClickDelay is used in Win32 click cmdlets like Invoke-UIAControlClick and Invoke-UIAControlContextMenu.
OnSleepDelay is used in specific cases as if your host is overloaded and there is no reason to try to get an AutomationElement without a delay.
When you are debugging the script, the Debug mode is much more useful. It sleep a second after the success and five seconds after a failure.
When you are leaving the office and the script will work unattended, set the Normal mode, where there are no delays at all.
Moreover, all the settings, delays and others, you are allowed to change manually.
Homework: try to play with profiles (modes). Change individual settings. Observe, what are results. Change modes over any individual settings were done. Try change them in any combination.

(MS) UIAutomation: all about a combo box


Task: automate combo box selection.

Task #2: provide a homework helping to learn the control.

Requirements: describe both getting the selection and selecting list items.

Solution: below is the preparation (services.msc is already running, select a service, open the context menu, click Properties):

Get-UIAWindow -Name Services | `
Get-UIADataGrid | `
Get-UIADataItem -Name Alerter | `
Invoke-UIAControlContextMenu | `
Get-UIAMenuItem -Name Properties | `
Invoke-UIAMenuItemClick;

If you are sitting against a Windows 7 box, select for our test another service, for example, BranchCache.

Now the combo box can be seen. It has three or four values (I’m again doing the testing on a Windows XP box as I care about working on older and the newest operating systems): Automatic, Manual, and Disabled. There might or mightn’t be two ‘Automatic…’ items.

How to get what is selected now? Do so:

$((Get-UIAWindow -p mmc | Get-UIAComboBox | Get-UIAComboBoxSelection)[0]).Current

Now, is there a way to set a selection? Who roared ‘SetFocus; SendKeys’? Shut up, it’s not a sport but a torture. If a control is healthy and developers’ hands are sewn directly to shoulders, we can and we must use patterns.

Get-UIAWindow -p mmc | `
Get-UIAComboBox | `
Invoke-UIAComboBoxExpand | `
Get-UIAList | `
Get-UIAListItem -Name Automatic | `
Invoke-UIAListItemSelectItem -ItemName Automatic | `
Invoke-UIAListItemClick;

Too long? Let’s shortenize:

Get-UIAWindow -p mmc | `
Get-UIAComboBox | `
Invoke-UIAComboBoxExpand | `
Get-UIAListItem -Name Automatic | `
Invoke-UIAListItemSelectItem -ItemName Automatic | `
Invoke-UIAListItemClick;

We managed to delete only the Get-UIAList cmdlet. Why?
1) All the Get- cmdlets can perform search from the point they’ve been given with. Given a window, a childwindow or a control, from any of that start, a Get- cmdlet (excluding Get-UIAWindow as it does not have the -InputObject parameter) can continue the search down the Automation tree.
2) All the Invoke- (and pattern-cmdlets Get- and Set- like Get-UIAComboBoxSelection. Recognize them in a way: Get/Set-UIA[ControlType][Pattern name or abbreviation]) cmdlets can’t perform any search. They should be fed with a pipeline from a Get- cmdlet or a cmdlet that passed thru (-PassThru’ed) the object of the AutomationElement type.

To shortenize even more, you may cut out the first call, Get-UIAWindow (as it’s always stored in the [UIAutomation.CurrentData]::CurrentWindow variable), but be very attentively doing so: if you/your test Get-s a window (the main or an active child), closes it and tries to work with subsequent calls of Get-UIAWindow, error definitely will appear.

Homework: practisize working with the combo box, try another values, get the selection (I suppose that you know what is the difference between the selected list item and the selected list item that is braced inside the dotted box), save selection to a file, change selection, read it back from the file you’ve created, set the saved selection to the combo box, and do it until a new post got published. 🙂

(MS) UIAutomation: how to be informed if a control disappears…


An interesting question has been raised yesterday: how UIAutomation help to deal with a control that is disappearing. As known, MS UIAutomation works with only controls that are visible. It’s either ideology and technical limitation.
How the button I mentioned can disappear? For example, this is a button serving a progress bar of a wizard with the label Stop on its face. The progress bar achieves 100% and the button hides.

Using such an awkward code is not a solution:

while (Get-UIAWindow -p processname | Get-UIAButton -n Stop)

Here comes the help again from MS UI Automation: StructureChangedEvent provides us with six types of EventID, among others is the one we needed – ChildRemoved.

Task: wait for a button to be out of sight.

Requirements: get a signal from the button that can triger the main code. Use StructureChangedEventHandler.

Solution: the following sample represents the use of StructureChangedEvent

ipmo .\UIAutomation.dll
[bool]$global:buttonVisible1 = $true;
[bool]$global:buttonVisible2 = $true;
[bool]$global:buttonAddedAgain = $false;
Get-UIAWindow -n ShowHideButton | Get-UIAButton -n Button | Register-UIAStructureChangedEvent -ChildRemoved -EventAction {"01" >> c:\report.txt; $global:buttonVisible1=$false; "1" >> c:\report.txt;} -Verbose
Get-UIAWindow -n ShowHideButton | Get-UIAButton -n Button | Register-UIAStructureChangedEvent -ChildRemoved -EventAction {"02" >> c:\report.txt; $global:buttonVisible2=$false; "2" >> c:\report.txt;} -Verbose
Get-UIAWindow -n ShowHideButton | Register-UIAStructureChangedEvent -ChildAdded -EventAction {$global:buttonAddedAgain=$true; "3" >> c:\report.txt;} -Verbose
#sleep -Seconds 5
#Write-Host "1" $global:buttonVisible1
#Write-Host "2" $global:buttonVisible2
#Write-Host "3" $global:buttonAddedAgain

while ($global:buttonVisible1 -eq $true -and $global:buttonVisible2 -eq $true){
	sleep -Seconds 1;
	Write-Host "1" $global:buttonVisible1
	Write-Host "2" $global:buttonVisible2
	Write-Host "waiting...";
}
Write-Host "completed";
Write-Host "1" $global:buttonVisible1
Write-Host "2" $global:buttonVisible2
Write-Host "3" $global:buttonAddedAgain

The sample works with a simple program that is a form with two buttons. One button is Show/Hide button, its click event shows/hides the second button.

Just run the program (samples 0.6.2 or newer), after that run the script (it subscribes to events) and click one or several times upon the Show/Hide button.
Soon or later you’ll see both global variables set to $false, and the third to $true.

Restrictions may apply, to read more about the event and which eventId can be used on a purpose, visit StructureChangedEventArgs Constructor

(MS) UIAutomation: administrative rights may be necessary on comtemporary operating systems


Problem: the framework behaves strangely.

Requirements: reclaim the stability.

Solution: run your powershell console or IDE under domain admin credentials or through Run As Administrator. When working with something like Administrative tools (services.msc or compmgmt.msc, or ADUC), or regedit, or your company’s administrative tool, there might be a waterfall of errors, sometimes amazing. The answer is simple: use administrative credentials while working with administrative applications through inter-process communication.

(MS) UIAutomation: performing a Win32 click on a VMware guest


Task: explain how to make your UI test script working in the VMware environment.

Requirements: provide scripters with recommendations.

Solution: sometimes you need to do a specific click. It might happen if a control does not support InvokePattern (thus, there’s no Invoke-UIA[ControlType]Cmdlet or there IS a cmdlet, but the control does not support the InvokePattern on its own). Usually it is not a problem: we have a Win32 click (Invoke-UIAControlClick) and cmdlets based on this click (Invoke-UIAControlContextMenu).

All of a sudden you may notice that surprisingly your click does not work. You are running the script on a VMware guest and the problem is that the cursor refuses positioning. In other words, the API function SetCursorPos does not work.

Theoretically, there might be used OpenInputDesktop, SetThreadDesktop and similar stuff. However, there is a much simplier way: just uninstall VMware Pointing device. With losing a bit of comfort, you’ll get your script working.

(MS) UIAutomation: using the script recorder


Task: start working with the script recorder.

Requirements: to be able to work with the recorder from the command line.

Solution: The script recorder is the cmdlet that was in the framework from almost its commencement. In fact, how to write the code? The scripter needs to have the following:
– a tool for recognizing AutomationID, ControlType and Name of controls that are under test
– a tool that can inform about supported patterns
– knowledge about cmdlets and parameters, some experience with the framework
– log problems

A tester needs significantly more:
– a way to get and store test results
– detailed log
– saving screenshots on any situation when it’s required
– custom parameters (the need to parametrize as much as possible to multiply the tests)

The script recorder helps in the needs of the scripter. The needs of a tester are resolved in the framework by different means. Issue the following command:

Start-UIARecorder -NoClassInformation -NoUI -NoScriptHeader -WriteCurrentPattern -Seconds 60

Now you can move the mouse cursor to the controls of your interest, performs actions like clicks and editing the text. The script recorder is far away of being finished, nevertheless its help may be very useful.
Remember that for now it records all it managed to record without separation from unnecessary windows and controls. Nevertheless, it might help at least in syntax.

%d bloggers like this: