Category Archives: TMX

Sample scripts online


Have I not mentioned yet that all sample scripts are constantly available online? Scripts that are honored to be public are always put in the folder that is checked in to the source control. Here is the root of the samples folder: https://github.com/apetrovskiy/STUPS/tree/master/samples

SamplesFolder

For instance, UIAutomation samples are here: https://github.com/apetrovskiy/STUPS/tree/master/samples/UIAutomation, TMX samples are here https://github.com/apetrovskiy/STUPS/tree/master/samples/TMX and so on.

Unfortunately, some scripts may become incompatible with the contemporary versions of binaries. Sometimes I refresh them, alternatively, you could initiate a pull request with a renewed version of a script.

 

Turned a year!


Just remembered that the PowerShell testing suite turned a year. As it believed, approx. on November 28th, 2011, the result of a programming mistake, of course (a piece of code didn’t work in PowerShell, just a usage mistake), the decision to write up C#-cmdlets has been made. 🙂

Since then, our frameworks have changed significantly. It was only UIAutomation which was the root of the suite. Now, there are several frameworks published and several versions upcoming.

Shortly, I’d drop a word about the Software Testing Using PowerShell team plans. From the features that are almost ready to those that are only in mind. What will the upcoming winter bring to us?

UIAutomation:

  • on fail, the Invoke-UIA[ControlType]Click cmdlets will try Win32 click by default. This behavior (the user wants it click and it clicks) is most awaiting (thanks to JohnQuest for the idea).
  • the Set-UIA[ControlType]Text cmdlets will check the result and try to use SendKeys on a failure
  • the Set-UIAControlKeys cmdlet will try several times to put text into the input field, checking the result
  • UIAutomationSpy are going to produce more useful code. Moreover, the code produced is going to be faster. How? When possible, UIAutomationSpy will set the -Win32 parameter, what, in certain situations, accelerates tests significantly (working with controls near grids and listviews is slow; this can be healed by using Win32 handles).
  • UIAutomationSpy will finally check code that it generates (sometimes what can be got by hovering over is not the same that can be got by searching in the Automation tree).
  • at least ten pages of documentation as what is published is like a mad mix
  • setup for the module
  • Java Access Brigde (that has not been tried yet)
  • we will try to use UIA2 if it’s possible to combine two UI Automation technologies

SePSX:

  • support for ChromeOptions, InternetExplorerOptions and FirefoxProfile
  • fast-working ‘named’ cmdlets Get-Se[TagName]
  • cmdlets for working with tables
  • SeleniumSpy
  • in a perspective, support for other browsers

TMX:

  • TestLink add-in (missing functionality)
  • TFS add-in
  • working with test cases in SQLite, TestLink, various files

AWSPSX:

  • EC2 and S3 cmdlets to allow automated test lab deployment and execution

ESXiMgmt:

  • a small set of cmdlets allowing us to start/stop/suspend/snapshot guests

Internals:

  • these days we are working on harnessing a Dependency Injection framework. By now, SePSX is partially sitting on Autofac 2.6 and we are also experimenting with NInject 3 in other projects. TLAddin has benefited from using Moq. The reasons we are working on not a business-understood features are such:
  1. the number of tests is growing and the testing cycle should be running every several minutes. Hundreds of tests we already have are for daily framework verification (they take twenty minutes or more, what is critical for us). On the opposite, a hundred of unit tests we have already had take less than five seconds (and around seventeen seconds does Gallio take to get the consciousness at the start of testing)
  2. we need more through-out testing of our code

Projects’ roadmap’s gotten some dates


The recently published roadmap now contains schedule for upcoming functionality. As time continues, more features’ll get exact dates where only month is written now.

Test Case Management: generating test results on the fly


The truth is that almost nobody writes test cases these days by hands. Okay, for big and serious, and requiring mind efforts things like scenarios, testers write. PowerShell frameworks are all about automation and demand as little manual work as possible. Like other GUI and Web testing tools, PowerShell frameworks generate test results by watching the code execution. Let’s go to the samples:

ipmo [path]\SePSX.dll
[SePSX.Preferences]::EveryCmdletAsTestResult = $true;
Start-SeFirefox | Enter-SeURL "http://google.com" | Get-SeWebElement -Name q | Set-SeWebElementKeys Cheese | Submit-SeWebElement;

What did we expect from this code? We set the EveryCmdletAsTestResult setting on, started an instance of Firefox, navigated to Google and submitted a query. As we saw no errors, we want to get our passed test results. How to do that?

[TMX.TestData]::CurrentTestScenario.TestResults | FL Name,Status

The output is as follows:
Name : Start-SeFirefox
Status : PASSED

Name : Enter-SeURL “http://google.com”
Status : PASSED

Name : Get-SeWebElement -Name q
Status : PASSED

Name : Set-SeWebElementKeys Cheese
Status : PASSED

Name : Submit-SeWebElement;
Status : PASSED

Name :
Status : NOT TESTED
Every cmdlet reported itself as Passed, and five results display this. The sixth result is a pre-generated test result that will be used in the immediately next cmdlet call.

This is too ideal, now we change our code to obtain real-life results. We are seeking for one of Google 2.0 controls, namely ‘q2’ (Marissa Mayer’s gone, hasn’t she?).

ipmo [path]\SePSX.dll
[SePSX.Preferences]::EveryCmdletAsTestResult = $true;
Start-SeFirefox | Enter-SeURL "http://google.com" | Get-SeWebElement -Name q2 | Set-SeWebElementKeys Cheese | Submit-SeWebElement;

The code failed (controls have not been renamed yet), where are our results?

[TMX.TestData]::CurrentTestScenario.TestResults | FL Name,Status

The output is below:
Name : Start-SeFirefox
Status : PASSED

Name : Enter-SeURL “http://google.com”
Status : PASSED

Name : Get-SeWebElement -Name q2
Status : FAILED

Name : Get-SeWebElement -Name q2
Status : FAILED

Name :
Status : NOT TESTED

The third and the fourth results have different exceptions in their Descriptions, though it’s a place where module’s code review needed…

Now, let’s see how time was consumed:

[TMX.TestData]::CurrentTestScenario.TestResults | FL Name,Status,TimeSpent

As can be seen, there is a default 500 milliseconds’ delay after starting the browser, 2 seconds were spent on navigation to the google page, and 5 seconds (the full time of [SePSX.Preferences]::Timeout) were spent on attempts to get the control of our interest.

In practice, testers are often interested in Failed results, much more often that in the list of Passed. 🙂 How to obtain such a list? First of all, we need to import the TMX module. Selenium and UIAutomation modules use TMX indirectly, as a library, but the advanced functionality is available as cmdlets:

ipmo [path]\TMX.dll;
Search-TMXTestResult -FilterFailed | FL Name,LineNumber,Code,Details

If the time of every test result is stored, what benefits do we have? Consider using the following query:

Search-TMXTestResult -OrderByTimeSpent -Descending -FilterFailed | FL Name,TimeSpent,Code

This lists Failed test results and time consumed in the descending mode (for what kind of user the contemporary versions of MS Excel have been written? Maybe, I’m wrong here? Okay, I’ll rewrite). This command lists Failed test results from bigger time spent to smaller.

Metro automation: navigating to a certain page


A typical Metro UI application is a set of pages. One is main, and there are several pages where we can go from the main page.

A Metro UI app is unlikely to a traditional app, which often starts from some starting point. A Metro UI app starts for the first time with its main page, but further is may start from wherever page the user left the app.

In our sample, we’ll be using the *Wizard* cmdlets. What are they?

1. the New-UIAWizard cmdlet creates a wizard object, provides it with name and contains a code to start the AUT (application under test), for example. Some initial code.

2. the Add-UIAWizardStep cmdlet creates a step, provides the step with a name, takes a wizard object as an input.

First action we should do on a step is to examine the step. The -SearchCriteria parameter does exactly this task.

After that, we run code for this step. There are two actions we can use, for forward movement and for backward. In out Metro UI app we will be using only forward steps for simplicity sake.

3. the Invoke-UIAWizard cmdlets runs the wizard (in fact, the code we added to the New-UIAWizard cmdlet). It’s usually used for starting the AUT. For purposes of Metro UI testing, we will use this action to show the Start screen and to open the app window.

4. the Step-UIAWizard cmdlet checks the -SearchCriteria (i.e., checks that controls are here and we are on the right page) and runs the step code.


Set-StrictMode -Version Latest

[UIAutomation.Preferences]::Timeout = 10000;
[UIAutomation.Preferences]::EveryCmdletAsTestResult = $true;
[UIAutomation.Preferences]::OnSuccessDelay = 300;

# names of the pages and buttons in the app menu
[string]$constPageMain = 'Home';
[string]$constPageFeaturedDestinations = 'Featured Destinations';
[string]$constPageDestinations = 'Destinations';
[string]$constPageFlights = 'Flights';
[string]$constPageHotels = 'Hotels';
[string]$constPageBestOfWeb = 'Best Of Web';

# names of wizard steps
[string]$constWizardName = 'TravelWizard';
[string]$constWizardStepHome = 'Home';
[string]$constWizardStepDestinations = 'Destinations';
[string]$constWizardStepFlights = 'Flights';
[string]$constWizardStepHotels = 'Hotels';
[string]$constWizardStepBestOfWeb = 'BestOfWeb';

# we need to clear the collection of wizards or delete
# the wizard we are playing with.
# Otherwise, the next run will throw an exception
# 'Wizard already exists'
[UIAutomation.WizardCollection]::Wizards.Clear();

# Creating a new wizard
New-UIAWizard -Name $constWizardName `
 -StartAction {
 # setting the Metro Start screen
 Get-UIADesktop;
 Show-UIAMetroStartScreen;

# clicking on the tile of the AUT
 Get-UIAListItem -Name 'Travel' | `
 Invoke-UIAListItemClick;

# the loading screen (if happened)
 try {
 Get-UIAWindow -Name 'Travel' | `
 Get-UIAProgressBar;
 #sleep -Seconds 5; # the app is loading
 }
 catch {}

 # start our tests from the Main page
 Show-UIAMetroMenu;
 Get-UIAWindow -Name 'Travel' | `
 Get-UIAMenuBar -Name 'App Bar' | `
 Get-UIAHyperlink -Name $constPageMain | `
 Invoke-UIAControlClick;
 } | `
 Add-UIAWizardStep -Name $constWizardStepHome `
 -SearchCriteria @{ControlType="Text";Name='Bing Travel'} `
 -StepForwardAction {
 #[System.Windows.Forms.MessageBox]::Show("StepHome");
 "<<<<<<<<<< On the Home page >>>>>>>>>>";
 try {
 Get-UIAText -Name 'Bing Travel';
 Close-TMXTestResult -Name "<<<<<<<<<< Navigating to the Home page >>>>>>>>>>" -TestPassed;
 }
 catch {
 Close-TMXTestResult -Name "<<<<<<<<<< Navigating to the Home page >>>>>>>>>>";
 }
 } -PassThru | `
 Add-UIAWizardStep -Name $constWizardStepDestinations `
 -SearchCriteria @{ControlType="Text";Name='Destinations'},@{ControlType="Text";Name='Region'} `
 -StepForwardAction {
 #[System.Windows.Forms.MessageBox]::Show("StepDestinations");
 "<<<<<<<<<< On the Destinations page >>>>>>>>>>";
 try {
 Get-UIAText -Name 'Destinations';
 Get-UIAText -Name 'Region';
 Close-TMXTestResult -Name "<<<<<<<<<< Navigating to the Destinations page >>>>>>>>>>" -TestPassed;
 }
 catch {
 Close-TMXTestResult -Name "<<<<<<<<<< Navigating to the Destinations page >>>>>>>>>>";
 }
 } -PassThru | `
 Add-UIAWizardStep -Name $constWizardStepFlights `
 -SearchCriteria @{ControlType="Text";Name='Flights'},@{ControlType="Text";Name='Schedule'} `
 -StepForwardAction {
 #[System.Windows.Forms.MessageBox]::Show("StepFlights");
 "<<<<<<<<<< On the Flights page >>>>>>>>>>";
 try {
 Get-UIAText -Name 'Flights';
 Get-UIAText -Name 'Schedule';
 Close-TMXTestResult -Name "<<<<<<<<<< Navigating to the Flights page >>>>>>>>>>" -TestPassed;
 }
 catch {
 Close-TMXTestResult -Name "<<<<<<<<<< Navigating to the Flights page >>>>>>>>>>";
 }
 } -PassThru | `
 Add-UIAWizardStep -Name $constWizardStepHotels `
 -SearchCriteria @{ControlType="Text";Name='Hotels'},@{ControlType="Text";Name='Check-in'} `
 -StepForwardAction {
 #[System.Windows.Forms.MessageBox]::Show("StepHotels");
 "<<<<<<<<<< On the Hotels page >>>>>>>>>>";
 try {
 Get-UIAText -Name 'Hotels';
 Get-UIAText -Name 'Check-in';
 Close-TMXTestResult -Name "<<<<<<<<<< Navigating to the Hotels page >>>>>>>>>>" -TestPassed;
 }
 catch {
 Close-TMXTestResult -Name "<<<<<<<<<< Navigating to the Hotels page >>>>>>>>>>";
 }
 } -PassThru | `
 Add-UIAWizardStep -Name $constWizardStepBestOfWeb `
 -SearchCriteria @{ControlType="Text";Name='Travel'},@{ControlType="Text";Name='Explore'},@{ControlType="Text";Name='Plan a Trip'} `
 -StepForwardAction {
 #[System.Windows.Forms.MessageBox]::Show("Best Of Web");
 "<<<<<<<<<< On the 'Best Of Web' page >>>>>>>>>>";
 try {
 Get-UIAText -Name 'Travel';
 Get-UIAText -Name 'Explore';
 Get-UIAText -Name 'Plan a Trip';
 Close-TMXTestResult -Name "<<<<<<<<<< Navigating to the 'Best Of Web' page >>>>>>>>>>" -TestPassed;
 }
 catch {
 Close-TMXTestResult -Name "<<<<<<<<<< Navigating to the 'Best Of Web' page >>>>>>>>>>";
 }
 };

# Start the wizard
[UIAutomation.Wizard]$wizard = Invoke-UIAWizard -Name $constWizardName;

# Click on the Destinations button in the app menu
Show-UIAMetroMenu;
Get-UIAWindow -Name 'Travel' | `
 Get-UIAMenuBar -Name 'App Bar' | `
 Get-UIAHyperlink -Name $constPageDestinations | `
 Invoke-UIAControlClick;

# Check that this is the Destinations page and run code (in the future sample)
$wizard | Step-UIAWizard -Name $constWizardStepDestinations

# Click on the Flights button in the app menu
Show-UIAMetroMenu;
Get-UIAWindow -Name 'Travel' | `
 Get-UIAMenuBar -Name 'App Bar' | `
 Get-UIAHyperlink -Name $constPageFlights | `
 Invoke-UIAControlClick;

# Check that this is the Flights page and run code (in the future sample)
$wizard | Step-UIAWizard -Name $constWizardStepFlights;

# Click on the Hotels button in the app menu
Show-UIAMetroMenu;
Get-UIAWindow -Name 'Travel' | `
 Get-UIAMenuBar -Name 'App Bar' | `
 Get-UIAHyperlink -Name $constPageHotels | `
 Invoke-UIAControlClick;

# Check that this is the Hotels page and run code (in the future sample)
$wizard | Step-UIAWizard -Name $constWizardStepHotels;

# Click on the 'Best Of Web' button in the app menu
Show-UIAMetroMenu;
Get-UIAWindow -Name 'Travel' | `
 Get-UIAMenuBar -Name 'App Bar' | `
 Get-UIAHyperlink -Name $constPageBestOfWeb | `
 Invoke-UIAControlClick;

# Check that this is the 'Best Of Web' page and run code (in the future sample)
$wizard | Step-UIAWizard -Name $constWizardStepBestOfWeb;

The dubious piece of code is getting a progress bar on loading the application. Sometimes, it is shown. Sometimes, there is no progress bar.

0.8.0 Preview 2: wanted richer output?


Again and again, it’s not going to end. People ask me ‘why Metro UI testing doesn’t work from PowerShell ISE?’

PowerShell ISE and powershell.exe are Microsoft’s tools, they have no relation to Metro UI testing. Technically, Microsoft guys can build ISE and powershell.exe to make them uiAccess’ible. However, don’t wait this: why should they build a host for my cmdlets? To help compete with their Coded UI? I sooner believe that they will write their own UI cmdlets in v3 or v4 (as they usually answer to PowerGUI features, for example), than they do something specific for 3rd party cmdlets.

The other side of coin is that I’m not going to write one more code editor with debugging capabilities. I’d like to, but this is a time-consuming task. Not to say that an editor of appropriate quality is not so easy to write up.

That’s a dilemma: we can run PowerShell over Metro UI, but we can’t work deeply with variables and output.

0.8.0 Preview 2 solves this partly. Now, output is partially supported. The list of what is supported:

  • string output (code: “string”)
  • string output for objects (code: (Get-UIAWindow 0n calc*).Current | Out-String)

Write- cmdlets are not supported in this version. Possibly, they will be supported later.

Here is a sample:

[UIAutomation.Preferences]::EveryCmdletAsTestResult=$true
[UIAutomation.Preferences]::OnSuccessDelay = 0;

Start-Process calc
Get-UIAWindow -n calc* | Get-UIAButton -n 1 | Invoke-UIAButtonClick;
(Get-UIAButton -n add | Invoke-UIAButtonClick).Current.AutomationId;
(Get-UIAButton -n 1 | Invoke-UIAButtonClick).Current | Out-String;
Get-UIAWindow -n calc* | Get-UIAButton | Set-UIAFocus | Set-UIAControlKeys -Text "1{+}1{=}"
'Get-UIAWindow -n calc* | Out-String; ->'
Get-UIAWindow -n calc* | Out-String;
'(Get-UIAWindow -n calc*).Current | Out-String;'
(Get-UIAWindow -n calc*).Current | Out-String;

"now test should fail"
Get-UIAWindow -n "non-existing window" -Seconds 2
Get-UIAWindow -n calc* | Get-UIAButton -n 10 | Invoke-UIAButtonClick;

Output is shown as light-blue lines. It returns:

  • the button 1 (Invoke-UIAButtonClick returns)
  • the automation id of the button Add (again, the same cmdlet returns)
  • AutomationElementInfo (i.e., .Current). This is the button 1 again.
  • further, after an empty string, there are string lines to help you compare what means to put an AutomationElement and an AutomationElementInfo to the Out-String cmdlet.

It’s not proper way to debug your scripts, though it’s the only way for now.

0.8.0 Preview 2: double squares help to visualize controls better


Controls belong to a big hierarchy. What may be the reason to highlight only two levels of controls if, in practice, there might be ten or twenty levels in the hierarchy?

The pictures show what the double-highlighting is. The inner square is over the control of interest, whereas the circumscribing one is over its parent.

However, things suddenly change when you put the cursor over WPF applications, its (WPF’s) derivatives and HTML:

This is a rather typical case: we are seeing a button or a link and, at the same time, the topmost control is a text.

The same story in Metro UI applications:

Here it is seen even better:

and here too:

The name of the button is not ‘Permissions’ as we could think, the name is ‘Back’.

At the end, there is a brilliant example:

There are TWO things on top of the button! This is the out-of-the-box Mail application.

0.8.0 Preview 1: UIARunner, UIAutomationSpy, and modules are published


Even though with some bugs and ugly-looking (it never was our intent to write GUI tools besides modules:)), UIARunner is published. As far as I know, this is the first GUI wrapper for PowerShell that looks like a software testing tool. 🙂

This preliminary version supports:

  • running one script at a time
  • using the UIARunner.ps1 file (in the application’s folder) as a configuration file
  • generating test report on the fly (in the GUI)
  • generating test report on the fly as a CSV file (both GUI and command-line versions produce such output)
  • unattended runs
  • the GUI version also shows how many test results are passed, failed and the velocity in newly introduced measures called ‘test results per second’ (trps)
  • of course, it supports basic sorting in the grid.

Known issues are numerous (again, this project lacks of professional GUI programmers :)):

  • bad multithreading
  • poor informing about what happened at the PowerShell level
  • and many, many lesser bugs.

Anyway, this version is worth experimenting with. Each package includes several simple scripts to try the runner. Scripts work similarly to other PowerShell hosts, note, however, that you need to use try/catch blocks to avoid terminating errors.

0.8.0 Preview 1: UIARunner as a graphical test tool


Today’s Preview 1 brings to us a new test runner: UIARunner for testing Metro UI and Windows UI.

This version, even though not a release, already can run a script and display test results. The current version supports only test results, i.e., events generated by the Close-TMXTestResult cmdlet or closed automatically through the parameter [UIAutomation.Preferences]::EveryCmdletAsTestResult set to $true.

The figure below shows how it works:

 

The picture was taken when UIARunner was working on a performance test: it should calculate the results of expressions 1 + 1 and 2 + 2 a thousand times, and thousand times try to find the 10 button (it’s a flaw that Microsoft didn’t manage to find more space to put this useful button on the form).

To obtain such a visual report, you need no more than start the application with default settings, open a script through the File -> Open menu item (or by pressing Ctrl+O) and run the script by clicking on the menu item Script -> Run or by pressing F5.

The application is not smooth and cool for its first version. Nonetheless, it already can be used for gathering script results.

Speaking about future improvements, the features that are planned to implement are:

  •  supporting hierarchical reports based on test suites, test scenarios and test results and shown in a grouped list view
  • probably, pausing the script
  • maybe, some events or alerts on conditions like ‘too many failures’ or ‘global timeout interrupted the script’.

 

Test Case Management: filtering and sorting the results


Let’s start from yesterday’s script. I added the code that never would work: getting a non-existing window and finding the “10” button on Calculator. Here is the script:

ipmo [path]\TMX.dll
ipmo [path]\UIAutomation.dll
Start-Process calc
Get-UIAWindow -n calc* | Get-UIAButton -n 1 | Invoke-UIAButtonClick;
Get-UIAWindow -n calc* | Get-UIAButton -n add | Invoke-UIAButtonClick;
Get-UIAWindow -n calc* | Get-UIAButton -n 1 | Invoke-UIAButtonClick;
Get-UIAWindow -n calc* | Get-UIAButton | Set-UIAFocus | Set-UIAControlKeys -Text "1{+}1{=}"

Get-UIAWindow -n "non-existing window" -Seconds 2
Get-UIAWindow -n calc* | Get-UIAButton -n 10 | Invoke-UIAButtonClick;

Stop-Process -Name calc

Search-TMXTestResult
Export-TMXTestResults -As HTML -Path c:\1\calc_test_results_generated.htm
c:\1\calc_test_results_generated.htm

The three lines copy-pasted from yesterday’s post do exactly what they did yesterday: generate thirteen test results. What do the lines I added? The first line tries to get the window that’s rarely seen in the real life (you may compile one, though). The next line searches for the 10 button (that Microsoft declined to add to the tool, with very oblique explanation).

Okay, what should we expect? The instruction ‘Get-UIAWindow -n “non-existing window” -Seconds 2′ fails after two seconds’ time, so does ‘Get-UIAWindow -n calc* | Get-UIAButton -n 10’.

What does it mean for the whole test? These test cases (i.e., commands) are considered as most time-consuming (several queries to the Automaiton tree until the timeout ends this) and, no doubts, failed.

All the test results that failed can be pulled out with the following code:

Search-TMXTestResult -FilterFailed

The worst test result, considering as the worst a test result with the greatest time spent, can be scooped up with the following piece of code;

Search-TMXTestResult -FilterFailed -OrderByTimeSpent -Descending

Test results that are generated automatically have names based on the code. This leads us to the opportunity to filter test results by a code snippet or a keyword:

Search-TMXTestResult -FilterNameContains Button

or

Search-TMXTestResult -FilterNameContains 10

All that described are going to be available today as part of 0.8.0. Preview 1.

%d bloggers like this: