Tag Archives: TestResult

Test Case Management: generating test results on the fly


The truth is that almost nobody writes test cases these days by hands. Okay, for big and serious, and requiring mind efforts things like scenarios, testers write. PowerShell frameworks are all about automation and demand as little manual work as possible. Like other GUI and Web testing tools, PowerShell frameworks generate test results by watching the code execution. Let’s go to the samples:

ipmo [path]\SePSX.dll
[SePSX.Preferences]::EveryCmdletAsTestResult = $true;
Start-SeFirefox | Enter-SeURL "http://google.com" | Get-SeWebElement -Name q | Set-SeWebElementKeys Cheese | Submit-SeWebElement;

What did we expect from this code? We set the EveryCmdletAsTestResult setting on, started an instance of Firefox, navigated to Google and submitted a query. As we saw no errors, we want to get our passed test results. How to do that?

[TMX.TestData]::CurrentTestScenario.TestResults | FL Name,Status

The output is as follows:
Name : Start-SeFirefox
Status : PASSED

Name : Enter-SeURL “http://google.com”
Status : PASSED

Name : Get-SeWebElement -Name q
Status : PASSED

Name : Set-SeWebElementKeys Cheese
Status : PASSED

Name : Submit-SeWebElement;
Status : PASSED

Name :
Status : NOT TESTED
Every cmdlet reported itself as Passed, and five results display this. The sixth result is a pre-generated test result that will be used in the immediately next cmdlet call.

This is too ideal, now we change our code to obtain real-life results. We are seeking for one of Google 2.0 controls, namely ‘q2’ (Marissa Mayer’s gone, hasn’t she?).

ipmo [path]\SePSX.dll
[SePSX.Preferences]::EveryCmdletAsTestResult = $true;
Start-SeFirefox | Enter-SeURL "http://google.com" | Get-SeWebElement -Name q2 | Set-SeWebElementKeys Cheese | Submit-SeWebElement;

The code failed (controls have not been renamed yet), where are our results?

[TMX.TestData]::CurrentTestScenario.TestResults | FL Name,Status

The output is below:
Name : Start-SeFirefox
Status : PASSED

Name : Enter-SeURL “http://google.com”
Status : PASSED

Name : Get-SeWebElement -Name q2
Status : FAILED

Name : Get-SeWebElement -Name q2
Status : FAILED

Name :
Status : NOT TESTED

The third and the fourth results have different exceptions in their Descriptions, though it’s a place where module’s code review needed…

Now, let’s see how time was consumed:

[TMX.TestData]::CurrentTestScenario.TestResults | FL Name,Status,TimeSpent

As can be seen, there is a default 500 milliseconds’ delay after starting the browser, 2 seconds were spent on navigation to the google page, and 5 seconds (the full time of [SePSX.Preferences]::Timeout) were spent on attempts to get the control of our interest.

In practice, testers are often interested in Failed results, much more often that in the list of Passed. 🙂 How to obtain such a list? First of all, we need to import the TMX module. Selenium and UIAutomation modules use TMX indirectly, as a library, but the advanced functionality is available as cmdlets:

ipmo [path]\TMX.dll;
Search-TMXTestResult -FilterFailed | FL Name,LineNumber,Code,Details

If the time of every test result is stored, what benefits do we have? Consider using the following query:

Search-TMXTestResult -OrderByTimeSpent -Descending -FilterFailed | FL Name,TimeSpent,Code

This lists Failed test results and time consumed in the descending mode (for what kind of user the contemporary versions of MS Excel have been written? Maybe, I’m wrong here? Okay, I’ll rewrite). This command lists Failed test results from bigger time spent to smaller.

Advertisements

Test Case Management: filtering and sorting the results


Let’s start from yesterday’s script. I added the code that never would work: getting a non-existing window and finding the “10” button on Calculator. Here is the script:

ipmo [path]\TMX.dll
ipmo [path]\UIAutomation.dll
Start-Process calc
Get-UIAWindow -n calc* | Get-UIAButton -n 1 | Invoke-UIAButtonClick;
Get-UIAWindow -n calc* | Get-UIAButton -n add | Invoke-UIAButtonClick;
Get-UIAWindow -n calc* | Get-UIAButton -n 1 | Invoke-UIAButtonClick;
Get-UIAWindow -n calc* | Get-UIAButton | Set-UIAFocus | Set-UIAControlKeys -Text "1{+}1{=}"

Get-UIAWindow -n "non-existing window" -Seconds 2
Get-UIAWindow -n calc* | Get-UIAButton -n 10 | Invoke-UIAButtonClick;

Stop-Process -Name calc

Search-TMXTestResult
Export-TMXTestResults -As HTML -Path c:\1\calc_test_results_generated.htm
c:\1\calc_test_results_generated.htm

The three lines copy-pasted from yesterday’s post do exactly what they did yesterday: generate thirteen test results. What do the lines I added? The first line tries to get the window that’s rarely seen in the real life (you may compile one, though). The next line searches for the 10 button (that Microsoft declined to add to the tool, with very oblique explanation).

Okay, what should we expect? The instruction ‘Get-UIAWindow -n “non-existing window” -Seconds 2′ fails after two seconds’ time, so does ‘Get-UIAWindow -n calc* | Get-UIAButton -n 10’.

What does it mean for the whole test? These test cases (i.e., commands) are considered as most time-consuming (several queries to the Automaiton tree until the timeout ends this) and, no doubts, failed.

All the test results that failed can be pulled out with the following code:

Search-TMXTestResult -FilterFailed

The worst test result, considering as the worst a test result with the greatest time spent, can be scooped up with the following piece of code;

Search-TMXTestResult -FilterFailed -OrderByTimeSpent -Descending

Test results that are generated automatically have names based on the code. This leads us to the opportunity to filter test results by a code snippet or a keyword:

Search-TMXTestResult -FilterNameContains Button

or

Search-TMXTestResult -FilterNameContains 10

All that described are going to be available today as part of 0.8.0. Preview 1.

Test Case Management: how to generate test cases from the code


Software engineers use test cases for serious testing. Test cases are written rules, written in words or in code, and intended to become test results. Test results reflect which test cases have passed and which have failed.

This means that test cases are theoretical, static part of testing, whereas test results are dynamic, practical part. Life cycle of a test case longs from several builds to several versions of the product the test case belongs to. Life cycle of test results is from run to run. Usually, a run is performed once per build. Sometimes, several runs (i.e., test results) are conducted per build.

Now, let’s take a look at how contemporary test tools publish results. Click’n’play tools generate results for every click. Is this valuable? I think so, because a test engineer often needs to investigate into ‘not clicked’ issue. To sum up, this is an industry-proven practice to write up test results automatically. In fact, nobody would write test cases like ‘click the Next button’ by hands.

Following the industry practices, we also added test results generation from the code. How it works? If the paramater [UIAutomation.Preferences]::EveryCmdletAsTestResult is set to $true, all the results brought to the pipeline will be also stored as test results.

Let’s take the following piece of code:

ipmo [path]\TMX.dll
ipmo [path]\UIAutomation.dll
[UIAutomation.Preferences]::EveryCmdletAsTestResult=$true
Start-Process calc
Get-UIAWindow -n calc* | Get-UIAButton -n 1 | Invoke-UIAButtonClick;
Get-UIAWindow -n calc* | Get-UIAButton -n add | Invoke-UIAButtonClick;
Get-UIAWindow -n calc* | Get-UIAButton -n 1 | Invoke-UIAButtonClick;
Get-UIAWindow -n calc* | Get-UIAButton | Set-UIAFocus | Set-UIAControlKeys -Text "1{+}1{=}"

Search-TMXTestResult
Export-TMXTestResults -As HTML -Path c:\1\calc_test_results_generated.htm
c:\1\calc_test_results_generated.htm

In the code above, we can see thirteen calls of UIA cmdlets. These thirteen calls generate thirteen test results. That’s all, no more manual work.

The Search-TMXTestResult cmdlets outputs all the test results to the pipeline, allowing you to process them in your own way. The Export-TMXTestResults cmdlet creates a simple HTML report (the projects seriously lacks of a HTML/CSS programmer).

When this way to obtain test result would be right:

  • you are playing with a small experimental script and you simply don’t want to spend time for creating test cases to a probe script
  • your test suite is huge, so huge that writing test cases manually makes no sense
  • you are not a professional tester, but test results are valuable for you

Daily automation: the great mathematical enigma


You may have heard that beyond our little world of PowerShell and software testing boundaries many other things exist. For example, the mathematical theorems that are waiting for its solution. One of them is ‘what is the answer to the expression 2 + 2 / 2?’. I’m not a mathematician, you are perhaps too, so that what’s the reason to argue would it be 2, 3 or even 42?

Well, we are going only to test a program to confirm the readiness to such mathematical things. Meet the candidate: calc.exe!

Task: demonstrate how to test an application.

Requirements: use UIAutomation and TMX modules to report the results.

Solution: below is the code we run against the application under test (AUT):

ipmo [path]\UIAutomation.dll;
ipmo [path]\TMX.dll;

# Let's create a new test suite
# to clean up the data that are possibly out of the current interest
[TMX.TestData]::ResetData();
[UIAutomation.Mode]::Profile = [UIAutomation.Modes]::Presentation;
New-TMXTestSuite -Name 'Calculator testing' -Id calc;
Add-TMXTestScenario -Name 'Calculator menu' -Id calcmenu;
Add-TMXTestScenario -Name 'Caclucator simple calculations' -Id maths;

# start the application under test (AUT)
Start-Process calc -PassThru | Get-UIAWindow;
# now we've got the window in the
# [UIAutomation.CurrentData]::CurrentWindow variable

# set the initial state: Standard
Open-TMXTestScenario -Id calcmenu;
# we are going to save our test result regarding how the menu works here

# the main menu here is expandable/collapsible
# what can be learnt by issuing the following line of code:
# (Get-UIAMenuItem -Name View).GetSupportedPatterns()
Get-UIAMenuItem -Name View | `
 Invoke-UIAMenuItemExpand | `
 Get-UIAMenuItem -Name Standard | `
 Invoke-UIAMenuItemClick -TestResultName 'The Standard menu item is available' `
 -TestPassed -TestLog;
# the TestLog parameters logs the last line of code
# the TestPassed parameter informs that we have gotten the result we expected
# the TestResultName parameter gives the name to our test result
# test result Id is generated as we don't need to set it manually, this is just result

function calculateExpression
{
 param(
 [string]$modeName
 )
 # produce the input
 Get-UIAButton -Name 2 | Invoke-UIAButtonClick;
 Get-UIAButton -Name Add | Invoke-UIAButtonClick -TestResultName "The Add button is clickable $($modeName)" -TestPassed;
 Get-UIAButton -Name 2 | Invoke-UIAButtonClick;
 Get-UIAButton -Name Divide | Invoke-UIAButtonClick;
 Get-UIAButton -Name 2 | Invoke-UIAButtonClick;
 Get-UIAButton -Name Equals | Invoke-UIAButtonClick -TestResultName "The Equals button is clickable $($modeName)" -TestPassed;

# now we need to get the result
 # there is a difficulty: the name of a Text element (i.e., Label in .NET) is its value
 # okay, let's find all of them and our module highlight each
 #Get-UIAControlDescendants -ControlType Text | `
 # %{Get-UIAText -AutomationId $_.Current.AutomationId; `
 # sleep -Seconds 2; `
 # Write-Host "this was " $_.Current.Name $_.Current.AutomationId;}
 # the last, with Id = 158, is ours
 [int]$result = (Get-UIAText -AutomationId 158).Current.Name;
 $result;
}

# we must determine where we are
# as testers, we never believe, we check
# you might, for example, investigate into the second level menu:
#Get-UIAMenuItem -Name View | `
# Invoke-UIAMenuItemExpand | `
# Get-UIAMenu | `
# Get-UIAControlChildren | `
# %{Write-Host $_.Current.ControlType.ProgrammaticName $_.Current.Name $_.Current.AutomationId;}

# the easy way is to simple calculate the buttons
# MS UI Automation never returns what it can't see (by design, as it is intended for screen readers)
# the Standard mode - 31
# the Statictics mode - 33
# the Programmer mode - 48
# the Scientific mode - 53
[string]$testResultName = 'The Standard mode is available';
# to ensure that after changing the mode
# we are working with the actual window
Get-UIAWindow -ProcessName calc;
if ((Get-UIAControlDescendants -ControlType Button).Count -eq 31) {
 Close-TMXTestResult -Name $testResultName -TestPassed;
} else {
 Close-TMXTestResult -Name $testResultName -TestPassed:$false;
 # do additional actions if needed
}

# now we are going to test calculations so that
# we are switching to appropriate scenario
Open-TMXTestScenario -Id maths;
[string]$modeName = 'in the Standard mode';

calculateExpression $modeName; # in the Standard mode

# test the same in the Scientific mode
Open-TMXTestScenario -Id calcmenu;
Get-UIAMenuItem -Name View | `
 Invoke-UIAMenuItemExpand | `
 Get-UIAMenuItem -Name Scientific | `
 Invoke-UIAMenuItemClick -TestResultName 'The Scientific menu item is available' `
 -TestPassed -TestLog;

# check the mode we are in now
$testResultName = 'The Scientific mode is available';
# to ensure that after changing the mode
# we are working with the actual window
Get-UIAWindow -ProcessName calc;
if ((Get-UIAControlDescendants -ControlType Button).Count -eq 53) {
 Close-TMXTestResult -Name $testResultName -TestPassed;
} else {
 Close-TMXTestResult -Name $testResultName -TestPassed:$false;
 # do additional actions if needed
}

Open-TMXTestScenario -Id maths;
[string]$modeName = 'in the Scientific mode';
calculateExpression $modeName; # in the Scientific mode

# test reports are still ugly!!!
Export-TMXTestResults -As HTML -Path C:\1\calc_result.htm

After all, what is the answer? Okay, to choose the answer is 2 or 3 is not our mission. We only tested it, without the ambition to solve the mathematical problem…

%d bloggers like this: