Automation Testing - 5 Fears and How to Overcome Them

This series addresses the fears faced by organisations developing on the Microsoft Power Platform who want to harness automation testing, and details how to overcome them.

Nov 22nd 2019·9 min read

Most of us have heard of automation testing on the Power Platform by now. But it doesn't seem like everyone's actually been able to put the tools into practice. This series goes beyond the introductory how-to guides (valuable as they are) to consider obstacles preventing your organisation from upgrading to automation and, crucially, suggests measures to overcome them.

Who should read this?

Perhaps you’ve read my previous or Wael Hamze’s EasyRepro get started guides, or watched Mohamed Radwan’s UI test introduction video. You know the tools are out there, maybe you have the skills to write the tests yourself, but for one reason or another you still don’t find your team automating tests for Dynamics 365, Portals and PowerApps.

Perhaps you’ve overcome one or more of the fears laid out here, in which case you can nod along knowingly and join the discussion; sharing you own experience. Let's start from the top.

1. Selenium is too complex

To begin with the basics, a common concern is that automating UI actions with Selenium is too complex. Selenium works through its “WebDriver”; allowing programmers to interface with a web browser by automating keyboard input and point and click mouse actions to selected HTML elements. Writing tests with Selenium requires the programmer to find and maintain these HTML element selectors, such as:

  • Dynamics 365 command buttons
  • Office 365 login prompts
  • lookup field search icons
  • ... and more

The list is extensive, and moreover the attribute used to select these HTML elements has been known to change between product versions (hence find and maintain).

In 2016 the EasyRepro project landed on GitHub; an open-source project from Microsoft that facilitates automation testing for Dynamics with a structured API that covers core commands. Here’s an example of a lead being qualified while I’m away from my keyboard making a cup of tea:

EasyRepro has been embraced by the open-source community: the project is updated regularly and there’s a support network to resolve issues, should you encounter any. There are also several walkthroughs on how to get started, such as this get started guide from Wael Hamze.

So we’ve addressed the concern of Selenium’s complexity with EasyRepro, but we’re still left having to write C# code to automate our UI tests. If this is also a concern, then read on.

2. Our testing team isn't proficient in C#

Our testing team is fantastic at testing; systematic, professionally pedantic and efficient at making things fall over. They’re familiar with the nuances of our product offerings and pick holes in the finer details of customisations, leading to higher quality software being delivered to our customers. But they aren’t proficient in C#.

Conversely our C# developers aren’t efficient at testing, and often wrote the code that needs to be tested. The question is then:

How do we merge skillsets in a way that is productive and feasible?

How do we approach both teams working closely together: testers providing the human eye and developers applying their C# experience to automate UI tests?

Behaviour-driven development (BDD) addresses this. It's a development approach that encourages collaboration between tech and non-tech stakeholders. It achieves this collaboration by writing tests before the coding begins, in a language that is user-focused and easy to understand. BDD is nothing new, but it is somewhat new for Dynamics 365 development.

How does BDD work with Dynamics 365?

In BDD, tests begin their journey as a scenario specifying some behaviour of the system. A scenario in Dynamics 365 might look like:

Feature: Lead qualification

	In order to qualify a lead as representing a potential deal
	As a sales executive
	I want to be able to qualify a lead to an opportunity

Scenario: Qualify a lead

	Given I login to CRM
	When I open a lead record
	And I qualify the lead
	Then an opportunity record should be created

How is a test scenario automated?

Using a tool like SpecFlow, we can bind each line of our scenario to a block of C# code written against the EasyRepro API. So “Given I login to CRM” would be handled in code by opening a web browser, navigating to an application’s URL and logging in.

public sealed class Login : XrmBinding
    [Given(@"I am logged in to CRM as (.*)")]
    public void GivenIAmLoggedIntoCrmAs(string userAlias)
        XrmApp.OnlineLogin.Login(TestSettings.CrmUrl,TestSettings.Username, TestSettings.Password);

Part 2 of this series takes a deeper look at BDD, going step-by-step through a working scenario and supporting source code. If you think BDD has potential to add value to your projects, then part 2 will ensure you have everything you need to get started.

3. UI tests take too long to run

If you’ve ever run a UI test, you’ll know that they can take some time to fully execute. A suite of test scenarios executing on your machine can sink hours of your time and hog system resources while you twiddle your thumbs patiently.

Fortunately, an update to EasyRepro in early 2018 added headless Chrome compatibility, meaning we can run our automatic UI tests from a Chrome browser on another machine, namely an Azure pipeline agent. We can configure the agent to run the tests at different times too, such as when a code pull request is created, or scheduled to run nightly. Running UI tests nightly has the benefits of:

  • offsetting the burden of UI tests taking a while to run by executing them on cloud resources outside of peak times
  • providing an email summary of failed tests each morning, specifying areas of regression introduced to the system by code changes made the day before

If you’re interested in configuring your UI tests to run automatically from Azure, my previous post covers a step-by-step to pipeline agent configuration.

vsts agent

4. UI tests lack traceability

A common complaint levelled against UI tests is their lack of traceability. If a suite of UI tests is run overnight, what feedback do we receive from our virtual machine’s console? Is the information provided sufficient to pass to the development team for fixing?

When a human tests a system’s behaviour they can:

  • note the exact area of the system at fault and provide steps to reproduce
  • associate the defect with a test case or a user story
  • screenshot or record the error message

We should expect nothing less from automation if we’re going to consider it feasible. The bad news is, by running EasyRepro out-of-the-box, your error logs won’t provide as much detail in your defect reports as a human can. The good news is there are tools available that address this, allowing you to configure EasyRepro to:

  • produce a test run report using ExtentReports
  • capture a video from a test run
  • capture a screenshot when a test fails

At the risk of regurgitating an already well written how-to guide, I'll defer the specifics to Jason Lattimer's reporting on EasyRepro test runs post from June. Jason's post details how to build the above three points in your automation tests and provides supporting source code.

Fortunately the effort for configuring these tools in your tests is a one-off "fire and forget" rather than having to configure them manually for each test. But, it's still more C# code that needs writing and maintaining. The next point looks to address this and offers a new alternative.

5. It's not feasible to maintain another C# repository

It't not often that a Microsoft software house is short of C# .NET developers. But what if the C# developers are flat out on other work ? Maybe we have a team of front-end developers who could get involved in writing automation tests if only there was a tool they could use written in their preferred language of JavaScript or TypeScript.

Well there are alternatives to the C# world of EasyRepro!

One emerging project is Florian Krönert's D365-UI-Test which is described as:

An unopinionated UI testing library for Dynamics 365 CE and Dynamics Portals. Powered by TypeScript and Puppeteer.

Whereas EasyRepro simulates user input by interacting with HTML DOM elements, D365-UI-Test interacts with the Dynamics 365 and Portals interface (yes it works with Portals) via the Xrm namespace. This means:

  • developers with a knowledge of the widely used Xrm namespace can contribute to the project and add their own bindings
  • the project requires less maintenance than EasyRepro, as the Xrm namespace will change less than Dynamics 365's DOM element selectors

How does it work?

First the package is installed in your JavaScript or TypeScript test project using:

npm install d365-ui-test

Then install your preferred testing framework if one isn't already installed such as Jest.

Now you can write your tests against the bindings available so far.

Here's an example that:

  • logs into a Dynamics 365 environment
  • navigates to a specified model-driven app
  • creates a new account record
  • asserts that the name of the account record was changed successfully
test("It should set account name and save", async () => {
  await xrmTest.Navigation.openCreateForm("account");

  await xrmTest.Attribute.setValue("name", "Test name");

  const value = await xrmTest.Attribute.getValue("name");
  expect(value).toBe("Test name");

  await xrmTest.Form.reset();

Running the test command npm run test specified in package.json opens a browser window and automatically runs the test:

d365 ui test

This project is in its infancy, but as you've seen it can already execute the basics of logging in, changing app and setting string fields. It can also:

  • set other basic field data types
  • read values and assert against them
  • assert visibility and read only states

So, you probably already knew Florian Krönert as the author of Xrm-WebApi-Client. If writing automation tests in TypeScript for Dynamics 365 and Portals sounds interesting, consider checking out his new D365-UI-Test project and even contributing!


Automation testing is the future of ensuring solutions built on Microsoft's Power Platform are built to specification and free from ongoing regression defects.

The tools available to teams building these solutions have become well known; EasyRepro is widely reported on by the community and has been further showcased by this article.

However, even armed with these tools, teams face challenges actually putting these tools to use, such as cultural misunderstanding of the approach required to make automation successful, or failure to sell the benefits of automation testing to senior project stakeholders.

This post has addressed the five main fears that I've seen facing teams of developers as they look to automate their testing pipeline for customisations on the Power Platform.

What do you think? Can you name five more?

Dave Clark IO