Acceptance and UI Testing with SpecFlow and Selenium

Basically, tests can be divided into three major groups:

• unit tests
• integration tests
• acceptance tests

Unit tests are those tests which verify the behavior of a unit under test in isolation. That’s why when a unit under test has dependencies, we often mock them or in other words replace by special test doubles. Unit tests are great, and the majority of tests should belong to this type of tests. This is because this type of tests is hard to break and easy to run (of course, if they are written in a good style).

On the contrary, integration tests verify the behavior of either a part of a system or a whole system. Saying “part of a system”, I mean any significant part which consists of a few units which communicate with each other. Integration tests are brittle, easy to break, hard to maintain and slow. The only one benefit is that they verify how many components interact with each other without any mocks. Components are going to be tested in the environment close to the production’s environment. That’s why integration tests can detect defects undetected by unit tests.

Acceptance tests verify the software from the user’s point of view. Acceptance tests are expressed in a regular English language. Yes, indeed, they allow customers to write tests in plain English just in a formal syntax. Of course, customers are not allowed to write novels, but writing acceptance tests is much easier than writing unit tests for customers. This way of writing tests allows to meet the requirements written by a customer himself. We will look into an example or a few of examples of how to write such tests in a corresponding section.

You should understand that each type of tests has its own pros and cons. Integrations tests are not better than unit tests or vice-versa. A wise approach to testing is to use different types of tests. Moreover, we usually end up with a pyramid of tests. Here is the pyramid:

There is no silver bullet regarding the right balance of different types of tests. That balance depends on the type of software you’re working on.

Integration and UI Testing

The most significant part of the pyramid consists of unit tests, then integration and acceptance tests go.

Integration tests verify the behavior of either a part of a system or a whole system. They verify how many components interact with each other without any mocks. Components are going to be tested in the environment close to the production environment. That’s why integration tests can detect defects undetected by unit tests. There is also a notion of UI tests. What are UI tests?

Remember that UI tests are integration tests in nature. UI Tests allow to drive the executing of an application by automatic actions performed on the UI. It allows saving much time since it doesn’t require manual testing. We just write code which performs all the actions on the UI. Assertions in UI tests verify what UI shows to users. Integration tests are not comprised of only UI tests. UI tests are only a part of the suite of integration tests.

Why Write Automated UI Tests?

Why we might want to write automated UI tests at all? The major benefits of writing UI tests are related to the benefits of any integration tests since UI tests are integration tests (in principle).

So, here are the main benefits of writing UI tests:

• UI tests can play a role of end-to-end tests which bring into play literally all the code layers of the application.
• UI glitches – UI tests can catch any UI-glitches which is very important for providing the best user experience. For example, UI tests can catch such bugs as a help message doesn’t appear, a button gets disabled or enabled inappropriately and so on.
• UI tests reduce (sometimes down to zero) the amount of work that needs to be done by manual testers.
• In the case of Web UI tests, you’ll be able to run the same tests against different web browsers such as Firefox, Chrome, Opera and so on. I bet you understand what value it brings to the table, especially if you’re a web developer.

I mentioned that UI tests can reduce the amount of work that needs to be done by manual testers. Actually, teams often try to compare automated UI testing versus manual UI testing. Let’s mark out some benefits of automated UI tests over manual tests:

• Automated UI tests run faster than manual tests and because of that, they provide feedback much faster. They also can be run at any point in time just by a couple of clicks.
• Executing a suite of automated UI tests, you ensure the fullness of UI tests while human beings very often forget to perform some tests even when they work with a check-list.
• Automated UI tests very often serve as a mechanism which is complementary to what manual testers do. I would say, that if your development team has manual testers, the software developers or sometimes even testers on their own can write all the automated UI tests which are simple to implement, leaving all the complex UI tests which are hard to implement in the code for manual execution. In such a case, UI-tests will free up human testers and allow them to use their skills more efficiently performing the so-called exploratory testing. Human beings are pretty smart and can discover such slick bugs that can’t be discovered automatically.

What is an Acceptance Test?

In general, acceptance tests are business readable automated tests. Why do we need them? There are several benefits of writing acceptance tests.

The first benefit is to bring people from the problem domain. In 99% of cases, people from the problem domain are not software engineers and know nothing about unit tests and coding process in general. They can only express the requirements in the natural language. So be it, acceptance tests allow non-technicians to write tests in the natural language. Not in the completely arbitrary style. There is a special form for writing scenarios. So, the first goal is to reduce or completely remove the gap between software developers and specialists from the problem domain. Another benefit is that acceptance tests form the documentation. And what even better, they form executable documentation. That documentation reflects:

• what features does the system have
• how do those features work
• what different usage scenarios does the system support

The good thing is that tests reflect the needs of the end users what means that developers can build the right thing and build it right. Indeed, acceptance tests are electronic incarnation of use cases written on cards within the Agile.

Another benefit is that acceptance tests always reflect current requirements. Either we keep acceptance tests always updated or ditch the whole idea completely. There is no something in between these states. Acceptance tests should always stay in sync with the actual code. Acceptance tests, as well as any tests, are backed up by a framework. The most popular framework to write and run acceptance tests is called SpecFlow.

Acceptance and UI-Automation Testing Frameworks

I would mark out the following UI-testing frameworks:

• Selenium
• Visual Studio Coded UI Test
• TestStack.White

Selenium is a framework for testing web-UI while VS Coded UI Test allows writing UI-tests both for web-UI and desktop-UI. Selenium and VS Coded UI Test frameworks record all your actions and generate code which replays them. Selenium, however, supports another approach which is more suitable for developers and that approach allows to write tests manually. Such tests, in general, are more customizable, readable, and maintainable.

TestStack.White doesn’t allow to record actions and replay them. It allows writing UI-tests manually in code. This approach takes more time, but it allows to write, as I already said, human-readable and well-maintainable code while the generated code is hard to understand and maintain. TestStack.White is faster than VS Coded UI Test roughly five times being tested on a WPF application.

SpecFlow is a mature framework for writing acceptance tests. SpecFlow allows us writing special features files in a semi-formal language called Gherkin. We express business requirements and use cases in feature files. A SpecFlow extension that we need to install in Visual Studio allows generating executable C# code from the feature files. That code automatically gets bound to feature scenarios. In the generated code we write actual testing code and write assertions with the power of almost any unit testing framework we want to.

We will look into SpecFlow and Selenium further in this series of posts.