An Overview of JavaScript Testing in 2018
TL;DR; Use Jest for unit and integration tests and TestCafe for UI tests.
This guide is intended to catch you up with the most important reasoning, terms, tools, and approaches to JavaScript testing in 2018. It combines information from many great articles, that are linked at the bottom, and adds from our own experience in Welldone Software Solutions where we implemented different testing solutions for different products for years.
Whoever reads and understands this guide, can safely assume they know the big picture of the state of JavaScript testing in the web development community for 2018. This is a great reason to share it with all your co workers, family and friends
- I did a pretty serious research for this article. If you see anything I missed or got wrong, comment about it and I’ll address it in no-time.
- Notice the links at the bottom of the page. Reading them will take you from just understanding the big picture, to be an expert (in theory).
- The best way to implement this guide is to choose whatever testing types you need, select several tools that looks suitable for you, and test them all. We live in a world of a great abundance of tools and practices. Your goal is to filter them and to get to the best set up for your individual cases.
Enjoy
Intro
Look at the logo of Jest, a testing framework by Facebook:
As you can see, their slogan promises a “painless” JavaScript Testing, but as “some guy from the comments” pointed out:
And indeed, Facebook have an excellent reason to use this slogan. In general, JS developers are not too happy with website testing. JS tests tend to be limited, hard to implement, slow and sometimes expensive.
Nevertheless, with the right strategy and the right combination of tools a nearly full coverage can be achieved and tests can be very organized, simple, and relatively fast.
Test Types
You can read about different test types in more depth here and here and here.
In general, the most important test types for a website are:
- Unit Tests - Testing of individual functions or classes by supplying input and making sure the output is as expected.
- Integration Tests - Testing processes or components to behave as expected, including the side effects.
- UI Tests - (A.K.A Functional Tests) Testing scenarios on the product itself, by controlling the browser or the website, regardless of the internal structure to ensure expected behavior.
Test Tools Types
Test tools can be divided into the following functionalities. Some provide us with only one functionality, and some provide us with a combination.
In order to get a more flexible functionality, it’s common to use a combination of tools even if one can achieve relatively the same.
- Provide a testing structure (Mocha, Jasmine, Jest, Cucumber)
- Provide assertions functions (Chai, Jasmine, Jest, Unexpected)
- Generate, display, and watch test results (Mocha, Jasmine, Jest, Karma)
- Generate and compare snapshots of component and data structures to make sure changes from previous runs are intended (Jest, Ava)
- Provide mocks, spies, and stubs (Sinon, Jasmine, enzyme, Jest, testdouble)
- Generate code coverage reports (Istanbul, Jest, Blanket)
- Provide a browser or browser-like environment with a control on their scenarios execution (Protractor , Nightwatch, Phantom , Casper)
Let’s explain some of the terms mentioned above:
Testing structure refers to the organization of your tests. Nowdays, tests are usually organized in a BDD structure that supports behavior-driven development (BDD). It often looks like this:
Assertion functions are functions that make sure that tested variables contain the expected value. They usually looks like this, where the most popular are the first two:
TIP: Here is a nice article about advanced Jasmine assertions.
Spies provide us with information about functions- How many times were they called, in what cases, and by whom?
They are used in integration tests to make sure that the side effects of a process are as expected. For example, how many times was a calculation function called during some process?
Stubbing or dubbing **(**like doubles in movies) replaces selected functions with selected functions to ensure an expected behavior on selected modules.
If we want to ensure user.isValid()
will always return true
during a test, where you test a different component, for example, you can do this:
It also works with promises:
Mocks or Fakes are faking certain modules or behaviors to test different parts of a processes.
Sinon can, for example, fake a server to ensure offline, fast and expected responses when testing a process.
Snapshot Testing is when you compare a data structure to an expected one.
The following example, from the official Jest documentation, shows a snapshot test of a certain Link
component.
It won’t actually render and take a picture of the component, but it would save its internal data in a separate file like this:
When the test runs, and the new snapshot is different from the last one, the developer is prompted to agree that the change is intended.
Notice: Snapshots are usually made to compare component representation data but they can also compare other types of data like redux stores and inner structure of different units in the application.
Browser or browser-like environment can be one of the three:
- jsdom —A Pure JavaScript environment that simulates a real browser. It has no UI and it renders nothing- It provides you with window, document, body, location, cookies, selectors and whatever you get when you run your JS inside the browser.
- Headless Browser Environment — A browser that runs without a UI for the purpose of making the browser respond faster.
- Real Browser Environment — An actual browser that opens and runs your tests.
Putting it All Together
We suggest using the same tools for all the test types if possible:
The same testing structure and syntax (1), assertion functions (2), result reporting, and watching mechanism (4).
We also suggest creating two different processes. One for running unit and integration tests and another for UI tests. This is because UI tests takes a longer time, especially if tested on different browsers and usually use an external service to provide you with different devices and browsers (this will be discussed later) and can cost money so you would probably prefer to run it much less then the first process. For example: only before a merge of a feature branch.
Unit Tests
Should cover all small pure units of the application- utils, services and helpers. Provide all these units with simple and edge case inputs and make sure their outputs are as expected using the assertion functions (3). Also make sure to use a coverage reporting tool (6) to know what units are covered.
Unit tests are one of the reasons to use functional programming and pure functions as much as possible- > The purer your application is, the easier you can test it.
Integration Tests
Old school tests were focused on unit testing and resulted in applications where many small parts are working but the processes as a whole keeps on failing.> Integration tests (including snapshots), on the other hand, can detect many unexpected failures where you fix one thing and break the other.> It is also important to remember that in the real world , for the reasons of imperfect design and the widespread use of black boxes, not all units are pure and not all units are testable- Some units can be tested only as part of a bigger process.
Integration tests should cover important cross-module processes. Comparing to unit tests, you would probably use spies (5) to ensure expected side effects instead of just asserting the output and stubs (5) to mock and modify the parts of the process that are not in the specific test.
Also, as opposed to unit tests, a browser or browser-like environment (7) could help with processes that are dependent on the window
and when part of the process is to render certain components or interact with them.
Component snapshot tests (4) fall into this category as well. They provide us with a way to test how processes affect selected components without actually render them or using a browser or browser-like environment.
UI Tests
Sometimes the quick and effective unit and integration tests are not enough.
UI tests are always running inside a browser or browser-like environment (7) that were discussed earlier.
They simulate user behavior on these environments (clicking, typing, scrolling etc…) and make sure these scenarios actually work from the point of view of an end user.
It is important to remember that these tests are the hardest to set up. Imagine yourself creating an environment to run a test on different machines, devices, browser types and versions. This is why there are many services that provide this service for you. And even more can be found here.
List of General Prominent Testing Tools
jsdom
jsdom is a JavaScript implementation of the WHATWG DOM and HTML standards. In other words, jsdom simulates a browser’s environment without running anything but plain JS.
As mentioned before, in this simulated browser environment, tests can run really fast. The drawback of jsdom is that not everything can be simulated outside a real browser (you can’t take a screenshot for example) so using it will limit your test’s reach.
It is worth mentioning that the JS community rapidly improves it and the current version is very close to a real browser.
Istanbul
Istanbul, a JavaScript test coverage tool
Istanbul will tell you how much of your code is covered with unit tests. It will report on statement, line, function and branch coverage in percentages so you will understand better what is left to cover.
Karma
Karma lets you run tests in browser and browser like environments including jsdom.
Karma hosts a test server with a special web page to run your tests in the page’s environment. This page can be run across many browsers.
This also means tests can be run remotely using services like BrowserStack.
Chai
Chai is the most popular assertion library.
Unexpected
Unexpected is an assertion library with a slightly different syntax from Chai. It is also extensible so assertions can be more advanced with libraries that are based on it like unexpected-react that you can read about more in depth here.
Sinon.JS
Sinon is a very powerful standalone test spies, stubs and mocks for JavaScript that works with any unit testing framework.
testdouble.js
testdouble is a less popular library that does what Sinon does, and claims to do it better. With a few differences in design, philosophy, and features that could make it useful in many cases. you can read about it here, hereand here.
Wallaby
Wallaby is another tool worth mentioning. It is not free, but many users recommend buying it. It runs on your IDE (it supports all major ones) and runs relevant to your code changes tests and indicates if anything fails in real time just alongside your code.
Cucumber
Cucumber help with writing tests in BDD by dividing them between the acceptance criteria files using the Gherkin syntax and the tests that are correspondent to them.
Tests can be written in a variety of languages that are supported by the framework, including JS that we are focusing on:
Many teams will find this syntax more convenient then TDD.
Choose Your Unit and Integration Tests Framework
The first choice you should probably make is what framework do you want to use and libraries to support it. It is recommended to use the tools your framework provides until a need for unique tools arises.
- In short, if you want to “just get started” or looking for a fast framework for large projects, go with Jest.
- If you want a very flexible and extendable configuration, go with Mocha.
- If you are looking for simplicity go with Ava.
- If you want to be really low-level, go with tape.
Here is a list of the most prominent tools with some of their charactaristics:
mocha
Mocha is currently the most used library. Unlike Jasmine, it is used with third party assertion, mocking, and spying tools (usually Enzyme and Chai).
This means Mocha is a little harder to set up and divided into more libraries but it is more flexible and open to extensions.
For example, if you want special assertion logic, you can fork Chai and replace only Chai with your own assertion library. This can also be done in Jasmine but in Mocka this change will be more obvious.
- Community - Has many plugins and extension to test unique scenarios.
- Extensibility - Plugins, extensions and libraries such as Sinon includes features Jasmine does not have.
- Globals - Creates test structure globals by default, but obviously not assertions, spies and mocks like Jasmine- some people are surprised by this seemingly inconsistency of globals.
Jest
Jest is the testing framework recommended by Facebook. It is based on Jasmine that we will discuss later. By today Facebook replaced most of its functionality and a added a lot of features on top of it.
After reading a huge amount of articles and blog posts, it’s incredible how people are impressed by Jest’s speed and convenience.
-
Performance - First of all Jest is considered to be faster for big projects with many test files by implementing a clever parallel testing mechanism (For example by us from our experience and in these blog posts: here, here, here, here).
-
UI - Clear and convenient.
-
Ready-To-Go - Comes with assertions, spies, mocks that are equivalent to libraries that do the same like Sinon. Libraries still can easily be used in case you need some unique features.
-
Globals - Like in Jasmine, it creates test globals by default so there is no need to require them. This can also be considered bad since it makes your tests less flexible and less controllable, but in most cases it just makes your life easier:
-
Snapshot testing - jest-snapshot is developed and maintained by Facebook, although it can be used in almost any other framework as part of the framework’s integration of the tool or by using the right plugins.
-
Improved modules mocking - Jest provides you with an easy way to mock heavy modules to improve testing speed. For example a service can be mocked to resolve a promise instead of making a network request.
-
Code coverage - Includes a powerful and fast built-in code coverage tool that is based on Istanbul.
-
Reliability - Although this is a relatively young library, throughout 2017 Jest stabilized and is now considered reliable. It is currently supported by all the major IDEs and tools.
-
Development- jest only updates the files updated so tests are running very fast in watch mode.
jasmine
Jasmine is the testing framework that Jest is based on. Why would you still use Jasmine? It has been around for a longer time and has a huge amount of articles and tools that were created by the community.
Also, Angular still suggests using it over Jest, although Jest is perfectly suitable to run Angular tests as well, and many people do it.
- Ready-To-Go - Comes with everything you need to start testing.
- Globals - Comes with all the important testing features in the global scope as well.
- Community- It has been on the market since 2009 and gathered a vast amount of articles, suggestions and tools that are based on it.
- Angular - Has widespread Angular support for all it’s versions and it is what recommended in the official Angular documentation.
AVA
Ava is a minimalistic testing library that runs tests in parallel.
- Ready-To-Go - Comes with everything you need to start testing (besides spying and dubbing that you can add in no-time). Using the following syntax for test structure and assertions, and runs in Node.js:
- Globals - As seen above, it does not create any test globals thus you have more control over your tests.
- Simplicity - Simple structure and assertions without a complex API while supporting many advanced features.
- Development- Ava only updates the files updated so tests are running fast in watch mode.
- Speed - Runs tests in parallel as separate Node.js processes.
- Snapshot testing is supported as part of the framework.
tape
Tape is the simple of them all. It’s just a JS file you run with node with a very short and “to-the-point” API.
- Simplicity - Minimalistic structure and assertions without a complex API. Even more than Ava.
- Globals - Does not create any test globals thus you have more control over your tests.
- No Shared State between tests- Tape discourages the use of functions like “beforeEach” to ensure test modularity and maximum user control over the tests cycle.
- No CLI is needed- Tape is simply run anywhere JS can be run.
UI Testing Tools
First of all, as mentioned above, here and here you can find great articles about service providers that would host the machines where tests would run and help you run these tests on different devices and browsers.
The number of permanent tools for the purpose of UI testing differs very much from each other both in their implementation, philosophy and API, so it is strongly suggested to invest time in understanding the different solutions and testing them on your product.
- In short, if you want to “just get started” with a reliable and simple to set-up cross-browser all-in one tool, go with TestCafe.
- If you want to go with the flow and have maximum community support, WebdriverIO is the way to go.
- If you don’t care about cross-browser support, use Puppeteer.
- If your application has no complex user interactions and graphics, like a system full of forms and navigations, use cross browser headless tools like Casper.
selenium
Selenium, automates the browser to simulate user behavior. It is not written specifically for tests and can control a browser for many purposes by exposing a server that simulates user behavior on a browser using an API.
Selenium can be controlled in many ways and using a variety programming languages, and with some tools even without any real programming.
To our needs, however, Selenium server is controlled by a Selenium WebDriver that serves as a communication layer between our NodeJS and the server that operates the browser.
Node.js <=> WebDriver <=> Selenium Server <=> FF/Chrome/IE/Safari
The WebDriver can be imported into your testing framework and tests can be written as part of it:
The WebDriver itself might be sufficient for you and indeed some people suggest using it as it is, but various libraries were created to extend it ether by forking and altering it or by wrapping it.
And indeed wrapping the WebDriver might add redundant code and could make debugging harder, whereas forking it might diverge it from the very active (as for 2018) ongoing development of WebDriver.
Still, some people prefer to not use it directly. Let’s look at some of libraries for selenium operating:
Appium
Apium provides an API similar to Selenium for testing websites on a mobile device using the following tools:
- iOS 9.3+ : Apple’s XCUITest
- Before iOS 9.3 : Apple’s UIAutomation
- Android 4.2+ : Google’s UiAutomator/UiAutomator2
- Android 2.3+ : Google’s Instrumentation. (Instrumentation support is provided by bundling a separate project, Selendroid)
- Windows Phone : Microsoft’s WinAppDriver
So if you use Selenium or Selenium based tools, you can also use Apium to test on mobile devices.
Protractor
Protractor is a library that wraps Selenium to add it an improved syntax and special in-built hooks for Angular.
- Angular- Has special hooks, although can be successfully used with other JS frameworks too. Angular official documentation suggests using this tool.
- Error reporting - Good mechanism.
- Support - TypeScript support is available and the library is operated and maintained by the huge Angular team.
WebdriverIO
WebdriverIO has it’s own implementation of the selenium WebDriver.
- Syntax - very easy and readable.
- Flexible - A very simple and agnostic from even being used for tests, flexible and extensible library.
- Community - It has good support and enthusiastic developer community that makes it reach with plugins and extensions.
Nightwatch
Nightwatch has it’s own implementation of the selenium WebDriver. And provides it’s own testing framework with a test server, assertions, and tools.
- Framework - Can be used with other frameworks too, but can be especially useful in case you want to run functional tests not as part of other framework.
- Syntax - looks the easiest and the most readable.
- Support - No typescript support and in general, this library seems to be slightly less supported than the others.
TestCafe
TestCafe is a great alternative to Selenium-Based tools. It was rewritten and open-sourced in the end of 2016.
There is still a paid version that offers non programming testing tools like a test recorder and a customer support, this is important because many outdated articles mistakenly state that it’s code is closed and count it like the libraries disadvantage.
It injects itself into the website as JavaScript scripts instead of controlling the browsers themselves like Selenium does. This allows TestCafe to run on any browser, including on mobile devices , and have a full control over the JavaScript execution loop.
TestCafe is JavaScript and test oriented. It is under massive development right now, although it is already considered stable and full of features.
- Fast to set up - You don’t need special browsers. Just open a browser and run your tests there.
- Cross Browser and Devices - Supports many browsers and devices and can be used with SouceLabs or BrowserStack that can provide the devices and the browsers to test on for you. This includes running of tests in Headless Chrome and Headless Firefox which will discussed later.
- Parallel Testing - TestCafe can run your tests on several browser instances at once. This practice can significantly decrease your testing time.
- Convenient Error Reporting
- Own Ecosystem - TestCafe uses it’s own test structure. It could be quite convenient, especially because UI tests usually run separately from other tests, but some people dislike it.
Cypress
Cypress is a direct competitor of TestCafe. They are doing relatively the same, which is injecting tests into a website, but they try to do it in a more modern, flexible and convenient way.
They are newer, and just moved from closed beta to public beta (At October 2017) but they already have many adopters.
- No Cross-Browser Support - Only support chrome right now (and not headless). They are working on it as this article is being created. (And we will update the article once they do it)
- Lacks Advanced Features - Parallel testings and several testing tools are still missing comparing to TestCafe but they are on the road-map of the product’s diligent team.
- Documentation - Solid and clear.
- Debugging Tools - Easy debugging and logging of the test process.
- Using Mocha as it’s test structure provider makes it’s use pretty standard and can make your UI tests be built in the same structure as the rest of your tests.
Puppeteer
Puppeteer is a Node.js library, developed by Google. It provides a convenient Node.js API to control Headless Chrome .
Headless Chrome is just a regular Chrome v59+ that is launched with the
--headless
flag. When chrome is run in headless mode, it exposes an API to control it, and as said before, Puppeteer is the JavaScript tool that Google provides to control it.
Here it is worth mentioning, that Firefox has also released their headless mode in the end of 2017.
Notice that different testing tools can also use Headless Chrome and Firefox. For example: TestCafe, Karma, .
- Puppeteer is relatively new, but it has a big community that uses and develops tools and wrappers around it.
- Since it is native, it is faster, it is native and uses the latest Chrome engine , unlike PhantomJS that is built on an older fork of WebKit. (We will discuss it in the next section)
- One major drawback of Headless Chrome (thus of Puppeteer as well) is that it doesn’t supports extensions like flash and probably wouldn’t in the near future.
PhantomJS
Phantom implements the chromium engine to create a controllable Chrome-like headless browser.
Since Google announcement of theirPuppeteer, It’s creator and maintainer Vitaliy Slobodin no longer works on it, so it is maintained and developed much slower since mid 2017, although it is still somewhat maintained.
Why would you use Phantom over Puppeteer?
- First of all because it is much more mature and has many great guides and tools.
- It also used by many useful tools like CasperJS that we will discuss later.
- It uses older WebKit so it can simulate older Chrome browsers.
- Also, as mentioned before, Phantom supports extensions like Flash as opposed to Headless Chrome.
Nightmare
Nightmare is a great UI testing library that offers a very simple test syntax.
It uses Electron which is similar to Phantom but uses a newer Chromium and actively maintained and developed, since Electron’s main purpose is to build cross platform desktop apps with JavaScript, HTML, and CSS.
Also, they are now [discussing(https://github.com/segmentio/nightmare/issues/1092) and experimenting with Headless Chrome as well.
Here how Nightmare code looks like vs Phantom code.
Casper
Casper is written on top of PhantomJS and SlimerJS(The same as Phantom but written using Firefox’s Gecko) to provide navigation, scripting and testing utilities and abstracts away a lot of the complicated, asynchronous stuff when creating Phantom and Slimer scripts.
Slimer was in widespread use for a long time although considered experimental, but in the end of 2017 they released their beta version:
1.0.0-beta.1
that uses the new Headless Firefox and currently working on stabilizing it and releasing version 1.0.0
.
Casper would probably migrate from PhantomJS to Puppeteer in the near future with their expected release of version 2.0 and become a great tool to test on both Headless Chrome and Headless Firefox. Stay tuned.
CodeceptJS
Like CucumberJS that was discussed above, Codecept provides another abstraction over different libraries’ API’s to make your interactions with tests using a slightly another philosophy that focuses on user behaviour.
Here is how it looks like:
And here is the list of libraries that can be executed using this code. All discussed above.
WebDriverIO, Protractor, Nightmare, Appium, Puppeteer.
If you believe this syntax is better for your needs, give it a shot.
Conclusion
We saw the most trending testing strategies and tools in the web development community and hopefully made it easier for you to test your sites.
In the end, the best decisions regarding application architecture today are made by understanding general solution patterns that are developed by the very active community of developers, and combining them with your own experience and by taking in account the characteristics of your application and it’s special needs.
Oh, and writing, and rewriting, and rewriting, and rewriting, and testing different solutions
Happy testings
Thanks
Suggested Articles
Best
- Javascript tools for end-to-end testing web applications (July 2017)
- Top 5 Most Rated Node.js Frameworks for End-to-End Web Testing (March 2017)
- Top UI Test Automation Best Practices You Should Follow (November 2017)
- How to set up E2E browser testing for your GitHub project (With TestCafe) (February 2017)
- Introduction to Headless Browser Testing (September 2017)
- Web Scraping in 2017 : Advanced Headless Chrome Tips & Tricks (august 2017)
- How GitLab switched to Headless Chrome for testing (From Phantom)(December 2017)
General
- The Top 8 Essential JavaScript Automation Frameworks (June 2016)
- JavaScript Testing: Unit vs Functional vs Integration Tests (April 2016)
- What are Unit Testing, Integration Testing and Functional Testing? (April 2015)
- Top 12 Browser Compatibility Testing Tools (June 2017)
- Top 10 Cross Browser Compatibility Testing Tools In 2018 (2018)
spies and stubs
- Bye-Bye, Sinon — Hello, testdouble (March 2016)
- JavaScript Testing Tool Showdown: Sinon.js vs testdouble.js (April 2017)
- testdouble.js vs. sinon.js (March 2016)
- enzyme vs unexpected-react (April 2017)
Testing Frameworks Comparison
- React Testing — Jest or Mocha? (May 2017)
- Testing React Applications with Karma, Jest or Mocha (August 2016)
- Let’s Compare Javascript Testing Framework (August 2016)
- Jasmine vs. Mocha, Chai, and Sinon (Updated February 2016)
- Jasmine vs. Mocha - JavaScript Testing Framework Comparison
(January 2017)
Jest
- How to Snapshot Test Everything in Your Redux App With Jest
(December 2016) - Migrating to Jest on the P2P team at PayPal (November 2016)
- Jest - From 0 to 100% Coverage Real Quick (January 2017)
- The Hidden Power of Jest Matchers (January 2017)
- Testing React components with Jest and Enzyme (December 2016)
- Testing a React-Redux app using Jest and Enzyme (January 2017)
- Snapshot Testing React Components with Jest (January 2017)
- Testing an AngularJS app with Jest (October 2016)
Ava
- Effortless unit testing with AVA (April 2016)
- Javascript Testing with AVA (February 2017)
- Unit testing front-end JavaScript with AVA and jsdom (November 2016)
- Testing the Bejeezus out of React Native Apps with AVA (June 2016)
Tape
UI Tests
- Hybrid Application Testing with Protractor and Appium (March 2017)
- Headless Chrome vs PhantomJS Benchmark (September 2017)
- Protractor vs WebdriverIO vs NightWatch (November 2016)
- End to End (e2e) Testing React Apps With Selenium WebDriver And Node.js is Easier Than You Think (April 2016)
- WebdriverIO - Using JavaScript to Drive Selenium Tests (August 2016)
- JavaScript Functional Testing with Nightwatch.js (August 2016)
- Node.js End-to-End Testing with Nightwatch.js (March 2017)
- Nightmare of End-to-End Testing (September 2016)
- Protractor vs WebdriverIO vs NightWatch (November 2016)
- Stack Overflow- What is the difference between nightwatch.js And webdriver.io? (March 2016)
- Comparison of JavaScript browser automation and test specification libraries (December 2015)
- What is PhantomJS and How is it Used? (June 2016)
- Setting up End-to-End Testing in Angular Project with CodeceptJS (December 2017)
TestCafe
- Which automated testing tool to pick — Testcafe or Selenium WebDriver? (January 2016)
- Introducing TestCafe — Open Source Testing Framework (Octobeer 2016)
- Why not use Selenium? (October 2016)
- Functional Testing With TestCafe (March 2017)
- UI Test Automation Frameworks Showdown: TestCafé versus Nightwatch.js (Fabruary 2018)
its such a nice article …thanks vitali