How to use Cucumber as Enterprise Class Test Framework
Cucumber is a behavior-driven (BDD) testing framework, and with it one can write test scenarios in native language. It enables efficient collaboration and effective communication among team members. Cucumber has already been adopted as an automated testing framework in many companies.
Cucumber also has implementations in different programming languages, such as Java, JavaScript, and Ruby etc. Dispite different implementations, they all have the same principle and use the same feature file format(a.k.a. gherkin).
We are using Cucumber.js to test our own application and websites, and it is easy to use and enables users to create the automation script quickly. When automation script grows become more complex, users may find some limitations in the tool, which if not solved, can limit its scalability and prevent it from being adopted on big enterprise class applications.
To better meet the automation test need in enterprise, we have made some enhancements or address some of its limitation:
- Report generation
- JSON file size
- Scenario management
- Subtitle to display running steps
Report Generation
When launch Cucumber.js from command line, by default it uses text format output. When use parameter --format json:filename.json, it saves the report data to a JSON file after the project finishes running. JSON file contains data for all the run results, it can then be formatted into HTML reports to meet different user's needs. Every user can apply their own html template to the report, therefore JSON output file is quite handy to use.
We know that JSON is not a streaming file, all the data must be ready before generating the target JSON file. When running, Cucumber.js stores all results in member, and finally after it finishes running, output data to a JSON file. White this way works, it creates a problem: If a project has many scenarios in it, it is time-consuming to run, and if stopped in the middle, no run result is generated. For example, suppose there are 200 test scenarios in a project. If you run this project, after quite a while, 190 of them completes and is close to get the report result. When runs the 191st scenario, the test script is stuck and does not proceed (e.g. the cause can be some error handling problem in your test script). If you stop the execution now by force, no JSON file is generated, and you lose all the run result data for this run. There is no JSON file is generated therefore no test report either, and the time spent on this execution is wasted. Quite upsetting, right? And you have to fix the problem in your script and start running from the beginning.
In CukeTest, which has built-in Cucumber.js engine, we addressed this limitation by streaming JSON data to a log file. During a project run, when each scenario completes running the corresponding JSON run result data is written out to disk file. If the execution is stop in the middle, it still show html test report from the data already generated. similiar to the following, which contains all the finished scenarios.
Automation engineers, in the process of development, can stop the project run manually when notice a problem, investigate the problems in the partially finished run report, and run it again. Any time, you can get the report for those scenarios that finishes running.
JSON File Size
When doing automation testing for Web, Mobile or Windows desktop application, one common practice is adding screenshots to the run report when a scenario is executed, as an evidence that the scenario runs well. It can also be used to locate potential UI bugs. Cucumber.js records all the data in a JSON file including screenshots, which is convenient. The screenshot images are also stored in this JSON file, in base64 format. If many images are embedded, this JSON file becomes huge. Node.js will throw an exception (something like "Invalid string length") when stringify the images to JSON when the data exceeds its limit. Besides, all the screenshots data is stored in memory before generating JSON file, which consumes a lot of memory and make the system less stable.
To eliminate this limitation, CukeTest provide an option to save the image to separate files. By selecting the option, even if you have hundreds of scenarios, or captured thousands of pictures, the test reports can be still generated immediately.
Because this is a switch setting, you still have all the flexbility:
- You can choose to attach all the images in the html report to get a single report file, or you can choose to store them separately.
- You can also choose to make your html report smaller by referring to the css/js resources from public CDN; or if you are in a disconnected network, choose to embedded all css/js resources inside your html file. The latter generates a bit bigger report file, but no longer need to worry about the network problem which may cause report fail to render.
Scenarios Management
When doing automated test projects, maintaining automated test scripts becomes a challenge as the test automation script grows larger. Cucumber has a good reputation of creating an automation scripts with good readability, by formats test cases with feature files and scenarios. However, the challenge still exists if the script grows to certain large size. Imagine you are developing test automation scripts for a big enterprise application, there can be dozens of features, hundreds of scenarios, how do you plan to manage all of them and quickly locate one test scenario and corresponding code for modification? Sometimes, some steps need to be reused in multiple feature files, which creates another dimension of complexity.
In Cucumber.js's coding tool, CukeTest, has built-in senarios browsing and searching functionality. When opens a project folder, it lists all the features, scenarios, and optionally steps on the left panel. you can click any of them to quickly jump to a scenario or steps. Also, one can type in the text to search for a scenario/step, when text is input, only the matching result are displayed, you can click the filter options to show only scenario, only feature, or only steps.
One can easily reuse the steps by simply drag a step from the left panel to your scenario:
In addition, in the CukeTest, by clicking each round button on each operation step, you can directly locate the code implementation part to facilitate development and debugging.
Subtitle to Display Running Steps
When running a Cucumber project, when run many scenarios, you not always know which scenario or which step is currently running. Or maybe you as an automation engineer knows it, but when demo the running to some other team members, they want to know what it is testing.
In CukeTest, when running a project, it can show the current running scenario as well as the running step on the screen, like the subtitle for a movie. It describe what it is exactly doing to people who watch the execution. Also, it shows the number of scenarios it is running, and the index of the running step. The following GIF shows you how the subtitle is displayed at the bottom of the screen when running a sample shopping web project:
the smaller white text on the upper left is the scenario, and the bigger white text in the middle is the step text.
You can also configure a run profile to record video for the test run. So that the execution can be played back later, sometime for trouble-shooting the problems identified during test run. The video already contains scenario and step information, so it is easy to jump to the scenario that the has the problem. The subtitle makes the recorded video more valuable for trouble-shooting and some other purposes.
Note: By the time of writing, Subtitle feature is only on Windows version, Mac and Linux users still need wait a few more days for a new release to get this feature.
Summary
With these enhancements, Cucumber becomes even more convenient to use, and is more qualified as an enterprise-class test automation framework. CukeTest as a BDD oriented automation tool, is now a choice in enterprises including financial companies, manufacturing and some research institutes.
