Parallel Testing

Context:

Who would not want to do parallel testing? It speeds up execution and cuts down overall time. Isn’t that nice. Also, the fundamental purpose of Automation testing is to report feedback as quickly as possible. So parallel testing accelerates the feedback loop isn’t it? While it is exciting to have the capability of being able to run tests in parallel, there are some nuances that we will need to consider. Some of them are:

  • Does the application support parallel sessions?
  • Are there max. number of sessions that application caps ? If so how do we mirror that in Automation Tests.
  • What about data structures in the code that are not thread-safe? 
  • How about asynchronous web application and apply parallel testing on top of that. That is very difficult to keep track in mind right ?

That said, we obviously cannot conquer the world right now with parallel testing, however we will make some assumptions right now and move on in terms of how Ruby, Cucumber has capabilities that help parallel testing. Most web applications support parallel testing right [For eg. we can open www.seleniumframework.com in multiple browsers at the same time and have operations performed on all instances]

Caution:

  • If our tests did not reach a stage where they can execute “successfully” in a sequence, then it would be an extremely bad idea to execute them in parallel
  • Parallel Testing is advisable only when the automation pipeline is established and consistently run with “pass” status sequentially.
  • Though we might talk about parallel tests on the UI layer here, in a n-layered application [UI-api-messaging-OR….DB], if parallelism is not supported at any layer in the call sequence, it will only result in frustration when we see our tests fail over and over again.

Relevance to Automation:

While we have been a little negatively biased in the above description, we would like to mention that there are obvious advantages of parallel testing given the assumptions above are taken into consideration. They are:

  1. Developers can execute tests in parallel in an ATDD environment before code check-ins and that saves time [especially on sanity tests that are run before code check-in]
  2. By increasing the number of parallel sessions, we are automatically replicating a real time environment where we have multiple parallel sessions from end-users [This touches upon the performance testing phase, though performance testing simulates the sessions and virtual users and so on]. Please do not consider this approach as a substitution for performance testing.
  3. We can maximize the usage of machine resources [memory, cpu cycles etc]

In this post though, we will talk specifically on how can we run our previously written Cucumber scenarios in parallel without actually changing much of code. Of course there are many options that we can create for parallel testing. However, I believe that the options below will give quick ROI. Once we know how to do parallel execution, we can always build upon the concept and introduce more options over time.

Parallel Tests Gem:

We will be leveraging a gem for parallel execution and it is called “parallel_tests” gem. You can find complete reference on github here. The gem supports parallel execution on Test::Unit, Rspec and Cucumber. So how does it run tests in parallel?

There are a bunch of us [including me], who would want to get into extreme details of what exactly “parallel” means. Some of the terms that come to my mind are multi-programming, multi-tasking and multi-processing. And how do each of these terms relate to multi-core processors [that silicon chip companies sell]. We all want “things” to run faster right? That all of us agree. However, if we peel the requirement into mapping on “how” to achieve speed, we would have to talk about the capabilities that software and hardware gives us ultimately to give us a “speedy” experience.

As part of making things faster, technically we would have to get into each of those detailed geeky terms because they apply at different layers [remember ISO-OSI model]. Here is a good post that alludes a little bit to the differences between the three. There are much much larger discussion on whether Ruby really implements multi-threading and that differs with various flavors of Ruby, however it is out of scope here. For folks interested, here it is. We will probably have to stop this discussion as it might digress us into other topics 🙂

Oh ! One last thing. We know that we want to run multiple browsers in parallel right, so we all agree on that requirement. So let’s believe that parallel tests gem has abstracted the implementation details for us and we will move forward agreeing to the principles of parallel_tests gem.

Parallel_tests gem has a sub component called parallel_cucumber that actually runs our Cucumber scenarios in parallel. By default, the number of parallel processes is equal to the number of cores that our CPU is made up of. We can of course change the default number of processes. One might ask, how do I know how many cores my CPU has ? Well, you can always go look up your model #, CPU information and search it up and read the technical specs. One other easy way[on windows machines] is to open your Task manager – Performance tab. The number of blocks we see on “CPU Usage history” is the number of cores. My machine has 4 cores as below.

4core_taskmgr

 

Agenda:

  1. Install and configure parallel_tests
  2. Enabling cucumber options for parallel execution
  3. Focus on key lines of code
  4. How to interpret test results and reports
  5. Execution Output
  6. Closing Thoughts

1) Install and configure parallel_tests

Add “gem parallel_tests” to Gemfile as below

gemfile_paralleltests

Run bundle update from Rubymine [Tools-bundler-update] or from command line as below from the project root.

bundler-update-cmd

That is it.

2)Cucumber options

Add the below line in cucumber.yml file. We are adding a cucumber profile here. We will discuss profiles in another post, but for now consider a cucumber profile as representing a bunch of options that we want to launch cucumber with. Tags was one option which we discussed previously.

cucumberyml_parallel

 

3)Key Lines of Code:

ENV[‘TEST_ENV_NUMBER’] is an environment variable that parallel cucumber automatically sets based on the number of cores. So on my machine, it will track 4 values in the environment. And we are asking the cucumber profile to create the test report [the html reports we have been creating so far as results.html] with the name report+<number>. So after execution we should see report.html, report1.html, report2.html and report3.html, each report representing the process that has run on each CPU core.

The parallel cucumber has another logic, which we would notice after a certain number of executions only. The logic is as follows.

  • Cucumber Features execute in parallel i.e. for example 4 .feature files will be picked up when we first start execution
  • Within each .feature file, the scenarios would run in sequence
  • So there are always 4 processes running and each process would run the entire feature file [in turn all scenarios in the feature file] sequentially.
  • Once all scenarios in a feature file complete execution, the next available feature file is picked up
  • This makes sense because when we look at Test Results, we want to see all scenarios belonging to a .feature file at one place [or in sequence]. In fact isn’t that the purpose, we grouped the scenarios in one feature file.
  • At the end of execution, we will have 4 reports and each report will contain the results [how many ever feature files that 1 of 4 processes ran] that it executed across multiple feature files.

4) Execute:

Method1: 

Open command prompt and from project root, execute “parallel_cucumber features”. As circled below, you will notice that 4 processes would be started and 4 browsers would pop up on your machine executing each scenario.

paralle_execution

Method2:

Another way to execute is through bundle. This is not specific to parallel testing, however when we execute through bundle, then the versions of gems that are locked down in gemfile.lock would be the ones that will be used to kick off this current execution. This is the reason we added parallel_tests to the Gemfile, otherwise we could have installed the gem outside of our project and kicked off the tests since we are anyways using command line.

paralle_execution_bundle

Method3:

In the previous two methods, test reports do NOT get generated since we have NOT used the parallel profile that we created in cucumber.yml. To generate test reports and analyse the results, we would have to use the parallel cucumber profile. Below is how we would use it. Observe the “-o” option

paralle_execution_bundle2

Existing Cucumber Options:

We might want to run the parallel scenarios but also would want to run it with our existing cucumber options like say “tags”. Remember we talked about Tags and how they can be helpful to slice and dice tests. OR you would want to specify the browser parameter which we have set in our code as described in headless testing section. By adding the “-o” option to the parallel cucumber, we can pass the regular cucumber parameters. The example below runs cucumber features in parallel with “firefox” browser and runs only the scenarios that are tagged with @beta3

paralle_execution3

5)Test Results:

As we mentioned above, there will be 4 reports generated in the project folder. Open the reports in browser and we will see the results.

parallel_reports

 

Output:

As we mentioned above, we have not written any new cucumber scenarios here. We just ran the already written scenarios using parallel_cucumber instead of cucumber. So the html reports and the way we have been interpreting those reports, with the screenshots embedded inside the reports [when failed], all of that experience should NOT change in any way. The difference is we see the execution time cut down and we see four reports now.

Closing Thoughts:

Can we run only with 4 processes? Absolutely not, in fact that parameter is configurable. For example, I can kick off 10 processes in parallel too by passing a configuration parameters to parallel_cucumber, that means technically, I should see 10 browsers popping up on my machine when I kick off the tests. However bear in mind that if you supply a large number that consumes all your machine resources, then the system might crash. Feel free to read the parallel tests gem documentation or write to us if you need more information.