What is Cross Browser Test Automation?

Background

It is a well known fact by this time that we are dealing with an exploding number of browser combinations for web applications. Every other day, a new browser and/or its version is released and the application has to deal with the browser/version/device which the end user is using. This also goes by the term ‘form-factor’. On the web application side, we have at least 400+ combinations of browser/versions and on the mobile side, we have at least 120+ combinations of browsers (browsers sitting on mobile devices). So how can we ensure that we test against all those combinations and more importantly, where are we going to get the infrastructure to install all those browsers and version and maintain the environments?

Crazy right ?

Some Analysis

Let’s first talk about some analysis on this. We know what a selenium grid does right ? It has been designed to exactly solve this type of problem i.e. have an execution environments with different browser combinations that we can re-direct our script to execute on. So why can’t we have the grid installed on various machines where we have the browsers and point our scripts to the grid to take care of execution?

Fair question ! Here are some of the challenges we might have maintaining a selenium grid (again it is NOT impossible, just that we have to have trade-off discussions whether it is really worth the ROI, yes return on investment;))

Can we really maintain 400+ combinations of browsers i.e. have at least 100 machines for example (hypothetically assuming that we install chrome, firefox, ie of some version on each machine)?
Can we keep updating the drivers i.e. chromedriver.exe, IEDriverServer.exe etc. on all those machines as and when the updates are released?
Can we keep the selenium grid processes (HUB and NODE JVM’s) up and running all the times?
What about memory leaks on the selenium grid jvm processes? What if it goes down, can we have a script that automatically restarts it detecting the down period
What about videos ? if we want to see the videos of execution
What about having a debugging environment where I can stop the script while it is executing and can investigate a potential bug
And so on…..

I think we get the point here right. It surely is possible to do all of the above if the number of combinations are very minimal, let’s just say 10 or below for example. We can maintain an internal environment of selenium grid and that is good enough. But if the business demands for more and more combinations, it is sometimes not feasible to maintain the cross-browser test infrastructure environment and at the same time be able to write test automation scripts and deal with all moving parts.

For example, we already talked extensively on how we can spin up a selenium grid in house / on premise in the section Selenium GRID. We talked about

All said, how nice it would be if we can have such a service that maintains the cross browser infrastructure for us and we ask the service to execute the scripts for us and send back the results?

Well, indeed there are many such cloud service these days available !

Some Solutions

There are couple of solutions that come to mind when we look for cross browser infrastructure. Obviously we are interested in access to infrastructure in first place, but more importantly, what platforms (selenium, appium, webdriver, continuous integration….) do they support, manual or automated access and what are all the attributes that really matter to us – end of day – how do they align with our existing continuous test automation principles. Below is a quick competitive analysis diagram (taken from stackio). You can obviously google search and read more from various forums.

Competitive Analysis

All of the above parameters we see in the diagram matters to some or the other organization, so it is extremely important for someone technical in the organization to do the competitive analysis and match it with the requirements that the organization has. Here is what I would suggest as steps for anyone looking to invest in the solutions above:

Sign up for a free trial on the websites and start playing with it (The next few sections ,we will have some videos to help you with that)
Generally you can use the above services – in a manual way OR automated way i.e. selenium/webdriver compatible
Once you have experimented enough with the services, you might have questions around private sites, data encryption etc. to satisfy the security needs of your organization
At this point, take help i.e. either call them up directly – them as in browser stack guys or sauce labs guys et al. They are generally very responsive because they are in growth mode
On this website, we provide training videos that help you understand where these kind of services will give you ROI and where it be too expensive in some cases

Read the next two sections on how we can automate cross browser testing