What are Frameworks?

A test automation framework is an integrated set of technologies, tools, processes and patterns brings organization of thought and clarity thereby simplifying and enabling a team perform Automation not only effectively but also efficiently. We would be focusing on the technology aspect of Framework on this website. I blogged about attributes of frameworks a while ago, however this section and tutorial is getting all the pieces together and connecting the dots. We have also talked about how do we choose an Automation solution at the beginning of Basic Tutorial and setting the stage and context. We touched upon frameworks there but more from the perspective of programming language, design patterns, process models et al. Here we would be focusing on the “design patterns”, otherwise sometimes referred to as technical frameworks in many contexts when we get tactical and implementation specific.

Among many things, a framework should model the application state and behavior [synchronous and asynchronous]

Why do we need one?

For the same reasons in the blog, I will try to keep it simple and move quickly to the coding aspects, however let’s spend some time on this because we need to know “why” we write the code we write in a framework.

1. Maintainability:  How do we ensure maintenance of scripts is optimal?  By that we mean, as we evolve by churning out feature files, step definitions, data and page objects, we need to have a strategy to be able to diagnose, version control and audit the artifacts produced. There should also be a tight couplingbetween the released version of source code and the corresponding set of acceptance tests which tested that released version.
2. Re-usability: What are the specific artifacts that can be re-used across feature teams? This is particularly helpful during integration testing when components/functions have to talk to each other. For example, data and page objects are re-usable artifacts
3. Scalability: How do we scale the acceptance tests and ultimately accelerate the feedback loop to verify/validate the software to release and market per deadlines?We can use Selenium GRID for parallel execution of tests. The necessary infrastructure required for a distributed parallel execution of tests needs to be discussed too. The technology needs for the same in the Technology section and any exceptions and dependencies on infrastructure (hardware/software) should be laid out.
4. Configurability: How do we configure the test automation framework and how much of it is convention vs. configuration?Environment configuration (switch environments),Logging (Detailed log to help debug)
5. Auditability: How do we keep track of work done and roll up metrics to serve QS dashboards or any other management required metrics. Reports (Historical vs. current), Auditability (System of record)

Which one to select?

Below are popular framework patterns in Test Automation space and I have tried my best to provide guidance to help select the framework based on situations.

1) Keyword Driven : Keywords are essentially blocks or functions that are named so that they can be consumed for a given functionality. At a simplest level, keyword takes in parameters and throws back some output. Keywords are written to abstract the complexity of repeatedly writing code. Think of Keywords as API’s that serve you (or your application) when called. In one of the frameworks I designed for a large company, Keywords.java was a Class file with list of Java methods with defined inputs and outputs. There is a very thin line between keywords and modules. We hear the term “modularization” or “modular framework” and so on and there is a intersection of spaces here.

So Keywords or modules would pretty much the same on this website. In the Ruby world, we can implement this kind of framework by defining modules and mixin them into strategic places. When we get into frameworks discussion, we will see that modules are crucial pieces to bake into the framework. Not exactly similar, but a module in Ruby is very close to an abstract class in Java, however there are differences too.

However Keywords/modules can get complicated too based  on data structures and layers in your test automation framework

2) Data Driven: If your applications is NOT too deep with pages , however each page can have scenarios that need to be tested with large datasets, you would want to write automation scripts with a focus on test data aka. data-driven. Tools like QTP already have excel sheet parsing etc that loops through rows and the same test case is executed for each data-set. It helps me to think in an easier way when I talk about excel for data driven, however please DO NOT think excel is the most optimum way to handle test data. In fact, excel is heavier than xml, yml or other data exchange formats. If you are at the beginning of designing a data-driven framework, go ahead and use a spreadsheet to manage your data, however as you mature and advance, try doing away with spreadsheets.

3) Page Object: This is a very popular pattern used these days for Test Automation Frameworks. If you think of a web based application as a network of pages that interact with each other AND there are large number of pages in your application that can change nature frequently, this pattern is the most favorable one to use.

Page-objects (or classes) are defined for each page, where a page contains the html locator information and also the methods that can operate on those page elements. Though we say page-objects (aka. classes) are written for each page, there are some nuances in terms of how we model a Web page into a page-object. A network of page-objects is supposed to model the state of the web application ultimately, however with asynchronous web applications, where parts of page get refreshed (as opposed to full web page. aka. DOM), we have to handle ajax calls based on the javascript library. Anyways, will mention the caveats, pros and cons, situations in which web application (page html source) to page-object modeling becomes the most important piece in your framework etc, will be dealt in a separate post at the end of frameworks tutorial.The advantage of this pattern from the decision making point is:

• If a html locator information changes for an element, changing it in the page-object reflects across all the scripts – nice right
• Gives us ability to define keywords too as we can define methods inside page-objects
• We can define navigation logic between the pages too as a way to tie the links between the pages

4) Hybrid: A Hybrid model is permutations and combinations of the above three frameworks described