Fitnesse using Watir
No automation framework is perfect for everyone, and Watir is no exception. There are a few public, open-source automation frameworks that people can adopt, and follow as a standard. They may also require adopting certain methods of automation to use them such as keyword-driven automation, etc. Those who don’t use these, follow one of two paths: Some don’t use frameworks at all, relying only on some commercial test automation tool. Perhaps because their automation needs are simple or they don’t have staff with expertise in implementing a framework for their automation.
Spigot uses several proven open-source and internal technologies. In such cases, there will always be design and implementation work to get things going, but a properly designed framework will minimize the amount of effort required for changes.
Spigot attempts to address all these issues by following these traditional design patterns:
A barebones, minimalistic framework, where most of the automation “heavy-lifting” is actually done by the various components within the framework.
Components of the framework are viewed as generic black box items. The framework doesn’t care what they do or how they work, as long as they are callable by the framework. Therefore, these components can be other automation tools that we use.
Components can run standalone, apart from the framework, as needed. This makes integrating an external tool into the framework more feasible.
ruby test runner [local]
SQAScriptRunner leveraging objective reference [grid]
Reporting (and notifications) is kept separate from the primary framework. In fact, the reporting piece is considered a component of the framework and optional. Therefore, one could use the framework for automation without any reporting. And changes to reporting do not directly affect the framework. You can also implement as many reporting components as needed; there is no mandate for just one reporting component. Reporting can also be bundled with other parts rather than be a standalone component. The spigot has two main reporting features. 1) Local, and 2) Grid.
No requirement for (internal) I/O formats of the components. You may also use whatever desirable methods you choose (keyword-driven automation (cucumber, fittnesse, excel, data-driven automation, cucumber scenarios, etc.). The only exception is that the component must be able to communicate with the framework via the defined framework I/O format and master test harness component invocation format.
Defines an extensible standard format for automation I/O in terms of common shared data (build version # and other build info, the system under test info, etc.) and output format of test results. The standardization makes it easier to collect and report results and has one central repository for data used by all components of the framework.
In a nutshell, Spigot is a framework that is highly abstracted (in implementation), loosely coupled, yet cohesive by structuring the (test) process into self-contained atomic components that are independent of each other but share a conventional data store and follow a standard reporting format. Additionally, Spigot can easily be extended to incorporate new behavior without modifying existing code. While the framework is intended for test automation, it can be adapted for other uses like software build automation, continuous integration testing, and other non-test automation tasks. The next section describes the framework architecture.
Framework Architecture
Spigot is in essence, a collection of components. See Figure 1 for a graphical representation of the Spigot component architecture (shown excluding the system under test).
A “driver” or master test harness (to execute subsystems, the “other” components)
The global data store (accessible by all subsystems)
Master reporting engine (to collect and report on results from all subsystems, and is yet another subsystem itself)
(Test) subsystems, each being self-contained (can be executed stand-alone, interface with the system(s) under test, and ideally should do its own test result reporting)
Support tools (interoperability wrappers, or tools for manual intervention, test progress monitoring, etc.)
Figure 1 — Spigot Architecture
Master Test Harness (Fittnesse\Spigot\Cucumber > Test::Unit — Ruby)
The master test harness is responsible for driving or managing all test subsystems and reporting engines that together make up the complete test process. To keep the framework loosely coupled, the master test harness is only responsible for starting and synchronizing the execution of test subsystems. With that in mind, the master harness should have no interaction (or minimal, if needed) with the global data store. The global data store is for use by the test subsystems.
Because not all tests require test results reporting (or if the framework was used for other purposes), the master reporting engine is optional and separate from the master test harness. In fact, the master reporting engine, itself is to be a test subsystem that is driven by the master test harness, as needed.
The master harness calls the individual test subsystems by reading from tabular data (table-driven automation). This data may be from a fit file, the global data store, or from a database query, etc. For any subsystem to be used with the framework, it must be callable, with parameters defined in a table row or record. The master harness input can also apply to the master reporting engine.
Best Practices for Script Design
The desired philosophy is to break down scripts into atomic, self-contained units of functionality to perform automation tasks. This approach advocates breadth in component design (over depth of features within a component (i.e. add more features and steps to a script for it to do more tasks). It is understandable that the breadth-wise approach is best suited for small and straightforward automation call flows and could be messy to maintain for complex automation that has long steps. So there will always be a trade-off. In those cases, Spigot recommends gravitating towards breadth in components wherever feasible. The reasoning is that it will always be easier to swap or “stub” out self-contained scripts in an automation call flow than to swap out scripts requiring nested dependencies, and it is always easier to read and navigate through 100 lines of steps in a self-contained script than to do the same for a 100 steps/methods within a 10,000 lines steps/methods. In this respect, we equate a module/component to a script, and when you add features, don’t bloat up your script, but instead create more libraries as needed. However, it is up to you to define how many feature additions to add to a script before you need to create a new script.