Avid’s QA team has chosen Selenium to satisfy its automated testing needs. It is a free, open source project that is well supported by a large userbase. With this popularity comes a great deal of online documentation and discussion; being able to draw from the collective experience of so many is one of its principle advantages. However, it is not without its limitations. For example, it does not support Flash at all. Further, it does not handle AJAX functionality as smoothly as one might hope. Nevertheless, Selenium has served us well thus far.
Development of our testing suite began by writing a framework library, intended to handle tasks such as reading configuration files, launching the Selenium server, executing test cases, writing to log files, emailing test summary reports, etc.
On top of that was written an API specific to the Ashley Madison site, in which functionality for interacting with the site is kept. Actions such as registering a new profile, logging in and out, subscribing, etc. are highly reusable in any number of test cases, so it makes sense for us to build an API rather than describe every step separately in each test case.
Finally we have a suite of individual test cases, wherein we describe specific site use scenarios. For example, one test case may register a new profile, login, buy a membership, and logout. Using the building blocks described in the API, we can play with any number of end user scenarios to verify that none of the steps along the way were broken. Test cases are run sequentially and without dependency on each other, so one test failing will not cause other tests to automatically fail.
The entire testing framework is setup on a spare machine that will automatically execute a smoke test suite after the dev team deploys code to a QA environment. A cron job also runs a full regression test suite nightly.
For the automated testing of some of our other properties, such as CougarLife and Established Men, we have chosen to use Selenium IDE, a Firefox extension. As with most decisions of this nature, this choice comes with several trade offs.
Perhaps the greatest advantage to using the IDE is the ease with which new test cases can be prototyped. Start its record feature, manually execute a user scenario, save it, and with minimal editing you have a completely reusable test case. This is particularly relevant when testing a new site feature; being able to quickly cover new functionality without resorting to a more involved API development process can leave more time for actually testing. Further, this ease of use puts the authoring of automated tests within reach for our entire team, whether or not they have significant expertise in programming.
Selenium IDE’s greatest advantage is also perhaps its greatest limitation. It is by nature a simple beast. The Selenese language it uses is not nearly as powerful as a full scripting language such as ruby. Particularly lacking is flow control of test execution, a shortcoming that can be mitigated somewhat with Flow Control, a Selenium IDE plug-in. Be warned however that extensive use of it can make for spaghetti code. Even so, most test scenarios on our sites do not involve a great deal of repetition, so the need for extensive flow control is minimal in our case.
We mentioned previously that we use the deprecated Selenium RC as opposed to its successor, Selenium 2.0/WebDriver. The principle reason for this relates to what was said at the outset: that one of Selenium’s greatest features is the amount of user generated documentation available online. However, most of this documentation is specific to the Selenium IDE and the older Selenium RC. Only a small fraction of what’s out there actually applies to newer Selenium 2.0/WebDriver. This alone is enough reason for us to stick with the older versions for now. Perhaps when the Selenium 2.0/WebDriver project and its documentation has matured somewhat, we will reassess this decision.
Ultimately, our test automation projects as we’ve developed them have served us well for the time invested. Indeed, the time it took to develop our tools to this point is trivial in comparison with the time saved smoke and regression testing all of our sites on a regular basis. They have also reduced the amount of repetition required by our manual testing processes, thus sparing the sanity of our QA team. Cheers to that!