History of a Large Test Automation Project using Selenium
There is much information available about how to begin automated UI testing projects. There is little information available about how to maintain successful, effective, long-term, large-scale UI testing projects.
Over the course of more than two years, my company Socialtext was able to grow a test automation project from a proof of concept of 400 test steps, run on demand, to nearly 10,000 test steps run automatically 24 hours a day, 7 days a week. This talk will cover test design, test architecture, test creation, test maintenance, and the project’s future steps.
There are any number of ways to go about designing User Interface (UI) tests. One can attempt to maximize code coverage; one can attempt to test for fixed defects; one can go element-by-element, screen-by-screen. Socialtext chose to design their UI tests as sets of interesting paths through the application. These paths were intended to exercise as many features of the application as possible. In retrospect, designing tests this way turned out to be an excellent choice for a number of reasons. The talk will provide illustrations illuminating this successful approach.
Socialtext has a highly customized framework wrapping Selenium-RC in Perl. Most of this framework is available from CPAN. This customization provides two critical abilities: the ability to keep test data as wiki pages, and thus able to be manipulated by all the tools available to the wiki; and the ability to create fixtures for common actions in the tests, thus saving testers from having to write thousands of extra steps to accomplish routine actions. The talk will demonstrate some fixtures available to test creators. If there is interest, we can also examine the source code for the fixtures.
Socialtext moved gradually from about 400 test steps to about 1000 test steps. At that point, having demonstrated the value of the automated tests, we wanted to really accelerate automating the backlog of UI tests. We moved from about 1000 test steps to about 4000 test steps by hiring an intern to spend a summer doing nothing but automating scripted tests. This turned out to be an excellent investment, but required careful preparation. The talk will show the difference between the manual test scripts given to the intern and the finished automated test data.
Socialtext found that at about 4000 test steps (for the product at that time), regression bugs released to production fell essentially to zero. At this point the automation effort moved to shoring up weak spots and to covering new features. Since the tests now took significantly longer to run, we wrapped the whole test suite in a simple Continuous-Integration-like script so that the tests run all the time, creating wiki pages describing individual test failures. The talk will show some of the nice points of that script, written in Ruby. Some of the source code for the CI-like script will be of interest.
Socialtext’s UI has undergone two significant overhauls since the inception of the automation project. The talk will address how we handled refactoring hundreds of tests with thousands of test steps when the underlying application changed radically (and had bugs!). If we have an internet connection, I can show bug reports in real time on a Socialtext wiki.
The talk will finish by discussing future directions for this successful project, including managing the testing in a multiuser/multiprocess fashion; large-scale refactoring, further integration via fixtures, and other aspects of the project.
- Large Scale Web Testing
- Selenium Frameworks
- UI Test Design
- UI Test Refactoring
- State of the Practice