When adding new features or making changes to an application, it’s important to ensure that the changes won’t break existing functionality. While you could manually run through the different scenarios, that would become a time-consuming task with a high chance of not testing every different aspect. Success in manual end-to-end (aka acceptance) testing has always inspired us to focus more on QA automation. We have used automated test tools like Selenium IDE in the past and had good test coverage for different features. Although Selenium’s recorded tests were helping us to some extent, it still had many limitations. Tests were hard coded and very specific to individual end-to-end use cases. They were not easily scalable and were getting hard to maintain. The challenge was growing with every new release of FoxyCart. So we decided to revisit our QA process to see how we can improve upon it.
Our Goals with Improving our Testing
Instead of writing huge amounts of tests that have a lot of maintainability cost, we wanted to write tests that are efficient, scalable, easy to write, easy to debug, and flexible enough to fit into different test scenarios. So with these five goals in mind, we started researching for a better solution for our needs. A few interesting options we got a chance to look into were PhantomJS, CasperJS, Sahi, Webdriver and Cucumber (for Ruby, which also inspired us for BDD).
Our Initial Attempts
We were amazed by the flexibility that CasperJS/PhantomJS offers. The power of a headless browser (phantomjs) and such an elegant API wrapper (casperjs) for controlling it, was a breeze. So we decided to use these tools for testing our next release. A week later, we had a good amount of test coverage that helped immensely in finding bugs for our next release. It had great flexibility, tests were easy to write and scalable but we noticed some inconsistency for our tests. A test that ran fine for the first few times, suddenly started to break. Especially if it’s a network issue or something else, it was hard to debug. Thus we were failing to achieve our “easy to debug” goal. Capturing a screenshot was helpful but was also very time consuming. There were some other ways to debug, for example you can even record a video of your tests by taking a lot of screenshots and running them with a specific frame rate using tools like ffmpeg. But it didn’t seem to be a very elegant solution. Another goal we were not able to achieve was efficiency. Browser testing comes with a cost of speed but there are ways, that I will talk about later, to minimise it.
We went back to Selenium, started to look at the Selenium webdriver and came across it’s PHP binding and Codeception’s php-webdriver. Being a PHP developer, a testing framework in PHP for acceptance tests sounded great! Another option we found in PHP for acceptance testing is Behat, a true BDD framework for PHP. The most inspiring thing about these frameworks for us is that your tests are not tightly coupled with a specific tool. You just write your tests, plug-in your favourite tool and run tests against them. Prefer PhantomJS over Webdriver ? No problem, just hook in your phantomJS extension/plugin and you are ready to go. You don’t have to worry about updating all of your test suites, which reduces maintainability cost to a great extent. Both are great frameworks and offer a lot of flexibility. We found codeception good for writing tests quickly for small projects. We have nothing bad to say about it, to be honest. It’s a great testing framework. In the end we chose Behat over Codeception because of it’s BDD style, which felt more natural to us. We would suggest to play with both and choose whatever feels better for your needs.
Behat was heavily inspired by Ruby’s Cucumber project. Since v3.0, Behat is considered an official Cucumber implementation in PHP, and is part of one big family of BDD tools. It is easy to learn, quick to use, and will put the fun back into your testing. Tests are easy to write, easy to debug and scalable. Test suites can be composed of one or many scenarios and the fun thing is, you can select from different browsers to use when running tests. You can write one scenario (where JS is not needed) against Goutte, PHPBrowser (or any other web scraper) and the test will run noticeably faster than running it on a real (or even headless) browser like phantomJS or Selenium server (which would load FireFox). Similarly, you can write another scenario to test against a real browser, if needed. It will make tests run a lot faster than running all of your test suites against a real browser, thus making your acceptance test more efficient. Also, having the ability to add logic against your different scenarios offers great flexibility. Behat and it’s gherkin syntax makes it possible to have a readable and up-to-date documentation of your codebase. Behat meets the requirements we were looking for, allowing us to have an efficient and easy to use testing suites, which allows us to make changes to our codebase with confidence.
Our Conclusion: Behat for BDD
Technologies are developing, so are techniques. Anything that feels good right now, may feel obsolete in future. But when it comes to testing, instead of waiting for the perfect solution, it’s best to pick what works best for you and start developing test suites. For our needs, we have chosen BDD and Behat for our acceptance tests and we are optimistic about it’s future.
