Debate on Automated Acceptance Testing
Part of agile software development is the practice of acceptance testing to ensure the appropriate business requirements, use cases, and ensure confidence in the delivered product. This process is commonly recommended to be automated so the tests can be easily run at a given time as well as be consistent. James Shore created and encouraged the agile community to use his automated testing framework (Fit) to create these tests.
The problem is that it was meant for the business to be creating these tests, not the developers, however, it turns out that the developers were tasked with creating them instead which is a “maintenance burden”.
Shore now believes that automating the acceptance testing is not a good idea, summarized by the following two points:
1. The real planned benefit of an automated acceptance tool, like Fit, was that the business folks (“customers”) would write executable examples themselves. History has shown that very rarely occurs. In a few cases testers do, but in the majority of cases these tests are written by the developers.
2. These tests often become a real maintenance burden, as they are slow and brittle, and often hard to refactor. On this point, that end-to-end “integration tests” present a higher cost than they are worth, JB Rainsberger has a great series of articles explaining his rational why.
What do you think?
Full article on InfoQ: http://bit.ly/a2002q
Selenium 1.x has absolutely failed me. When it comes to driving Ajax & DHTML features of the modern web application, Selenium has very inconsistent and perplexing behavior in all browsers. I am looking forward to trying Selenium 2 (now in beta), which is a merged effort between Google’s WebDriver and Selenium. I know WebDriver took a smarter approach to supporting the various browsers, but I haven’t heard much about the quality of the merged effort as yet. Old Selenium code will not be directly portable, but in many case it’ll be a job of simple search & replace.
@Greg, I too tried Selenium 1 once and did not like it all. Are you familiar with Fit?
this is tough. so if end-to-end “integration tests” present a higher cost than they are worth and acceptance test become a maintenance burden to keep up to date. Is there another option. I know that a handful of our devs love Selenium and that we are working towards building a library of scripts that take items through various stages of the workflow. The theory being that the entire test does not need to be rewritten if part of the workflow is changed.
@Kevin, I’ve recently started using Selenium again and I really like it. They’ve improved the system a lot.
In answer to your question, the objective is to find a happy balance. You want to cover as much ground as you can with automated tests that are going to replicate the normal user functions. For abnormal functions and edge cases, it’s ideal to have a QA team to pound the system. If not, then you have to test it yourself.