June 12, 2018

Testing Rant

(It looks like I wrote most of this back in 2015. I've dusted it off and present it for your edutainment.)

It's time to rant about a technology issue close to my heart. I realize that the gold standard for technology rants has been set by Steve of Stevey's Drunken Blog Rants and the incomparable Zed Shaw. This means I should either swear often or only write after consuming significant quantities of alcoholic beverages. Ever the rebel, I'm going to dispense with both of these requirements and enjoy a nice cup of tea instead.

At Benevolent Employer, they are trying really hard to fix things. As is often the case with large companies which have been around for many decades, there is much to fix. I wholeheartedly endorse fixing stuff. The problem is that they are trying to fix testing, particularly unit testing, as if it was a standalone thing that they could upgrade by creating enough PowerPoint slides and new policies. What they need to fix first is their understanding of what testing is and then they can proceed on to the step of testing better.

Companies with I.S. divisions generally have a department within I.S. called Quality Assurance. Interestingly, they usually never speak the full name, referring to them instead as QA. Thus testing is seen as a QA thing. This is unfortunate because if they called them by their full name, they might realize that the department is wrongly named. It is my careful observation that QA departments perform no Quality Assurance. Rather, they are acting as Quality Control and there is a significant difference. Quality Control is something that manufacturing companies use. Quality Control is the department that ensures the company's manufactured product meets all of its stated specifications. That the product is the correct size or weight within tolerances and that it operates in the correct manner before it leaves the factory and is shipped to customers. This is the exact role that most QA department fulfill, that last ditch effort to catch bad code or wrong system functionalities before they are delivered to the users. Don't get me wrong here, a certain amount of Quality Control is appropriate, as it helps to catch issues in the manufacturing process and the production line can be halted and all machines and tooling can be inspected to bring them back within tolerances. But Quality Control is not, and will never be, the same thing as Quality Assurance.

True Quality Assurance takes place at the other end of the production line. Quality Assurance is involved with the creation of the process of manufacturing with the explicit goal of eliminating future problems by designing them out. Their department name describes their objective well, they are to assure that quality is built-in to the process that they help design.

I'm sure by this point you are ready to burst and explain to me that unit testing is not part of the Quality Assurance process. To which I say that when Quality Assurance is allowed to help assure quality, then unit testing is certainly within their focus. Most people do not see Quality Assurance as a highly skilled technical role. This is unfortunate and it brings negative consequences. The most obvious negative consequence is that the Quality Assurance department is then not expected to handle skilled technical actions and so is not staffed with skilled technical people. Further, because Quality Assurance people are seen as non-technical, and thereby less expensive than "real" technical people, the tendency is to pull folks in as contractors when you need them and then let them go after the project needs have been completed. This sets in place a rotating door effect in the Quality Assurance department and guarantees that the current staff will be inexperienced in the ways of your company and the context that the system they are testing runs within. And any system context and knowledge they do somehow pick up will leave with them after the project and be promptly forgotten.

And having said all of that, just changing the operating mode of your Quality Assurance department is not the answer. It's part of the answer, but not the whole answer. The real answer is that you need a culture of excellence. And you can only create a culture with people, ideas and examples. No amount of unit tests will ever create a culture of testing. The good news is that a culture of excellence will inspire many and excellent unit tests!

Creating a culture of excellence seems to be difficult for many large companies. They have lost their roots and have exchanged agility (in the classic sense) for economies of scale. Economies of scale by their very definition preclude agility and necessitate strict adherence to written procedures. This fits the Quality Control model far better than the Quality Assurance model of testing.

Tags: Observations