Iterative Development

“At last, my current practice of writing no automated tests has the blessing of science! See, TDD doesn’t do anything!” That’s how Fucci et al.‘s 2016 conference paper An External Replication on the Effects of Test-driven Development Using a Multi-site Blind Analysis Approach was introduced to me.

And, indeed, it concludes like so:

Despite adopting such countermeasures, aimed at reducing researchers’ bias [when replicating a prior, baseline study], we confirmed the baseline results: TDD does not affect testing effort, software external quality, and developers’ productivity.

Takeaways:

• All coding is debugging
  • Work in small steps
  • Stay grounded in observed outputs
  • Keep good notes (tests or REPL session logs)
• TDD won’t slow you down at steady state
  • Changing how you code to be more intentional and iterative might to start.
  • What will definitely slow you down: Learning your tooling and the impact of that iterative approach on the code you produce (expose those probe points for external testing! add indicator LEDs via assertions!)

Now that we’ve got the conclusion out of the way, keep reading to see how I got there. :)

Martin: It Tells Us Nothing: Keep on keeping on with TDD!

In the time since I’ve had “I should really write up my response to this” on my to-do list (a few months…), Bob Martin wrote up his take: TDD Doesn’t Work.

His article concludes the study made a distinction without difference and so naturally found no difference: Basically, that the folks involved were still practicing TDD, but where they actually wrote the tests – rather than conceived of and directed their coding efforts to them – was altered slightly.

Me: Iterativeness Is The Key!

But, check this:

Control treatment: the baseline experiment and its replication compared TDD to a really similar approach, labelled as TLD. Under this [sic] circumstances, we might be focusing on the incorrect part of development process (i.e., whether write tests first or not), and disregard the part of the process in which the a substantial effect might lie (i.e., the iterativeness of the process). Accordingly, the tasks used for both experiments were designed to fit the iterative nature of both treatments — i.e., isolate the process itself from the cognitive effort required to break down a complex problem into sub-problems. Pančur and Ciglarič [33] made a similar claim reporting the inconclusive results of a similar experiment. (bold emphasis added)

Fail Fast, Focus on Observed Outputs

The authors are on to something: I’ve seen people new to TDD stumble over learning to work in checkable baby steps. That iterative, fail-fast approach is where a lot of the time savings comes from; the other savings comes from being very intentional and focused about the concrete change you’re attempting to effect, or the specific knowledge you’re trying to elicit through experimentation. This same mindset also pays off in spades in debugging.

TDD or REPL, Just Use One!

We know you can learn this mindset via TDD, but a dev loop based around a REPL can work just as well. It ends up as TDD without the durable byproduct – once the session scrolls away, all those tests you wrote during bring-up are gone.

TDD Won’t Slow You Down

There’s another positive takeaway from the paper’s conclusion of no substantive difference:

• TDD does not affect testing effort, software external quality, and developers’ productivity. As long as you’re working iteratively and actually writing tests, you’re going to write working software and be as productive as you can under the circumstances.

If you don’t practice TDD already, fear not: TDD is not going to slow you down.

…But Learning to Use Test Frameworks Might, To Start

Getting the hang of working iteratively, and actually writing tests yourself, on the other hand – those will take a bit of time. And then save you far more over time.

Missing

The Long View: Maintenance Burden

It would be interesting to see experiments comparing TDD or ITLD (iterative, test-last development) and development where one of those two constraints is relaxed:

• Either drop the iterative bit, or
• drop the test-writing bit.

This was a small dev task, so I bet you’d see productivity go up as quality goes down. Put another way, I bet small tasks naturally lead people to ditch both of the things that make up TDD/ITLD.

This short scale approach doesn’t assess two real-world challenges that we are concerned with as software maintainers:

• Responding to changes over time.
• Not breaking stuff on timescales longer than a single workday in codebases larger than what one person can turn out in a workday.

Style: Internal Code Quality

The study also did not assess internal quality (is it readable, navigable, maintainable code?) in any way. Out of scope for their purposes, rather important for those of many professional developers, as the wide spread of PR-based code review processes and the flourishing of adjuvants like SwiftLint and Danger reflect.

Conclusion

Reread the intro. (Or, as the grimly satirical Scarfolk Council would say: “For more information, please reread.") I’m saving you having to read the abstract and then page to the conclusion this way. Go forth, and do the same for others. ;)