Q & A with Jez Humble: Continuous Delivery

After reading Jez Humble & David Farley's book "Continuous Delivery" Jesper Boeg, Trifork's Agile coach, had a chance to interview Jez Humble:

 

Q: Continuous Delivery has gained quite a lot of attention during the last 1-2 years and I am guessing a lot of people have bought the book you wrote together with Dave (David Farley, ed.). Are we beginning to see this attention turn into real change or is it still mostly a theoretical concept that people like but have difficulty implementing in real life?

A: It's definitely not just theoretical. Of course, many forward-thinking organizations such as Amazon, Facebook, Etsy, Flickr and so forth have been using these techniques for some time. But the book was written based on experiences that Dave, me, and our colleagues at ThoughtWorks had on a number of projects, some of them mission critical systems in complex brownfield environments. So it's all been proven to work. I think what the book contributes is to tie everything together and give it a name, and hopefully also to show that you don't need to be a start-up to apply patterns like the deployment pipeline - they work at scale too.
 
Q: Lean Startup, DevOps and Continuous Delivery are all concepts that have been getting a lot of attention lately and many people seem to use different words to describe the same concepts. How does DevOps, Continous Delivery and Lean Startup fit together are they different views on the same concept?

A: There's a bunch of overlap between these movements, but they have different goals and target audiences. Continuous Delivery is focussed on business outcomes - faster responsiveness to customers and lower risk releases - which really appeal quite broadly. It's something you can talk about to everybody from the CEO of a Fortune 500 company to a DBA. It applies to embedded systems, software products, or web-based systems. DevOps meanwhile is primarily about improving the way we build and run web-based systems. If you want to achieve continuous delivery and you're working on that kind of system, you will definitely need to consider the kinds of things we focus on in the DevOps community - infrastructure as code, monitoring, better collaboration between dev, test and ops - in order to succeed, and we discuss these issues in the book. But my guess is, for better or worse, you'll have more luck talking about continuous delivery to the CEO than DevOps. Meanwhile the lean startup movement is focussed on innovating efficiently through getting feedback as fast as possible from customers - and to do that, you need to use the technical practices continuous delivery discusses. I think what unites all of us is a focus on those technical practices - things like continuous integration, the importance of comprehensive configuration management, and automation of the build, test, provisioning and deployment process.
 
Q: What skills do you need to establish continuous delivery in your organization and what should you do in case you haven't got them?

A: I really think that all you need is smart people who are focussed relentlessly on their customers - intellectually curious, always trying to improve things. With those kind of people, you can do anything, and without enough of them, you will get nowhere. You also need the kind of organization that rewards that kind of activity. Unfortunately these two ingredients are rather hard to find in real life. The good news is that people and organizations can change - I often say that continuous improvement, through the vehicle of regular retrospectives, is the most important lean / agile process. Every week look at what you're doing - what could we do to make things better? How could we run an experiment to test these ideas? Then come back next week and see what the results of the experiments were. Usually the biggest impediment is getting the time to do retrospectives regularly, and the slack time to run the experiments you need to.

Q: Some refer to continuous delivery as 2. Generation Agile. What is the reason for this and is there some truths in that statement?

A: In many ways I don't think there's anything new about the idea of continuous delivery. Dave has been active in the agile movement from before it was a movement, I am heavily influenced by XP, and much of what we say is just building on that. I think what's changed though is that in the last ten years the patterns and tools that enable you to achieve that original goal - "continuous delivery of valuable software" - have emerged and made it possible to achieve that original vision, even at scale in complex, heterogeneous environments. I think there is a way in which that statement is true though - over the last few years Scrum has really been the popular face of Agile, and - while I have no problem with Scrum - it's not the whole story (and the Scrum people would acknowledge that). To achieve the goals of continuous delivery you have to focus on the basic technical practices - continuous integration, automated testing, comprehensive configuration management. All the movements you mention above are united on the central importance of these technical practices, and of creating a culture in which everyone works together to harness them in order to get rapid feedback so as to make sure what we're doing is valuable. We've seen too many failed "agile adoptions" where organizations just ended up doing cargo cult agile - stand-up meetings, planning meetings, giving people funky new titles - but nothing really changed. People were still working on death-march multi-year projects that delivered poor quality software that didn't deliver the expected return on investment. That's simply not acceptable. There's another way to do agile adoption - focus on your definition of done. Make sure you can actually get work done - meaning you are getting real feedback based on working, releasable software, not that your story was "dev complete" - rapidly, repeatably and reliably.

 

Q: In your book you state that all companies are able to do continuous delivery if they want to. Is that really true or are there situations where companies are dysfunctional either in terms of culture or technical issues making it close to impossible.

A: I honestly think it's true that any organization can do continuous delivery. But there's no doubt it might be very hard, depending on your starting point. Yuriy Zubarev recently posted a hilarious "Cynical Agile and Scrum Dictionary" in which he defined "impediment" as "a manifestation of organizational culture". The good news is that you can move to continuous delivery incrementally - indeed, that's the only good way to do it, the worst thing you could do is start up a project to do a big-bang continuous delivery "implementation". If you're focussed on continuous improvement, you should find that over time you end up adopting many of the practices we discuss naturally. There's a book coming out this year ("A Practical Approach to Large-Scale Agile Development") by the folks who work on the HP LaserJet firmware who ended up doing continuous delivery - although of course they didn't call it that at the time - through exactly this process. 
 
Q: Ideally continuous delivery increased both quality, fitness for purpose, feedback, effectiveness and communication. What would you say is the most important improvement?

A: I think feedback is the most important - assuming you listen to it. Only your users can determine quality and fitness for purpose. They're not things you can calculate up front, you can only guess (that's really the whole premise of the agile movement). Jerry Weinberg once defined quality as "value to some person", which even applies to internal code quality (which is valuable to the developers who have to maintain and evolve the codebase). So really the only way to achieve the highest quality and best fitness for purpose is to optimize your feedback loop. One of the most important prerequisites to this is communication. If you need to raise a ticket and wait days to get an integrated testing environment, you're dead in the water, so you'll certainly have to work on that. But the primary thing you should be measuring is cycle time for your software delivery process.
 
Q: Reading your book it seems a daunting task to establish a continuous delivery workflow. How do you approach the situation in terms of "change management" when helping companies make the transition.

A: Our typical approach is to start by doing an assessment of their current state and where they want to be using a tool based on the maturity model in the last chapter of the book, and then help them put together a plan to get there. The key (and I think this is very generally applicable) is to focus on achieving measurable results in the space of a few weeks to a few months. Pick the low hanging fruit. Once that's gone, something else will be the low hanging fruit, so go and look for that (this is really just an application of what Weinberg calls "Rudy's Rutabaga Rule" - "Once you eliminate your number one problem, number two gets a promotion.") I would question your assertion that it's a daunting task to establish a CD workflow - if you know what you're doing and you're starting on a new project it's pretty easy. Most of the time we're not in that situation of course, but if you focus on one thing at a time it doesn't have to be daunting. Mary and Tom Poppendieck told me that they have been asking a number of companies how long it took them to achieve continuous delivery, and they often get the answer "about a year". The tools, patterns and practices are out there.
 
Q: There are a lot of technical aspects mentioned in your book and yet it seems that a lot of the talks we hear at conferences mainly deal with the cultural or more abstract value focused aspects. What do you think is the reason for this?

A: I've seen plenty of talks and discussion around the technical aspects too. I suggest signing up to the devops and devops toolchain mailing list on Google Groups, or going to one of the DevOpsDays or CITCON conferences - there's plenty of meaty technical discussion to be had. There are definitely some areas where I don't see enough material: automated testing, especially automated acceptance testing which some of the agile luminaries (mistakenly, I believe) gave up as a lost cause, and continuous delivery with systems that talk to COTS such as SAP or the Oracle packages for example. I have to say though that the cultural stuff can't be ignored. For example, people often complain it's hard to create maintainable suites of automated tests, which is true. The only way I've seen it done successfully is when teams practice TDD/BDD (including ATDD). But TDD is still not very widespread - in fact, I often ask people who come to my talks how many practice TDD, and usually it's around 25% of the audience (even in Silicon Valley, where I'd expect it to be standard practice). I find that horrifying, and fundamentally I believe it's a cultural problem, because this is 2012 for goodness sake - we know that TDD works when it's done right. It's not like some big secret you have to be initiated into - there's more than enough good material out there on how to do it right.
 
Q: Is there one specific technical aspect of continuous delivery that companies find hard to overcome or is every situation different?

A: Of course every company is different. However there are impediments that are very common, such as people working on branches that don't regularly get integrated into mainline, working on systems whose architecture makes it difficult to test the software without an expensive integrated environment, and working in organizations where it's slow and expensive to get that kind of integrated environment. In these kinds of situations, you're certainly going to have to do some hard work. Often a good place to start is continuous integration - which means developers splitting up their work into smaller steps so they can check in to mainline at least once a day, having some fast, reliable automated tests that can run against the system to detect problems, and making sure that teams prioritize keeping the build green over completing work. You're also going to need to make changes to your architecture so you can do end-to-end automated acceptance tests without having to provision an expensive integrated environment. Again though, the key is to be incremental. Architectural changes can be made incrementally while checking in to mainline using the branch by abstraction pattern. Automated test coverage should be increased incrementally - I'll take ten tests that aren't flaky and reliably detect bugs over 1,000 tests that are so unreliable the developers completely ignore failures.