Technology, Cloud

IT is like salad dressing, sometimes you need to shake it up

Blog-post by,
HP Blogger

The simple act of dressing a salad demands chefs bend the laws of chemistry. They have to coax two substances that barely get along, oil and vinegar, to blend harmoniously. The only way to get them to play nicely is to apply vigorous agitation, and even then the ingredients separate the minute external energy is removed.

The CIOs role is not dissimilar to that of a master chef. Getting modern Web 2.0 applications to market quickly, efficiently and reliably requires a finely balanced blend of deeply skilled professionals. This is easier said than done, especially when their behaviors focus more on meeting the needs of their immediate team than the ultimate outcome. The challenge is that without a “master chef” coaxing them into harmony, and a system for keeping them that way, functionally aligned teams are more likely to split into their separate “tribes” - just like oil and vinegar - creating an unsavory mess.

A systematic approach to IT

We can’t allow this to continue. With industry experts estimating that over 95 percent of all capital projects rely on IT, the majority of IT leaders I speak with find themselves becoming an innovation bottleneck, preventing new initiatives from being delivered on time. The pressure to push under-baked projects into production is often overwhelming.

Forced to rush projects through the system, two in three of these projects fail to deliver against expected outcomes. The root cause is frequently poorly documented requirements, cursory attention to security and QA processes, and an operations team who’d be forgiven for thinking the rest of the organization was running the a secret society for all the notice they’d been given of the project’s production requirements. We need to perform better and I believe this starts by breaking down the silos between the governance, development, QA and operations functions.

I imagine some of you might be thinking, “Hey! Didn’t Adam Smith teach that by specializing and dividing labour a bucket of pins could be produced more efficiently than a group of generalists create the whole product themselves?”  Well, yes. I’m not suggesting that your development team should plan, build and support each application on their own any more than I would suggest that a Toyota worker should assemble an the entire car themselves.

What I am saying is that, just like the Toyota Production System, it’s important to recognize that IT delivery is a system. A system where up and down-stream activities impact the finished product so dramatically that we need a connected approach to goals, tools and KPIs that aligns and enables our teams, rather than divides them. 

The DevOps Movement

This isn't a new problem, but it is getting harder. I’ve spent the better part of the last two decades helping organizations transform by bringing all IT stakeholders to the table around an aligned set of processes, tools and a measurement system. Starting with Service Management and itSMF and then subsequently the governance, security and application delivery worlds, I've found that each group has its own specialized systems for thinking about its problems largely disconnected from the needs of the others.

The great news is that there are several grass roots movements working to bring together the "tribes" of IT, bringing the opportunity to clear the project backlogs at lower cost, higher quality and greater compliance. Backed by pragmatic standards and technology innovations such as model based automation, rich platform stacks and cloud services, I believe IT leaders have a career defining opportunity to transform IT delivery for the 21st century by changing the way they think about the application and service lifecycle.

The most vocal group by far would have to be the DevOps movement. First coined by Patrick Debois, the DevOps movement and principals aim to recognize and codify the practices and workflows of developers, QA and operations teams to facilitate the rapid build and automated deployment of higher quality applications.

Born in the web operations worlds of the likes of Flickr, Kaching (now Wealthfront), Netflix and their larger cousins, DevOps takes a systems thinking approach to IT delivery, including the full lifecycle of a service. Most powerfully DevOps acknowledgesthat the majority of IT projects are never “done” and instead are in a continuous state of change, which is one of the reasons you’ll often hear about Continuous Delivery, Agile and DevOps mentioned in the same breath. In a tip of the hat to the Toyota Production System and Eric Reis’ book The Lean Startup, you’ll even hear the term Lean IT thrown in for good measure.

Breaking Down Silos In The Real World

 A great example of the pay-off of breaking down silos was demonstrated to me by a customer and friend who heads up Systems Management at a major health insurance provider. In his 20 years in IT, he’s been both a developer and operations leader. It’s fair to say there’s little he doesn’t know about building and monitoring mission critical, composite applications.

He recently shared how he’d been able to reduce the number of production incidents by 90 percent. Even better, when problems do occur he now only requires half the number of staff and a fifth of the time to isolate and resolve the problems than in the past.

He shared what I thought were three actionable principals that work for him:

  • Operations has a voice in requirements - it’s important that the supportability of the application be part of its non-functional requirements. The Operations teams are best positioned to form those requirements.
  • App development and QA should be part of operations escalations - this helps to build a shared understanding of real-world use cases for fault isolation and diagnostics.
  • Establish common tooling between the two - in his case he focused on common synthetic transaction and load generators, diagnostics and configuration management tools. This not only prevented his development and operations teams doubling up on tools that essentially did the same thing, but facilitated a common bond between Dev and Ops and that helped bridge what was previously a gap.


You get what you inspect

What makes DevOps such a powerful concept is that it brings the voice of operations into the development and requirements process.  It acknowledges and formalizes the requirements of the developer as a customer of operations. And most important, it anticipates the high velocity, high rate-of-change enterprise that I believe will be IT’s new normal. However, I’ve learned the hard way that “you get what you inspect, not what you expect” - blending two teams that often define themselves by their differences requires more than new tools and processes. IT leaders have to establish KPI’s that drive the right behavior across as well as within functional IT teams, for example, looking at unauthorized changes for signs that governance has lapsed.

If you’ve not already looked into DevOps, I recommend that you do. However, it’s worth noting that as a grass roots movement, the DevOps community are wary of vendors misappropriating the term and how to refer to DevOps is itself a subject of much heated discussion, Cameron Haight from Gartner provides an excellent objective summary. The one thing everyone does agrees on is that establishing a DevOps model is not a problem that can be solved by tools alone, a principal I agree with wholeheartedly.

If you’ve had experience, positive or negative, bringing together development, QA and operations, I’m sure everyone would like to hear more.  If nothing else, it’s food for thought!

Photo by sffubs -

(3) (3)

Would you like to comment on this content? Log in or Register.
Gene Kim 2 Points | Tue, 10/25/2011 - 18:26

A great summary of the need and the promise of DevOps, Paul!  Like you, I've been studying high and low performing IT organizations, and continually scratch my head at how the business often doesn't see (or refuses to see) the incredible value creation or destruction capability that IT has.

I believe that DevOps will help show how the business is culpable and enables the problem when IT is perceived to be in the way of business goals, or even actively destroys business value, either by preventing critical business projects from being completed, or disruption of operations.

Over the past couple of years, I've sought after and studied with the smartest practitioners and researchers I know from the domains of Lean, the Toyota Production System, Service Management, DevOps, Operations, Agile, Information Security and Compliance, Systems Thinkers and other exotic forms of jujitsu.

I have come to conclude that the underpinning of DevOps is the logical deriviation of where IT needs to go, based on the need for end-to-end systems thinking, destruction of siloes, lasting countermeasures that reduce WIP and decrease cycle times.

The benefit to the business and the IT industry could be as large as that of the manufacturing movement of the 1980s.  Idealistic?  Maybe.  But, for the sake of IT workers everywhere, I hope it's true and am actively helping in whatever way I can.

Gene Kim

Paul Muller 119 Points | Tue, 10/25/2011 - 20:51

Thanks Gene - I like the work your compatriots at the IT Performance Institute ( are doing to further the science of the debate. It would be great to have you share some orginal research here.

Warren Burns 11 Points | Mon, 10/24/2011 - 11:11

I like it.  I have been a fan of Devops as a service management concept for a while now and I can see it becoming more mainstream.  One trend I have noticed is that Cloud (platform or Infrastructure as a service) is playing a disruptive role in new projects.  If your infrastructure guys are either slow coming to the party or uncompetitive they are quickly bypassed in order to get a project off the ground.  It has placed a healthy amount of pressure on the preformance of this part of an IT organisation.  Developers are even coming to meetings with ideas they have thrown onto an instance in order to demonstrate their idea better.  I love the idea of disposable IT.

John Dodge 1535 Points | Mon, 10/24/2011 - 14:51

Warren, Have you checked out Patrick Debois' 118 slide presentation with the BBQ and slaughterhouse themes? It's pretty amazing...especially if you were fortunate enough to hear his presentation along with it. There's a bunch of Youtube videos with Patrick Debois.

Here's a DevOps primer... 

Paul Muller 119 Points | Fri, 10/21/2011 - 22:20

Here's a great Slideshare presentation on DevOps by Patrick that you might find fun.\
Douglas Stevenson 2 Points | Fri, 10/21/2011 - 12:30

I think that IT organizations NEED to be shaken up, stirred, and reformed. What has happened in many cases is that processes have been created in IT that are designed to inhibit change and new service delivery. This happens in part, because of several factors:

Lack of experience and maturity of Management

An unwillingness to change

Comfort and job security


Political positioning

I find that many CxOs and Sr. VPs have no clue whats happening in the organizations below them. Part of this is the propensity to manage to the green.  Another is the avoidance of negativity to protect political stature.  In some instances, you have VPs that are not native to Operations or Engineering.

I have considered doing a consulting service where I bring in staff augmentation in several departments, then reporting on the politics, silos, and roadblocks to change, back to the CIO or Sr. VP.  All under an NDA with the company. I'd be willing to bet, by inserting into the processes actual work, you see and can document the "long poles in the tent" and the dead end processes as they occur. And you give the Executive the ability to react armed with information they previously wouldn't have had.  Ongoing, they get a chance to see the effects of change without the "Manage to the Green" obfuscation.

In less than 6 months, the value this could create could be immense. 

Does your organization have a reputation for not delivering?  Do you have the "No can do" org?

I live in the ENSM/OSS space.  My danger flags I look for are:

Multiple products that do the same thing.

Multiple like product implementations owned by different organizations.

Bandaid implementations. (Something like Nagios deployed for a small subset of servers)

In house developed products that are clearly short of COTS products, yet do not die.

EOLed products still in Production

Products that are not integrated (Help Desk not integrated into Fault Management, etc.)

Partial implementations of COTS. 

Shelfware that is being supported but not used or implemented.

While this is not all inclusive, it is a lot more common than many may realize. People are the most important IT asset a company may have. A single poorly selected Manager can single handedly kill true creativity in the Engineering organizations, kill off process improvement, and thwart change and evolution, with little to no visibility upward.  And in the end, company performance suffers.


Paul Muller 119 Points | Fri, 10/21/2011 - 18:37

The point you make about people is spot on - people are "the killer app" of change.

John Dodge 1535 Points | Fri, 10/21/2011 - 13:27

Welcome to the ECF, Douglas. What do you mean by "manage to the green?" 

As Bill Laberis points out in this video, change is the hardest thing to do.
Douglas Stevenson 2 Points | Sat, 10/22/2011 - 02:29

By Manage to the Green I mean they only show good things northbound in the organization.  When a demo is provided, in many cases it is scripted or isolated such that management does not see problems.

It originally came from being mandated to provide demos of Enterprise Management applications and hiding the long term issues and conditions that could raise concerns for upper management.  In effect, we turned the maps and event displays green so as to not be negative with regards to management.

When I was at AOL, I worked under a VP who told us to make good mistakes.. But only once and manage the risks.  Brilliant vision - especially for Engineering. We did things that other organizations only dream of. 

Paul Muller 119 Points | Sun, 10/23/2011 - 01:01

And this is the one great challenge of dashboards and executive scorecards. The most common mistake is to want metrics to look good to your manager, but to want visbility of the red underneath you - problem is, you can't have one without the other. Excellent idea for a post or perhaps Skype round table?

John Dodge 1535 Points | Sat, 10/22/2011 - 12:48

The significance of Manage to the Green would be a great blog well as the lessons from your AOL experience. I could feature it on the home page and push it out on social media.

Pearl Zhu 90 Points | Thu, 10/20/2011 - 17:06

Hi, Paul, nice blog to well address the Systematic IT and DevOps methodology and principles, yes, many complaining about IT's slow delivery may cased by 20st century's heavy-weight multiple years IT projects, also because the inflexibile methodology or over hierachical structure, I think DevOps methodology intend to leverage the Agile with holistic view of IT delivery discipline, to overcome the silo and over-specilized project layout, and to enhance the communication both within the team, but also cross the teams, and with customers & partners as well. thanks