DevOps Zone is brought to you in partnership with:

I’ve been involved in everything from hands-on systems administration, managing systems implementation teams, launching new data centers, running project management and service delivery teams. For the last 18 months I have managed the build and release management organization which evolved into the leadership of a software engineering team for our devops automation platform; supporting 2,100 test environment deployments and 300 production releases monthly. Paul has posted 4 posts at DZone. View Full User Profile

Why DevOps is Doomed! Ops teams are lost! (1 of 3)

01.17.2012
| 7953 views |
  • submit to reddit
The problem between dev and ops is primarily a terminology, communication and respect problem resulting in poor operational support.  The two organizations say common things backed by different definitions that are not in agreement. For example, would ops define an “application” in Puppet the same way dev would define an “application” in Hudson? If not, how would you automate or even communicate between the two for automated application deployments? Dev and Ops really have no concept of each other’s world, yet they assume the other side understands their view, or they expect that the other side should understand their view.

I love the concept of DevOps and I am very optimistic about the movement’s value. However, I’m also very concerned about traditional IT leadership’s capacity to focus on the right goals to make DevOps successful. Bridging development and operations is NOT about dev teams utilizing a continuous integration tool like Hudson or Bamboo. And it’s NOT about ops teams standing up a configuration management tools like Puppet or Chef. Both may be needed for your automation efforts, but DevOps is about bringing dev and ops teams together so people and tools from both realms are communicating with common terminology, data sources and objectives. As always, communicating and working together for a common goal is the challenge!

  • Developers tend to think infrastructure is pretty straightforward. “I can stand up a server at Amazon in seconds. These clowns at work take forever with the simplest requests.”
  • Systems Administrators tend to expect developers to understand the infrastructure their applications run in. “The developer said it worked on his dev server, so obviously we screwed it up in production. The dumbass doesn’t understand firewalls or our company’s network.”

On average, developers know application code architecture and think they know systems architecture, but they DO NOT. On average, systems and network administrators have good diversity and know a lot of different infrastructure disciplines, and think they know application code architecture, but they DO NOT.

So why would DevOps be doomed for failure?

Web applications, services architecture and cloud providers have destroyed any hope of success for the traditional IT leadership sold on yesterday’s operational support model. There has to be a fundamental change to recognize that systems and applications are no longer static, documented operational models; they are dynamic release-time architectures. And there has to be a systematic way for dev teams to communicate application architectures so ops teams understand them.

Have you ever been asked to document application dependencies? If so, could you? If so, how long was it valid? Documenting a traditional three-tiered application is pretty easy. Documenting an application in a service-oriented architecture is only valid until the next code release  –As each release may utilize a new service end-point, dependent on a new network segment, dependent on a new database, dependent on a new data center in a different region. Good luck on managing the relationships for your ops teams!

Application designs no longer have a universal hierarchy; the diversity and rate of change can not be easily modeled in a traditional database schema. Enterprise IT tools used to manage the environment provide little help as they expect a static hierarchical application model. ITIL and service catalog implementations also tend to expect a static hierarchical application model. The three-tiered app is gone with the introduction of web application, service architectures and cloud providers. It’s game over if you can’t define your applications, model it, and use that same data to automate the build, deployment and operations life cycle.

The Bottom Line

App maps look like a circuit board.

Operations teams are lost and have no idea what an application looks like, how to model it, or how to support it. Nor have traditional enterprise IT solutions provided the tools to help model the web app and cloud era. Today’s dependency maps look like circuit boards.  If you zoom in, you only see some components of your applications dependencies.  If you zoom out, you see the circuit board but can’t read or understand any details.

Let’s say your web application renders a page. For that simple transaction, your application calls multiple service applications, each with multiple endpoints, each with multiple database dependencies.  Some databases may be dependent on nightly ETL jobs to provide valid data for your functionality.  Maybe the UI is rendered by a separate UI platform with its own application, service dependencies and databases.  Now, let’s say the relevant applications, services, and databases are developed by five different dev teams across three different states.

An event: some functionality in your application fails intermittently.  How does your ops team troubleshoot the problem and resolve it?  Is the “application” just the part your dev team developed, or is the application the whole “circuit board” of dependencies?  Can your app be described effectively in a knowledgebase, KB article, or wiki site?  Can the “circuit board” be effectively described in a CMDB or support tools?  If so, who out of the five dev teams is accountable for maintaining changes to it?  Is your ops team relegated to calling in subject matter experts from each team for troubleshooting?  Is your ops team able to be effective without a clear understanding of the application?

To be successful, we have to enable our ops teams to manage the dynamic changes and complexity of today’s applications. Manual communication processes will fail, so we need to redefine the minimum bar for “automation.” Systems Administrators creating a bunch a scripts and standing up Puppet or Chef is not automation. Developers using Hudson or Bamboo for continuous integration builds is not automation.  Automation has to link the application, build, and configuration management together.

  • “Automation” needs to be an architecture platform, not an individual tool or effort.
  • Automation “platforms” must bridge the technical communication gap between development and operational lifecycle tools, thus enabling organizational DevOps efforts.

The key is establishing common data models and service architectures that enables the automation and a common communication language at a very technical level. If you have been following Willie’s posts on skydingo.com, then it should be clear why we think a CMDB architecture using an unstructured NoSQL technology like Neo4j is so valuable:  Why I’m pretty excited about using NEO4J for a CMDB backend.

In part 2 of this series I will illustrate an application example providing details on how it lacks hierarchical structure, and why the term “application” creates so many problems for DevOps in enterprise organizations.  Then I’ll describe how we are working to solve the problem with our automation platform.

Source:  http://skydingo.com/blog/?p=472

Published at DZone with permission of its author, Paul Jenson.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Jilles Van Gurp replied on Tue, 2012/01/17 - 8:51am

The nice thing about the devops movement is that it is about tearing down the often artificial barriers between developers and operations people. Being in different departments and being accountable for different, often conflicting goals can lead to strange situations. For example the ops team is incentivized to resist any form of risk, like deploying a new version of an application that delivers business value that is likely to come with a minor risk attached. If they do their job, the devs get the credit. If something goes wrong, everybody is looking to them to fix it. So, a minor risk can easily and up blocking an important change. So, people get frustrated and things escalate to management. Do that a couple of times and you have a classic us-them situation where both sides resist the other.

This can lead to situations where the devs take things into their own hands, often in an amateurish fashion. This creates a lot of duplicate work. Often this is the seed for a devops situation to emerge because sooner or later the devs will get somebody with a clue to manage things for them. 

I've had good and bad experiences in both camps. The bottom line for me is that there needs to be a level of mutual trust and understanding about each other's goals. Most importantly, those goals need to be aligned. Ops is not just about keeping things running. It's also about making sure things get delivered. It is about supporting developer teams doing that.  

At least where I work, there are some weird tradeoffs being made. For example, we are running an ancient version of tomcat with known, and long fixed bugs (including a few that would allow people to bring down a server), because ops insists on having a "tested" version of the damn thing. Worse it is running on an ancient version of centos for the same reason. From a developer point of view, updating either to their latest versions is a no brainer (low risk, obvious benefits). From an ops point of view it is low priority work with a pretty high impact on a full backlog. So, it doesn't get done. Three years and counting now.

This kind of conservatism is sometimes appropriate but when you have a lot of it, it starts to stiffle progress.  The whole point about amazon is that it is a managed environment that costs a some dollars but that is much more customer oriented than internal ops teams. So, amazon and a few good devops people can deliver a lot of value in situations where a traditional ops team is basically moving around deck chairs on a sinking titanic rather than delivering value. 

Willie Wheeler replied on Tue, 2012/01/17 - 11:49pm in response to: Jilles Van Gurp

Hi Jilles. These are great insights that you and Paul are sharing.

A lot of people are still trying to figure out exactly what devops is. It is a mistake to think of it as developers learning to become great sysadmins and vice versa. While it's great for people to have awareness of what happens in adjacent areas, it makes much more sense for me to focus on being a great developer instead of being half-baked at dev and half-baked at system administration.

Instead devops is exactly as you say--move away from misaligned objectives (business and development wanting to push, push, push and ops wanting to throttle change) toward alignment around delivery in a rapid but predictable fashion. The team wins or loses as one. The best way to achieve both desiderata together is to get serious about configuration management and automation, and that requires both developers and ops.

Matt Watson replied on Wed, 2012/08/29 - 8:34pm

All the things you describe are so true. Developers and System admins have different views of things and usually don't get along very well. 

 I started Stackify to try and solve a lot of the problems you describe. We can give developers more visibility to their applications and production servers. This helps developers do a lot better job at troubleshooting problems with their applications and being more involved in operations. We can also tell the system admins what applications are installed on their servers. 

 We fill that hole that happens after an application is deployed. The developers no longer have visibility or access to their apps, but yet they ultimiately are responsible for ensuring they work and fixing problems with them. We have basically invented Devops Support.

 Check out our website to learn more: http://www.stackify.com

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.