DevOps is a set of practices that combines software development (Dev) and information-technology operations (Ops) to shorten the systems development life cycle and provide continuous delivery with high software quality.
So Why DevOps?
The word “DevOps” was coined in 2009 by Patrick Debois, who became one of its gurus. The term was formed by combining “development” and “operations,” which provides a starting point for understanding exactly what people typically mean when they say “DevOps.” Notably, DevOps isn’t a process or a technology or a standard. Many devotees refer to DevOps as a “culture”—a viewpoint that New Relic favors. We also use the term “DevOps movement” when talking about topics such as adoption rates and trends for the future, and “DevOps environment” to refer to an IT organization that has adopted a DevOps culture.
So, to understand the need of this evolution of the Software Development Life Cycle (SDLC), we first must understand the motivation behind such a change. First, back in the day, there was the “Waterfall method” for the SDLC.
The Waterfall...
To avoid belaboring the issue, this methodology required many hours of research and planning to layout and crystal ball what would be needed to develop an application. Development time could be up to a year or beyond. Hence, in some cases, as evidenced by Nordstrom's business case.
Nordstrom’s development model still included a waterfall delivery method, big batch releases, and lots of shared services when it undertook rewriting its in-store clienteling application. When it finally launched the program two and a half years later in 2011, it was already irrelevant. "It was a big wake up call for us as an organization that we had to figure out a way to deliver in this new context," says Courtney Kissler, vice president of e-commerce and store technologies.
Agile to the Rescue
Agile can be viewed as addressing communication gaps between customers and developers, where development efforts were continuously generated and subdivided into iterations, or “sprints” where feedback would be solicited after each iteration from the interim delivery.
Limitations of Agile
This was the key to getting code rapidly out the door while being mindful of the end-users needs. On the other hand, ignoring the conventional objections about this methodology, documentation, training, re-useable artifacts, etc. These items are at least addressable in this process; however, the overarching show stopper is the total exclusion of the operations side of the equation. Oddly enough, there are TWO (2) teams involved in a successful software delivery, the development team and Operations team.
Yes, the Operations team, the unsung heroes who actually test, inherit, and support the end product.
Sure, in Agile, the end-users get their say in the requirements, but the success may DOA in the deployment of said software.
A quick example is a developer uses a set up a set of libraries, in this case he uses PHP 5.6. Now when this program has to be given to someone else, an operations guy. The operations guy gets the code – places it in his system where he has all the same supporting code. However, due to the prevailing corporate policy of keeping upgrades and patches current for security reasons, his version of PHP is 7.0; hence, the code doesn’t run, so Operations sends it back faulting the code when it was only a version conflict of a third-party package.
In come DevOps
How do these disparate groups join forces? By subscribing to a common set of principles that transcends traditional discipline boundaries and roles, for example:
- Set expectations and priorities and the fundamental beliefs that guide them.
- Collaborate both within and between teams on problem solving.
- Automate common and repetitive processes to free up time for higher-level work.
- Integrate feedback into the work, measuring everything that is moved into production.
- Share the data with everyone involved to foster a more effective culture of working well together across different skills and specialized knowledge.
By the Numbers...
Adopting DevOps
Eighty percent of respondents have either already adopted a DevOps approach (47 percent of people) or plan to over the next two years (33 percent). That left just 20 percent saying they weren’t planning to change to DevOps in the next two years. The bigger the company, the more likely the organization is to move to DevOps -- 59 percent of companies with over 1,000 employees had already switched to the methodology.
According to DevOps.com, the adoption rate increased significantly from 2015 to 2016.
- 2015: 66 percent of organizations were adopting DevOps, 19 percent weren’t adopting DevOps, and 15 percent were undecided.
- 2016: 74 percent of organizations were adopting DevOps, 16 percent weren’t adopting DevOps, and 10 percent were undecided.
Don’t Take Our Word: 10 companies that use DevOps
1. Amazon
Within a year of Amazon's move to AWS, engineers were deploying code every 11.7 seconds, on average. The approach also reduced both the number and duration of outages, resulting in increased revenue.
2. Netflix
Netflix has continued its commitment to automation and open source, and today engineers deploy code thousands of times per day. This year, Netflix was unanimously selected for the JAX Special Jury Award, with JAXenter editor Coman Hamilton declaring, "The rate at which this entertainment game-changer has adopted new technologies and implemented them into its DevOps approach is setting new standards in IT."
3. Target
Inside Target, several groups have been evangelizing DevOps for years. According to technical architect Dan Cundiff, what "started out in small corners of development and infrastructure teams has since caught on like wildfire."
He's not exaggerating. These days DevOps not only powers the development of projects like Cartwheel, Target's mobile savings app, but also has transformed the organization's culture. Target now hosts DevOpsDays for its internal teams, featuring demos, open labs, lightning talks, breakout sessions, and guest keynotes. It also continues to spread the good word through the business community by sponsoring Minneapolis DevOpsDays meetups.
4. Walmart
While Walmart is the king of big box retailers in the American heartland, online it has always struggled in the shadow of Amazon. To gain ground, it assembled a cutting-edge team through several tech acquisitions and founded WalmartLabs, the retailer's technology innovation and development arm, in 2011.
WalmartLabs has taken a decidedly DevOps approach to its mission. It incorporated OneOps cloud-based technology, which automates and accelerates application deployment.
5. Nordstrom
Nordstrom's development model still included a waterfall delivery method, big batch releases, and lots of shared services when it undertook rewriting its in-store clienteling application. When it finally launched the program two and a half years later in 2011, it was already irrelevant. "It was a big wake up call for us as an organization that we had to figure out a way to deliver in this new context," says Courtney Kissler, vice president of e-commerce and store technologies.
Nordstrom's customer mobile app team was the groundbreaker for its DevOps makeover. After surfacing the reasons behind mobile's 22- to 28-week lead time, the team broke down the divide between dev and product support and organized squads around value. The company also migrated to continuous planning and moved to a single backlog of work. As a result, bugs went down, throughput went up, and releases went from twice per year to monthly. More importantly, Nordstrom realized these methods could work for any team, and it's continuing to apply them across the organization.
6. Facebook
Facebook helped change the way we think about software development. Many of the tenets it adopted early on, including code ownership, incremental changes, automation, and continuous improvement, were DevOps in all but name. Its approach has matured over the years, and it recently migrated its entire infrastructure and back-end IT to the Chef configuration management platform (and made some of its cookbooks available to the public).
Facebook's accelerated development lifecycle continues to reshape consumers' expectations of software. Its recently announced bi-weekly app updates effectively served notice that constant, rapid refreshes for mobile apps are the new normal, and any company that can't keep up risks getting left behind.
7. Etsy
For its first several years, Etsy struggled with slow, painful site updates that frequently caused the site to go down. In addition to frustrating visitors, any downtime impacted sales for Etsy's millions of users who sold goods through the online marketplace and risked driving them to a competitor.
With the help of a new technical management team, Etsy transitioned from its waterfall model, which produced four-hour full-site deployments twice weekly, to a more agile approach. Today, it has a fully automated deployment pipeline, and its continuous delivery practices have reportedly resulted in more than 50 deployments a day with fewer disruptions. And though Etsy has no DevOps group per se, its commitment to collaboration across teams has made the company a model of the DevOps framework.
8. Adobe
Adobe's DevOps transformation began five years ago when the company moved from packaged software to a cloud services model and was suddenly faced with making a continuous series of small software updates rather than big, semi-annual releases.
To maintain the required pace, Adobe uses CloudMunch's end-to-end DevOps platform to automate and manage its deployments. Because it integrates with a variety of software, developers can continue to use their preferred tools, and its multi-project view allows them to see how a change to any one Adobe product affects the others.
The move has enabled faster delivery and better product management, and according to the Wall Street Journal, Adobe has already been able to meet 60 percent more app development demand (subscription required).
9. Sony Pictures Entertainment
Sony Pictures Entertainment's Digital Media Group (DMG) faced significant challenges delivering a software system to manage entertainment assets for end users. Manual processes and other hurdles typically resulted in a months-long delay between completion of software development and delivery.
To smooth out this "last mile," DMG implemented an automated cloud delivery system composed of open source tools and SaaS solutions. Since adopting a continuous delivery model, DMG has cut down its months-long delivery time to just minutes. This allowed developers to focus on adding features and reduced idle resources and associated costs.
10. Fidelity Worldwide Investment
Like many enterprises, Fidelity Worldwide Investment had several business units developing software applications and was burdened with legacy release processes that placed huge demands on its teams. Apps were deployed manually across hundreds of servers, with each app requiring customization. Manually introduced errors frequently broke the process.
When it came time to develop a critical trading application with a firm launch date, the organization knew its error-prone manual process would jeopardize the project. Fidelity used the opportunity to embrace a DevOps approach and implement an automated software release framework that would enable it to meet the rollout schedule.
That solution resulted in more than $2.3 million per year in cost avoidance for that app alone. Since then, the Fidelity team has automated the release of dozens of applications, reducing release times from two to three days to one to two hours and decreasing test-team downtime. The process has also made it easier to display regulatory compliance and has enabled predictable release schedules that stakeholders can rely on.
Origins
Despite the mythical tone of some of the stories about its origins, DevOps was not created out of whole cloth. Rather, the seeds of DevOps were planted long ago and have been nurtured by forward-thinking IT experts in a number of disciplines. The primary antecedents of DevOps are:
Agile
Agile and DevOps serve complementary roles: several standard DevOps practices such as automated build and test, continuous integration, and continuous delivery originated in the Agile world, which dates (informally) to the 1990s, and formally to 2001. Agile can be viewed as addressing communication gaps between customers and developers, while DevOps addresses gaps between developers and IT operations / infrastructure.[24] Also, DevOps has focus on the deployment of developed software, whether it is developed via Agile or other methodologies.
ArchOps
ArchOps presents an extension for DevOps practice, starting from software architecture artifacts, instead of source code, for operation deployment. ArchOps states that architectural models are first-class entities in software development, deployment, and operations.
TestOps
Continuous delivery
Continuous delivery and DevOps have common goals and are often used in conjunction, but there are subtle differences.
While continuous delivery is focused on automating the processes in software delivery, DevOps also focuses on the organizational change to support great collaboration between the many functions involved.
DevOps and continuous delivery share a common background in agile methods and lean thinking: small and frequent changes with focused value to the end customer. Lean management and continuous delivery are fundamental to delivering value faster, in a sustainable way.
DataOps
The application of continuous delivery and DevOps to data analytics has been termed DataOps. DataOps seeks to integrate data engineering, data integration, data quality, data security, and data privacy with operations it applies principles from DevOps, Agile Development and the statistical process control, used in lean manufacturing, to improve the cycle time of extracting value from data analytics
SRE (Site-reliability engineering)
In 2003, Google developed site reliability engineering (SRE), an approach for releasing new features continuously into large-scale high-availability systems while maintaining high-quality end-user experience. While SRE predates the development of DevOps, they are generally viewed as being related to each other
WinOps
WinOps is the term used for DevOps practices for a Microsoft-centric view
How Does DevOps “Work”?
Like all cultures, DevOps incorporates many variations on the theme. However, most observers would agree that the following capabilities are common to virtually all DevOps cultures: collaboration, automation, continuous integration, continuous delivery, continuous testing, continuous monitoring, and rapid remediation.
Collaboration
Instead of pointing fingers at each other, development and IT operations work together (no, really). While the disconnect between these two groups was the impetus for its creation, DevOps extends far beyond the IT organization, because the need for collaboration extends to everyone with a stake in the delivery of software (not just between dev and ops, but all teams, including test, product management, and executives)
“The foundation of DevOps success is how well teams and individuals collaborate across the enterprise to get things done more rapidly, efficiently and effectively.”
—Tony Bradley, “Scaling Collaboration in DevOps,” DevOps.com
Automation
DevOps relies heavily on automation—and that means you need tools. Tools you build. Tools you buy. Open source tools. Proprietary tools. And those tools are not just scattered around the lab willy-nilly: DevOps relies on toolchains to automate large parts of the end-to-end software development and deployment process.
Caveat: Because DevOps tools are so amazingly awesome, there’s a tendency to see DevOps as just a collection of tools. While it’s true that DevOps relies on tools, DevOps is much more than that.
Continuous Integration
You usually find continuous integration in DevOps cultures because DevOps emerged from Agile culture, and continuous integration is a fundamental tenet of the Agile approach:
“A cornerstone of DevOps is continuous integration (CI), a technique designed and named by Grady Booch that continually merges source code updates from all developers on a team into a shared mainline. This continual merging prevents a developer’s local copy of a software project from drifting too far afield as new code is added by others, avoiding catastrophic merge conflicts.”
—Aaron Cois, “Continuous Integration in DevOps,” DevOps blog, Software Engineering Institute, Carnegie Mellon
The continuous integration principle of agile development has a cultural implication for the development group. Forcing developers to integrate their work with other developers’ work frequently—at least daily—exposes integration issues and conflicts much earlier than is the case with waterfall development. However, to achieve this benefit, developers have to communicate with each other much more frequently—a process that runs counter to the image of the solitary genius coder working for weeks or months on a module before she is “ready” to send it out in the world. That seed of open, frequent communication blooms in DevOps.
Continuous Testing
The testing piece of DevOps is easy to overlook—until you get burned. As Gartner puts it, “Given the rising cost and impact of software failures, you can’t afford to unleash a release that could disrupt the existing user experience or introduce new features that expose the organization to new security, reliability, or compliance risks.” While continuous integration and delivery get the lion’s share of the coverage, continuous testing is quietly finding its place as a critical piece of DevOps.
Continuous testing is not just a QA function; in fact, it starts in the development environment. The days are over when developers could simply throw the code over the wall to QA and say, “Have at it.” In a DevOps environment, quality is everyone’s job. Developers build quality into the code and provide test data sets. QA engineers configure automation test cases and the testing environment.
On the QA side, the big need is speed. After all, if the QA cycle takes days and weeks, you’re right back into a long, drawn out waterfall kind of schedule. Test engineers meet the challenge of quick turnaround not only by automating much of the test process but also redefining test methodologies:
“Continuous testing creates a central system of decision that helps you assess the business risk each application presents to your organization. Applied consistently, it guides development teams to meet business expectations and provides managers visibility to make informed trade-off decisions in order to optimize the business value of a release candidate.”
—Continuous Testing for IT Leaders, Parasoft
Although it may come as a surprise, the operations function has an important role to play in testing and QA. Operations can ensure that monitoring tools are in place and test environments are properly configured. They can participate in functional, load, stress, and leak tests and offer analysis based on their experience with similar applications running in production.
The payoff from continuous testing is well worth the effort. The test function in a DevOps environment helps developers to balance quality and speed. Using automated tools reduces the cost of testing and allows test engineers to leverage their time more effectively. Most important, continuous testing shortens test cycles by allowing integration testing earlier in the process.
Continuous testing also eliminates testing bottlenecks through virtualized, dependent services, and it simplifies the creation of virtualized test environments that can be easily deployed, shared, and updated as systems change. These capabilities reduce the cost of provisioning and maintaining test environments, and they shorten test cycle times by allowing integration testing earlier in the life cycle.
Continuous Delivery
The team at Amazon Web Services defines continuous delivery as a DevOps “software development practice where code changes are automatically built, tested, and prepared for a release to production. It expands upon continuous integration by deploying all code changes to a testing environment and/or a production environment after the build stage. When continuous delivery is implemented properly, developers will always have a deployment-ready build artifact that has passed through a standardized test process.
The actual release frequency can vary greatly depending on the company’s legacy and goals. High-performing organizations using DevOps achieve multiple deployments per day compared to medium performers who release between once per week and once per month.
Exactly what gets released varies as well. In some organizations, QA and operations triage potential releases: many go directly to users, some go back to development, and a few simply are not deployed at all. Other companies push everything that comes from developers out to users and count on real-time monitoring and rapid remediation to minimize the impact of the rare failure. And it’s important to note that because each update is smaller, the chance of any one of them causing a failure is significantly reduced.
Continuous Monitoring
Given the sheer number of releases in a continuous delivery shop, there’s no way to implement the kind of rigorous pre-release testing typically required in waterfall development approaches. In a DevOps environment, failures must be found and fixed in real time. How do you do that? A big part is continuous monitoring
With continuous monitoring, teams measure the performance and availability of software to improve stability. Continuous monitoring helps identify root causes of issues quickly to proactively prevent outages and minimize user issues. Some monitoring experts even advocate that the definition of a service must include monitoring—they see it as integral to service delivery.
Like testing, monitoring starts in development. The same tools that monitor the production environment can be employed in development to spot performance problems before they hit production.
Two kinds of monitoring are required for DevOps: server monitoring and application performance monitoring. Monitoring discussions quickly get down to tools discussions, because there is no effective monitoring without the proper tools. For a list of DevOps tools (and more DevOps-related content), visit the New Relic DevOps Hub.
Here are the top five reasons why the industry has been so quick to adopt DevOps principles:
1. Shorter Development Cycles, Faster Innovation
When development and operations teams are in separate silos, it’s usually difficult to tell if an application is ready for operations. When development teams simply turn over an application, the operations’ cycle times are extended needlessly.
With a combined development and operations team, applications are ready for use much more quickly. This is important, since companies succeed based on their ability to innovate faster than their competitors do. In fact, Kevin Murphy from Red Hat estimates that shorter development cycles translate to bringing an application to market 60 percent faster than with traditional approaches.
2. Reduced Deployment Failures, Rollbacks, and Time to Recover
Part of the reason teams experience deployment failures is due to programming defects. The shorter development cycles with DevOps promote more frequent code releases. This, in turn, makes it easier to spot code defects. Therefore, teams can reduce the number of deployment failures using agile programming principles that call for collaboration and modular programming. Rollbacks are similarly easier to manage because, when necessary, only some modules are affected.
Time to recover is an important issue, because some failure has to be expected. But recovery is much faster when the development and operations teams have been working together, exchanging ideas and accounting for both teams’ challenges during development.
3. Improved Communication and Collaboration
DevOps improves the software development culture. Combined teams are happier and more productive. The culture becomes focused on performance rather than individual goals. When the teams trust each other, they can experiment and innovate more effectively. The teams can focus on getting the product to market or into production, and their KPIs should be structured accordingly.
It’s no longer a matter of “turning over” the application to operations and waiting to see what happens. Operations doesn’t need to wait for a different team to troubleshoot and fix a problem. The process becomes increasingly seamless as all individuals work toward a common goal.
4. Increased Efficiencies
Increased efficiency helps to speed the development process and make it less prone to error. There are ways to automate DevOps tasks. Continuous integration servers automate the process of testing code, reducing the amount of manual work required. This means that software engineers can focus on completing tasks that can’t be automated.
Acceleration tools are another opportunity for increasing efficiency. For example:
- Scalable infrastructures, such as cloud-based platforms, increase the access the team has to hardware resources. As a result, testing and deployment operations speed up.
- Build acceleration tools can be used to compile code more quickly.
- Parallel workflows can be embedded into the continuous delivery chain to avoid delays; one team waits for another to complete its work.
- Using one environment avoids the useless task of transferring data between environments. This means you don’t have to use one environment for development, a different environment for testing, and a third for deployment.
5. Reduced Costs and IT Headcount
All of the DevOps benefits translate to reduced overall costs and IT headcount requirements. According to Kevin Murphy from Red Hat, DevOps development teams require 35 percent less IT staff and 30 percent lower IT costs.
Here, Be Ye Dragons...
Skills Are the Greatest Challenge
The greatest obstacle to moving to a DevOps way of working was a perceived lack of appropriate skills (cited by 21 percent of respondents), ahead of a lack of alignment between the development and operations teams (20 percent). Only nine percent said that their executive leadership didn’t support the shift. However, for those not planning to adopt DevOps 35 percent of respondents said a lack of awareness of the business benefits was the greatest challenge.
When it came to integrating database changes into the DevOps process, consistency was top of the list of obstacles, with 32 percent believing that synchronizing application and database changes would be the biggest challenge, followed by overcoming different approaches to application and database development (25 percent). Nineteen percent said preserving and protecting business critical data would be an issue.
Companies Are Integrating Applications and Databases
The old picture of companies made up of silos of developers and database administrators that don’t work together is far from the truth. Fifty-eight percent of companies said the integration between the two was either good (38 percent) or great (20 percent), meaning teams work together on either a planned or ad hoc basis. Just 12 percent classed integration as poor, with DBAs working in silos.
Whether they had adopted DevOps or not, three quarters (75 percent) of companies had teams that included developers who were working across databases and applications, with just 25 percent having dedicated database developers. In 60 percent of companies developers were responsible for building database deployment scripts, compared to 35 percent of DBAs, although in the majority (52 percent) of organizations DBAs deployed database changes to production systems.
Reducing Risk Key to Moving to DevOps
When asked what the major challenges were with traditional database deployment practices, over a quarter (26 percent) pointed to an increased risk of failed deployments or downtime when making changes. This was followed by slow development and deployment cycles and an inability to respond quickly to changing business requirements (both 18 percent). Sixteen percent cited poor visibility of the change management process as another issue. This shows the vital importance of fast, successful database changes and deployments to business today.
For companies with 5,001-10,000 employees slow development and release cycles were seen to be more of a drawback than application downtime, while for organizations with more than 10,000 employees, the inability to respond quickly to changing business requirements is deemed to be a slightly more significant downside.
With DevOps providing a way of overcoming these issues, no wonder it is gaining traction so quickly. Twenty-six percent said the main driver for adopting it would be to increase the speed of delivery, while 17 percent thought it would free up developers for more added value work. Comparing results from developers and DBAs did highlight some differences -- developers wanted to be able to focus on added value work, while DBAs saw it as a way of reducing application downtime and to improve collaboration.
Practices Are not Consistent Between the Database and Applications
Despite the levels of integration within organizations, there is still a gap between how companies operate and automate when it comes to applications and databases. For example, while 81 percent of respondents use version control with their applications, just 58 percent apply it to database development. Even amongst those who are most advanced with DevOps practices are not applied equally across both. While the majority of those who have adopted DevOps are doing issue tracking, continuous integration, test automation and automated deployment for their application code, only a fifth of respondents apply the same to their database development. Clearly, there is still more that can be done to bring the two together through common practices and use of third party tools.
As our survey shows, applying DevOps across both applications and databases speeds up software delivery, frees developers to do more value-added work and increases the rate at which database changes are delivered. At a time when IT organizations need to be operating in a faster, more seamless manner to help companies to thrive in more competitive markets, it is time for everyone involved to find out more about how introducing DevOps can transform how they operate. What’s stopping you exploring database DevOps?
Conclusion
A decade into the great DevOps experiment, the data is clear: DevOps is here to stay—and for some very good reasons. Many thought it impossible, but DevOps has succeeded in integrating business users, developers, test engineers, security engineers, and system administrators into a single workflow focused on meeting customer requirements. Why would they willingly do so? Because there’s something in it for everyone. Developers and system administrators stop arguing and start supporting each other, lowering blood pressures all around. Business managers are happy because they get the software products that they need to sell products and services. Executives watch their beloved dashboard metrics—revenue, customer satisfaction, system reliability—heading steadily north. And everyone is able to deliver the best results and overall experience possible to the customer.
Gains like these, however, don’t come easily. To successfully deploy code more frequently while keeping your systems humming, you need the ability to accurately monitor all the changes going on in your environment. The New Relic platform gives developers and operations full-stack visibility—from the digital customer experience to the applications and dynamic infrastructure, through integrated alerts and dashboards—which can help everyone within an organization enjoy a shared understanding of how software is deployed and performs in real time.
Share and enjoy...
___________________________________________
References
______________________________________________________ __________________
“Once more unto the breach, dear friends, once more;”
About Rick Ricker
An IT professional with over 23 years experience in Information Security, wireless broadband, network and Infrastructure design, development, and support.
“Once more unto the breach, dear friends, once more;”
About Rick Ricker
An IT professional with over 23 years experience in Information Security, wireless broadband, network and Infrastructure design, development, and support.
No comments:
Post a Comment
Thanks for your input, your ideas, critiques, suggestions are always welcome...
- Wasabi Roll Staff