Production Support and the Art of Motorcycle Maintenance

I have a passion for motorcycles. Over the years, I’ve owned several and have put thousands of miles on them. One of the bikes I currently have in my stable is a Bulgarian 1959 Balkan — an absolutely beautiful old motorcycle.

A few years ago, I made the long journey from Bulgaria all the way to the Stella Alpina Rally in Italy on that 1959 Balkan. The trip was well-planned and I completely rebuilt the bike before we embarked. Given that I’d be riding an almost 60-year-old two-stroke bike, I planned for small repairs and maintenance while on the road. And it’s a good thing for me that I planned in advance.

Even a machine that runs perfectly will break down eventually, often at the most inopportune time.

In my case, I broke done on the side of the highway a few times throughout the trip. But, because I was prepared, I was able to quickly fix things and get back on the road, causing little delay to my trip. If I hadn’t prepared ahead of time and properly maintained my bike, I’d be telling a much different story about my trip right now.

So, What Does This Have to Do With Production Support?

Being prepared to face a crisis in a production environment is not much different from being prepared for problems with an old bike. Both are complicated ecosystems that must be maintained and kept in optimal shape to perform properly. When you drive a motorcycle off the lot, it’s up to you to keep on that maintenance. Sure, the dealership might send out reminders but how you choose to customize and maintain your bike is all up to you.

The same can be said of production environments. Cloud providers like Amazon Web Services (AWS) and Azure provide the infrastructure to run technology platforms and workloads but leave everything else up to the customer. This includes setting it up and building it out as well as securing it in addition to ongoing maintenance.

Production support encompasses many different tasks that, when performed regularly, ensure that your environment is maintained, up to date, and free from production issues.

Maintenance Plan

In order to properly maintain your production environment, you need a plan. Just like maintaining a bike requires different tools, keeping the production environment in good shape requires a number of tools and skills.

A maintenance plan includes strategies around updates and patching but also backups, disaster recovery, monitoring and alerts, and how your team addresses the security of the environment. Each of those activities can be covered with a number of different tools. You get to decide which will be best for your environment and implement it but the maintenance doesn’t stop there.

Mother Nature has a simple definition of life: there is no “at rest” in life. You either progress or regress. Knowing that means your production environment should be evolving over time — hopefully progressing, of course.

In keeping up with the progress, you’ll likely find that some of the tools become obsolete or insufficient. If that happens, replace them with other tools better suited for your current situation.

Disaster Recovery Plan

Have you thought about disasters? Will your business survive if your production environment is offline for 5 minutes? How about 5 years?

Having a solid Disaster Recovery plan is not only about creating backups once in a while but also about planning how to react in the case of a disaster. Who will recover the data from the backups? Do you have a well-documented procedure that has all the steps in it? Have you tested it lately?

In order for your business to stay “on the road”, you need to have recent backups of your production environment and a tested way to restore the environment from those backups. Having a high availability cluster with redundant disk storage reduces the chances of failures but it won’t save your data from loss in case of disaster. Neither will a copy of your data in a subfolder on the same hard drive for that matter.

Official support of the software in your production environment.

Every bike has an era where it was at the peak of technology. 60 years after that peak, it will still be functional but its brakes will be inferior, its acceleration will be lacking, and its gas mileage won’t be anything to write home about.

Similar things happen to software. It ages. But it ages way faster than bikes. The software world has way more components changing over time than a bike. While software security wasn’t an issue when my Balkan was built, it certainly is now.

Newer versions of different applications and operating systems are released every day. Patches to existing versions too. But, at some point, most of us migrate from older software to a newer version. At that point, the vendor decides the software is obsolete and stops maintaining it. (OK, in most cases there is a planned EOL)

To put it simply, if you’re still using the older version of a piece of software, you’re driving an old bike for which no one produces brake pads anymore. The brakes (or the software’s security system) will fail sooner rather than later. And, with no patches or replacement brake pads available, you are quickly heading towards a disaster.

Integrate intrusion detection and prevention into your security protocol.

Intrusion detection and prevention tools are sophisticated. It takes time to integrate them with the business logic of your app and its associated backend to fully protect the data. Adding them after development is underway only serves to increase the complexity and cost of the effort.

The security of your solution isn’t limited to the code itself. It also must extend to the production environment. We recommend a 24/7 intrusion detection and prevention system. This guards against sniffing, brute forcing, and hackers employing a combination of methodologies to gain access to your tool and its data.

Plan on security patches as often as monthly, and quarterly at a minimum, to continue protecting and fortifying your solution. Security vulnerabilities can happen to any piece of software in any industry. You don’t have to be covered by HIPAA, GDPR, or PCI for security to be an issue for your software.

Vulnerabilities aren’t limited to just your product, either. Any vulnerabilities found in a third-party’s system can also wreak havoc on the safety of your software and its users.

It’s important to know what your market expects and how to proactively develop and test to protect information. There are many security and monitoring solutions available that can test your software before and after release depending on the platform, hosting services, and industry.

And don’t forget to put your SSL certificate renewal and installation on your calendar!

Final Thoughts

Production support, and motorcycle maintenance for that matter, both require constant vigilance to be successful. It might all seem overwhelming but it’s important to remember that you don’t need to handle all of your own production support. Our team is here to help maintain and support your infrastructure as well as optimize it for performance, efficiency, and cost. We might not carry spare motorcycle parts at work but we do carry a lot of knowledge and experience to help us solve even the toughest situations.

To continue reading about how Production Support fits into our greater Continuation Engineering model, download our eBook, 10 Ways to Support Your Software Ecosystem After Launch.

Original post can be found here.

Authored by Nikola Donev:

As the SysOps & DevOps Lead on our Continuation Engineering team, Nikola is the go-to person for the most technical of questions. Nikola’s nicknames vary from “The Walking Linux Encyclopedia” to “The Security Nerd.” An integral part of our CE practice, he built the Ops side of the team from scratch by not only seeking out the right people but also making the CE team use our knowledge efficiently. He creates and refines the processes within the CE team from an Ops perspective and is involved in projects involving FedRAMP and SOC2 as well as HIPAA compliance.

Nikola is intent on proving that a deeply technical person can also be a good leader. As such, he enjoys the technical challenges of his job just as much as leading people. On the technical side, he finds troubleshooting to be the most exciting part while sharing knowledge and motivating others are the most rewarding aspects of his leadership role. Outside of work, you’ll likely find Nikola riding one of his motorcycles or working on home improvement projects.