I’m going to tell you the tale of two software shops.  Both shops have incredible Engineers,  great Product Managers and excellent processes.  The only thing that separates these two shops is release night.  One shop knocks out the change in minimal time, the other is working through the weekend to fix issues.  This anomaly has nothing to do with platform choice, complexity of the stack or location.  The problem is universal across all of these.  The difference between these two shops, a release engineer.

Typically a release engineer is a person, or team of people, who are familiar with the build, deployment and configuration of a product.  This is important because one thing I can tell you for sure, your production environment is not exactly the same as your lower environment.  Even more sure is that your production support team has even less knowledge on how to trouble shoot release issues.  You absolutely need someone who is familiar with your configuration.

Unfortunately release engineers are somewhat of a luxury for many shops.  They are an unappreciated component of a successful software shop. If you can’t get the budget to hire one I have a suggestion that has worked for me time and time again.  Appoint one of your developers (and a backup) as the designated release engineer.  This makes on person solely responsible for the success of your pushes. 

There is a cost associated with dedicating on of your developers.  I generally find it amounts to 10 hours a week as the developer builds and defines the process, approximately 6 weeks total.  After that, maybe 1 hour a week to maintain the process.  It sounds like a modest investment, but it will pay off the first time you have 6 developers working all week to make a deployment successful.