Cascading Defects & Automated Migrations

We all know how annoying it is when you’re surfing the web and you’re greeted with a cascade of never ending pop up windows.  I saw the same thing happen yesterday at work, but instead of with popups it was with defects.

The goal of the day was to finish the last of the defects, push the fixed assemblies them to the system integration test environment and freeze the code before a release to production.  I wasn’t overly involved with the process (I’m already hard (ly) at work on the next major release of the software), but I did see the effects of it on the team.  These guys came in to work that morning with a lot of energy and optimism, but by lunch time they looked like they had been marched to hell and back and they still had a late night in front of them.

So what happened?  Well, apparently bug fixes were ticking right along and then the first of many migrations to system integration testing occurred.  Even with a strong team and a good (not great though, but that’s for another post) set of source control tools there were problems with getting the correct set of files into the build and then getting the build deployed correctly.  I don’t know the specifics, but it is safe to say that myself and the other team leads have been pointing to this process as a huge gap for a number of months.  It finally bit our collective asses.

A while back I did a post on environment reproducibility where I discussed the need to be able to quickly and effectively create development and testing environments.  Even with solid and reproducible environments you need a solid and reproducible way to get your software into them.  Without an automated migration process you are relying on the installation gods and the last time I checked they didn’t respond well to chicken sacrifices.  We have tools to ensure that we can deploy components to the desktop in a consistent manner such as ClickOnce and MS Updater Blocks.  Why don’t we have great tools for the movement of the system from development to the locations that are required by these tools?

How can you do this?  I’d love to be able to give you an end to end solution, but I think that too much depends on the environment you work in and the technologies that you’re using.  Deploying assemblies, html and other files is as easy as creating a batch file and using xcopy.  What about database DDL upgrades or Stored Procedure changes though?  Some environments preclude you from doing this without providing a script to an administrator.  If that’s the case, then you might not be able to perform an end to end migration process.  If you’re like us, you probably can’t automate the deployment of the JCL or COBOL either.

What I will suggest is that you automate as much as you possibly can.  You’ll never know about the troubles that you saved yourself.  On the other hand, you’ll know all about the troubles that can be present with the stuff you don’t have automated.