Continuous integration with extreme builds

A known best practice in software development is Continuous Integration. It is better to integrate often than it is to integrate rarely because the effort to resolve integration issues grows exponentially with the volume of change that is integrated.

Now let's do some math. Suppose we have a project that runs a complete build (including a smoke test) in 10 minutes. Given an exponential growth of integration issues with a growing volume of changes, suppose the following average times to resolve integration issues:

• 1 hour interval: 10 minutes to resolve integration issues
  
• 2 hour interval: 40 minutes
  
• 3 hour interval: 90 minutes = 1 hours 30 minutes
  
• 4 hour interval: 160 minutes = 2 hours 40 minutes
  
• 8 hour interval: 640 minutes = 10 hours 40 minutes
  

So when running a nightly build, it is not possible to resolve the integration issues the next day anymore.

Now suppose we have a large system where the build takes 1 hour to run. So the minimum build interval is 2 hours. But a large system also has a large development team, so the volume of change in 2 hours results in more than 40 minutes to resolve integration issues. Let's say it takes 50 minutes. So the build runs for 1 hour and the repairs take 50 minutes. So you have 4 runs per day, some of them will not even result in a successful build before the next run.
You can probably imagine what happens if the build+repair time starts exceeding the build interval: the project will become instable because nobody can trust the build anymore. Increasing the build interval will make things exponentially worse!

This may seem like a theoretical problem, but I have worked for several organizations where the build run takes more than 1 hour to run, sometimes even 4 hours. There you have the challenge to make the organization productive in software development that I am involved with...

• Software Development in Healthcare
  
• Nightmare of component-based development (part 1)
  
• Nightmare of component-based development (part 2)
  
• Nightmare of component-based development (part 3)