Nightmare of component based development (2)

Before going into some ideas for the integration process, let me tell you something about the history. The previous project was a monolythic system developed on a single branch. Although the intention was to have a daily build, practice showed that it was not feasible to make a stable system configuration every day. So, the cycle was delayed to a weekly cycle.

However, since we were facing a large development team (and a large system), all changes made in a week was too much to make a stable configuration on a weekly basis. Delaying more would probably make it worse.

So that is why we came up with the idea to reduce the development scope to subsystems rather than the whole system. Architecture was changed and SCM was changed too, into subsystems. But now we are facing the problem that there are too many trajectories running in parallel and delivering every week a new subsystem configuration for system integration. In addition, the number of dependencies between subsystems became unmanageably large.

So we came up with the idea to reducing the amount of configurations by clustering those subsystems, not into a single system (as we hard in the past) but into four or five subsystem clusters. For each cluster, we can make a weekly or even daily build and for integrating those clusters together we can make a weekly or (bi-)monthly cycle.

However, this approach implies that witin the clusters there is early integration, while across the clusters there is late integration. This implies that the clusters must be chosen such that the number of inter-cluster dependent changes are minimal. However, when we have an inter-cluster dependent change, it must be developed within two or more clusters at the same time and pre-integrated (and tested) before it is officially integrated at overall system level. But crossing the boundaries of clusters will increase the managerial complexity again, possibly forcing clusters to be deliver simultaneously to system integration.

As you see, the problem is not very simple. If you have any idea how the next step could look like, don't hesitate to send me a comment.