SCM tools are not configuration management tools

The main reason why most so-called SCM tools are not really SCM tools is because they don't support managing software configurations. Making software is more than writing source code and converting them into executable code and data models for databases. A real SCM tool would be able to capture everything that is important for deploying and maintaining the software. This includes requirements, designs, models, sources, tools, infrastructure, knowledge, skills, test scripts, test data, manuals, scripts and other information.

Most SCM tools are able to capture files in a structure and control changes to the files and the structure. New files and new versions of existing files are all merely new files. Typically, the structure is 3-dimensional:

• Directory (or folders) structure
  
• Version (or revision) structure
• Branching structure
  

The directory structure may seem to be a 2-dimensional structure (i.e. nested directories and directory next to eachother at the same nesting level), but if we consider the pathname + filename as the single identifier for a file, then the directory may be considered as 1-dimensional. The version structure is 1-dimensional: the successive versions supersede their predecessors. How about parallel versions? Parallel versions are partial contributions to a single succesor version. The actual successor is a merge of these partial contributions. Branches look similar to parallel versions, but the essential difference is that parallel versions are partial contributions to a single successor while branches are full contributions (version structures) to alternative successors.

If we look at the implementation of these dimensions, then the most simple implementation is 1-dimensional: all 3 dimensions are projected onto the same implementation, e.g. as directory structure (or path+filename). The "version control tool" could be an ordinary file system. For example:

main/gui/generic/foo-v1.c
main/webgui/unix/foo-v1.c
main/webgui/unix/foo-v2.c
main/webgui/unix/foo-v3.c
main/webgui/winxp/foo-v1.c
main/webgui/winxp/foo-v2.c
R1.0.0.0/gui/generic/foo-v1.c
R1.0.0.0/webgui/unix/foo-v2.c
R1.0.0.0/webgui/winxp/foo-v2.c

The problem with this version control system is that path+filenames changes for every new version. So users and the build process have to do more work to figure out which version they should us.

The next better implemention would be a 2-dimensional implementation: directory and branching are combined into a single dimension (path+filename), and versioning is the other dimension. Simple version control tools like Subversion works this way. For example:

main/gui/generic/foo.c (versions: 1)
main/webgui/unix/foo.c (versions: 1, 2 and 3)
main/webgui/winxp/foo.c (versions: 1 and 2)
R1.0.0.0/gui/generic/foo.c (versions: 1)
R1.0.0.0/webgui/unix/foo.c (versions: 2)
R1.0.0.0/webgui/winxp/foo.c (versions: 2)

Advantage is that the path+filename remains the same for all versions within a branch. But for different branches, the path+filename is different. And since directories and branches are resolved in the same dimension (the path), it is not possible to distinguish between a directory and a branch. For example, are main and R1.0.0.0 different branches? Are unix and winxp different branches? Are gui and webgui different branches? Or are they different directories within the same branch? So users have to make agreements about naming conventions to distinguish between branches. The SCM tool only takes care of deciding (automated) which version is used.

One step further is a 3-dimensional solution, where directory (path+filename), version and branch are independent of each other. More advanced version control tools like ClearCase or Synergy are needed. For example:

gui/foo.c (versions: 1 on branch: main)
webgui/foo.c (versions: 1 and on branch: main; versions: 2 and 3 on branch: unix; versions: 2 on branch winxp)

Advantage is now that the path+filename remains the same for all versions and all branches. This simplifies the implementation of an automated build process and the description in design documents and models. But the counterside is that SCM has to have the information to decide which branch a user is working on in order to select the correct version for the user to work on. And the user may be unaware of the branch he is working on - introducing the risk that he is working on the wrong branch.
So on one side SCM makes life easier for the user and the organization (e.g. automation), but on the other side it introduces extra work to reduce the risk mistakes or to repair them.

As you can see, I have left out the baseline (R1.0.0.0) from the last example. In the first and second example (1 and 2 dimensional), the baseline was combined with the directory dimension. In the last example, the baseline could be combined with the branch dimension, but it could also be implemented as a 4th dimension: labeling or tagging.

And this brings me to the point where version control enters the domain of configuration management. An essential feature of configuration management is to identify dependencies. A dependency defines which objects belong together. There are many different dependencies that can be (or need to be) identified, for example:

• Directory dependency: all files within the same directory tree
  
• Branch dependency: all files on the same branch
• Version dependency: all latest versions
  
• Status dependency: all files with the same status (e.g. release R1.0.0.0)
• Content dependency: all files with compatible content (e.g. requirements-design-code consistency)
  

The status dependency is typically modeled in a so-called promotion model. All files go through a predefined series of statuses, and files (versions) with similar status and context (e.g. branches, directories) belong together as a configuration. These statuses are for exampe: working, integration testing, system testing, released. Tools support promotion by branching (e.g. ClearCase/UCM by deliver and rebase, or Subversion by "smart" copying of directories called branching) or by selection rules (e.g. Synergy by reconfigure property templates).

But one of the biggest shortcomings in SCM tools that I know is the absense of support for content dependencies. How do you identify the impact of the change of the design on code, requirements and tests? How do you identify the impact of code changes on other code, interfaces, design models? How to maintain the content information efficiently? How to you know that the content dependency is compromised? How do you know that release 3.2 of product X does work with release 1.2 of the framework, but not with release 1.1 of the framework? How do you know that release 6.1 of product Y cannot work with product X because it does not work with framework 1.2?

Another big shortcoming of SCM tools is that they only support control on file level. They don't control requirements, components in the design model, test cases in a test specification, tool versions (e.g. compilers, IDEs, webservices), hardware versions (e.g. 32-bit architecture). Consequently, many organizations try to capture those items in files, e.g. by creating a requirement specification document that "baselines" a set of requirements. But then again, those individual requirements - although versioned in a requirement management tool - cannot be identified as separate objects in the SCM tool, let alone that dependencies on requirements level can be identified or that individual requirements can be identified to a baseline.

The only solution that I am aware of that comes close to an "SCM tool" is the Jazz platform, starting the Rational Team Concert, but integrated with the requirements, test and project management applications. Since all information is stored in a composite repository, where information objects are actually identified as objects (not as files), it becomes possible to identify relationships (such as dependencies) between objects (not only files). Yet, I doubt whether it will be capable of identifying dependencies between configurations, e.g. content dependencies between software packages.

Deliverable planning and control

For many people Configuration Management is little more than controlling the versioned storage of work products, sometimes extended with Change Control which is little more than controlling change requests and problem reports. But both boil down to something like an sophisticated database with a sophisticated tool, sometimes not even sophisticated.

But the real added value of Configuration Management is that it helps manage and control the project. Let's have a look of a project in general first. In short every project is an enterprise, a journey, to establish a particular result of a particular quality, within the limits of particular resource constraints (typically money, time, people, technology, methodology, etcetera).

Every project starts with planning. Planning the result means expressing the results in items that will be delivered - if the project is executed according to the plan. Those items need to be identified (Configuration Identification), controlled (Configuration Control) and must be of a certain quality level expressed by a status (Configuration Status Accounting). So the first order planning of - the result of - a project already involves configuration management.
Secondly, these resulting items don't fall out of the sky; they need to be developed. So every item will require at least one task in the planning that produces the item. Some tasks may even product multiple items, that's OK. But every task needs input. So there we have a second order of configuration items.
Thirdly, these second order items need to be produced also, or they are inputs to the project from an external source. These can be requirement definitions, architectural definitions or even complete subsystem implementations from another project or a previous project.

Anyway, project planning strongly involves configuration planning, which is a responsibility of configuration management. In other words, configuration management is part of project management, at least during planning.

After the planning has been built up, which involves a lot of other things to be planned as well, the project is executed according to the plan. The tasks are being executed and the (planned) items are being produced as the result of the tasks. Some items may be produced iteratively, thus producing multiple versions of it. Anyway, these items (i.e. the versions) are stored in the CM system and status is assigned to indicate the progress of the item (how far is it "ready"? which is a quality attribute) and the maturity (how good is it? which also is a quality attribute). But controlling the versions and their status - and even the status of integrated items, which is a different thing than the status of the individual parts. This is all Configuration Management, which contributes to the Project Monitoring and Control of the project.

All I am saying is that the deliverable planning and tracking processes are both part of Configuration Management and of Project Management. That's why we should not consider CM in terms of the tools, databases, licensing issues, disk or network performance issues, branching issues, access control, etc. Those issues are not configuration management, but they are IT systems management issues. Configuration management is a project management discipline that is concerned with managing the physical work products of the project.

The Green Shift Anti-Pattern

In Dr Dobb's Agile Newsletter of July 2006, Scott Ambler writes about the "Green Shift". I was reminded of the article by Jason Yip.

The problem is that the project status "green shifted" as it rose through the various levels of management. The people on the ground were very clearly reporting a red project status, our project manager wanted to play it safe so reported yellow status, and his manager was more political yet and reported green status.

Green-shift is a form of information "erosion". When rocks erode under the influence of wind, rain, sand and dust, the sharp edges of the rocks disappear first. Next the shape of the rocks will change and finally the rock will crumble into pieces losing its original form entirely.

Erosion of information is a natural phenomena in communication. Every time a story is told, it is changed a bit. This way insignificant events may become spectacular (red-shift) and alarming reports may become uplifting (green-shift).

I like the solution that Scott proposes that by holding scrums the green-shift can be reduced. Three elements may play an important role in this. First, direct communication instead of passing the information over several layers. Less layers means less erosion or green-shift. Second, frequent communication reducing the delays. Less delays means less (time for) erosion. And thirdly, a scrum adds non-verbal communication that (written) reports are lacking. This improves the interpretation of the information, and interpretation is one of the sources of information erosion.

Unconcious thinking

Have you ever tried to look at a very faint light, a star or so? And did discover that you can see it better by looking next to it?

In a science edition of the NRC (Dutch newspaper) I read an article about unconcious thinking. In the article it was claimed that the concious mind is only suitable for simple decision, like buying toothpaste. But when it comes to really significant decisions, the complexity is too large for the concious mind. The unconcious mind is able to think much quicker and take much more complex thinking patterns than the concious mind. Decisions like buying a car or a house, changing your job or starting your own company are best left to the unconcious mind.

According to the experiment, a number of people were asked to listen to a large amount of unrelated information. Two groups were asked questions about the information after half an hour to verify what they remembered of it. One group was left to memorize the information and the other group conducted a numbering game that required them to use their rational abilities. It appeared that the second group was able to answer more questions correctly than the first group.

I was amazed by it, but it concurs with my own way of learning. If I absorb an amount of information and go do something else that takes my mind off the subject (playing on the piano or even studying a piece, for example), it is much easier to remember and see all kind of relationships in what I learned than when I try to stay concentrated and focussed. Sleep on it!

Technorati tags: Mind, Unconcious, Thinking

Managing complex projects

Next Thursday, November 18th, there will be an event organised by Bits&Chips about managing complex projects. Sure, it will be mostly in Dutch but it will be interesting anyway.

The challenge for software projects is to increase the productivity significantly; not by 10% or 50% but with 10 to 100 times or more. Will current process improvement initiatives achieve that? I doubt it. In my opinion, we need to reduce the size of projects instead of increasing it, we need to replace manual work and communication by machine work.