How to plan software as a configuration item

Yesterday I got a question from a colleague about configuration items. In our projects we use a deliverable list to identify which items are being delivered by the project and which are being received from outside the project. To help projects plan their configuration items we have defined a deliverable list template.

The deliverable list template lists documents like various types of requirement specs, design specs, test specs, manuals, training materials, etcetera. It also contains an single entry for Software. Now the question was:

What software deliverables (and receivables) should a project plan for?

Obviously, the deliverable list template does not give much clues. But we know that the software consist of components, some of which are received from other projects or third party suppliers - and possibly modified by the project - and some of which are created by the project. Then, how do you know which software deliverables the project is going to deliver?

The answer is that it depends on the agreement with the customer of the project. The requirements agreed with the customer state what he will get, in terms of functionality, performance, quality and deliverables. And these deliverables should be listed in the deliverable list. If the requirements don't state the (software) deliverables, then apparently the customer does not care as a long as the functional and non-functional requirements are satisfied.
In that case, the architecture or high level design determines the software deliverables. Even more, the architecture determine which of the software components are reused from other projects or third party suppliers and which are created anew.

Okay, now that requirements and architecture/design define which software deliverables and receivables are applicable,

how do we plan in which release they are delivered?

For projects with a single (final) release, it is simple: everything is released and delivered at the end. For projects with multiple releases, like incremental development or iterative development, it is more complicated but still quite simple.

The total work of a project is divided in "work packages", chunks of work that deliver a number of (possibly internal) deliverables. These work packages are assigned to a release and this way it is determined that the deliverables of those work packages are delivered at that release. More precisely, the deliverables are packed together as a baseline and the baseline is released.

Now in many software development organizations, the work packages are handled just like problem reports and change requests. They are registered in the change control tool/database, and assigned to people and releases (or iterations). We could call them "work items"; the list of work items planned for a release or iteration (in a release plan or iteration plan) determines which software deliverables are released. You could list those deliverables in the deliverable list, but it is more practical and more accurate to generate a report from the change control database listing the work packages, problem reports and change requests assigned to a particular release or iteration.

SCM tools are not configuration management tools

The main reason why most so-called SCM tools are not really SCM tools is because they don't support managing software configurations. Making software is more than writing source code and converting them into executable code and data models for databases. A real SCM tool would be able to capture everything that is important for deploying and maintaining the software. This includes requirements, designs, models, sources, tools, infrastructure, knowledge, skills, test scripts, test data, manuals, scripts and other information.

Most SCM tools are able to capture files in a structure and control changes to the files and the structure. New files and new versions of existing files are all merely new files. Typically, the structure is 3-dimensional:

• Directory (or folders) structure
  
• Version (or revision) structure
• Branching structure
  

The directory structure may seem to be a 2-dimensional structure (i.e. nested directories and directory next to eachother at the same nesting level), but if we consider the pathname + filename as the single identifier for a file, then the directory may be considered as 1-dimensional. The version structure is 1-dimensional: the successive versions supersede their predecessors. How about parallel versions? Parallel versions are partial contributions to a single succesor version. The actual successor is a merge of these partial contributions. Branches look similar to parallel versions, but the essential difference is that parallel versions are partial contributions to a single successor while branches are full contributions (version structures) to alternative successors.

If we look at the implementation of these dimensions, then the most simple implementation is 1-dimensional: all 3 dimensions are projected onto the same implementation, e.g. as directory structure (or path+filename). The "version control tool" could be an ordinary file system. For example:

main/gui/generic/foo-v1.c
main/webgui/unix/foo-v1.c
main/webgui/unix/foo-v2.c
main/webgui/unix/foo-v3.c
main/webgui/winxp/foo-v1.c
main/webgui/winxp/foo-v2.c
R1.0.0.0/gui/generic/foo-v1.c
R1.0.0.0/webgui/unix/foo-v2.c
R1.0.0.0/webgui/winxp/foo-v2.c

The problem with this version control system is that path+filenames changes for every new version. So users and the build process have to do more work to figure out which version they should us.

The next better implemention would be a 2-dimensional implementation: directory and branching are combined into a single dimension (path+filename), and versioning is the other dimension. Simple version control tools like Subversion works this way. For example:

main/gui/generic/foo.c (versions: 1)
main/webgui/unix/foo.c (versions: 1, 2 and 3)
main/webgui/winxp/foo.c (versions: 1 and 2)
R1.0.0.0/gui/generic/foo.c (versions: 1)
R1.0.0.0/webgui/unix/foo.c (versions: 2)
R1.0.0.0/webgui/winxp/foo.c (versions: 2)

Advantage is that the path+filename remains the same for all versions within a branch. But for different branches, the path+filename is different. And since directories and branches are resolved in the same dimension (the path), it is not possible to distinguish between a directory and a branch. For example, are main and R1.0.0.0 different branches? Are unix and winxp different branches? Are gui and webgui different branches? Or are they different directories within the same branch? So users have to make agreements about naming conventions to distinguish between branches. The SCM tool only takes care of deciding (automated) which version is used.

One step further is a 3-dimensional solution, where directory (path+filename), version and branch are independent of each other. More advanced version control tools like ClearCase or Synergy are needed. For example:

gui/foo.c (versions: 1 on branch: main)
webgui/foo.c (versions: 1 and on branch: main; versions: 2 and 3 on branch: unix; versions: 2 on branch winxp)

Advantage is now that the path+filename remains the same for all versions and all branches. This simplifies the implementation of an automated build process and the description in design documents and models. But the counterside is that SCM has to have the information to decide which branch a user is working on in order to select the correct version for the user to work on. And the user may be unaware of the branch he is working on - introducing the risk that he is working on the wrong branch.
So on one side SCM makes life easier for the user and the organization (e.g. automation), but on the other side it introduces extra work to reduce the risk mistakes or to repair them.

As you can see, I have left out the baseline (R1.0.0.0) from the last example. In the first and second example (1 and 2 dimensional), the baseline was combined with the directory dimension. In the last example, the baseline could be combined with the branch dimension, but it could also be implemented as a 4th dimension: labeling or tagging.

And this brings me to the point where version control enters the domain of configuration management. An essential feature of configuration management is to identify dependencies. A dependency defines which objects belong together. There are many different dependencies that can be (or need to be) identified, for example:

• Directory dependency: all files within the same directory tree
  
• Branch dependency: all files on the same branch
• Version dependency: all latest versions
  
• Status dependency: all files with the same status (e.g. release R1.0.0.0)
• Content dependency: all files with compatible content (e.g. requirements-design-code consistency)
  

The status dependency is typically modeled in a so-called promotion model. All files go through a predefined series of statuses, and files (versions) with similar status and context (e.g. branches, directories) belong together as a configuration. These statuses are for exampe: working, integration testing, system testing, released. Tools support promotion by branching (e.g. ClearCase/UCM by deliver and rebase, or Subversion by "smart" copying of directories called branching) or by selection rules (e.g. Synergy by reconfigure property templates).

But one of the biggest shortcomings in SCM tools that I know is the absense of support for content dependencies. How do you identify the impact of the change of the design on code, requirements and tests? How do you identify the impact of code changes on other code, interfaces, design models? How to maintain the content information efficiently? How to you know that the content dependency is compromised? How do you know that release 3.2 of product X does work with release 1.2 of the framework, but not with release 1.1 of the framework? How do you know that release 6.1 of product Y cannot work with product X because it does not work with framework 1.2?

Another big shortcoming of SCM tools is that they only support control on file level. They don't control requirements, components in the design model, test cases in a test specification, tool versions (e.g. compilers, IDEs, webservices), hardware versions (e.g. 32-bit architecture). Consequently, many organizations try to capture those items in files, e.g. by creating a requirement specification document that "baselines" a set of requirements. But then again, those individual requirements - although versioned in a requirement management tool - cannot be identified as separate objects in the SCM tool, let alone that dependencies on requirements level can be identified or that individual requirements can be identified to a baseline.

The only solution that I am aware of that comes close to an "SCM tool" is the Jazz platform, starting the Rational Team Concert, but integrated with the requirements, test and project management applications. Since all information is stored in a composite repository, where information objects are actually identified as objects (not as files), it becomes possible to identify relationships (such as dependencies) between objects (not only files). Yet, I doubt whether it will be capable of identifying dependencies between configurations, e.g. content dependencies between software packages.

Rational ClearCase is not an SCM tool

If Telelogic Synergy is not an SCM tool and Subversion is not an SCM tool, it is easy to conclude that IBM Rational ClearCase is not an SCM tool either. Not even ClearCase/UCM is an SCM tool.

Telelogic Synergy is not an SCM tool

In Januari I talked about Subversion being not an SCM tool. Now let's have a look at Telelogic Synergy, which has been acquired by IBM recently. Until recently, I did not have much practical experience with Synergy and most of it slipped away. Now, I am working with it quite intensively and I came to hate and love it.

So let me explain why I think Synergy is not an SCM tool either.

Promotion model
Since the introduction of the task-based, more or less as a replacement of the ancient object-based approach, Synergy has adopted the approach of change sets. Tasks are actually change sets.

In Synergy you can define almost any configuration regardless of how, where or when the objects are changed. There are a few predefined change models, e.g. Collaborative Development and Insulated Development. The difference is that with Collaborative Development all completed tasks come available to other developers, while with Insulated Development tasks are promoted manually by the build manager.

Using queries on the database you can define almost any configuration. Through (semi-)automatically updates of the configurations, objects are "promoted" to the configuration. So for this Synergy qualifies as an SCM tool.

Real baselines
With the introduction of Rapid Baselining, you can freeze a configuration of multiple components and subcomponents instantly. The resulting configuration are immutable.

So again, Synergy qualifies as an SCM tool.

No projects, teams and streams
Synergy structures a database into (what Synergy calls) projects. In fact these are not projects, but components that can be developed and released independently of eachother, of hierarchically if you wish. Calling them "projects" is misleading because they do not constitute an enterprise for an goal within an agreed timeframe, using a set of resources. Moreover, you cannot even assign people to a project (only to a database) or define a particular scope to a project.

Synergy does not know the concept of teams, or roles assigned to people in a project. It does define "roles" to assign tool privileges, but no roles in the sense of a project (e.g. project leader, librarian, developer, tester, release manager, etcetera).

It may sound strange, but Synergy does not know the concept of streams. You can create parallel configurations (called "parallel projects") and parallel versions, but the concept of "a flow of changes" (like a river meandering through woods and valleys) is missing. Configurations are more like cities (being somewhere, changing constantly but going nowhere) than like rivers (changing constantly and moving constantly). You can define concepts that look like branches or streams using the release value, purpose and platform, but they remain configurations not streams.
Even worse, Synergy is quite "parallel intolerant". There is no distinction between checking out the same version twice (with the same release/purpose/platform value) and checking out versions in completely different projects (with different release/purpose/platform value. Why would you be need to be warned that someone in another project is changing the same object as you are? I can understand why you need a warning if it happens within your own project team.

Change sets
As a wrote above, tasks (or change sets) are in the core of Synergy. It's more powerful and more comprehensively implemented than any other tool I know.

Components
At first sight, the so-called projects may be considered as components. However, since Synergy tries to be "smart" in selecting sub-project versions (or sub-component versions) there is a clash between using baselines of released components and using a development configuration of a sub-project/sub-component. Due to this smartness, Synergy sometimes selects the "wrong" configuration.

No roles
Yes, Synergy does define roles but only to distinquish between different tool privileges (e.g. developer, build_mgr, ccm_admin). Roles to separate responsibilities and authority, for supporting promotion models and release management are absent.
Although... you can define a change promotion model in Change Synergy that grants authority to promote requests. So in Change Synergy (control of requests) roles can be defined but in CM Synergy (control of configurations) you cannot.

Although some things can be customized, Synergy does not quality for SCM tool here.

Limited branching
Some branching can be implemented using release, purpose and platform, but it is rather a way to distinguish between different configuration than between different branches. Also, Synergy supports parallel versions but it is hardly sufficient to call it branching.

Conclusion
Taking all the pros and cons into account we can conclude that Synergy is a very powerful tool and it comes close to a SCM tool. Yet, it lacks several important concepts.

Why SCM is not sexy, and SCM-ers are nerds

Software development is inherently difficult. One of the reasons is that computers are inherently stupid and persistent in doing the same thing over and over again. Computers do exactly what they are told (programmed) to do, regardless of whether that is what you expect them to do. They are not human! To make computers do what you expect them to do, you need to be very meticulous and persisting in telling them what to do and how to do it.

Software configuration management is also inherently difficult. One of the reasons is that people are inherently clever and flexible in doing the things differently every time. People do what they think they must do, regardless of whether that is what you expect them to do. They are not computers! To make people do what you expect them to do, you need to be very convincing and careful in telling them what to do and how to do it.
Another reason is that software configuration management has a lot to do with computer systems, which are inherently stupid and persistent. You need to be very meticulous and persisting in telling them what to do and how to do it.

Try being meticulous and persisting in telling people what to do and the effect will be exactly opposite of what you might expect: resistance against your attempts to convince them of anything.

"computers cannot be convinced and people cannot be programmed"

A software configuration manager needs to be blessed with both talents to be really successful in SCM. But these skills are almost opposites, so the combination of both in a single person is rare.
Many SCM-ers like writing scripts, setting up and adjusting SCM tools and processes, performing software integrations and builds reporting results on websites and tuning systems performs and interactions with other systems. These SCM-ers are systems oriented, skilled with abstraction and logic. If they are good are likely to become the technical nerds of an organization.
Other SCM-ers like working with people, communicating, coaching, leading, defining strategies, giving trainings and presentations, coordinating the work in projects and process improvements. These SCM-ers are people oriented, skilled with social intelligence. If they are good they are likely to become the inspirator to bring the organization on a higher professional level. This group of SCM-ers is considerably smaller since they are likely to move on to other - more respected and better payed - jobs.
And finally, there is the extremely small group of SCM-ers with skills at both ends of the spectrum. They have a large interests in systems, structure, organization and tools, and at the same time a strong urge to inspire people, coach, guide and lead them.

Now if its true that the majority of the SCM-ers lack the necessary social skills to "sell" SCM and many people needing SCM lack the necessary technical skills to "buy" SCM (from these non-sellers), then its not so surprising that SCM is not seen as a popular profession.

Development speed vs accelleration

Some days ago, I had a discussion with some of my colleagues about what percentage of a project budget should be spent on "overhead" activities, such as planning, tracking, reporting, documenting, process improvement or configuration management. Some argued that it should be less than 10%, other said it could be up to 50% of the total effort spent.
The argument is that a development project is supposed to development a product, so the core activity should be on engineering activities. Sounds reasonable to me.

Then I read the article about process improvement at Toyota, where it is claimed that the core activity of the organization has become "improving the process of making products" and the production/development of the product is a side-effect of it. The idea is simple: you will win from any competitors if you improve faster and/or longer than they do. Or put differently:

• Speed wins over distance.
• Acceleration wins over speed.
• Accelerated acceleration wins over acceleration.

Now I revisited the original question: "What percentage of the budget should we spend on 'overhead'?" The answer became very simple: increase the percentage if it increases your business value. When it reaches 50%, or 60% or 90%, it does not matter as long as it is good for business. As with Toyota, this may imply that the core competence of the company will change (from making cars to improving the way cars are made). If for example only10% of the company are technical experts contributing to making the product, the company becomes a non-technical company.

When extrapolating this to the career challenges, it does not really matter whether I am good at what I do or not. Good technical skills, good social skills are nothing to be jealous about. Good learning skills, especially the learning skills to improve your learning skills, are ultimately what's going to make the difference.

Development is not alike manufacturing

After reading the article in the Rational Edge about Lean software delivery I am very much upset. The article regards software development as if it is a manufacturing process. For business applications this may be true, but for the technology world this is absolutely not so.

Being on the bleeding edge of technology means exploring new grounds, new possibilities, new opportunities and inventing new products. Manufacturing is about executing the same actions to create the same products and doing it over and over again. Development is about doing (possibly the same, but more likely) different actions to create a different product and doing it once. I would say that development is more like a social economy.

But what I am most upset about is that IBM/Rational and many other tool vendors are persistently ignoring the technology world and pretend that (software) development is about business processes and business applications. Of course, that's where the most money is so that's where tool vendors can make the most money from. But the technology world has very pressing demands for flexibility, development speed and control, and desparately need better tools allow the companies to keep up with the increasing speed of technological evolution.

"Waste" is not "any activity that does not directly contribute to the added value of the end product". If you eliminate all of those, a development organization in the technology world will propably not survive release 2, let alone be able to develop a product line.

Professional software development

Recently I renewed my interest in photography. I read the Strobist blog with great interest, in particular his Lighting 101 series.

One of the great virtues of digital photography is that it has become very easy and very cheap to make many pictures, and that you have instant feedback about the result. But the effect on me - and on many others - is that I make more pictures than I used to, but also worse pictures than I used to. I take less time to make a picture, but as a result it takes me much more time to make good picture. Why? Because shooting 10 bad pictures doesn't make any of them a good picture, while spending the same time on carefully shooting 2 pictures does.

I see a similar effect in software development projects. The tools we (can) use to make software becomes increasingly powerful, making it easier to model requirements, designs and tests, write code and perform static and dynamic analysis on it, generate testcase from the models and analysis reports. We produce more requirements, more models, more designs, more code, more complexities, more integrations, more variability, more platforms and the projects require more people, more teams, more sites, more departments, more companies, more processes, more reporting, more feedback and more historical data,

... More, more, more... Moores law rules!


But like more photographs don't make better pictures, more software does not make better software. But also, when making software becomes easier -- to make us more productive in terms of lines-of-code -- we teach people to do without thinking or understanding. Making a photograph becomes a point-n-click, and making a software systems becomes... point-n-click.

Slowly, we lose our ability to be patient enough to think and think carefully about what we are doing and what we are making. Lighting and composing a picture is about using available ambience and organizing things (strobes, angles, positions, attitudes, etc.) to achieve a desired effect, not about installing the equipment and press the button for a split second. Similarly, making software is not about installing and configuring the tools, dragging around objects and diagrams and pushing a button to build and run the software.

It's all about thinking carefully about what you want to achieve and what the effect may be of your lighting and composition choices. It's okay to try and retry, to run (incomplete) test shots to see effects like background, lights and shadows. But ultimately, we're better off producing less results with better quality than an unmanagable volume of crap.

Is software development a commodity

IBM goes for open source. At first I could not believe that a company like IBM would ever consider giving away anything to anyone. But with Eclipse they do and by donating a subset of the Rational Unified Process they are doing it again. Why?

It's like fishing. They throw out the bait to catch the big fish.

IBM goes for the big money. And where is the big money: information management. A way to achieve that is (1) by penetrating enterprises at corporate executive level and (2) making companies totally dependent on products of IBM. One of the most powerful ways to create total dependency is by giving away for free. But IBM is not giving away Eclipse or RUP, they are giving away software development technology, what used to be Rational. IBM is making software development an engineering commodity, like a mobile communication (which started off by giving away mobile phones) or car navigation systems.

This is a very smart move. Microsoft has become big by giving away commodities like Internet Explorer or Windows Media Player, and by condoning illegal copying of Windows and Office for a long time. Now Microsoft is trying to make money with .NET and IBM is slowly giving away software development technology. Which one to choose: pay for Microsoft or use for free from IBM. Giving away technology and binding customers to your products that make use of that technology, that's a way of creating a competitive edge!

Technorati tags: Software development, Methodology

It's all a game

In a blog, I read how IBM enters the gaming world and this brought me the idea that there may be more commonality between software development and online gaming then we might expect, such as

• Team game
• Strategic approach
• Online community
• Common goal
• Common set of rules
• Computer infrastructure
• Continuous learning

One of the nice things about strategic games is that the game vendor has created a virtual world that contains almost every capability to face the game challenges in order to achieve the ultimate goal of the game. You can give instructions, set assignments, get status report of all kind of aspects, do what-if analysis, and much more. Everything is represented in colorful, 2-D or 3-D graphics, accurate reposiories of artifacts, drag-n-droppable, interactive, on-line multiplayer and joyful to do. And how much does this game cost? A few dollars or euros. How much time to install it, get it working, learn it? A few hours, or may be a day or two.

Compare this to the software development environment that an average software development organization provides to its developers and the costs and efforts that are involved in setting this up, determining the rules of the "game" (procedures, practices, workflows, quality criteria, etc.). Expensive CM systems, SDE tools, a lot of in-house made scripting to achieve automation and reporting in spreadsheets and intranet sites.


What if we would create a software development environment à la Command & Conquer or Pharao? What if we would make an interactive, on-line multiplayer game for software development? My first intuition says that it can't be done or else it would have been done before. My second intuition says that if we spend billions on movies and games in the entertainment industry and find millions of people enthousiastic about it, then apparently that's the way to get people moving.

Software development in healthcare

In the Rational Edge e-magazine of November 2005, Tony Bosselait, IBM Business Executive for Worldwide Healthcare Industry Software Sales, and Dr. Ben Amaba, IBM Rational Public Sector Industry Solutions Executive are interviewed.

Although it is an interesting article about healthcare IT, it struck me that the real challenging side of (software or system) development in the healthcare arena is avoided. The advanced technological challenges concerning systems for diagnostics, imaging, treatment and therapy are left out of the equation in the article.

It is in this area that extremely complex real-time systems, advanced technology and advanced clinical application meet the critical health and safety requirements of patients, medical staff and scientists. It is in this area that business, technology and society truely meet.
One of the challenges that IBM (and software tool vendors in general) should address is not only to provide the tools to do modeling, architecture & design, coding, integration, verification and validation, configuration management, etcetera, but to do it with extremely large datasets of system assets/artefacts, in an extremely flexible way (e.g. refactoring, rapid prototyping), for extremely large systems with extremely complex, real-time behaviour, for extremely large development projects that cross not only geographic areas but also enterprise boundaries (multi-company collaboration to bundle pluriform competences) and extremely transparent across all of those disciplines (business modelling, requirement management, etcetera).

It has been evident to me that IBM is avoiding these real challenges and focusses more on the "big money" business (i.e. enterprises). Stakeholders like that! It is a large market and there is a lot of money going around. However, technologically large and complex systems are emerging into society with agressive speed (such as health, home/entertainment, automotive/aeronautics, semiconductor, (tele)communications, etc.) and e-commerce depends on it for its growth. Afterall, you can't sell what you can't make. Society is more and more relying on protection of health and safety in a automated, transparent, even invisible way, not only in a military way (e.g. counter terrorism) but also in an everyday situation like healthcare or traffic-safety (e.g. car safety). More and more, we rely on the correctness of automated guidance systems, surveillance systems, monitoring systems and even systems that take over our actions to cope with emergency situations.

Development of those systems require the most sophisticated and advanced development tools / methods. Tool vendors like IBM Rational do have a responsibility to society to enable the technological means for (software) development. I do understand that IBM rather goes where the most money is, but ultimately the industry needs far more sophisticated tools and methodologies than what is currently available in the marketplace.

A vision on future software development environments

Recently I contributed to a discussion (on a restricted discussion group) about the (lack of) user-friendliness and user-required functionality of a discussion forum website. It is the website of a tool vendor that sells software development tools. Apparently, this company is more interested in selling software tools than in applying them to give their user community a good user experience with them. You probably guess which company I am talking about, or at least consider two candidates.

One of the things I was wondering was that if they are not willing to invest in providing a strong, flexible and properly equiped website facilities, but the users do want a strong, flexible and properly equiped website facilities, why not let the users themselves develop it. This would perfectly fit in the open-source approach that this company (claims to) support.
Even more, if this company would provide all their development tools (requirement tool, modelling tool, design tool, configuration management tool, defect management tool, test tools, deployment tools, project management tools, etc.), in an integrated software development environment, fully operational with licenses and databases, then they only have to find trusted users who are willing to invest their free time in it (or their bosses time).

This is one side of the story that is about to come. The other side is that there are a lot of companies in need of good (software) developers and there are a lot of companies that hire good software developers on contract basis to those companies. The problem is that with the globalisation (of which outsourcing to India and China is an exponent) the "good" people are available at the wrong place. So either they have to move or travel, or they have to be able to work remote (or by multisite replication and data interchange). This brings an extra burden on the software development environment (and the management) and since it is not a core business process the senior management is unlikely to invest a lot of time, effort and money on it.

But... if we look at the open-source development community, they have a different approach which I envision as the future for (commercial) software development too. For open-source development, the "foundation" either provides a downloadable software development environment or a website (or conglomerate of websites) that provide a remote software development environment. This way, the foundation provides the means for software development and brings it to the people working in software development, rather than bringing the people to the software development environment. This way, the development community can be (and usually) is distributed globally, yet forming a single development group.

Given that it is becoming more and more difficult to setup, maintain and evolve a mature software development environment, with state-of-the-art development methodologies, state-of-the-art tools, perfect integration and interaction between the various parts of the facilities, streamlined information exchange between different data bases, I expect that the future of software development lies in software development hosting companies. These companies are similar to email hosting, website hosting, blogger hosting, search hosting, portal hostings, etc.

• SDE hosting companies provide a complete software development environment, consisting of tools and database for business modelling, requirement management, architecture and design modelling, coding, testing, deployment, configuration management, project management, continous integration/builds, etc. These tools are licensed, integrated and able to exchange and share information and upgrade with the pace of the tool vendors.
  
• SDE hosting companies provide a (probably limited) set of development paradigms, from RUP to Prince2, Agile, SCRUM, XP, Waterfall, Spiral, Incremental and other "standardized" methodologies. They watch the trends and upgrade with the pace of knowledge/competence development in those areas.
• SDE hosting companies work for a (probably limited) set of application domains or market, such as web-development, financial products, health, security, automotive/aeronautics, etc. so they can tailor the methodologies and tools towards it. They will probably also specialise in small, medium, large or extremely large (and complex) products.
• SDE hosting companies provide (restricted and secure) access to the software development environment from all over the world, through web-clients, VPN or other secure means, possibly with multisite replication behind the scenes to improve local performance. This way, multinational companies, teleworkers and travellers and open-source works can make use of it.
  

In particular technology companies often require quite fundamental hardware development (new chips, dedicated equipment, etc.), mechanical, optical and physics developments (miniturising in semiconductors, radiology science in health and treatment, encryption and recognition in security). So the SDE hosting companies must also provide hooks to (automatically) transfer, install and test on in-house equipment outside the SDE host environment and feedback information to the SDE host environment.

Client customers, i.e. companies that actually do the software development to make their products, will hire the SDE from a SDE hosting company and will be relieved of setting up all the different tools, databases, exchanges between database, etc. and don't have to "invent" and maintain the methodology based on "some guy who has read the book about it" or a consultant who is more interested in making managers happy, avoid company politics and get his money than in achieve the best solution and overcoming the resistence and politics do achieve that (and risk his reputation and the reputation of the company he is representing).

So the SDE hosting companies will only provide the means in terms of tools, databases and processes, but do not do the actual software development with it. Because they are specialised (or will become more and more specialised) in tools, databases and processes, and since they service various different customers, they can pay explicit attention to optimizing the software development environment by using specially configured servers, performance tuning, load-sharing facilities, fault-tolerance and fall-back, etc. while spreading the costs over a multitude of customers. The customers will save the overhead of "all those servers doing almost nothing for 70% of the time and being overloaded for 30% of the time", outage due to maintenance, upgrading, troubleshooting, etc.

Using this approach, we will separate how to develop software from what software to develop. For decades, many large and complex companies have been so arrogant to say that they are "different" because the software they make is so much different, and thus refusing to comply with the "same" way of software development. For decades, the tool vendors have been so arrogant to say that their tools and the methodologies that lay behind them are the solution to make software development more productive, more reliable and predictable, cheaper and faster, without actually being able to help when even with the tools the customer is not able to get it out properly ("You are using it in an inefficient way, because of ... your architecture ... your processes ... your company culture ..."). I think the time has come to accept that how to develop and what to develop are two different fields of expertise.

Software is knowledge, not a product

In a post on the Extreme Programming Yahoo! group there was a mention about an article The one minute risk assessment tool. In this article it is claimed that one of the main risk for failure of software projects is the perception that software is a product instead of an embodyment of knowledge.

Software Factories

Brad Appleton reminded me of 4 articles about Software Factories on the Chicago Agile Developers forum.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/softfact3.asp
http://theserverside.net/articles/showarticle.tss?id=SoftwareFactories
http://allconsuming.net/item.cgi?isbn=0471202843
http://www.sei.cmu.edu/SPLC2004/

In particular the article from Jack Greenfield of Microsoft seems interesting. It is a 4 part series, so soon I will have a look at the other parts.

If it's possible, it's allowed

Last week I had a meeting with a customer that is in the process of implementing a new configuration management system. We had a tough discussion about building in security measure to protect the system from malpractice of software developers.

Surprisingly, they were very much concerned about deliberate, intentional misbehaviour of their software developers. For them it was even more an issue than intentional, accidental faults. The argument was that intentional misbehaviour may be very difficult to detect because the engineers try to be extremely smart on that. The risk they are concerned about is that after delivery of those changes, the integrator may not discover problems in the system through the build and smoke test. So the problem is propagated to system level and may be even to the customer.

What they try to do is maximise the security mechanisms to make it is impossible for developers to bypass it. In my opinion, it is absolutely foolish to even try it. First, you will never outsmart those saboteurs and try it takes extreme costs and efforts. A better approach is to build in logging and detection mechanisms. Then, if someone runs in an unintentional malpractice you can build in protection mechanisms, but if someone deliverately hacks the system he should be warned first and fired next regardless of his other competences and knowledge level. It is better to spend the money on finding the right people and on try to secure the system against those terrorists.

What do you think we should do about it?

The Art of Software Creation

To me it is highly arguable whether software creation is an art. In the early days, the challenge of software was to find the most efficient or the most effective algorithms to crack a mathematical problem. Nowadays, most of the creative art has made place for crafted engineering: integrating existing concepts and technologies into new systems.

Nerds and wizkids, able to work in isolation and inventing the most unexpected solutions have been replaced by architects and engineers working together in efficient social structures to build and integrate predictable results in a predictable way. Beauty and cleverness have been replaced by control and productivity.

Did all the fun disappear then? No, it has changed. Instead of being a soloistic challenge (like athletics), software development has become a team challenge (like football). And instead of breaking world records time after time, the objective is to play in the champions league beating the competition. It requires completely different strategies and tactics. To me, being in a winning team is more fun than being better than everybody else.

Software should be about change

It has been said many times before: "If there is one constant in nature, it is change."

Yet, in many software development organization people are striving for reproducibility, predictability and (management) control. Plans are made and approved like they are carved in stone. The software architecture is designed to prepare for extension of the product, allowing it to scale for more functionality, but the software architecture itself is hardly ever designed for change. A a certain point in time, a major refactoring is needed to get the architecture going again.
The software implementation is often rigid and optimizations for performance have often lead to various unexpected side-effects when the implementation is changed. However, to accomodate for new and extended functionality, the software must be able to change.

And then the people. Products are becoming so complex and extensive that nobody has a total overview of the system. Only a few veterans still know the original ideas behind a lot of the concepts and principles that are used and never questioned, and above all never changed. And they are not changed because you should not fix it unless it is broken.
Not even the crew working on the product development can bear much change. The learning curve of newbies is too long and the risks of losing valuable knowledge are too high. And manu people resist change of habits simply because they don't like to change their habits.

What was the advantage of software again? It is much easier to change than hardware, much more flexibility. It is software. But in practice, software is hard to change. And when it needs to change, business demands a solid statement of profitability. Entrepreneurs are hardly found under established organizations.

In some occasions companies are lucky: they get into business trouble through the way they are organized and the lack of change. Managers are forced to accept change, to invest in entrepreneurship. But again, only very few learn from it to design the product and the organization for future change.

Developers want to be dumb

Today I got another confirmation that developers aren't dumb, they want to be dumb.

We have defined a development process with several promotion levels. The system is divided into subsystems and baselines of the subsystem need to be promoted to published before other subsystems are allowed to use it. Today, one of the integrators came to me asking whether it was possible to prevent developers from using baselines at a lower level.

"Why", I asked. The reason is that if developers use baselines at a lower level, they may submit extra problem reports about the subsystem they use. "But they know they use a baseline that has not been published for use yet!", I replied. But developers simply do not care! They will take any baseline from the other subsystem, regardless of it being released for use or not. And if they run into problems doing that, they will blame SCM for it:

Motto: "If it is possible, it is allowed"

Developers aren't dumb, they just act dumb and do not care about empowerment, responsibilities or plain professionalism. How can we ever protect them against themselves?