My most recent post on cultural change at Microsoft included a reference to the legacy of Bill Gates, and suggested that he should be recruited to help in this endeavour. I thought it was useful to expand on why I think that is the case.
Recent events in southern Alberta have led to an interesting discussion for the City of Calgary: what to do with $52 million in surplus tax money. The surplus was due to a number of factors (which don’t really need closer explanation here). The thinking was to find a way to “return” the money to Calgarians, in the form of a projects or initiative that otherwise might not be funded. A consultation process, using town hall-style meetings and web-based surveys, was created and conducted to decide which project should be chosen and funded. But the recent flood have many calling for the city to use the $52 million for flood relief and related activities. Some city councillors disagree, being (stubbornly?) held to a process that was created and held prior to the floods. Just how closely should any organization adhere to some process, particularly in light of dramatically changed circumstances?
This is the conclusion of a multipart series on changing core technologies in a software system or software-based service. Part 1 discusses the costs, part 2 discusses the risks and part 3 discusses the roadblocks to change.
Change is inevitable. Requirements for your business and your customers will change over time. Technologies and vendors come and go. At some point, you will have to make a major change in your technology in your system. That simply cannot be helped. The key to surviving that change is to:
- Understand the risks associated with the change
- Understand the costs that come with change
- Know the rewards that the change will provide
- Balance the cost and risk against the rewards as well as you can
Recognize that, no matter how much you analyze the risks, costs and rewards, you will always miss something. The goal is to have a plan that not only addresses the technology change and how it will occur, but how you will deal with anything that was overlooked or unanticipated. That isn’t very different than any other project or business plan, and is hopefully part of your corporate culture.
I hope that this series was helpful in outlining some of the factors involved, and providing some ideas for how to make a technology change successful.
This is part 3 of a multipart series on changing core technologies in software systems and software-based services. Part 1 talked about the costs of change, and part 2 discusses the risks associated with change.
In some organizations, there are cultural roadblocks to change (or to avoiding change). Let’s face it: many people don’t like change. Change can be a cause of stress and anxiety. Some organizations have built up an inventory of processes and procedures that not only track and manage change, but in some cases work to prevent it. Not all groups and companies do this, but some do, and you need to be aware of the cultural and procedural issues within your own organization that will try to prevent or subvert a change.
For people who are averse to learning something new, changing technologies can threaten or scare them. This can be particularly true when a technology currently in use has been there for a very, very long time. For some, their entire career could be staked on a particular technology. The best you can do is try to motivate them, train them and help them along. You don’t necessarily want to “discard” someone because they outright refuse to “play along”, but in the worst case, you may have no choice. If the technology change is mandatory in some way, then so is the team’s need to move along with it.
The other threat, though, is some variant of the “rabid fanboy”. For them, some element of the current technology can take on a mythic status in their mind, turning them into something resembling a religious zealot. Some element of the technology you use today becomes an extension of themselves, and removing it means a repudiation of them as a person. They oppose the change simply because it doesn’t fit within their current technological belief system. To be honest, I don’t know what you can do about this type of person. Hopefully you don’t have someone like this, but I don’t see much that can be done.
For some organizations, there is a tremendous amount of process and procedural overhead. I have seen this in action in the past, and I only found a couple of ways around it. One was to simply grit your teeth, come to grips with the “rules of the game” and work through the process. Do the documentation, put together the justifications and analysis and simply grind your way through it. Even in the worst systems, if you can show how this will save money, improve profitability or avoid business failure, it should work out. It may not always, so be prepared for a long, uphill battle.
The other way (which can be career-limiting at a company, even if you succeed) is the “beg for forgiveness, rather than ask for permission”. If you do this, though, then the change had better work out with limited or no downside. You have to be 200% right, and you will have to justify the means the end no matter how good they are. This approach will only work for a select few people, and failure can have ugly consequences.
If you have the luxury of working on a small, motivated team at a smaller organization, then (hopefully) you can rely on rational thought and intelligent discussion to resolve any reservations about making a major technology change. This can be where prototypes can help again. Rather than relying on someone else’s case studies or a vendor’s claims, you may have to “prove it” to the rest of the team by actually showing the new technology in action.
No matter how good a change in technology will look on paper, at some point you will find someone who will oppose change. Their reasons may not appear rational, but it may force you to make sure you do your homework on all of the other aspects, and take away the opposition due to cost or risk.
This is part 2 of a multipart series on changing core technologies in software systems and software-based services. Part 1 discussed the costs of change. This discusses the risks of change.
No change comes without risk. There is always risk, no matter how small. The risks can be broken down into a few main categories:
- technology risk
- project & delivery risk
- business risk
- operational risk
Some of the technology risks are obvious: this is a new product or device that you and your staff are unfamiliar with. You will find things that don’t work as you would have expected, and this can have an impact on your product or service. You will have to find ways to work around behaviours that are unwanted or unexpected (but may be “as designed” for the product) or problems that exist in the technology. Some of the more subtle risks, though, can occur when you depend on side-effects in the software. In some cases, you and your team may not be aware that the “feature” you are using is actually a side-effect or a bug. Where you get burned is if the side-effect is changed or removed, and now your system doesn’t work. This can happen with technology you already have, but presumably you are already familiar with your current technology and are in contact with the vendor or author. In the case of new technology, the features are unexplored territory which might not come with a full set of maps and guideposts.
Generally, technology risk can be managed and overcome, even if it requires brute force. The risk to the project during transition, and delivery of the product or service, can be harder to manage. Unexpected behaviours that require time for workarounds, or require a delay while the vendor fixes the problem, will have an impact on the schedule. You can try to build slack into the schedule to try to allow for this, but unforeseen issues are called that for a reason: you didn’t see them coming. You can “anticipate” by saying “we expect there will be problems”. But that doesn’t mean you will know, in advance, if these problems will be numerous or not, and how much of an impact they will have on the schedule.
The schedule risk ultimate affects your ability to deliver in a reasonable or useful timeframe. If you are up against a hard, fixed deadline, this kind of change can effectively kill and product or service if the unforeseeable becomes the unrepairable. Whatever the impact on the schedule, impacting delivery can impact business. Now you are into territory where the project can have a negative impact on revenue, cash flow and profitability. If the technology change is significant enough, you could be betting the business on this change.
How do you mitigate this? Up until this point, a lot of the risk can be minimized through prototyping, and tight iterative schedules. Don’t let the team go away for multiple months, with little or no visibility into the on-going effort, only to find out shortly before a deadline that the system won’t be ready. Building prototypes up front, prior to committing to schedules and deadlines, is essential. It gives the team a chance to try to understand the new technologies they are selecting. It also gives you the opportunity to abandon a change if you realize that what you need from the technology simply isn’t available, or isn’t going to be as advantageous as you had expected.
There are other business risks: the risks associated with dealing with a new vendor. As part of risk mitigation, you should be looking at the financial stability of your newly selected technology provider. They may have amazing products that will “solve all your problems”, but if they are going to be in bankruptcy protection next week, that will spell trouble for you. This isn’t to say you only buy products from extremely stable vendors. Its a question of balancing the risks of your vendor not being there, or not being large or mature enough to handle your day-to-day needs, relative to the benefits of using their technologies.
So, the system is built, installed and ready to run. Now you get into operational risk, and this can be mitigated early as well. The teams that will run the new system should be involved during the prototyping stages. They should be given a chance to see the technology in action, and start to identify the tools they will need to be able to manage and monitor the system. Leaving the operations component until after the system is built can mean problems when the system is running. If the operations staff don’t have an idea of how the system works, of what they should be monitoring, and what they should be looking for in terms of problems, then you can have an impact on the service (which can be a further impact on the business). Customers don’t like systems that are unreliable, unpredictable or have extended downtimes, and this can result if the operations team doesn’t have the tools and training they need to do their job.
No change is risk-free. Presumably you are changing technologies for some gain, either in system performance, stability or operations cost. As with anything, there is a balance between the risks associated with making the change and the rewards that come with it.
Part 3 continues with a discussion on cultural roadblocks to change.
This is part 1 of a multipart series on changing core technologies in a software system, or software based service. This part discusses the costs of change.
No change in core technology in a system comes for free. No matter how small, there is always a cost. There are the obviously dollar costs associated with acquiring new technologies, but there are other costs that tend to get overlooked. Even when looking at the monetary cost, there are elements to the initial purchase that can sometimes be forgotten.
When changing technologies, the first and most obvious cost is the money needed to buy or licence the new technology. Obviously, new hardware and/or new software means an outlay of cash for the product plus support. What you need to remember to budget, though, is for the ancillary costs, specifically for training. Unless your staff is already familiar and experienced with the new technology you are bringing in, you will need to train people on how to use it, how to run it and how to maintain it. Training isn’t generally free, and while you can learn something from available sources like books and on-line knowledge bases, at some point you will have to send people on training courses. You may also have to bring in specialists for a short period of time to bring your team up to speed. All of this needs to be factored into the budget.
For product companies, a technology change also imposes these costs on your customers. Either you have to build the cost into your product (through the initial licence fees or through maintenance fees), or the customer will also have to go out and obtain the new technology themselves. They, too, will require training. These costs will have to factored into your customer’s budgets when they review their licencing and maintenance fees with you.
So, you’ve spent the money, bought the new products you need and paid for your training. The full bill for your change has still not been paid. Now you have to take the time to migrate to these new platforms and products, or integrate them into your system. That takes time, and unless you are in the unique situation where your existing products and services do not require any sort of care and attention, you either have to split your team’s time between keeping the old stuff going while working on the new system, or you have have to increase your staffing to try to do both in parallel. Either way, you have to pay these people. Depending on circumstances, this can also impact delivery dates for existing and new product. This can result in lost time for customers experiencing problems with current product, and lost opportunity for the new product if the delivery is slipped out far enough.
Another cost that can be overlooked is the cost of tools for administration, management and monitoring of the system (either used by your staff or by your customers). Sometimes a new technology comes with everything it needs, but on occasion it may not. That means you have to step in, and spend development time and dollars building the tools you need to be able to watch the system when it is running, be alerted to issues as rapidly as possible, and diagnose the problem to fix it.
For systems that you run as service, these changes also can cost during incidents with the service once the technology is deployed. Your operations staff won’t be as familiar with the new technology. The tools and processes they used to use may not work, or may work differently. That means more time to try to figure out what is happening when something is going wrong, and more time to correct the problem. This issue should diminish over time, as the system becomes more mature (and presumably more stable) and the operations team become familiar with the tools and diagnostic processes. But initially, resolving incidents simply will take longer. Depending on the impact and severity, that in turn can hurt the business in the short term as customers pull back on their use of the service (or pull out entirely in some cases).
The last set of costs are those associated with the transition to the new product or service. This can mean spending to keep the current technologies in use while you move to the new technologies you’ve selected. The result can be duplicate servers, duplicate storage or duplicate software/maintenance fees for a period of time.
Moving to different technologies is not free, and it isn’t necessarily cheap. It can take months or even years to make back the costs associated with making a change. But it may also be worth it for other reasons, or may be required simply because a technology you depend on is no longer available. No matter the reason, a thorough review of all the costs is a good idea, so that you don’t have too many surprises either for the budget, the project plan or during operations.
Continued in Part 2 discussing the risks of change.
This is a 3 part series discussing issues around replacing core technology in a software product or software-based service. My intention is to lay out the risks and issues, and illustrate both some of the obvious issues, but also discuss some of things that are overlooked. When I talk about core technology, I am talking about changing things like:
- The underlying operating system support
- The hardware the software runs on
- Messaging fabric and middleware
- The programming language the system is implemented in
This isn’t a discussion about areas around architecture or design change (although many of the same issues apply), or around upgrades to existing components. This is about fundamental change of underlying technology.
When a system is first built, a number of decisions are made around the technology. Some of the decisions are based on cost, some on available resources (including people who know the technologies) or simply because “it looks cool”. These decisions offer opportunities and benefits, but they also close off certain avenues for your product or service. How many avenues are closed depends on how far down the stack the technology is. Selecting Java, for example, opens up a wide variety of platforms, but tends to limit you to Java-based and Java-enabled technologies. Selecting a particular operating system or hardware platform can be more limiting. If you were to select Windows, then you are limited to Intel-based machines. Selecting Linux would allow the use of Intel or PowerPC.
There are times, though, when there is a significant shift in requirements or the business environment. Perhaps you have a system that was geared to a modest number of users and lower performance targets, and the customers decide that they want orders of magnitude more users and significantly higher throughput. The technologies you selected could limit how far you can scale up, and to go further requires a major change.
As another example, perhaps the vendor you are using is discontinuing a piece of technology, or going out of business altogether. Unless you are comfortable running on unsupported technologies, this too requires you select an alternative and move to it.
The next three parts will discuss the following topics around change:
- The Cost of Change, including more than just upfront dollar costs
- The Risk of Change, talking about areas like business and technical risk
- Cultural Roadblocks that can delay change, or make it difficult to accomplish
Finally, the series will wrap up what has been discussed.