2023

There’s hidden treasure in tracking and capturing business benefits from change but it is not always easy and often neglected

I’ve tried to resist the temptation to start this piece with a gruff and throaty “Aye, me hearties!”, the topic being about searching for hidden treasure, or rather more prosaically, benefits realization.

You would have thought that after all the hard work running a project and implementing change people would be keen to make sure the promised benefits are actually banked. Sadly, no, this is often neglected and does not happen. There are probably a number of reasons:

• it’s not in the “plan” / budget, and nobody is on the hook for it…
• tracking benefits can be hard and requires discipline to stick with it as things plays out over a longer game…
• implementing actual change means changing some ingrained behaviours, and it is easier to not to look…
• there was more promise than reality in the proposition and nobody wants to get called out…
• nobody wants their pet project to be starved of funding so don’t want anybody looking too closely…
• tidying up is boring and not glamorous…
• “not my job”…
• the project objectives were to deliver a load of features (what), and benefits (why) weren’t clearly defined…
• can’t measure / see the effect of the changes made…

To get in context and do some setup for the topic, we can define three levels of business, like this…

The Strategy level guides the direction of the business; Business Change actually makes changes to the business; and the BAU operations layer gets on with actually doing the business, in between being guided and changed. To see where the delivery of benefits actually needs to get tracked, we need to project that out over the “Think-Build-Run” lifecycle of change and operation, thus…

You can see that “X” marks the spot for where this should happen in the top right hand corner in the intersection where Strategy meets Run, bubbling up from the monitoring of impact in the business change level. Depending on how thin the oxygen is at that exalted level, Strategy types might think they do not have to get their hands dirty, but in business and battle planning terms, it makes a lot of sense to check out whether you are actually winning or just staggering along from one non-event to the next.

In an ideal world, the benefits tracking would have a gimlet-eyed, CSI-like forensic analysis of the results of the strategy as it plays out, like this…

…and, indeed sometimes it can be like that, where the change is intimately linked to the business performance. Things like company or product revenue, or operating costs are usually good things to look at, although sometime you need proxy measures and targets for the effect of the change itself (like, conversion rates) which do then track into actual business performance changes.

Just a quick anecdote from times past on when that link is strong, I had a discussion with an EVP of a global financial services company during a technology investment portfolio prioritization programme.  We were discussing him providing a proforma  business case and NPV for a project to futz with the FX rates on some of their cards.  He just said "I'm not doing that, I know we'll make sh*tloads of money", so that was that...the project featured high on the priority list

But, as often as not, tracking the impact is more like looking into a murky fog through the wrong end of a telescope…

…and seeing meaningless confusion (or a donkey).

This is a frequent problem for parts of an organization that are not well connected to customers and revenue, or where the effect is unclear, e.g., for developing enabling infrastructure, where direct business benefits cannot be linked to the change (Hint: try to get the business front-end on the hook for some actual business benefit for changes like that, makes the NPV positive, and the whole change becomes more meaningful from a business perspective)

The challenge often derives from the measurability of the impact on performance brought about by the change (if that was actually defined, a different challenge, of course). To measure that you need to look at the difference in some metric that should be impacted, which is relatively easy when you have a clear baseline before and can measure the performance after the change, or you can test the effect in the present by parallel testing of systems with and without the feature, like A/B testing for conversion rates with different customer journeys / click-flows on a web-site of mobile app. Those are the first two cases like this…

…however the breakdown occurs in the second two cases where, either:

• there is no “before” baseline, obviously not good, but can be fixable or perhaps reference external experience to determine the likely impact; or,
• worse, the comparison would be between either of two possible future paths. This final case is the most difficult as you would (in theory) have to compare performance between:
  • A – How we would have performed in the “Path Not taken”; compared with
  • B – How we actually perform on the path we have taken

The case of the “Path Not Taken” is quite typical of commercial changes to the technology development process itself, something of a meta-topic perhaps (changing the change process…). Technology development has a huge discretionary element and there are many ways to waste money with it, so an important and essential question like “Are we doing technology development more effectively and efficiently now?” requires discerning analysis and thought.

However, to start with the basics across the broad spectrum of change, you need to set things up for success. One of the foundations is to start thinking in terms of outcomes and handing out the investment funds against outcomes rather than budgets by department/cost center or whatever. Delivering outcomes is typically multi-functional and cuts across organizational lines in order. There are a number of key elements to the recipe which you can see below.

Having measurable outcomes is fundamental to success in having real benefits to track, and the level at which they are defined sets the scope and breadth of their impact across the business. The higher up the performance hierarchy the more likely they will directly impact the fundamental performance of the business…

Outcomes need to be specified properly at the start of the any change journey, something like this:

• Aim is to define the achievement of a specific improvement brought about by a series of coordinated actions in near term scope, say, up to 3 months out;
• Outcomes should generally be SMART (Specific, Measurable, Actionable, Relevant, Timebound), or whatever your version of this acronym happens to be, “measurable” is not negotiable though.  They are specific, reasonably sized/feasible incremental beneficial results we are looking to achieve which contribute to the higher level business goals;
• They need to be focused on specific improvement, so wording needs to be stricter in definition, and should be in this syntax: “<Improvement verb> <some Attribute(s)> of <some Thing(s)> by <some Target Measure(s)>”, using
  • <Improvement verbs> like: Improve, Streamline, Optimise, Tune, Reduce, Accelerate (but not Do, Create, Assess, Evaluate, Analyse, Synthesise, Perform, Enumerate, Distribute, Communicate &c – these are “doing” action words which are part of the “How”)
  • <Attributes> like: Quality, Accuracy, Effectiveness, Awareness, Speed, Timeliness, “Fit”, Cost, Usability, Accessibility, Reliability…
  • <Things> are whatever entities of which we need to improve the attributes
  • Specific <Target measure(s)> of success (percentage, absolute value, etc.) of the metric (or metrics) meaningful for the improvement of the Attributes 

There are many ways to define improvements, here are a few examples…

When it comes to setting up the processes to track benefits, then we can slightly redraw the business level-lifecycle continuum from figure 2, and pick out the feedback loops…

The outermost Strategy & Investments loop (loop 1) is all about investing in the right things and typically runs on a quarterly cycle. The Business Change inner loops are all about prioritizing work to deliver the right features (loop 2) and delivering quality code (loop 3) which run bi-weekly and greater than daily frequency (or multi-quotidian, if you want to look that up in the dictionary). The Operations continuous improvement loop (loop 4) feeds upwards into the higher levels.

You can translate that conceptual model into an actual time-scape which integrates the loops, so that you have a rational approach to developing changes needed with quality delivery and proper tracking of the benefits to support regular investment review to accelerate or throttle funding…

The features of this particular time-scape are:

• Regular major releases of functionality, with content prioritized according to business need, maybe from outputs from a number of sprints
• Periodic feedback from the “market” (however that is defined), both on features and benefits delivery, into investment review
• Opportunity to reprioritize deliveries to market to focus on higher value features/service elements
• Opportunity to stop spending on a individual stream at any point

Obviously you can roll your own version, and also rework it for more generic business applicability. So…

(gruff and throaty) Aye me hearties, there’s treasure to be had!

Earnback of service credits in technology service agreements is an absurd concept that should be rooted out of contracts

As often seen, service credits are commonly used in technology service agreements to compensate clients for a service provider’s failure to meet a committed level of service. Whilst service credit mechanisms can range from simple to byzantine, earnback of service credits is one of the more egregious pieces of nonsense you can see in a service level agreement.

To take a step back, there is a social concept that you can somehow repay your bad actions to society by doing good works. It is a form of utilitarian philosophical thinking where the greater good can outweigh the bad; perhaps industrialised in the sale of indulgences back in Medieval times. Conceptually, a murderer can go to prison for many years to “pay their debt to society”, but they still killed somebody, nevertheless. So, the concept is based on some possibly dubious principles and which does not always survive robust scrutiny. Indeed, as often is the case with socio-philosophical inventions like this, there are conditions under which the simple equations fail, and the answer they produce is nonsense.

As an aside, and I hesitate to bring it up, but if you want to test boundary conditions and how a rule, social or otherwise, works or doesn't work in extremis, then try applying the Jimmy Savile Test (OK, yuck).  If you don't know, Jimmy Savile was a uniformly horrible person who curried favour with rich and famous people in the high strata of society with his apparent "good works" as a cover for his obscene behavior.  
The test is this:  if somebody proposes a generic rule then you think to yourself "what about Jimmy Savile?", and if the answer is "Euw!", "No way!", "Disgusting!", then the rule is probably a bust.

Back to earnback, which in the context of a commercial service agreement, means the service provider doing some good works to overcome below par performance at some point in time. Those good works then exempt them from paying compensation to the client which would otherwise be payable for the poor performance. Timing-wise, the performance bump can be ex-ante where previous good works put money in the bank to credit against claims, or ex-post where the good works occur after the default. In the generality, it is all about the push and shove of risk transfer between client and service provider; earnback pushes back risk to the client.

As a principle, earnback of sorts could make sense for a development activity or manufacturing piecework process creating widgets where poor rate of production in one month can be offset by increased output in later months so that the same expected pile of widgets is created in the overall period, meeting the expected weighted average production rate.

However it makes little sense in an operating service where the service delivered is of its moment and the experience is transient. In the service world, a dog, as they say, is only as good as its last trick – not an average of its good and not so good tricks. Sometimes it makes even less sense…

Consider this typical earnback clause that you might find in a service agreement…

This says what it says which is that if the service provider behaves for three months following a service level default then they can play their “get out of jail free” card and not pay the service credit for their default. Graphically, it looks like this…

The story is: the service provider commits to deliver to a service level in the contract. They then default on that service level in Month N, and proceed to meet the service level in Months N+1 to N+3.

The earnback clause magically absolves them of paying for the default…by doing the job they are already being paid to do. This is patently absurd: why does doing that job same as any other time exempt them from paying the compensation? Thanks for nothing!

In the fetid atmosphere of the negotiating rooms when these types of clause were first invented (maybe the 1980s?) it probably all seemed to be a very clever manipulation of risk and incentives. You can see other examples of “cleverness” where complex mechanisms have been designed that don’t make sense in the cold light of today (e.g., service level escalator ratchets which don’t work in a multi-vendor SIAM setup).

In the hierarchy of performance metrics many of the older concepts like earnback of service credits fiddle in the bottom tier of technical performance measures whilst user experience burns: the client suffering an “All-Green Hell” SLA dashboard for a service that users hate passionately. Better to focus design efforts on performance regimes further up the pyramid…

So, root out earnback from your service agreements, let the service provider make its penance at the time of default and move on!