Something Is Happening

I enjoy speaking at these events because they are a forcing device. They require me to summarize what I think I know about a topic and how to communicate it clearly to an audience. I also love meeting new people and understanding and debating their perspectives.

Fortuitously, while I was preparing for the event, I also had the opportunity to attend the NVIDIA GTC developer conference and be inspired by many of the sessions. The timing was perfect, as I could incorporate those new insights and inspirations into my talk themes.

I have structured this post around the three questions I find most executives have:

1. What is the impact of generative AI, and AI more broadly?
2. What are the challenges that get in the way?
3. How can organizations address the challenges to capture real impact at scale?

1. What is the impact of generative AI, and AI more broadly?

Impact, and therefore return, from AI is the critical place to begin.

In the initial phases of the generative AI boom, it was both exciting and gratifying to be “doing something” with generative AI. CEOs did not want to look like technology laggards, boards were challenging them to do something, and investors were initially positive on early news of generative AI application.

However, without real impact, there are only so many quarters you can spin the story before market sentiment sours. Impact is the oxygen that fuels any AI-enabled transformation.

It is great that we have generative AI. It has captured the public imagination and fueled a whole new investment wave in AI. However, generative AI is beginning to feel overhyped. A recent BCG survey of CXOs revealed that more than 90% felt it was more hype than substance and wanted to separate the hype from reality.

The reality is that generative AI adds a new set of valuable tools to the AI toolbox. However, there is a risk that we get dazzled by the new tools and forget the older but proven ones, such as gradient boosting, to name just one.

The irony of being dazzled by generative AI is that organizations risk substituting more proven AI techniques with shiny new objects that are less proven to deliver impact and are riskier. This reduces the overall return from AI investments. In fact, it is unclear if organizations are actually spending more on AI because of generative AI. Many surveys point to AI budgets remaining the same, with money being diverted to generative AI.

It makes more sense to talk about the overall impact from AI, with generative AI as a subset of the available AI tools. Most valuable AI is an ensemble technique, one that pulls in multiple techniques and disciplines to focus on delivering a business result.

What is different about generative AI?

As a small aside, I would like to take a moment to reflect on what is different with generative AI, where it is being used effectively and impactfully, and where it is probably being used ineffectively.

Generative AI was popularized by ChatGPT, OpenAI’s large language model. LLMs, based on transformer architecture, are a subset of generative AI and captivated the popular imagination with their human-like language capabilities.

LLMs are also useful as foundation models, which are basically pretrained AI models on steroids. What is nifty about foundation models is that you have AI that works out of the box.

As a trivial example, you may go to Hugging Face and take an open-source model such as FLAN or Llama, and within minutes create a simple sentiment analysis tool for customer feedback. Is the sentiment negative or positive? What aspect of the product or service is the customer commenting about? What could potential next actions be to address the issues raised?

This is incredible. In the pre-LLM world, you would need tons of labeled data and weeks, maybe months, of effort for a small team to create a similar result. This compression of time, effort, and cost should be a significant game changer for AI adoption.

The challenges of generative AI are reasonably well characterized and hotly debated: IP risks, hallucination, security, bias, energy consumption, and more. One aspect that I find gets less attention is what I view as the seductive but unfortunately wrong-footed intent to use generative AI to replace search.

The idea is that if we move from a world of information retrieval to information generation, we will eliminate waste from retrieving incorrect data. Parts of this idea seem valid. Generative AI generally seems better at understanding our intent than search engines. Try Googling and asking ChatGPT the same question, “which countries are not Japan,” to see how LLMs are better than current search technology at understanding intent.

However, LLMs fare worse on two dimensions.

First, they are not great at getting the right facts. Personally, I struggle to trust anything they come up with. LLM outputs tend to require robust and independent verification before they can be relied upon.

Second, generation is incredibly resource intensive compared with simple retrieval.

Retrieval augmented generation, or RAG, is an increasingly popular way to address some of the deficiencies in LLMs. RAG is necessary to provide company-specific context to LLMs by connecting them into proprietary data stores. However, RAG is new and brittle. How far it can resolve the intrinsic retrieval deficiencies of LLMs is still an important unknown and an area of frantic development.

One area of LLMs that I hope will develop more is their reasoning capability.

Current LLMs are good at generating hypotheses to develop and test decisions. For example, what courses of action are available in this situation? What would have to be true to choose each course of action? How do I analyze and test whether each condition is true?

This prompting of an LLM can yield a reasonably structured logic chain to support business decision-making. Combining this hypothesis generation with RAG-type approaches to test hypotheses based on available data, and perhaps finally some kind of actuator to act based on the most likely hypothesis, could be an exciting development in getting LLMs to do useful work.

The impact on technology companies

Back now to the question of impact. When I answer this question, I prefer to separate the discussion of impact on technology companies from non-technology companies.

It seems obvious that the impact of AI on technology companies has been profound. They are some of the world’s most valuable companies, and many of their business models could not exist without a strong role for AI, either as digital labor or as an integral part of their AI-powered products and services.

On generative AI, a recent Economist survey of the top 100 companies leading generative AI showed that $8 trillion of market capitalization growth had occurred since the launch of ChatGPT on November 30, 2022. Even accounting for broader secular market trends that have been propelling the market, and almost certainly an AI bubble, that is a staggering amount of value creation.

Not surprisingly, the infrastructure players, with NVIDIA leading the pack, have garnered the lion’s share of market capitalization growth. This is expected. During a period of new technology adoption, the infrastructure part of the stack needs to be built out first, and those players are the first to benefit. Also, in a gold rush, those making the shovels usually have the best risk-weighted returns.

The impact on the rest of the economy

How about the rest of the economy?

That question is critical for technology and non-technology companies alike, because without sustained value creation from AI on Main Street and Wall Street, the gains made by technology companies will not be sustainable. We must be able to do something hugely valuable with these new technologies.

The record of impact from AI in non-technology companies is mixed and hotly debated. Returning to the same BCG executive survey mentioned earlier, around two thirds of executives surveyed were dissatisfied with the business impact of their AI investments. Other surveys have placed the return on AI investments at around the 5% mark, below the cost of capital for most companies I know.

However, these averages often obscure a more complex reality. True, many AI investments do not yield much, and I will dive deeper into why in the next section. But part of that may be expected in the current immature technology adoption stage. Also, when you de-average the portfolio of AI investments, you find a wide spread of positive and less positive, or negative, returns.

AI business cases can be some of the most attractive you will find anywhere.

The contours around AI business cases are well established in areas of tried and tested adoption such as customer engagement. For example, 70% customer interaction deflection and automation with a 90% cost reduction for deflected and automated calls is common. Payback is typically in the six to nine month period, and 18-month ROIs can range from 5x to 10x.

With generative AI, it is even possible to push beyond this and obtain 90% or more interaction handling by the technology, juicing the business case even further. Total cost savings in the range of 50% to 60% are possible, and the impacts are also beyond cost, including improvements in customer experience and revenue uplift.

As an extreme example of this trend, Klarna, a Swedish fintech company, used generative AI to automate its customer interactions, freeing up hundreds of FTEs of effort and saving tens of millions in operating costs in a matter of months.

Klarna is a private company, but the market reaction to the announcement was notable. Teleperformance, a market leader in business process outsourcing for customer interactions and a service provider to Klarna, suffered a large drop in its stock price as a result.

Large enterprises typically struggle to achieve Klarna-like results with generative AI. Most large companies are still experimenting with generative AI and have not yet found a sustainable path to scale.

However, when it comes to AI more broadly, some large companies invest hundreds of millions of dollars in AI, have thousands of data scientists and data engineers, drive hundreds of use cases, and deliver billions of dollars of bottom-line impact. As impressive as this is, the billions of dollars of impact correspond only to around 5% to 10% profit uplift for the current winners of the AI race.

At the current level of adoption, it feels like we are just scratching the surface. I am cautiously optimistic that there is more to come.

One area I am most excited by is how AI can drive new scientific and engineering breakthroughs. We have so many unsolved problems that it will be a wonder for the world if AI can help progress our understanding.

Fresh from attending NVIDIA GTC, one thing became crystal clear to me: when we bring data, AI, and compute together, we can uncover new insights.

For example, in biology, AlphaFold was able to predict the structure of all known and imagined proteins, creating the possibility of intelligent nano-engineering of new drug compounds. Only last week in the UK, AI was demonstrated to be able to detect micro-tumors that doctors were unable to detect, leading to better outcomes for patients.

Something is definitely happening, but this underlying potential is yet to be fully tapped.

In the next section, I will discuss why.

2. What are the challenges that get in the way?

Most enterprises are struggling to fully tap the potential of AI.

It used to be said that less than 10% of data scientists’ output makes it into production. More recently, I saw a similar statistic on generative AI mentioning that only around 10% of pilots are scaling to production. Even allowing for an expected amount of experimentation and learning, I find these adoption numbers shockingly low given the scarcity and cost of AI talent.

Why does this happen?

Most organizations could probably list ten or more challenges that get in the way of AI implementation at scale. For simplicity, I boil them down to three principal challenges: value, data, and people.

Value

Many organizations have an unfocused and fragmented portfolio of AI initiatives that reflects a poor understanding of where AI can create value for the enterprise.

Such a set of AI initiatives might unkindly be called random acts of AI, and unfortunately such a portfolio yields only marginal returns at best.

An executive I spoke to recently bemoaned the fact that the best idea his team had come up with was summarizing emails. Instinctively, he felt that this would lead to little tangible incremental benefit, and sadly he was correct.

Data

It is generally said that value in the IT stack has been migrating from infrastructure to applications and now to data. This is an overly simplistic view of the world, but also a helpful generalization.

When we bring data, AI, and compute together, it is almost always possible to create new and valuable insights. Comparatively speaking, the AI and compute are the easy parts. For most valuable applications of AI, the data part is the most difficult to overcome.

If AI is the new jet engine of business, then data is the oil that powers the jet engine. Oil may in fact be a good analogy, as oil is not that useful until it is refined, in the case of a jet engine, into aviation-grade kerosene. In the same way, raw data in the enterprise often requires refining to make it able to power AI.

Many organizations find that they can execute small AI pilots with limited data sets, but as they begin to scale their pilots, data becomes a critical bottleneck. There is both a technical side to this, namely how to extract and transform the data to make it AI-ready, and a data cleansing and governance challenge.

In many organizations, data is not regarded as an asset, so it is siloed, fragmented, and of poor quality.

Significant investments on the business side are also required to clean data. I recently witnessed a data cleansing initiative where, despite two years of effort to cleanse enterprise data, it was still riddled with errors.

To be fair, data cleansing is not an energizing topic for individuals or enterprises. I feel it is the corporate equivalent of filling in your tax return: important, necessary, but probably not the most energizing.

Breaking this cycle is a challenge that must be overcome to scale AI sustainably in any enterprise. The costs, time, and organizational energy required to do so are almost always underestimated. In my experience, for every dollar an organization spends on AI, it spends at least several more dollars on getting the data right.

People

Turning finally to the people challenges, I view these as the most difficult part of scaling AI in an enterprise.

I like to consider people challenges through three lenses: talent, change, and stakeholders.

Firstly, on talent, most organizations are struggling to augment their AI skillsets, and it seems you can never have too much of this talent if you are serious about AI. In a recent discussion with an executive who had built an AI, analytics, and data team of several thousand people, the executive mentioned that talent was still the largest challenge.

Some talent profiles are extremely scarce. For example, there may be only several hundred people in the world who can build cutting-edge LLMs. Thankfully, most organizations do not require that grade of talent. They do at least require AI engineers with real practical experience of getting meaningful AI done at scale. That is scarce talent, and it can be difficult for companies to weed out the fakers from the talent pool in recruiting.

I have seen initiatives delayed by years because the people running them did not have a practical understanding of the complex sausage making required to make AI scale.

If the right talent is acquired, it will not be nurtured or retained unless it is set up to have impact in the enterprise. That includes working on the right tasks and having a real seat at the table. There are many tales of disgruntled data and AI engineers spending all their time cleansing data or doing BI as opposed to AI.

Secondly, the broader change challenge is a massive one. AI might be able to create superior insights, but value is only realized when the organization can turn these insights into business value. This may require a complete redesign of the business process, automation at scale, or a change to incentives. It is critical to think holistically about what enablers are required to monetize the new insights from AI.

Finally, people challenges extend beyond the boundaries of the enterprise to include the full set of governmental and societal stakeholders.

Getting AI wrong can increasingly jeopardize a company’s license to operate, either through infringement of regulations or by reputational impact on a broader set of stakeholders. Faced with these challenges, some companies are paralyzed, emphasizing the downsides and uncertainties, sometimes using regulation as an excuse for inaction. On the other hand, some have recklessly charged in, releasing immature AI-enabled products and services that create unacceptable regulatory and reputational risks.

Companies must find the happy path between these two extremes.

By addressing these challenges in the areas of value, data, and people, enterprises can enhance their ability to successfully implement and scale AI solutions across the organization, unlocking the full potential of these transformative technologies.

3. How can organizations address the challenges to capture real impact at scale?

While few organizations have comprehensively mastered scaling AI for business, some are successfully moving the needle on their initiatives.

What I share here is more work in progress than a tried and tested prescription for scaling. These thoughts are based on a composite view of what I have seen working and hopefully provide some hints for executives to impactfully progress their AI agendas.

Value

AI initiatives must be informed by and focused on delivering the company’s strategic priorities. This sounds obvious, but in practice rarely happens, usually because many organizations make the mistake of trying to apply AI to everything.

Only tight Business-AI alignment will ensure sufficient organizational focus and allocation of funding, both prerequisites for success. In addition, a level of boldness is required in getting the impact aspiration right, for example, 20% uplift in revenue or 50% cost takeout.

The initiatives must move the needle enough for people to care about them. At the same time, outrageous and unsubstantiated claims of value are not helpful either.

Impact quantification is preferred over qualitative factors. However, prediction of the future impact of AI initiatives, or any initiative, is always difficult. Typically, in any well-designed AI portfolio, around one third of initiatives will far exceed expectations, one third will deliver as expected, and one third will probably yield little.

Put another way, if some of your initiatives are not failing, you are probably not pushing the envelope enough. It is therefore important to consider a portfolio view and adapt to the learnings gathered from successful and unsuccessful initiatives.

Another trap to avoid is “implement and forget.” This requires upfront setting of impact KPIs and baselines, including controlling for externalities that might create undeserved windfalls or penalties. It is never possible to fully define and control for externalities before the fact, but this does ensure maximum integrity around the impacts. Unless something fact-based can be said about the impact realized from AI initiatives, funding for them is unlikely to continue.

Leading organizations are increasing their flexibility around the financing of AI initiatives.

This takes several forms. One is moving away from arbitrary caps such as the commonly seen 15% of IT budgets. Instead, they recognize that it is absurd to constrain AI investment as an IT cost when, in effect, it is a digital labor cost. This opens a more flexible approach to investment, one that is no longer constrained within a fixed IT budget.

Another approach is creating more dynamism around how costs are allocated to scaling up successful initiatives. This is a challenge in the world of fixed budgets and capital allocation. It can probably only be solved by taking a portfolio approach and being more flexible across initiatives.

Finally, one accounting challenge in particular is difficult to overcome, namely how to stop failed initiatives that have been capitalized. This often results in a write-down, not something that wins anyone friends in the corporate world. Frankly, I am not an accounting expert here, but I have seen accounting rules around write-downs form a barrier to more logical portfolio decisions, and I recognize this as an area where more progress is needed.

Data

On the data side, it is critical first to understand and recognize the scale of the problem. Too often, I have seen data as the afterthought that derailed the program.

The fact that data is difficult is perhaps perversely good. We need data, AI, and compute to obtain new insights. The AI and compute part of the equation is becoming rapidly commoditized, but the data part is the secret sauce that large incumbent enterprises have in spades and probably have not yet converted into a true competitive advantage.

That is precisely where the opportunity lies. The key is to define that opportunity and make staged investments in data quality to successively realize value at increasing scale.

One of the biggest challenges around data is how far ahead of the curve to make investments in getting the data foundations right. Getting too far ahead and building data infrastructure before it is needed usually results in slow time to impact, a data swamp, and highly disgruntled and skeptical business stakeholders.

Getting too reactive on data leads to AI initiatives practically grinding to a halt. The answer lies somewhere in the middle. Be a little ahead of initiatives, and anticipate and focus on the data objects that are likely to matter to specific initiatives.

People

On the people side, it is critical to define what skill levels are required to build and implement the AI portfolio. These profiles should be neither under-specified nor over-specified.

Talent needs must be assessed against a robust skill baseline. However, it is challenging to inventory the skill base of an organization.

Many companies start with manager or self-appraisals. This has many biases, including the fact that managers may not be good at recognizing the level of skill and self-appraisals are inherently rosy.

An alternative is to use AI itself to create a robust skill baseline from objective data such as ticket data. No technique is foolproof or bias-free, but it is critical to get as many triangulation points as possible and establish a robust baseline to measure progress against.

Acquiring and onboarding AI talent is also never straightforward. Certain profiles demand market-based salaries that could be out of reach. Great talent will not stay if the AI initiatives are only window dressing and do not deliver real business impact. The best AI talent wants to have a seat at the table to shape how AI is adopted in the enterprise.

Will the AI function be a high-powered one or a medium to low-powered one? All of this defines the talent you need and whether it can be retained.

The broader change challenges are also formidable. Companies need to foster a cultural change to become more AI-driven. However, rank and file employees and managers alike may rightly fear that AI will take their job or usurp their role in decision-making. Driving this cultural change is critical, but incredibly difficult.

Driving successful AI scale-up requires a top-down, enterprise-wide approach led by committed, visionary leadership. CEOs and C-suite executives must champion AI as a strategic imperative, allocating sufficient resources and clearly communicating the compelling business case and long-term vision.

Role modeling by executives is key here. For example, if executives push AI tools down but retain white-glove assistance for themselves, they will not be credible.

Equally important is breaking down organizational silos and fostering cross-functional collaboration. AI initiatives should be embedded into the core operations of the business, not siloed in an R&D lab or IT department. Agile, multidisciplinary teams comprising business stakeholders, data scientists, and change management experts are key to driving holistic, sustainable transformation.

To support cultural change, companies should invest heavily in AI literacy and change management. Comprehensive training and upskilling programs can help employees understand the capabilities and limitations of AI, while also highlighting the opportunities AI presents for their roles and the organization as a whole.

Proactive communication, empathetic leadership, and incentive structures that reward AI adoption are also crucial.

Finally, addressing the broader stakeholder challenges linked to a company’s license to operate requires increasing management attention and focus. Companies should prioritize the development of AI systems that are more transparent and explainable. This may mean sacrificing some model performance in exchange for greater interpretability, or exploring novel AI architectures and techniques that are inherently more explainable.

Rigorous testing, monitoring, and human oversight are also essential to ensuring the safety and reliability of scaled-up AI.

Companies will also need to develop and implement a comprehensive data privacy and ethics framework, including policies, processes, and technical controls to ensure responsible AI practices. Many companies are establishing cross-functional AI ethics committees to develop and enforce guidelines, policies, and processes for the responsible and ethical development and deployment of AI systems.

It will be important to develop a risk-based approach to avoid being paralyzed by the complexity and uncertainty in this area. This approach must consider the latest regulatory and legal trends, societal impacts, and implications for reputational risk. No perfect answer exists here, and every organization needs to find an acceptable risk-mitigated path forward.

Conclusion

Implementing AI at scale is undoubtedly a daunting challenge, but the potential rewards are immense. Companies that can successfully navigate the technical and organizational hurdles stand to gain a commanding competitive edge.

The key is to approach AI scale-up as a strategic, enterprise-wide transformation, not a narrow technology initiative. This requires visionary leadership, cross-functional collaboration, robust technical foundations, and a concerted effort to win hearts and minds across the organization.

It is a complex, multi-year journey, but one that can unlock transformative business value for those willing to take it on.