The Causal AI Revolution is Underway
The Causal AI Revolution is Happening Now
“Machines’ lack of understanding of causal relations is perhaps the biggest roadblock to giving them human-level intelligence.”
Judea Pearl, Turing Award winner and AI pioneer
“Causality is very important for the next steps of progress of machine learning.”
Yoshua Bengio, Turing Award winner and “Godfather of Deep Learning”
“Causal AI is a key enabler of the next wave of AI, where AI moves toward greater decision automation, autonomy, robustness and common sense.”
Gartner, Analyst Firm
In a Nutshell
Causality is one of the key missing ingredients that’s needed to unlock real progress in AI. This is fast becoming the consensus view, within academia and industry.
AI luminaries, like Pearl and Bengio, have been arguing for equipping machines with the ability to reason about cause and effect for many years.
There’s been a recent explosion of research activity in causality in AI. Big tech companies including Microsoft, Amazon, Facebook, Google, Netflix and Uber have all started investing heavily in Causal AI.
And now industry analysts are recognizing the benefits of this breakthrough technology for businesses from all sectors. Preeminent technology analysis firm Gartner recently listed Causal AI as one of 25 emerging technologies (out of 2,000+) that offer transformational potential for businesses (see Fig. 1).
A race is on to build mature Causal AI, adopt it in industry, and harness its unique benefits. We sketch why a causal revolution is taking place right now.
Link
Why Causal AI?
Why is causality so important for AI? In essence, an appreciation of cause and effect is a necessary requirement for true intelligence.
Beyond data
Models of the world that include causality are able to efficiently identify the critical information in datasets while discarding all the irrelevant and often misleading correlations. This creates simpler and more powerful models that make superior predictions.
More than this, causality goes beyond data. “You are smarter than your data”, as Pearl says, “data don’t understand cause and effect; humans do”. We use causal knowledge to anticipate how our actions will impact the environment, and thereby formulate plans and strategies. We depend on causal reasoning to imagine possible worlds that differ from our own, a competence that allows us to diagnose why something happened as it did. AI that understands cause and effect can harness these powerful abilities too.
“You are smarter than your data. Data don’t understand cause and effect; humans do.”
Judea Pearl
Solving open challenges in AI
Applying the principle of causality and effectively utilising Causal AI for enterprises promises to unlock progress on some of the biggest open problems in AI. Current AI systems crash when faced with novel data. They are bad at transferring learning to new tasks. They fail in deployment when the world changes unexpectedly. And they are susceptible to adversarial attacks.
These problems stem from current machine learning systems’ inability to make reasonable generalizations. Researchers believe causality can solve this deep challenge. “Generalizing well out the [training] setting requires learning not mere statistical associations between variables, but an underlying causal model”, say a group from Google Research and the Montreal Institute for Learning Algorithms (MILA).
“Causal models have a unique combination of capabilities that could enable a deeper understanding of complex systems and allow us to better align decision systems with society’s values.”
DeepMind
Human-machine symbiosis
Most current AI systems are black boxes that cannot be understood, even by their programmers. And they are often poorly aligned with human values. Causal AI is at the cutting edge of “XAI” (eXplainable AI) and AI fairness. Unlike conventional AI, which trades off transparency for accuracy, causality-based models deliver high performance and explainability. Causal AI also enables tech developers to maintain better control over AI systems, ensuring they behave as intended in production.
Causal AI has a “unique combination of capabilities”, DeepMind argues, “that could enable a deeper understanding of complex systems and allow us to better align decision systems with society’s values.”
Why Now?
There are several emerging trends that are changing how AI is used by businesses, all of which favour the adoption of Causal AI for enterprises.
Dissatisfaction with current machine learning
Firstly, there is growing dissatisfaction within industry with current machine learning algorithms. Nine in ten businesses fail to generate meaningful financial returns from AI investment, according to a recent global survey. 87% of machine learning projects never make it beyond an experimental phase into production, according to Forbes. Pandemic-related disruption has accelerated this trend, with machine learning models failing to adapt to changing conditions.
Prioritising explainability
Second, there is a growing recognition that the explainability, safety and fairness of AI systems are critical in business settings. This global trend has been boosted by a recently inked EU proposal for new regulation of AI systems. The wide-ranging incoming legislation mandates that businesses should provide explainability reports, apply stress tests and ensure humans remain “in the loop”.
Fostering human-machine partnership
Third, companies that benefit most from AI now tend to foster human-machine partnerships to generate new kinds of value. These leading organizations are outcompeting those that use AI in machine-only applications. Companies that prioritize human-machine partnerships have a 73% greater chance of reaping the biggest financial benefits from AI, a recent MIT Sloan & BCG global survey finds.
Navigating the world of big data
Fourth, the total volume of data is vast and increasing exponentially. However many datasets are correlated, expressing similar facts about the world. Businesses increasingly place a premium on identifying the right data to answer their questions and achieve their goals. To do so they need technology that can penetrate beneath correlations to identify distinctive marginal value in data.
These accelerating AI trends — away from standard machine learning and towards explainable, human-compatible AI that goes beyond correlations — are providing extra momentum to the Causal AI revolution. “Causal inference may be the new frontier as we migrate from association-based analysis only”, says Jeffrey Bohn, Chief Research Officer at Swiss Re Institute.
“Causal inference may be the new frontier as we migrate away from association-based analysis only.”
Swiss Re Institute
How are Businesses being Impacted?
Early adopters of Causal AI benefit from remarkable improvements in business decision making. Take, for example. Retail AI.
E-commerce platform eBay found that conventional AI wildly overestimates advertising effectiveness. They found that conventional Retail AI estimated the ROI of advertising clickthrough as 1400%, but causal analysis revealed the true ROI to be negative 63%.
A Harvard Business School study found that businesses using AI for churn prevention would achieve an additional reduction of 4.1-8.7% in churn rate by using causal methods as compared to standard predictive analytics.
Business-oriented social network LinkedIn conducts 20,000+ experiments each year, but some A/B tests can be unethical or impractical. LinkedIn turned to AI to simulate infeasible experiments and found that correlation-based methods produced errors ranging from 46-250% outside the true values. LinkedIn’s researchers concluded that, “It is dangerous to assume correlational results are causal… Causal studies, when properly applied, make the best use of the available data to improve decision-making ability.”
Beyond LinkedIn, most big technology companies are starting to benefit from adopting Causal AI. Netflix is a good example. Recognizing that “causal inference and machine learning have a symbiotic relationship that is growing deeper”, Netflix Research have “made a large investment in increasing the performance of causal inference algorithms”. Similarly, Uber Labs are doubling down on causality and say that “causal inference provides information that is critical to making business decisions through better understanding the impact of key initiatives.”
“Causal inference and machine learning have a symbiotic relationship that is growing deeper… We have made a large investment in Causal AI.”
Netflix Research
Causal AI-enabled counterfactual analysis, or “artificial imagination”, is also beginning to make an impact on AI in healthcare. Causal AI achieves expert clinical accuracy in medical diagnosis through counterfactual analysis. Causal AI places in the top 25% of doctors when benchmarked against humans while conventional machine learning has middling performance, a Nature article finds.
Further examining the efficacy of Causal AI in healthcare, counterfactuals have been applied in several high impact use cases for social good. These include diagnosing the causes of childhood disease in Pakistan, and combatting hesitancy to visit hospitals among women in India’s rural north. The Harvard-affiliated team who worked on these applications concludes, “We may yet find that an ounce of causal AI is worth a pound of prediction.”
Counterfactual analysis has enormous transformative potential beyond healthcare, not least for AI in finance and business. Riffing on the power of artificial imagination, Head of Hedge Funds at APG Michael Weinberg states, “what most excites me is seeing managers deploying new techniques like causality… new techniques and data are really beginning to expand the potential for AI strategies in a meaningful way”.
Many other leading portfolio managers agree on the merits of causality in revolutionizing current finance AI applications. Michael Grady, Head of Investment Strategy at Aviva Investors, finds that “Causal AI plays an ever-more important role in our investment analysis. It empowers our strategists and portfolio managers to generate alpha by identifying new causal relationships in economic, financial and alternative data.” Similarly, Chris Udy, CIO of Tibra says that “causality-based techniques and automation of quantitative workflows help us discover more orthogonal signals faster while discarding spurious correlations.”
Zooming out from these individual examples of impact, a recent global survey found that 87% of CTOs and senior data scientists agree that Causal AI can obtain a more robust model of their business environment. Senior data scientists are also gearing up to make deeper investments in causality. 83% believe causal inference in machine learning will be of increasing importance for data-driven decision making in the future and 60% intend to make a significant investment in Causal AI.
We have highlighted a variety of trends that are fuelling the Causal AI revolution. Increased research attention, increased investment within industry and increased disillusionment with conventional machine learning are all driving wider adoption of Causal AI for enterprises. With early adopters already seeing impactful results, we believe that the Causal AI revolution has only just begun.