In the race to adopt artificial intelligence, too many enterprises are flooring the brakes while neglecting the accelerator.
As the saying goes, “AI may not be coming for your job, but a company using AI is coming for your company.”
The pressure to integrate AI solutions is becoming intense, and organizations that have missed early adoption windows are increasingly turning to external vendors for quick fixes. The longer enterprises wait, the faster and riskier it becomes when they are forced to adopt AI. By delaying, they have to learn fast how to do it with no experience under their belt. This article explores the significant risks of unchecked AI deployment and offers guidance for navigating the challenges.
Unpredictable AI: When AI Tools Go Rogue
Remember the UK Post Office Horizon scandal? A conventional software system led to hundreds of innocent people being prosecuted, some imprisoned, and lives utterly destroyed. That was just normal software. The AI tools your organization might be preparing to unleash represent an entirely different beast.
AI is like an adolescent—moody, unpredictable, and occasionally dangerous. Consider Air Canada’s chatbot debacle: it confidently provided customers with incorrect bereavement policy information, and the courts ruled that Air Canada had to honor what their digital representative had erroneously promised. While in this case one might argue the chatbot was more humane than the company’s actual policies, the financial implications were significant.
The critical question is: will your AI tool be trusted to behave and do its job, or will it go on a rampage and wreck your business? Learning how to deploy AI with robust oversight is a critical skill organizations must master for successful AI deployments, and not to play Russian roulette. Companies starting now, are getting a significant edge in learning how to control this critical technology.
The Zillow Cautionary Tale
Zillow’s failed foray into real estate flipping highlights the dangers of AI relying solely on past data. The algorithm, confident in its predictions, failed to account for rapidly changing market conditions, such as a drop in demand or nearby property issues—it could take months for Zillow’s algorithm to recognize the impact on valuation. Meanwhile, savvy sellers capitalized on this, unloading properties to Zillow before Zillow detected the prices plummeting, costing the company 10% of its workforce.
The problem? Zillow’s AI was backward-looking, trained on historical data, and unable to adapt to dynamic environments. This same issue plagues stock-picking algorithms and other systems. that perform beautifully on historical data but collapse when faced with new market conditions. If your AI is making decisions based solely on past data without accounting for environmental changes, you’re setting yourself up for a Zillow-style catastrophe.
To mitigate this risk, ensure your AI’s training data represents current and anticipated future conditions. Consider the risks carefully! This is particularly crucial for financial systems, where tail risks are more frequent than models predict. Medical applications, like analyzing skin conditions, are much less susceptible to changing environments, as long as the AI is trained on a representative sample of the population.
Startup Corner-Cutting: From Unicorns to Bankruptcy – Hidden AI Risk with Vendors
Your vendor might be cutting corners. While they may not be another Theranos, the risk is real. Take Builder.ai, the UK tech unicorn that recently collapsed into bankruptcy amid financial reporting discrepancies. It has now emerged that Builder.ai was a fraud, and people using the service are left with orphaned applications.
Startups face intense pressure to deliver results, which can lead to critical oversights with inconvenient truths often getting swept under the rug. One common pitfall is bias in training data. When your system makes judgments about people, inherent biases can lead to discriminatory outcomes, and can even perpetuate and amplify discriminatory outcomes.
Even tech giants aren’t immune. Amazon attempted to build an AI resume screening tool to identify top talent by analyzing their current workforce’s resumes. The problem? AWS, their massive cloud division, was predominantly male, so the AI learned to favor male candidates. Even after purging overtly gender-identifying information, the system still detected subtle language patterns more common in men’s resumes and continued its bias.
If you’re using AI to determine whether someone qualifies for financing, how can you be sure the system isn’t perpetuating existing biases?
My advice, before deploying AI that makes decisions about people, carefully evaluate the data and the potential for bias. Consider implementing bias detection and mitigation techniques. Better yet, start now with an internal trial to see the problems that bias in the data might cause. Those organizations getting hands on experience right now, will be well ahead of their peers who have not started.
The Hallucination Problem: A Critical AI Risk to Manage
Then there are “hallucinations” in generative AI—a polite term for making things up, which is exactly what’s happening. Just ask Elon Musk, whose chatbot Grok fabricated a story about NBA star Klay Thompson throwing bricks through windows in Sacramento. Sacramento might be bland, but it did not drive Klay to throw bricks through his neighbor’s windows. Such fabrications are potentially damaging to reputations, including your company’s.
How can you prevent similar embarrassments? Keep humans in the decision loop—at minimum, you’ll have someone to blame when things go wrong. It wasn’t the AI you purchased from “Piranha AI backed by Shady VC” that approved those questionable loans; it was Johnny from accounting who signed off on them.
A practical approach is designing your AI to show its work. When the system generates outputs by writing code to extract database information, this transparency, or “explainable AI”, approach allows you to verify the results and logic used to arrive at them. There are other techniques that can reduce or eliminate the effect of hallucinations, but you need to get some hands-on experience to understand when they occur, what they say, and what risk this exposes your organization to.
The Economic and Societal Costs of AI Failures
The costs of AI security and compliance failures extend far beyond immediate losses:
- Direct Financial Costs: AI security breaches can lead to significant financial losses through theft, ransom payments, and operational disruption. The average cost of a data breach reached $4.45 million in 2023, with AI-enhanced attacks potentially driving this figure higher.
- Regulatory Penalties: Non-compliant AI systems increasingly face steep regulatory penalties. Under GDPR, companies can be fined up to 4% of annual global revenue.
- Reputational Damage: When AI systems make discriminatory decisions or privacy violations occur, the reputational damage can far exceed direct financial losses and persist for years.
- Market Confidence Erosion: Systematic AI failures across an industry can erode market confidence, potentially triggering investment pullbacks and valuation corrections.
- Societal Trust Decline: Each high-profile AI failure diminishes public trust in technology and institutions, making future innovation adoption more difficult.
The Path Forward: Reducing AI Risk
As you enter this dangerous world, you face a difficult reality: do you delay implementing AI, and then have to scramble to catch up, or are you more cautious and start working on AI projects now. The reality is that your competitors are likely adopting AI, and you must as well in the not-so-distant future. Some late starters will implement laughably ridiculous systems that cripple their operations. Don’t assume that purchasing from established vendors guarantees protection—many products assume you will manage the risks. Trying to run a major AI project with no experience is like trying to drive a car with no training. Close calls are the best you can hope for.
The winners will be companies that carefully select the best AI systems while implementing robust safeguards. Don’t assume established vendors are immune to the risks. Consider the following steps:
- Prioritize Human Oversight: Implement robust human review processes for AI outputs.
- Focus on Data Quality: Ensure your training data is accurate, representative, and accounts for potential biases.
- Demand Explainability: Choose AI systems that provide transparency into their decision-making processes.
- Establish Ethical Guidelines: Develop clear ethical guidelines for AI development and deployment. Alternatively, an AI consultancy can provide guidance. However, vet them carefully or you might end up with another problem rather than a solution.
- Apply Proper Security and Compliance Measures: This isn’t just good ethics—it’s good business.
In the race to AI adoption, remember: it’s better to arrive safely than to crash spectacularly before reaching the finish line.
Those who have already started their AI journey are learning valuable lessons about what works and what doesn’t. The longer you wait, the more risky your position becomes. For everyone else, all you can hope for is more empty chambers in your Russian roulette revolver.
