|
2024 Highlights!
Dec 20, 2024
2024: The Year of Generative AI in Action and the Revolution Ahead- From hype to impact
As 2024 draws to a close, I am more optimistic than ever about the potential for Generative AI to positively impact our societies. While headlines focus on the struggles of enterprise AI rollouts, a silent transformation occurs across workplaces worldwide. From frontline workers accessing AI via text messages to researchers accelerating breakthroughs, millions are finding practical ways to harness AI's power.
The story of 2024 isn't about technology challenges—it's about understanding how AI can empower everyone, not just those with access to computers, and how we must address the trust, security, and ethical challenges this new way of working brings.
🎧 Dive deeper into this week’s insights in our “AI in 60 Seconds—The 10-Min Podcast” version. Available on Apple Podcast, Spotify, Amazon Music, or your favorite app.
📊 2024 by the Numbers
| 71% |
88% |
4X |
| Organizations using AI |
Superuser satisfaction with AI tools |
Productivity gain for trained users |
| 3X+ |
18% |
69% |
| Return on AI investment |
Level of Trust on AI providers |
Employees using off the shelf AI tools (BYOAI) |
We've documented over 134 widely adopted use cases across industries, seven key applications dominate enterprise adoption:
| Top 7 Categories |
Adoption |
Key Benefits |
| Support Chatbots |
38% enterprise adoption |
• 24/7 knowledge-based support for employees and customers
• 30% to 58% reduction in service costs
• Higher user satisfaction |
| Content & Meeting Summarization |
41% of active enterprise users |
• Highest adoption across all roles and sectors
• Average time savings: 2-4 hours per week |
| Sales & Marketing Personalization |
78% use AI for market analysis and outreach |
• Double-digit increase in response rates
• Consistent ROI across company sizes |
| Data Analysis |
35% adoption |
• 80% accuracy for superuser
• Average users face 30% error rates due to poor prompts or LLM math/statistics challenges |
| Design & Creative Work |
69% of creative teams use AI for concepts |
• 2X faster iteration cycles in product design
• 83% say AI successfully augments creative work |
| Code Development |
58% of software developers |
• 75% of senior developers see 2X–4X productivity
• Median 3x improvement for experienced devs
• Junior devs face 40% error rates due to poor prompts |
| AI-powered Knowledge Management |
40% pilot/deep adoption |
• Startups like CustomGPT & Perplexity AI lead
• Agents built in <10 mins without code
• Solid hallucination mitigation |
💳 The $29 Revolution vs Enterprise Struggles
While large enterprises debate AI strategy in boardrooms, individuals are achieving remarkable ROI with minimal investment thanks to tens of thousands of Gen AI tools built for one specific use case:
| Chat GPT Enterprise, Google Gemini, Microsoft Copilot |
Niche AI Apps |
| Base: $29/mo + the cost of the productivity suite + $580 average training cost per person |
Base: $29/mo + self-training |
| 3+ months to achieve proficiency |
Days or weeks to proficiency |
| <30% of company-wide trials are successful |
+65% of group trials are successful |
| 30% user retention after 3 months |
70%+ user retention after 3 months |
| 70% of executives struggle to measure ROI |
Individuals report savings of 20 hours per month per tool used |
| +3x ROI after 3-6 months |
+3x ROI within weeks |
| Company governance over data |
+69% of tools are brought by employees paying with their credit cards, unclear data governance |
| ~48% average user satisfaction |
~81% average user satisfaction |
🌟 3 Personal Stories - 2024
During a night shift power outage, Teresa used an AI assistant via text message to navigate complex FDA protocols in Spanish, making critical food safety decisions in real time. The AI, trained on 600 pages of regulations, provided clear guidance without requiring special technology. During the trial phase of this tool, 100 frontline workers had 5,000 public-health-related chats with a 96% resolution rate.
Teresa - Clerk - Grocery Store - CA
|
Using off-the-shelf enterprise LLMs such as ChatGPT Enterprise and Claude Enterprise to accelerate the development of an enterprise-grade AI solution for pharmaceutical workflows, research, development, and manufacturing, Dr. Leon's team improved their outcome by 2x.
Dr. Leon - AI for Scientific and Medical Research - WA
|
At Samurai Labs, they combine LLMs with expert human knowledge to detect and prevent suicidal crises and abusive behavior such as cyberbullying and internet aggression. Their AI solution augments the capacity of human experts and moderators. They have supported 25,000 people in suicidal crises.
Patryja - Samurai Labs - Poland & CA
|
🎯 The Implementation Reality Check
Our year-end analysis reveals a complex landscape: while AI tools show tremendous potential, three critical challenges emerged in 2024:
🔒 The Trust Crisis
Our tracking shows trust in AI vendors has reached an all-time low:
- 82% of leaders express concerns about data handling and security
- 60% of CTOs and CDOs report strong concerns with training data practices
- 80% of nonprofit leaders report decreased trust after reading AI disclosures
The root cause? Obscure terms of service and auto opt-in policies that mirror the worst practices of early social media platforms. 🤔 Understanding AI "Hallucinations": Beyond Technical Limitations
Our analysis of over 5,000 user interactions with Microsoft Copilot, ChatGPT, Google Gemini, and private chat agents built by third parties using Retrieval Augmented Generation (RAG) shows:
- 30% failure rate in mathematical and statistical queries across all major LLMs; this is inherent to the AI model limitations and not to user error. Claude Sonnet is the top performer among the non-open-source models.
- Most private chat agents hallucinate 25% of the time, even when focusing on a small dataset, due to poor implementation and developer over-reliance on prompts.
-
Around 40% of incorrect responses are due to bad prompts:
- Users unknowingly embed biases in financial analysis prompts.
- Development teams report that AI systems are "fixing" code that isn't broken due to bad prompts.
- Even experienced users struggle with unintentionally biasing AI responses.
💡All the LLMs we tested, GPT-4o and 01, Claude Sonnet 3.5, Google Gemini, and Llama 2, frequently experience the classic behavioral economic biases we attribute to humans. The most commonly observed are Anchoring Bias, Confirmation Bias, Present Bias, Loss Aversion, Status Quo Bias, and Framing Effect. Understanding these biases is crucial for creating good Generative AI Solutions and Experiences and designing any Training program on Prompt Engineering and Generative AI use at work and school.
💡 The Path Forward: Three Strategic Pillars
-
People and Training
- Build organization-wide AI literacy with a focus on prompt engineering
- Implement security and privacy awareness programs
- Results: 50% increase in AI use and double-digit satisfaction gains after training
-
Technology and Design
- Replace generic chatboxes with guided, use-case-specific experiences
- Build in guardrails to mitigate hallucinations
- Implement transparent data governance and user-friendly disclosure practices
- Ensure data sovereignty and security
-
Clear Success Metrics
- Track meaningful productivity and accuracy metrics
- Monitor user confidence and retention
- Measure ROI through tangible business outcomes
🔮 One More Thing...
The real story of 2024 isn't about AI's limitations—it's about human adaptation. Every successful implementation, from Teresa's food safety decisions to Dr. Leon's research breakthroughs or Patryja's impact on mental health, shows that designing AI around human needs and capabilities positively transforms our work.
Looking ahead to 2025:
Entrepreneurs building AI tools for specific uses will continue to dominate
- Purpose-specific AI tools designed from the ground up will continue to dominate
- Generic chat interfaces hastily added to existing software will struggle with adoption
Mentoring Agents powered by Generative AI will be a leading use case
- AI mentoring agents will transform education, health, workforce development, and citizen services
- Personalized learning experiences will become mainstream
The Great Software & Internet Reimagining - past the initial hype
- Entrepreneurs and Venture Capitalists will focus on reinventing the software and internet experiences created over the past 50 years.
- Mobile-first, text-based interactions will democratize AI access to decades of great content created that almost no one accesses due to poorly designed interfaces.
A harsh year for our privacy and our trust
- Successful vendors will prioritize clear data policies, and Privacy-first solutions will gain market share
- We will perpetuate the mistakes made with online privacy and the well-designed but ineffective regulations and laws. It is unclear how this will shape AI adoption and established societal rules.
The pioneers of 2024 showed us that AI's true potential lies in reimagining how we work. As we enter 2025, the focus will shift from adding AI features to fundamentally rethinking how humans and technology can work together. The revolution is in the thousands of practical solutions that simplify our days, one task at a time.
📚 Dive Deeper
Resources
Luis J. Salazar
Founder | AI4SP
Sources:
Our insights are based on over 250 million data points from individuals and organizations that used our AI-powered tools, participated in our panels and research sessions, or attended our workshops and keynotes.
📣 Feel free to use this data in your communications, citing "AI4SP" and linking to AI4SP.org.
📬 If this email was forwarded to you and you'd like to receive our bi-weekly AI insights directly, click here to subscribe: https://ai4sp.org/60
|