Brilliant and unreliable at once

In a 2023 experiment, 758 consultants at Boston Consulting Group worked on realistic tasks, some with an AI model and some without. On 18 tasks, the AI group finished 12.2% more work, 25.1% more quickly, at higher quality. One task sat just outside the model’s strengths. On it, the AI group was 19 percentage points less likely to get it right.

Three years later, the edge is still jagged:

• Math vs. puzzles. In July 2026, OpenAI said GPT-5.6 Sol solved 83% of FrontierMath Tier 4, research-level math. On ARC-AGI-3, puzzle games people can solve, it scored 7.78%.
• Contests vs. chores. Stanford’s 2026 AI Index ran the headline “AI Can Win a Mathematical Olympiad But Can’t Tell Time.” Robots succeeded at only 12% of real household tasks.
• Rides vs. buses. Waymo gave about 500,000 paid driverless rides a week in March 2026. Three months earlier, it recalled software on 3,067 cars after they passed stopped school buses.

Jaggedness is also why experts argue about AGI (show definition), artificial general intelligence. There is no single definition, so “Are we close?” has no single answer.

• OpenAI’s charter defines AGI as “highly autonomous systems that outperform humans at most economically valuable work.”
• A 2025 paper by 33 researchers defines it as matching a well-educated adult’s range and skill across ten areas of thinking. On their scale, GPT-4 scored 27% and GPT-5 scored 57%.
• In the 2024 survey from Step 1, researchers put even odds on machines beating humans at every task by 2047. For every job being automated, they said 2116.

Same people, two framings, 69 years apart.

Treat any confident AGI date as an opinion. Ask which definition it uses, and who gains if you believe it.