AI Agent Cost and Efficiency Experiment
A 20-person team at a Silicon Valley AI training/inference company ran a $1M experiment with unlimited token access, revealing that AI use raised costs above human costs and degraded team members’ grasp of project details. - The team found that organization-wide efficiency gains plateau after initial AI adoption, as existing structures are designed for human workflows. - Team members increasingly outsourced thinking to AI, losing independent recall of project specifics and needing to query AI to answer colleagues’ questions. - The experiment concluded that token-unlimited access is impractical at scale, costing more than the human team itself.
Key insights
Unlimited AI Access Can Increase Costs Without Proportional Efficiency Gains
The experiment showed that AI costs exceeded human labor costs when tokens were unrestricted, challenging the assumption that more AI always yields higher productivity. Efficiency improvements were strongest from zero to initial AI adoption, but diminishing returns set in quickly under current organizational designs.
Excessive AI Reliance Weakens Human Expertise and Detail Recall
Team members reported losing the ability to answer questions without first consulting AI, as decision-making and detail tracking were effectively outsourced. This erosion of individual mastery could undermine long-term team capability.