Limits to AI Productivity Gains
Amdahl's law describes how much you can gain from AI
Amdahl’s Law, from 1967, is more relevant today than ever. Why? It perfectly describes limits to AI productivity gains. It explains why some people think productivity is way up and some see negligible impact. What is Amdahl’s law? Developers learn about Amdahl’s law when writing parallel code. Parallel code finishes faster with a task, because more CPU cores are used to compute the result. The more CPU cores you add, the faster the code finishes. But the time the code runs is more and more limited by the parts of the code, that can’t be parallelized. These parts start to dominate the time it takes the code to finish.
Amdahl’s law states the gain from optimizing code depends on the portion that you can’t optimize, or more general:
“the overall performance improvement gained by optimizing a single part of a system is limited by the fraction of time that the improved part is actually used”
But Amdahl’s has wider ramifications and applies to many other things beside optimizing code.
Take German railroads as an example. Amdahl’s law is the reason why the German railroad no longer aims for speeds above 300km/h and more or less settled on 250 km/h - because the German railroad system is limited by the many stops it has. The fastest train in Germany from Berlin to Frankfurt (called “Sprinter”) is not so astonishing because it goes so fast, but because it does not stop. If you can’t remove the stops, there is no need to go faster. The duration of your travel time is dominated by the parts that can’t be sped up, which are the stops. The gains of faster trains become smaller and smaller while the costs of ever faster trains rise and rise.
Today AI is making everyone more productive. Productivity gains are limited, because again of Amdahl’s law:
Your productivity gains are determined by the ratio of time you spend with AI and the parts of your work that you can’t use AI for.
Take my coding as an example. When I’ve started coding with Cursor, a programming environment with tight AI integration, the my speed from idea to code doubled or tripled. I think of an idea, I formulate an intention what needs to be done, and the AI does most of the code writing. I want something, I tell the AI, and <tab><tab><tab> - done.
As a coder I’m much faster, finishing a task or feature much earlier than before and without AI. I deliver at least twice the features in the same amount of time compared to before. The time spent on coding goes down:
Looking at the picture, I’m three times faster, but I deliver only twice as many features in the same time (three blocks of green coding become one block, and there are now two blocks in the same time span)? I’m limited by the time when I don’t use AI, my time between AI usage. Discussions with product managers is limiting my productivity gain. What do I want? What needs to be done? The time I don’t use AI is limiting my productivity gain by AI (Amdahl’s law!).
This is of course not limited to development, but to all AI assisted tasks. Be it in sales, or finance, HR or marketing. Amdahl’s law applies to all of them.
What am I going to do?
I just use more AI, and speedup my coding sessions even more, with more and better AI. This works, but the gains flat out, as can be seen in the graphs below. Your coding portion (“Parallel portion” in the graph) needs to be really big for the speedup. With coding being 75% of the work, you can’t reach a four times speed up, and it flats out real early. With 50% of your time coding, the maximal speedup from AI is even less.
The effect isn’t limited to coding. We see this going on in other industries or departments too:
Legal industry: Lawyers do accelerate document review or case law review with AI but remain limited by strategy, and client discussions.
Sales Teams: Here AI can automate lead qualification, email outreach, and scheduling, significantly speeding up routine sales activities. However, the overall sales productivity gain is constrained by relationship-building activities, trust establishment, and complex negotiation processes that still depend heavily on - currently - human interactions.
With this in mind we can optimize our work, processes and environment, so the non-AI parts do not dominate our productivity gains:
Batching and minimizing interruptions: Focused AI-intensive work sessions to minimize context-switching, as context-switching increases the non-AI part of Amdahl’s law.
Workflow optimization: We need to structure work processes around clear checkpoints and handoffs, keeping in mind maximizing AI time.
Improving decision-making speed: Fast decision making is already a super power for companies. With AI, this becomes crucial. Companies need to shorten cycles between ideation, decision-making, and task execution so AI-assisted tasks are not waiting on decisions.
Role transformations: We need to combine roles that - because of work load and skills - were separated. AI reduces workloads and gives additional skills, so roles can merge. One merging of roles people suggest is that of the
Product Engineer, merging product management and engineering.
As for coding, Amdahl’s law implies I need to be more focused on fast decision making, creating and maintaining TODO lists from which I can pull work from, optimize my workflow, so I can get faster back to using an AI - instead of spending time away from it. Only the time spend with an AI increases my productivity.
Doing this, I get three times more coding output with being three times faster:
Long term, the part that limits AI productivity gains will shrink until it is gone, with AI being able to do more and more things for me. The near future will see management decisions, project planning, and creative brainstorming become more automated, further decreasing the non-AI portions. Finally discussions and decision making goes away and AI is used for that too.
Your takeaway: all discussions about AI and productivity gains need to take Amdahl’s law into account, the tasks where you can’t and don’t use AI limit your overall productivity gains.