Programming with LLMs: Part 2
Some more thoughts on programming with LLMs, sparked by this article. This is a lightly edited version of this video on my YouTube channel.
Is Copilot is Making Programmers Worse?
The article is titled “Why Copilot is Making Programmers Worse at Programming,” and right at the top, the author acknowledges that there’s a trade-off here. These tools are really helping programmers increase their productivity, but as he says, there’s a trade-off because he’s concerned that they may have unintended consequences on the quality and skill set of programmers.
Eroding Core Programming Skills?
The first argument he makes against using AI for programming is that it can erode core programming skills. If you have the AI write code for you and you don’t think through the problem yourself, you’re gradually going to lose that skill. You’re going to gradually lose the ability to use critical thinking and deep reasoning.
There’s some truth to this, but if you look at history, there has been a long series of such arguments going all the way back to Socrates. Socrates didn’t like the idea of writing because he thought that if we didn’t hold these ideas in our heads and repeat them by speaking them out loud, we would forget them and not understand them as well. So, he did not like this idea of externalizing thoughts by writing.
Now, I don’t find this particular argument very compelling because it depends on what you mean by core programming skills. Would you have said that programming without an IDE is a core programming skill? That you should not have to rely on all the conveniences of an IDE? You could go lower and lower down the ladder of abstraction. Would you have said that assembly language is a core programming skill? I don’t think anyone misses programming in assembly today.
So, I think this is a matter of abstraction. Even if you have a new tool that is now writing the code for you, all that it means is that there’s a layer of abstraction on top of it—just like compilers compile your high-level language code down to assembly and machine code.
Over-Reliance on Auto-Generated Code
The next argument is really interesting: over-reliance on auto-generated code. I think there is a lot to this point. I like to make the parallel with how airline pilots use autopilot. There has been a ton of literature about how automation dependency, because of airline pilots using autopilot, has caused their flying skills to atrophy over time.
So, there’s a fine line to walk here because you definitely want to get the benefits of these new tools. They allow you to do more, faster. They allow you to build faster. Just like overall, autopilot really had a drastic influence on improving the safety of air travel, but it does come with this problem. It comes with this problem of automation dependency and pilots forgetting the fundamentals of how to fly.
So, this is somewhat related to the previous point, and I’m not sure what a good solution to this is, other than simply, as some have put it, disengaging the autopilot from time to time. So, programming without AI from time to time just to see if you can still do it, just to keep your skills sharp.
Lack of Ownership and Responsibility
The next point is about lack of ownership and responsibility. The author says that if you, as an individual developer, don’t stand behind the code you write, it might lead to sloppy or buggy code. After all, AI-generated code could have bugs or could just be flat-out incorrect.
This is definitely true now, but I think it will be less and less true as time goes on. It also depends a lot on your domain. If your domain is well-trodden, like building a web app backed by a database, my guess is that the error rate of LLM-generated code in that kind of well-understood domain is rapidly going to get to zero. In other domains that are not as well-trodden, the human programmer will still have a big role to play.
The other point I want to make here is that, in at least an industrial setting, if there are bugs, you should look more at the overall process rather than at individual programmers. This goes back to the idea of blameless post-mortems, which started at Google SRE. The main tenet is that when a system breaks, when there’s an outage or a bug, do not blame the individual teams or programmers. Instead, look at the root causes and the process that allowed such a bug to pass through undetected.
The same thing could happen with programmers who aren’t using these tools either. So, at least on this point, I think it goes back more to the process than to the individual programmers.
Reduced Learning Opportunities
Now, this is actually the one point in this piece that I disagree with most strongly. I feel like it’s exactly the opposite: LLMs lead to way more learning opportunities than before. They democratize the learning of programming more than anything else I’ve seen in two decades of programming.
Why is that? Because you can use the LLM to learn programming. You can use the LLM as a tutor to help teach you programming. If you’re a programmer, you can use the LLM to explain code to you, to review code for you, to suggest alternatives to code that you might have written. And if you just use the LLM as a co-pilot during the act of programming, you can learn a lot, especially if you’re new to a language and don’t know all the idiomatic ways of expressing common things in that language.
Narrowed Creative Thinking
Programming is as much about creativity as it is about logic, says the author. For this one, I actually talked about this specific point in my previous rant about creativity and programming with AI, so I’ll just leave a link to that in the description. There are interesting arguments both ways, so go take a look at that video.
Dependency on Proprietary Tools
There’s definitely something to this, right now, but the dependency on proprietary tools is rapidly decreasing because of all the open-source models—things like Llama, Phi, and Gemma, and so on. Right now, they’re not up to the level of the proprietary models like GPT-4, Gemini, or Claude, but the gap between open models and proprietary models is rapidly shrinking. I think very soon the open models will be good enough, if not just as good, so this point is gradually going away.
A False Sense of Expertise
A developer might feel proficient in programming because they quickly generate working code with the help of Copilot, even if they don’t fully understand the code.
I think this is mitigated a lot if the developer has the right attitude. If the developer just uses Copilot-generated code without understanding it at all, yes, they can do that—they won’t learn much. It’s similar to a high school or college student using an LLM to write an entire essay and just turning it in. But if instead, a developer uses the LLM to actually explore the problem, explore the solution, have a dialogue with the LLM about alternatives and about understanding the code, I think that might actually help them. Like I mentioned a couple of points ago, it might actually help them learn more, understand more, and develop more expertise.
Conclusion
To wrap up, I think the author makes some good points. He’s looked at this problem from a lot of different angles, and this debate is still going on. I myself have a bias towards being a heavy adopter of these tools. I think you should embrace them if you’re a programmer. There’s no going back. These tools will only get better over time, and they’ll only be adopted more and more over time.
So, if you, as a programmer, want to stay relevant, you have to realize the risks and shortcomings of these tools but not ignore them. You have to embrace them and learn them to the best of your ability—just like a couple of decades ago when IDEs became mainstream, and you had to learn how to wield your IDE in order to be a productive, modern programmer.