I feel this – we had a junior dev on our project who started using AI for coding, without management approval BTW (it was a small company and we didn’t yet have a policy specifically for it. Alas.)
I got the fun task, months later, of going through an entire component that I’m almost certain was ‘vibe coded’ – it “worked” the first time the main APIs were called, but leaked and crashed on subsequent calls. It used double- and even triple-pointers to data structures, which the API vendor’s documentation upon some casual reading indicated could all be declared statically and re-used (this was an embedded system); needless arguments; mallocs and frees everywhere for no good reason (again due to all of the un-needed dynamic storage involving said double/triple pointers to stuff). It was a horrible mess.
It should have never gotten through code review, but the senior devs were themselves overloaded with work (another, separate problem) …
I took two days and cleaned it all up, much simpler, no mem leaks, and could actually be, you know, used more than once.
Fucking mess, and LLMs (don’t call it “AI”) just allow those who are lazy and/or inexperienced to skate through short-term tasks, leaving huge technical debt for those that have to clean up after.
If you’re doing job interviews, ensure the interviewee is not connected to LLMs in any way and make them do the code themselves. No exceptions. Consider blocking LLMs from your corp network as well and ban locally-installed things like Ollama.
It should have never gotten through code review, but the senior devs were themselves overloaded with work
Ngl, as much as I dislike AI, I think this is really the bigger issue. Hiring a junior and then merging his contributions without code reviewing is a disaster waiting to happen with or without AI.
It used double- and even triple-pointers to data structures
(old song, to the tune of My Favourite Things)
🎶 “Pointers to pointers to pointers to strings,
this code does some rather unusual things…!” 🎶
I’ll quote myself from some time ago:
The entire article is based on the flawed premise, that “AI” would improve the performance of developers. From my daily observation the only people increasing their throughput with “AI” are inexperienced and/or bad developers. So, create terrible code faster with “AI”. Suggestions by copilot are >95% garbage (even for trivial stuff) just slowing me down in writing proper code (obviously I disabled it precisely for that reason). And I spend more time on PRs to filter out the “AI” garbage inserted by juniors and idiots. “AI” is killing the productivity of the best developers even if they don’t use it themselves, decreases code quality leading to more bugs (more time wasted) and reducing maintainability (more time wasted). At this point I assume ignorance and incompetence of everybody talking about benefits of “AI” for software development. Oh, you have 15 years of experience in the field and “AI” has improved your workflow? You sucked at what you’ve been doing for 15 years and “AI” increases the damage you are doing which later has to be fixed by people who are more competent.
from some time ago
It’s a fair statement and personal experience, but a question is, does this change with tool changes and user experience? Which makes studies like OP important.
Your >95% garbage claim may very well be an isolated issue due to tech or lib or llm usage patters or whatnot. And it may change over time, with different models or tooling.
At this point I assume ignorance and incompetence of everybody talking about benefits of “AI” for software development.
deleted by creator
Don’t give yourselves to these unnatural men - machine men with machine minds and machine hearts! You are not machines! You are men!
Paolo Nutini’s song ‘Iron Sky’ samples this same speech
Reading the paper, AI did a lot better than I would expect. It showed experienced devs working on a familiar code base got 19% slower. It’s telling that they thought they had been more productive, but the result was not that bad tbh.
I wish we had similar research for experienced devs on unfamiliar code bases, or for inexperienced devs, but those would probably be much harder to measure.
I don’t understand your point. How is it good that the developers thought they were faster? Does that imply anything at all in LLMs’ favour? IMO that makes the situation worse because we’re not only fighting inefficiency, but delusion.
20% slower is substantial. Imagine the effect on the economy if 20% of all output was discarded (or more accurately, spent using electricity).
I’m not saying it’s good, I’m saying I expected it to be even worse.
Does that imply anything at all in LLMs’ favour?
Yes it suggest lower cognitive load.
1% slowdown is pretty bad. You’d still do better just not using it. 19% is huge!
Would ai coders even get faster over time or just stay stagnant since they aren’t learning anything about what they’re doing
People are a bad judge of their own skill and overrely on tools and assistants when present. See also: car adas systems making drivers less skillful. More news at 11.
Now the interesting question is what it really means when less experienced programmers think they are 100% faster.
Someone told me the best use of AI was writing unit tests and I died on the inside.
Why? That is a great use for AI. I’m guessing you are imagining that people are just blindly asking for unit tests and not even reading the results? Obviously don’t do that.
Of course that’s what they’re doing. That’s the whole point. Generate a bunch of plausible-looking BS and move on.
Writing one UT (actually writing, not pressing tab) gives you ideas for other tests.
And unit tests are not some boring chore. When doing TDD, they help inform and guide the design. If the LLM is doing that thinking for you, too, you’re just flying blind. “Yeah, that looks about right.”
Can’t wait for this shit to show up in medical devices.
That is like claiming people are directly copying from university books and implementing whatever they get without checking.
Of course there are nitwits like that, but they are few and far in between.
Anyone seriously using LLM prompts double checks their work.
I’ve caught “professionals” pasting code from from forums and StackOverflow. Of course people are just blindly using LLMs the same way. Incredibly naive to think people aren’t already and won’t do so more in the future.
Damn there are so many AI critics who have clearly not seriously tried it. It’s like the smartphone naysayers of 2007 but much much worse.
Ad hominem.
🎵Had a great day out,
Callin’ my name like Ferris Bueller,
Time to wrap this up,
I’m getting 19℅ slower! 🎵
Well that’s a strangely deep cut Ken Ashcorp ref
Men of culture i see
I am honestly shocked to see a reference in the wild to Ken Ashcorp.
I’m honestly shocked that multiple people got the reference.
The only time it really helps me is when I’m following a pretty clear pattern and the auto-complete spares me from copy-pasting or just retyping the same thing over and over. Otherwise I’m double-checking everything it wrote, and I have to understand it to test it, and that probably takes most of my time. Furthermore, it usually doesn’t take the entire codebase into account so it looks like it was written by someone who didn’t know our team or company standards as well as our proprietary code.
If there’s a clear pattern, regex is your friend. I use it for complex find-and-replace actions, or to generate code based on a template and a list of values (find a value, replace it with the template including the value). Full control over the output, more reliable even than manual copy-pasting.
It’s hard to even call them specialists, they are at the level of cashiers, for whom the computer does everything, and sometimes they do something at the level of communicating with clients and that’s all. I’m certainly not a professional, but I think the main message is clear.
It would be interesting to see another study focusing on cognitive load. Maybe the AI let’s you offload some amount of thinking so you reserve that energy for things it’s bad at. But I could see how that would potentially be a wash as you need to clearly specify your requirements in the prompt, which is a different cognitive load.
I seem to recall a separate study showing that it just encouraged users to think more lazily, not more critically.
My velocity has taken an unreasonable rocket trajectory. Deploying internal tooling, agent creation, automation. I have teams/swarms that tackle so many things, and do it well. I understand there are issues, but learning how to use the tools is critical to improving performance, blindly expecting the tools to be sci-fi super coders is unrealistic.
I wish AI was never invented, but surely this isn’t ture.
I’ve been able to solve coding issues that usually took me hours in minutes.
Wish it wasn’t so, but it’s been my reality
LLMs making you code faster means your slow not LLMs fast
I doubt anyone can write complex regex in ~30 seconds, LLM’s can
yes they can. I regularly do. Regexes aren’t hard to write, their logic is quite simple. They’re hard to read, yes, but they are almost always one-offs (ex, substitutions in nvim).
Regexes aren’t hard to write, their logic is quite simple.
He did say complex regex. A complex regex is not simple.
yes, “complex” regexes are quite simple too. Complex regexes are long, not difficult. They appear complex because you have to “inline” everything. They really are not that hard.
This is stupid pedantry. By that logic literally nothing is complex because everything is made up of simple parts.
cryptic != complex. Are they cryptic? yes. Are they complex? not really, if you can understand “one or more” or “zero or more” and some other really simple concepts like “one of these” or “not one of these” or “this is optional”. You could explain these to a child. It’s only because they look cryptic that people think they are complex. Unless you start using backreferences and advanced concepts like those (which are not usually needed in most cases) they are very simple. long != complex
I’m not trusting a regex written by AI
That’s why you write tests
tests can never prove correctness of code. All they can prove is “the thing hasn’t failed yet”. Proper reasoning is always needed if you want a guarantee.
If you had the llm write the regex for you, I can practically guarantee that you won’t think of, and write tests for, all the edge cases.
Proper reasoning is always needed if you want a guarantee.
You formally verify your regexes? Doubtful.
No, which is why I avoid regexes for most production code and also why I would never use one written by a pathological liar and always guessing coder like an LLM.
LLM is great when you’re coding in a pure fictional programming language like elm and are using loss of custom types to make impossible states unrepresentable, and the function you’re writing could have been derived by the Haskell compiler, so mathematically the only possible way you could write it wrong is to use the wrong constructor, then it’s usually right and when it’s wrong either it doesn’t compile or you can see it’s chosen the wrong path.
The rest of the time it will make shit up and when you challenge it, out will happily rewrite it for you, but there’s no particular reason why it wouldn’t make up more nonsense.
Regexes are far easier to write than to debug, which is exactly why they’re poison for a maintainable code base and a really bad use case for an LLM.
I also wouldn’t use an LLM for languages in which there are lots and lots of ways to go wrong. That’s exactly when you need an experienced developer, not someone who guesses based on what they read online and no understanding, never learning anything, because, my young padawan, that’s exactly what an LLM is, every day.
Watch your LLM like a hawk.
You don’t have to. You can read it.
Depends on what you need to match. Regex is just another programming language. It’s more declarative than traditional languages though (it’s basically pattern matching).
Pattern matching is something I already do a lot of in my code, so regexes aren’t that much different.
Regardless, the syntax sucks. It takes some time to get familiar with it, but once you get past that, it’s really simple.
The experienced developers in the study believed they were 20% faster. There’s a chance you also measured your efficiency more subjectively than you think you did.
I suspect that unless you were considerably more rigorous in testing your efficiency than they were, you might just be in a time flies when you’re having fun kind of situation.
Ai-only vibe coders. As a development manager I can tell you that AI-augmented actual developers who know how to write software and what good and bad code looks like are unquestionably faster. GitHub Copilot makes creating a suite of unit tests and documentation for a class take very little time.
Try reading the article.
The article is a blog post summarizing the actual research. The researchers’ summary says:
We do not provide evidence that: AI systems do not currently speed up many or most software developers. We do not claim that our developers or repositories represent a majority or plurality of software development work.
The research shows that under their tested scenario and assumptions, devs were less productive.
The takeaway from this study is to measure and benchmark what’s important to your team. However many development teams have been doing that, albeit not in a formal study format, and finding AI improves productivity. It is not (only) “vibe productivity”.
And certainly I agree with the person you replied to: anecdotally, AI makes my devs more productive by cutting out the most grindy parts, like writing mocks for tests or getting that last missing coverage corner. So we have some measuring and validation to do.
The research explicitly showed that the anecdotes were flawed, and that actual measured productivity was the inverse of what the users imagined. That’s the entire point. You’re just saying “nuh uh, muh anecdotes.”
I said it needs to be measured. But few teams are going to do that, they’re building products not case studies.
This study is catnip for the people who put “AI” in scare quotes and expect those of us who use it to suddenly realize that we’ve only been generating hallucination slop. This has not been the lived experience of those of us in software development. In my own case I’ve seen teams stop hiring because they are getting the same amount of work done in less time. But those are anecdotes, so it doesn’t count.
It’s entirely possible to measure metrics.
Enjoy your slopware.
We do measure metrics around sprint productivity, code quality, reworks, and more. However, we do not build case studies for clickbait articles. This isn’t a productive conversation as you just want to ridicule those using AI. Blocked.
LOL. Doing a good job of that all by yourself, mate.
I did, thank you. Terms therein like “they spend more time prompting the AI” genuinely do not apply to a code copilot, like the one provided by GitHub, because it infers its prompt based on what you’re doing and the context of the file and application and creates an autocomplete based on its chat completion, which you can accept or ignore like any autocomplete.
You can start writing test templates and it will fill them out for you, and then write the next tests based on the inputs of your methods and the imports in the test class. You can write a whole class without any copilot usage and then start writing the xmldocs and it will autocomplete them for you based on work you already did. Try it for yourself if you haven’t already, it’s pretty useful.
I read the article (not the study only the abstract) and they were getting paid an hourly rate. It did not mention anything about whether or not they had expirence in using llms to code. I feel there is a sweet spot, has to do with context window size etc.
I was not consistently better a year and a half ago but now i know the limits caveats and methods.
I think this is a very difficult thing to quantify but haters gonna latch on to this, same as the study that said “ai makes you stupid” and “llms cant reason”… its a cool tool that has limits.
One interesting feature in this paper is that the programmers who used LLMs thought they were faster, they estimated it was saving about 20% of the time it would have taken without LLMs. I think that’s a clear sign that you shouldn’t trust your gut about how much time LLMs save you, you should definitely try to measure it.
The study did find a correlation between prior experience and performance. One of the developers who showed a positive speedup with AI was the one with the most previous experience using Cursor (over 50 hours).
