Code analysis firm sees no major benefits from AI dev tool when measuring key programming metrics, though others report incremental gains from coding copilots with emphasis on code review.
I sent a PR back to a Dev five times before I gave the work to someone else.
they used AI to generate everything.
surprise, there were so many problems it broke the whole stack.
this is a routine thing this one dev does too. every PR has to be tossed back at least once. not expecting perfection, but I do expect it to not break the whole app.
that depends on your definition of what a “terrible dev” is.
of the three devs that I know have used AI, all we’re moderately acceptable devs before they relied on AI. this formed my opinion that AI code and the devs that use it are terrible.
two of those three I no longer work with because they were let go for quality and productivity issues.
so you can clearly see why my opinion of AI code is so low.
Some tools deserve blame. In the case of this, you’re supposed to use it to automate away certain things but that automation isn’t really reliable. If it has to be babysat to the extent that I certainly would argue that it does, then it deserves some blame for being a crappy tool.
If, for instance, getter and setter generating or refactor tools in IDEs routinely screwed up in the same ways, people would say that the tools were broken and that people shouldn’t use them. I don’t get how this is different just because of “AI”.
Okay, so if the tool seems counterproductive for you, it’s very assuming to generalize that and assume it’s the same for everyone else too. I definitely do not have that experience.
It’s not about it being counterproductive. It’s about correctness. If a tool produces a million lines of pure compilable gibberish unrelated to what you’re trying to do, from a pure lines of code perspective, that’d be a productive tool. But software development is more complicated than writing the most lines.
Now, I’m not saying that AI tools produce pure compilable gibberish, but they don’t reliably produce correct code either. So, they fall somewhere in the middle, and similarly to “driver assistance” technologies that half automate things but require constant supervision, it’s quite possible that the middle is the worst area for a tool to fall into.
Everywhere around AI tools there are asterisks about it not always producing correct results. The developer using the tool is ultimately responsible for the output of their own commits, but the tool itself shares in the blame because of its unreliable nature.
Using a tool to speed up your work is not lazy. Using a tool stupidly is stupid. Anyone who thinks these tools are meant to replace humans using logic is misunderstanding them entirely.
You remind me of some of my coworkers who would rather do the same mind numbing task for hours every day rather than write a script that handles it. I judge them for thinking working smarter is “lazy” and I think it’s a fair judgement. I see them as the lazy ones. They’d rather not think more deeply about the scripting aspect because it’s hard. They rather zone out and mindlessly click, copy/paste, etc. I’d rather analyze and break down the problem so I can solve it once and then move onto something more interesting to solve.
sometimes working smarter is actually putting the work in so you don’t have to waste time and stress about if it’s going to work or not.
I get Dreamweaver vibes from AI generated code. Sure, the website works. looks exactly the way it should. works exactly how it should. that HTML source though… fucking aweful.
I can agree, AI is an augment to the tools you can use. however, it’s being marketed as a replacement and a large variety of devs are using it as such.
Same. AI seems like yet another attempt at RAD just like MS Access, Visual Basic, Dreamweaver, and even to some extent Salesforce, or ServiceNow. There are so many technologies that champion this…RoR, Django, Spring Boot…the list is basically forever.
To an extent, it’s more general purpose than those because it can be used with multiple languages or toolkits, but I find it not at all surprising that the first usage of gen AI in my company was to push out “POCs” (the vast majority of which never amounted to anything).
The same gravity applies to this tool as everything else in software…which is that prototyping is easy…integration is hard (unless the organization is well structured, which, well, almost none of them are), and software executives tend to confuse a POC with production code and want to push it out immediately, only to find out that it’s a Potemkin village underneath as they sometimes (or even often) were told the entire time.
So much of the software industry is “JUST GET THIS DONE FASTER DAMMIT!” from middle managers who still seem (despite decades of screaming this) to have developed no widespread means of determining either what they want to get done, or what it would take to get it done faster.
What we have been dealing with the entire time is people that hate to be dependent upon coders or other “nerds”, but need them in order to create products to accomplish their business objectives.
Middle managers still think creating software is algorithmic nerd shit that could be automated…solving the same problems over and over again. It’s largely been my experience that despite even Computer Science programs giving it that image, that the reality is modern coding is more akin to being a millwright. Most of the repetitive, algorithmic nerd shit was settled long ago and can be imported via modules. Imported modules are analogous to parts, and your job is to build or maintain the actual machine that produces the outcomes that are desired, making connecting parts to get the various components to interoperate as needed, repairing failing components, or spotting the shoddy welding between them that is making the current machine fail.
No, shitty devs are enabled by piss-poor hiring practices. I’m currently working with two devs that submit mind bogglingly bad PRs all of the time, and it’s 100% because we hired them in a hasty manner and overlooking issues they displayed during interviews.
Neither of these bad devs use AI to my knowledge. On the other hand I use copilot constantly and the only difference I see in my work is that it takes me less time to complete a given task. It shaves 1-2 minutes off of writing a block/function several times an hour, and that is a good thing.
They rather zone out and mindlessly click, copy/paste, etc. I’d rather analyze and break down the problem so I can solve it once and then move onto something more interesting to solve.
From what I’ve seen of AI code in my time using it, it often is an advanced form of copying and pasting. It frequently takes problems that could be better solved more efficiently with fewer lines of code or by generalizing the problem and does the (IMO evil) work of making the solution that used to require the most drudgery easy.
Typical lack of nuance on the Internet, sadly. Everything has to be Bad or Good. Black or White. AI is either The best thing ever™ or The worst thing ever™. No room for anything in between. Considering negative news generates more clicks, you can see why the media tend to take the latter approach.
I also think much of the hate is just people jumping on the AI = bad band-wagon. Does it have issues? Absolutely. Is it perfect? Far from it. But the constant negativity has gotten tired. There’s a lot of fascinating discussion to be had around AI, especially in the art world, but God forbid you suggest it’s anything but responsible for the total collapse of civilisation as we know it…
If it didn’t significantly contribute to the cooking of all lifeforms on planet Earth, most of us would not mind. We would still deride it because of its untrustworthiness. However, it’s not just useless: it’s also harmful. That’s the core of the beef I (and a lot of other folks) have against the tech.
Having to deal with pull requests defecated by “developers” who blindly copy code from chatgpt is a particularly annoying and depressing waste of time.
At least back when they blindly copied code from stack overflow they had to read through the answers and comments and try to figure out which one fit their use case better and why, and maybe learn something… now they just assume the LLM is right (despite the fact that they asked the wrong question and even if they had asked the right one it’d’ve given the wrong answer) and call it a day; no brain activity or learning whatsoever.
I was lucky enough to not have access to LLMs when I was learning to code.
Plus, over the years I’ve developed a good thick protective shell (or callus) of cynicism, spite, distrust, and absolute seething hatred towards anything involving computers, which younger developers yet lack.
Sorry, you misunderstood my comment, which was very badly worded.
I meant to imply that you, an experienced developer, didn’t get “scammed” by the LLM, and that the difference between you and the dev you mentioned is that you know how to program.
I was trying to make the point that the issue is not the LLM but the developer using it.
Wow, this was somehow considered offensive enough for a mod to remove, but we’re also allowing obvious paid trolls on the platform. Lemmy is not what we want it to be.
I want to believe people arent this dumb but i also dont want to be crazy for suggesting such nonsensical sentiment is manufactured. Such is life in the disinformation age.
Like what are we going to do, tell all Countries and fraudsters to stop using ai because it turns out its too much of a hassle?
We can’t do that, nobody’s saying we can. But this is an important reminder that the tech savior bros aren’t very different from the oil execs.
And constant activism might hopefully achieve the goal of pushing the tech out of the mainstream, with its friend crypto, along other things not to be taken seriously anymore like flying cars and the Hyperloop.
I’m talking about people criticizing LLMs. I’m not a politician. But I’ve seen a few debates about LLMs on this platform, enough to know about the common complaints against ShitGPT. I’ve never seen anyone on this platform seriously arguing for a ban. We all know it’s stupid and that it will be ineffective, just like crackdowns on VPNs in authoritarian countries.
The reminder is the tech itself. It’s yet another tech pushed by techbros to save the world that fails to deliver and is costing the rest of the planet dearly in the form of ludicrous energy consumption.
And by activism, I mean stuff happening on Lemmy as well as outside (coworkers, friends, technical people at conferences/meetups). Like it or not, the consensus among techies in my big canadian city is that, while the tech sure is interesting, it’s regarded with a lot of mistrust.
You can take LLMs seriously if you’d like. But the proofs that the tech is unsound for software engineering keep piling up. I’m fine with your skepticism. But I think the future will look bleaker and bleaker as times goes by. Not a week goes by without its lot of AI fuckups being reported in the press. This article is one of many examples.
Theres no particular fuck up mentioned by this article.
The company that conducted the study which this article speculates on said these tools are getting rapidly better and that they arent suggesting to ban ai development assistants.
Also as quoted in the article, the use of these coding assistance is a process in and of itself. If you arent using ai carefully and iteratively then you wont get good results with current models. How we interact with models is as important as the model’s capability. The article quotes that if models are used well, a coder can be faster by 2x or 3x. Not sure about that personally… seems optimistic depending on whats being developed.
It seems like a good discussion with no obvious conclusion given the infancy of the tech. Yet the article headline and accompanying image suggest its wreaking havoc.
Reduction of complexity in this topic serves nobody. We should have the patience and impartiality to watch it develop and form opinions independently from commeter and headline sentiment. Groupthink has been paricularly dumb on this topic from what ive seen.
Nobody talked about banning them, once again. I don’t want to do that. I want it to leave the mainstream, for environmental reasons first and foremost.
The fuckup is, IDK, the false impression of productivity, and the 41% more bugs? That seems like a huge deal to me, even though I’d like to see this study being reproduced to draw real conclusions.
This, with strawberrries, Air Canada’s chatbots, the 3 Miles Island stuff, the delaying of Google’s carbon neutrality efforts, the cursed Google results telling you to add glue to your pizza, the distrust of the general public about anything with an AI label on it, to mention just a few examples… It’s starting to become a lot.
Even if you omit the ethical aspects of cooking the planet for a toy, the technology is wildly unsound. You seem to think it can get better, and I can respect that. But I’m very skeptical, and there’s a lot of people with the same opinion, even in tech.
Also, when a tool increases your productivity but your salary and paid time off don’t increase, it’s a tool that only benefits the overlords and as such deserves to be hated.
I use a 13 year old PC because a newer one will be infected with Windows 11. (The company refuses to migrate to Linux because some of the software they use isn’t compatible.)
I mean, you’re clearly using them because they still work, not because of a hatred for increasing productivity for the overlords. Your choice was based on reasonable logic, unlike the other guy.
I truly don’t understand the tendency of people to hate these kinds of tools. Honestly seems like an ego thing to me.
I sent a PR back to a Dev five times before I gave the work to someone else.
they used AI to generate everything.
surprise, there were so many problems it broke the whole stack.
this is a routine thing this one dev does too. every PR has to be tossed back at least once. not expecting perfection, but I do expect it to not break the whole app.
Like I told another person ITT, hiring terrible devs isn’t something you can blame on software.
that depends on your definition of what a “terrible dev” is.
of the three devs that I know have used AI, all we’re moderately acceptable devs before they relied on AI. this formed my opinion that AI code and the devs that use it are terrible.
two of those three I no longer work with because they were let go for quality and productivity issues.
so you can clearly see why my opinion of AI code is so low.
I would argue that it’s obvious if someone doesn’t know how to use a tool to do their job, they aren’t great at their job to begin with.
Your argument is to blame the tool and excuse the person who is awful with the tool.
my argument is that lazy devs use the tool because that’s what it was designed for.
just calling a hammer a hammer.
Some tools deserve blame. In the case of this, you’re supposed to use it to automate away certain things but that automation isn’t really reliable. If it has to be babysat to the extent that I certainly would argue that it does, then it deserves some blame for being a crappy tool.
If, for instance, getter and setter generating or refactor tools in IDEs routinely screwed up in the same ways, people would say that the tools were broken and that people shouldn’t use them. I don’t get how this is different just because of “AI”.
Okay, so if the tool seems counterproductive for you, it’s very assuming to generalize that and assume it’s the same for everyone else too. I definitely do not have that experience.
Have you read the article? It’s a shared experience multiple people report, and the article even provide statistics.
It’s not about it being counterproductive. It’s about correctness. If a tool produces a million lines of pure compilable gibberish unrelated to what you’re trying to do, from a pure lines of code perspective, that’d be a productive tool. But software development is more complicated than writing the most lines.
Now, I’m not saying that AI tools produce pure compilable gibberish, but they don’t reliably produce correct code either. So, they fall somewhere in the middle, and similarly to “driver assistance” technologies that half automate things but require constant supervision, it’s quite possible that the middle is the worst area for a tool to fall into.
Everywhere around AI tools there are asterisks about it not always producing correct results. The developer using the tool is ultimately responsible for the output of their own commits, but the tool itself shares in the blame because of its unreliable nature.
Using a tool to speed up your work is not lazy. Using a tool stupidly is stupid. Anyone who thinks these tools are meant to replace humans using logic is misunderstanding them entirely.
You remind me of some of my coworkers who would rather do the same mind numbing task for hours every day rather than write a script that handles it. I judge them for thinking working smarter is “lazy” and I think it’s a fair judgement. I see them as the lazy ones. They’d rather not think more deeply about the scripting aspect because it’s hard. They rather zone out and mindlessly click, copy/paste, etc. I’d rather analyze and break down the problem so I can solve it once and then move onto something more interesting to solve.
sometimes working smarter is actually putting the work in so you don’t have to waste time and stress about if it’s going to work or not.
I get Dreamweaver vibes from AI generated code. Sure, the website works. looks exactly the way it should. works exactly how it should. that HTML source though… fucking aweful.
I can agree, AI is an augment to the tools you can use. however, it’s being marketed as a replacement and a large variety of devs are using it as such.
shitty devs are enabled by shitty tools.
Same. AI seems like yet another attempt at RAD just like MS Access, Visual Basic, Dreamweaver, and even to some extent Salesforce, or ServiceNow. There are so many technologies that champion this…RoR, Django, Spring Boot…the list is basically forever.
To an extent, it’s more general purpose than those because it can be used with multiple languages or toolkits, but I find it not at all surprising that the first usage of gen AI in my company was to push out “POCs” (the vast majority of which never amounted to anything).
The same gravity applies to this tool as everything else in software…which is that prototyping is easy…integration is hard (unless the organization is well structured, which, well, almost none of them are), and software executives tend to confuse a POC with production code and want to push it out immediately, only to find out that it’s a Potemkin village underneath as they sometimes (or even often) were told the entire time.
So much of the software industry is “JUST GET THIS DONE FASTER DAMMIT!” from middle managers who still seem (despite decades of screaming this) to have developed no widespread means of determining either what they want to get done, or what it would take to get it done faster.
What we have been dealing with the entire time is people that hate to be dependent upon coders or other “nerds”, but need them in order to create products to accomplish their business objectives.
Middle managers still think creating software is algorithmic nerd shit that could be automated…solving the same problems over and over again. It’s largely been my experience that despite even Computer Science programs giving it that image, that the reality is modern coding is more akin to being a millwright. Most of the repetitive, algorithmic nerd shit was settled long ago and can be imported via modules. Imported modules are analogous to parts, and your job is to build or maintain the actual machine that produces the outcomes that are desired, making connecting parts to get the various components to interoperate as needed, repairing failing components, or spotting the shoddy welding between them that is making the current machine fail.
No, shitty devs are enabled by piss-poor hiring practices. I’m currently working with two devs that submit mind bogglingly bad PRs all of the time, and it’s 100% because we hired them in a hasty manner and overlooking issues they displayed during interviews.
Neither of these bad devs use AI to my knowledge. On the other hand I use copilot constantly and the only difference I see in my work is that it takes me less time to complete a given task. It shaves 1-2 minutes off of writing a block/function several times an hour, and that is a good thing.
From what I’ve seen of AI code in my time using it, it often is an advanced form of copying and pasting. It frequently takes problems that could be better solved more efficiently with fewer lines of code or by generalizing the problem and does the (IMO evil) work of making the solution that used to require the most drudgery easy.
Removed by mod
Typical lack of nuance on the Internet, sadly. Everything has to be Bad or Good. Black or White. AI is either The best thing ever™ or The worst thing ever™. No room for anything in between. Considering negative news generates more clicks, you can see why the media tend to take the latter approach.
I also think much of the hate is just people jumping on the AI = bad band-wagon. Does it have issues? Absolutely. Is it perfect? Far from it. But the constant negativity has gotten tired. There’s a lot of fascinating discussion to be had around AI, especially in the art world, but God forbid you suggest it’s anything but responsible for the total collapse of civilisation as we know it…
If it didn’t significantly contribute to the cooking of all lifeforms on planet Earth, most of us would not mind. We would still deride it because of its untrustworthiness. However, it’s not just useless: it’s also harmful. That’s the core of the beef I (and a lot of other folks) have against the tech.
Oh for sure. How we regulate AI (including how we power it) is really important, definitely.
I think you nailed it with everything you just said.
Having to deal with pull requests defecated by “developers” who blindly copy code from chatgpt is a particularly annoying and depressing waste of time.
At least back when they blindly copied code from stack overflow they had to read through the answers and comments and try to figure out which one fit their use case better and why, and maybe learn something… now they just assume the LLM is right (despite the fact that they asked the wrong question and even if they had asked the right one it’d’ve given the wrong answer) and call it a day; no brain activity or learning whatsoever.
That is not a problem with the ai software, that’s a problem with hiring morons who have zero experience.
No. LLMs are very good at scamming people into believing they’re giving correct answers. It’s practically the only thing they’re any good at.
Don’t blame the victims, blame the scammers selling LLMs as anything other than fancy but useless toys.
Removed by mod
Did you get scammed by the LLM? If not, what’s the difference between you and the dev you mentioned?
I was lucky enough to not have access to LLMs when I was learning to code.
Plus, over the years I’ve developed a good thick protective shell (or callus) of cynicism, spite, distrust, and absolute seething hatred towards anything involving computers, which younger developers yet lack.
Sorry, you misunderstood my comment, which was very badly worded.
I meant to imply that you, an experienced developer, didn’t get “scammed” by the LLM, and that the difference between you and the dev you mentioned is that you know how to program.
I was trying to make the point that the issue is not the LLM but the developer using it.
And I’m saying that I could have been that developer if I were twenty years younger.
They’re not bad developers, they just haven’t yet been hurt enough to develop protective mechanisms against scams like these.
They are not the problem. The scammers selling the LLM’s as something they’re not are.
Ah, gotcha, and I agree
Carbon footprint. Techbro arrogance. Not sure what’s hard to understand about it.
Removed by mod
Of course you know me better than myself.
I guess you wanted an answer but decided upfront you weren’t gonna like it no matter what? Not much I can do about that.
Removed by mod
Whether or not I’ve used Copilot is entirely irrelevant to the points I was making. Please remain on topic.
Removed by mod
Removed by mod
Wow, this was somehow considered offensive enough for a mod to remove, but we’re also allowing obvious paid trolls on the platform. Lemmy is not what we want it to be.
Trying to belittle or offend others is against rule 3, it is not being excellent to eachother. Let’s do better my friend.
deleted by creator
Its really weird.
I want to believe people arent this dumb but i also dont want to be crazy for suggesting such nonsensical sentiment is manufactured. Such is life in the disinformation age.
Like what are we going to do, tell all Countries and fraudsters to stop using ai because it turns out its too much of a hassle?
We can’t do that, nobody’s saying we can. But this is an important reminder that the tech savior bros aren’t very different from the oil execs.
And constant activism might hopefully achieve the goal of pushing the tech out of the mainstream, with its friend crypto, along other things not to be taken seriously anymore like flying cars and the Hyperloop.
You are speaking for everyone so right away i dont see this as an actual conversion, but a decree of fact by someone i know nothing about.
What are you saying is an important reminder? This article?
By constant activism, do you mean anything that occurs outside of lemmy comments?
Why would we not take LLMs seriously?
I’m talking about people criticizing LLMs. I’m not a politician. But I’ve seen a few debates about LLMs on this platform, enough to know about the common complaints against ShitGPT. I’ve never seen anyone on this platform seriously arguing for a ban. We all know it’s stupid and that it will be ineffective, just like crackdowns on VPNs in authoritarian countries.
The reminder is the tech itself. It’s yet another tech pushed by techbros to save the world that fails to deliver and is costing the rest of the planet dearly in the form of ludicrous energy consumption.
And by activism, I mean stuff happening on Lemmy as well as outside (coworkers, friends, technical people at conferences/meetups). Like it or not, the consensus among techies in my big canadian city is that, while the tech sure is interesting, it’s regarded with a lot of mistrust.
You can take LLMs seriously if you’d like. But the proofs that the tech is unsound for software engineering keep piling up. I’m fine with your skepticism. But I think the future will look bleaker and bleaker as times goes by. Not a week goes by without its lot of AI fuckups being reported in the press. This article is one of many examples.
Theres no particular fuck up mentioned by this article.
The company that conducted the study which this article speculates on said these tools are getting rapidly better and that they arent suggesting to ban ai development assistants.
Also as quoted in the article, the use of these coding assistance is a process in and of itself. If you arent using ai carefully and iteratively then you wont get good results with current models. How we interact with models is as important as the model’s capability. The article quotes that if models are used well, a coder can be faster by 2x or 3x. Not sure about that personally… seems optimistic depending on whats being developed.
It seems like a good discussion with no obvious conclusion given the infancy of the tech. Yet the article headline and accompanying image suggest its wreaking havoc.
Reduction of complexity in this topic serves nobody. We should have the patience and impartiality to watch it develop and form opinions independently from commeter and headline sentiment. Groupthink has been paricularly dumb on this topic from what ive seen.
Nobody talked about banning them, once again. I don’t want to do that. I want it to leave the mainstream, for environmental reasons first and foremost.
The fuckup is, IDK, the false impression of productivity, and the 41% more bugs? That seems like a huge deal to me, even though I’d like to see this study being reproduced to draw real conclusions.
This, with strawberrries, Air Canada’s chatbots, the 3 Miles Island stuff, the delaying of Google’s carbon neutrality efforts, the cursed Google results telling you to add glue to your pizza, the distrust of the general public about anything with an AI label on it, to mention just a few examples… It’s starting to become a lot.
Even if you omit the ethical aspects of cooking the planet for a toy, the technology is wildly unsound. You seem to think it can get better, and I can respect that. But I’m very skeptical, and there’s a lot of people with the same opinion, even in tech.
Also, when a tool increases your productivity but your salary and paid time off don’t increase, it’s a tool that only benefits the overlords and as such deserves to be hated.
Oh, so do you use a 13 year old PC because a newer one increases your productivity without increasing your salary and paid time off?
I use a 13 year old PC because a newer one will be infected with Windows 11. (The company refuses to migrate to Linux because some of the software they use isn’t compatible.)
Personally… I do
I could request a new one, but why? This one works, it’s just slow as all hell.
Gives excellent argument for requesting a new one:
I’m paid by the hour, I don’t care
I mean, you’re clearly using them because they still work, not because of a hatred for increasing productivity for the overlords. Your choice was based on reasonable logic, unlike the other guy.
Removed by mod
Removed by mod
Removed by mod
Removed by mod
Removed by mod
Removed by mod
Removed by mod