Why "use AI" may not be the answer to boosting software productivity

A surprising new study finding that AI tools can reduce programmers' productivity is upending assumptions about the technology's world-changing potential.Why it matters: Software runs our civilization, and AI is already transforming the business of making it — but no one really knows whether AI will decimate programming jobs, or turn every coder into a miracle worker, or both.Driving the news: The study by METR, a nonprofit independent research outfit, looked at experienced programmers working on large, established open-source projects.It found that these developers believed that using AI tools helped them perform 20% faster — but they actually worked 19% slower.The study appears rigorous and well-designed, but it's small (only 16 programmers participated, completing 246 tasks).Zoom out: For decades, industry visionaries have dreamed of a holy grail called "natural language programming" that would allow people to instruct computers using everyday speech, without needing to write code.As large language models' coding prowess became evident, it appeared this milestone had been achieved."The hottest new programming language is English," declared AI guru (and OpenAI cofounder) Andrej Karpathy on X early in 2023, soon after ChatGPT's launch.In February, Karpathy also coined the term "vibe-coding" — meaning the quick creation of rough-code prototypes for new projects by just telling your favorite AI to whip up something from scratch.The most fervent believers in software's AI-written future say that human beings will do less and less programming, and engineers will turn into some combination of project manager, specifications-refiner and quality-checker. Either that, or they'll be unemployed.Zoom in: AI-driven coding tends to be more valuable in building new systems from the ground up than in extending or refining existing systems, particularly when they're big.While innovative new products get the biggest buzz and make the largest fortunes, the bulk of software work in most industries consists of more mundane maintenance labor.Anything that makes such work more efficient could save enormous amounts of time and money.Yes, but: This is where the METR study found AI actually slowed experienced programmers down.One key factor was that human developers found AI-generated code unreliable and ended up devoting extra time to reviewing, testing and fixing it."One developer notes that he 'wasted at least an hour first trying to [solve a specific issue] with AI' before eventually reverting all code changes and just implementing it without AI assistance," the study says.Between the lines: The study authors note that AI coding tools are improving at a rapid enough rate that their findings could soon be obsolete. They also warn against generalizing too broadly from their findings and note the many counter-examples of organizations and projects that have made productivity gains with coding tools. One notable caution that's inescapable from the study's findings: Don't trust self-reporting of productivity outcomes. We're not always the best judges of our own efficiency. Another is that it's relatively easy to measure productivity in terms of "task completion" but very hard to assess total added value in software-making. Thousands of completed tickets can be meaningless — if, for instance, a program is about to be discontinued. Meanwhile, one big new insight can change everything in ways no productivity metric can capture. The big picture: The software community is divided over whether to view the advent of AI coding with excitement or dread."Quitting programming as a career right now because of LLMs would be like quitting carpentry as a career thanks to the invention of the table saw," the AI developer and blogger Simon Willison recently wrote on Bluesky. Or "it could be like quitting agriculture as a career thanks to the invention of tractors and combine harvesters instead," industry veteran Tom Coates replied.

Comments

World news

Psychosurgery is back. But these are not the ice-pick-through-the-eye-socket lobotomies of the past

National Post

Comments

World news

Psychosurgery is back. But these are not the ice-pick-through-the-eye-socket lobotomies of the past

1,500 tarantulas seized at airport after being smuggled in chocolate spongecake boxes

Gov. Gavin Newsom unloads on ‘son of a b***h’ Trump over ‘Newscum’ moniker

Watch moment Utah wildfire turns into huge fire vortex

Mike Waltz to be grilled over Signal chat saga at hearing for UN role

Aid cuts put vital work to protect millions in Ethiopia from climate crisis at risk

I burnt my house down and was left homeless as a child, says Italian PM Giorgia Meloni

Why Russian markets jumped after Trump’s tariff threat against Putin

Young subway surfers caught travelling on top of moving train by police drone team

Lord Darzi robbed of £175,000 Richard Mille watch on Capri holiday

From ‘The Salt Path’ to my fake columns, we can’t recognise truth anymore

Drought declared for millions in England as more areas face hosepipe bans

Masterchef is bigger than Wallace and Torode and ‘will survive’, says BBC boss

Three Lions selection headaches that will be keeping Andy Farrell up at night

Four unanswered questions about John Torode’s MasterChef future

Which cars will be eligible for the EV scheme – with Tesla set to miss out

Why it’s taking LA so long to rebuild

The 9 best books of the year so far

Equatorial Guinea urges UN to block the sale of a Paris mansion seized by France

Photos show Bastille Day celebrations in France

MoD data breach put lives of around 20,000 Afghans 'at risk of serious violence'

4 charged after allegedly forcing their way into Toronto MP’s constituency office

Canadians face air quality, heat warnings as some stuck indoors from both

Missed Prime Day? You can still snag these Amazon discounts

Ontario trustee has ignored requests to repay share of $145K Italy trip, government says

Ontario students targeted by hundreds of antisemitic incidents, federal report finds

How Melania Trump emerged as one of Zelensky’s key allies in the White House

Yulia Svyrydenko: Ukraine’s new prime minister who masterminded key Trump deal

Republicans block effort to force release of Epstein files in Congress

Jon Stewart mocks Trump’s attempt to distract MAGA base from Epstein debacle in epic takedown

Nextdoor social site, looking for a revival, pins hopes on partnership with local news providers

Flash floods swamp New York City subway station as horrified commuters watch on

Why Europe’s only indigenous tribe started a rap festival in the Arctic circle

Trump ‘asks Zelensky if he could strike Moscow with US long-range weapons’

British tourist, 25, fighting for life after fall from supermarket car park in Ibiza

Yorkshire Water ‘rewarded’ £800k since 2020 despite losing 21% of water to leaks

I’ve already spent £100k trying to get my kids SEND support

Seven explosive moments that lit up England vs India thriller at Lord’s

Ask a urologist: Am I going to the loo too much?

We bought a £25k fixer-upper in France – now we live our dream life mortgage-free

My son is autistic – his school is proof the SEND system can work

The UK has utterly failed SEND kids – but the solution is simple

MAGA's siege mentality shadows Trump's "Golden Age"

Trump admin faces GOP blowback in Congress over Epstein

Trump's new wall: His push to oust legal U.S. residents

Gen Z's broken school-to-work pipeline

Where Americans walk and bike most often

Trump's dueling economic realities will inevitably collide

Sycamore Gap vandals face their fate in English court for felling beloved tree

Tesla goes to trial over Autopilot crash that killed stargazing student

Netanyahu’s government on edge as ultra-Orthodox party quits coalition over military draft law

Texas floods latest: Over 130 dead as report claims Camp Mystic leader received flood warning hour before disaster

Firebrand evangelical pastor John MacArthur dies at 86

Unreleased Beyoncé music stolen from car on Cowboy Carter tour

Why Marco Rubio’s Asia Visit Was a Total Bust

Why "use AI" may not be the answer to boosting software productivity

That blank look you're getting? It's called the "Gen Z stare"

Rap festival near the Arctic Ocean delights crowds under the midnight sun

Rescuers search for 11 missing after speedboat capsized in western Indonesia

Tesla debuts in India with upscale showroom launch in Mumbai