Scale AI documents show Google extensively using ChatGPT to improve its AI chatbot: 'Make it BETTER than GPT'

Getty Images; Alyssa Powell/BIScale AI contractors used ChatGPT to improve Google's AI chatbot responses, documents showed.Hundreds of internal Google Docs about the effort were left public by Scale AI.Scale AI and Google denied using ChatGPT to train models.In 2023, Google was in a race to catch up with ChatGPT — and it turned to ChatGPT itself to do it.Hundreds of documents obtained by Business Insider reveal that Google's contractors at Scale AI systematically used ChatGPT to improve Bard, Google's own chatbot at the time. When it launched earlier that year, Bard, which has since been renamed Gemini, was internally mocked as "rushed" and "botched."Scale AI contractors generated thousands of responses from ChatGPT and compared them to their own "rewrites" of Bard's answers. They then improved their rewrites to exceed or at least match ChatGPT, feeding all the data back to Google.Scale AI managers wrote in detail how ChatGPT's answers tended to have better formatting and more interesting facts. They ordered workers to "explain why gpt4 is better" and "make it BETTER than GPT." A single spreadsheet flagged dozens of contractors for writing responses "consistently worse than GPT4." In one instance, the document said contractors could get a 15% bonus for their responses performing better than ChatGPT.Scale AI is a San Francisco startup that does crucial AI grunt work for Big Tech. It uses an army of human contractors to do things like labeling images and, as was the case with Google, rewriting chatbot responses. Meta is reportedly investing $15 billion in Scale AI as part of a blockbuster AI deal to buy almost half the company and hire its CEO, Alexandr Wang, for an in-house "superintelligence" team.The documents obtained by BI showcase how closely Google monitored its chief rival's work.OpenAI's terms of service at the time prohibited others from using its output "to develop models that compete with OpenAI." Scale AI and Google did not respond to a question about whether they got permission from OpenAI for these detailed comparisons and rewrites.Scale AI told BI that the ChatGPT outputs weren't used to train Google's or any others' models and were part of routine "evaluations," which it said are industry standards."Scale did not, and does not, use ChatGPT responses to train Gemini or any models," a Scale AI spokesperson said in a statement. The spokesperson said that the documents describe "standard side-by-side evaluations, not the use of ChatGPT or any third-party model outputs for training.""Doing side-by-side competitive evals is standard practice for the industry and those evaluation results are not used to train models," the spokesperson said.Similarly, Google said, "Any suggestion that we have used other companies' models to train Gemini is inaccurate."Experts told BI that this kind of comparison is indeed common at some top AI labs. Open AI, which is reportedly in partnership talks with Google Cloud, didn't respond to repeated requests for comment.Project 'Bulba'Scale AI gave Bard a catchy codename, "Bulba," after the Pokémon Bulbasaur. The mission was clear: compare Bulba's answers with ChatGPT's to make them better.Scale AI never mentioned Google by name in the documents, referring instead to its anonymous "client." It references Bard over a dozen times in a private Google sheet titled "bard rewrite comparison with gpt4," and a slide in one training document includes Google's logo.Scale AI founder Alexandr Wang.Jeff Chiu/APIn July 2023, a manager ordered workers to study GPT-4's responses closely and figure out why they outperformed Bard's. "Try to come up with feedback that we can share so that experts can write responses better than GPT4 or at least the same," the manager wrote.Scale AI also created a spreadsheet that compared 1,729 Bard rewrites directly to ChatGPT in October 2023. Each rewrite was rated with labels like "worse than GPT4" or "Needs Some Fixes." In one example, a worker rewrote a Bard review of a nursery chair that managers stamped "worse than GPT" because it "lacks detail compared to GPT4."Another contractor's review of a Charleston history museum didn't make the cut either — a manager wrote that ChatGPT's version was "much better."Scale AI also used ChatGPT to improve Bard's responses in specific domains, like engineering or physics. In an update from August 2023, Scale AI managers wrote that they would have staff "redo" Google's AI answers for engineering-related questions "with GPT4 guidance."The documents showed that Scale AI and Google barred its contractors from copying and pasting ChatGPT responses directly into their rewrites, though, an issue many contractors were flagged for.Scale AI says comparisons weren't for trainingThe internal documents BI reviewed described the project's goal as helping "train" Bard to give it more specific and complete answers, and refer to efforts to "improve the model."Google did not answer follow-up questions on whether those comparisons influenced training. Scale AI said that there's a clear line between evaluating a model's performance and training it — and that ChatGPT outputs were only used for the former."There is a difference between training data and evaluation data," a spokesperson said. "Evaluation data is not ingested by a model to train it, but rather used to measure how well a model is performing."Matthew Guzdial, an assistant computer science professor at the University of Alberta, says evaluation data can still influence an AI model."Even if all they're doing is looking at those outputs and rating that information to adjust the structure of the model, you could still make the argument that it's involved in the training process," he told BI.The documents were left publicScale AI, which has not previously made public details about its work with Google, left an over 300-page Google Doc public.It contains dozens of links to other Google Docs, many of which are also public and contain sensitive information, including contractors' compensation details, personal email addresses, and performance reviews, along with still-functioning passwords to internal training sessions. Some of the Google Docs can still be edited by anyone who has the link.Scale AI told BI that it is "actively investigating" how the document "may have been accessed" and is "taking steps to ensure any inadvertent exposure is remediated."More than two days after BI told Scale AI about the public Google Doc, it was still online and available for anyone with the link to download.Google is ahead on AI againGoogle CEO Sundar Pichai.Klaudia Radecka/NurPhotoThe documents don't specify how effective the comparison efforts were. Since its Bard flub in 2023, Google has rebranded Bard to Gemini and transformed into an AI shipping machine. Last month, it launched over 100 new AI products and features at I/O, its annual developer conference.Google CEO Sundar Pichai began his speech at I/O by rattling off the industry benchmarks that Gemini is topping, touting the company's latest AI achievements."We are shipping faster than ever," Pichai said onstage.Have a tip? Contact this reporter via email at [email protected] or Signal and WhatsApp at 628-282-2811. Use a personal email address and a nonwork device; here's our guide to sharing information securely.Read the original article on Business Insider

Comments

Business News

Jeff Bezos and Lauren Sanchez to spotlight Venice's artisanal heritage during upcoming nuptials

washington times

I bought a house using a VA loan, becoming the only family member to own a home. I feel guilty that my siblings never will.

about 2 hours ago

26 photos of the worst hurricanes to have hit the US

insider

When you should (and shouldn't) take out a prenup: Top divorce lawyer VANESSA LLOYD PLATT on how to protect your assets when tying the knot

about 4 hours ago

Eat your beans — 1 cup a day cuts inflammation and bad cholesterol, scientists say

insider

I lost my job and picked up decluttering as a side hustle. I'm happy that I can make money on my own — and make it right away.

about 6 hours ago

Here's what a poolside cocktail could cost you on average at these popular vacation destinations, from Maui to Miami

insider

Comments

Similar News

I invested $30,000 to scale my European-based tote bag business in the US &mdash; then tariffs hit

Scale AI CEO Alexandr Wang Leaves to Join Meta Following $14.3B Deal

Meta invests $14.3B in AI firm Scale and recruits its CEO for 'superintelligence' team

5 things to know about Alexandr Wang, the buzzy Scale AI founder

Meta Invests Nearly $15 Billion in Scale AI to Kick-Start Superintelligence Lab

Meta Eyes Scale AI, a Quiet Data Powerhouse, in $10B A.I. Push

Meta in talks over Scale AI stake that could top $10B, Bloomberg reports

Trump indicates openness to scaling back SALT relief in GOP tax bill

Rosebud lands $6M to scale its interactive AI journaling app

Trump Threatens ‘Large Scale Fines’ After Transgender Girl Competes In California Track Competition

Thames crisis underlines scale of water industry turnaround task

Marriott targets budget travelers with new mid-scale extended-stay option

Scale AI hires team behind remote developer recruiting platform Pesto AI

Startups Weekly: AMD acquisition and other moves to scale AI startups

European companies cut costs, scale back investments in China as its economy slows

European companies cut costs, scale back investments in China as its economy slows

Scots accountancy chief underlines business 'scale up' challenge

Alt Carbon scores $12M seed to scale carbon removal in India

Serve is betting that food delivery and access to public markets are the keys to scaling robotics

Lowe’s Scales And Optimizes Its Online Marketplace For Vendors And Customers

Business News

Jeff Bezos and Lauren Sanchez to spotlight Venice's artisanal heritage during upcoming nuptials

DWP confirms people born before exact date will receive 2025 Winter Fuel Payments

Twin federal proposals threaten provider taxes, key source of Medicaid funding for states

I'm a hot sleeper, Simba's 'cool touch' duvet sends me to sleep in a heatwave

I bought a house using a VA loan, becoming the only family member to own a home. I feel guilty that my siblings never will.

26 photos of the worst hurricanes to have hit the US

Tech execs, Uncle Sam wants you for the US Army

News diary 16-22 June: US Tiktok sell-or-ban deadline, Chris Brown in UK court

‘How To Train Your Dragon’ Just Set A Rotten Tomatoes Audience Score Record

When Is “How To Train Your Dragon” Coming To Streaming?

Higher Oil Prices Mean Less GDP

UFC Tonight: What Time Does The UFC Atlanta Fight Card Start?

The Aaron Civale Trade Is A Win For The Brewers And The White Sox

'We sit in the dark to save money on electricity'

Two groups of people 'need to apply' for £300 DWP winter fuel payment

Copy Kate Middleton’s designer jewellery for less with pearl earrings just like her go-to pair

The scariest thing about AI might be the way your boss is talking about it

Drone overload: Too many people want to sell drones to the US military

My dad and I spent a week traveling together without our phones. It was one of the best trips I've ever taken.

A Great ‘Reacher’ Season 4 Release Date Update, Book Adaptation Confirmation

Russian Strikes On Ukrainian Hotels Silencing The Press

Why RFK Jr.’s Purge Of Vaccine Advisors May Increase Your Health Costs

Ex-FC Barcelona Star Dembele Makes Ballon d’Or, Lamine And Messi Confessions

The Ad Industry’s A.I. Reckoning

After Ambiance Apparel raid, Fashion District businesses, workers wait in fear

High street lender Metro Bank receives takeover approach

Home Bargains reduces £87 makeup kits to just 99p each

'A lot of people' to get DWP PIP benefit cuts with 13-week rule coming

Princess Kate pays tribute to Diana as she copies her turquoise look for Trooping The Colour

Amazon's £28 Molton Brown Father's Day gift set includes next day delivery

When you should (and shouldn't) take out a prenup: Top divorce lawyer VANESSA LLOYD PLATT on how to protect your assets when tying the knot

Eat your beans &mdash; 1 cup a day cuts inflammation and bad cholesterol, scientists say

The No. 1 thing that makes interns stand out, according to a Google Cloud exec

I've been on the low-FODMAP diet for 6 weeks. Here are 6 of my favorite Trader Joe's grocery staples.

I just took a 3-week vacation across Europe without my partner. I prefer to travel alone rather than with her.

Doctors stunned to see strict exercise regimen is as good as medicine for colon cancer

I got breast cancer at 30. My treatment means I'll need to delay having kids for 5 to 10 years.

I own a grocery store. A big supplier outage revealed problems with our food system that customers rarely see.

Rare earth minerals are the biggest card China can play in its negotiations with Trump

Economic anxiety or not, Americans are still prioritizing Euro summer travel

Thomas Frank Has Risked His Reputation On Tottenham Hotspur Move

‘Final Destination Bloodlines’ Arrives On Streaming This Week

Harrods plots legal action against estate of former owner al-Fayed

'I tried a viral tan remover that works in 'just 1-minute' and I didn't expect these results'

7-foot climbing Wisteria that blooms vibrant flowers twice a year plummets to under £60 in sale

The Big Stay is finally paying off: Quitting to job-switch is worse for wage growth than sticking it out

What you need to know about 'fractional leadership' &mdash; and why this CEO thinks it's the future

From frustration to elation: What Wall Street thinks about the potential death of the private equity recruiting race

My wife and I were both managers. When we retired, we set 3 marriage rules so we wouldn't micromanage each other.

The best place to live in the US is this small town in Texas

I lost my job and picked up decluttering as a side hustle. I'm happy that I can make money on my own &mdash; and make it right away.

Here's what a poolside cocktail could cost you on average at these popular vacation destinations, from Maui to Miami

I own a small solar company in Montana and might have to lay off most of my employees. I'm not hiding that from them.

I've been waking up before 3 a.m. for 30 years. I love my solo time, and my morning routine sets me up for a good day.

The hottest tech job candidates in America &mdash; and what you need to become one &mdash; according to top recruiters

I invested $30,000 to scale my European-based tote bag business in the US &mdash; then tariffs hit

FC Barcelona Legend Suarez Praises Lamine Yamal

I invested $30,000 to scale my European-based tote bag business in the US — then tariffs hit

Eat your beans — 1 cup a day cuts inflammation and bad cholesterol, scientists say

What you need to know about 'fractional leadership' — and why this CEO thinks it's the future

I lost my job and picked up decluttering as a side hustle. I'm happy that I can make money on my own — and make it right away.

The hottest tech job candidates in America — and what you need to become one — according to top recruiters

I invested $30,000 to scale my European-based tote bag business in the US — then tariffs hit