YouGov publishes scientific validation of AI twins
Summary by AI BETAClose X
YouGov publishes scientific validation of its AI twins – and challenges industry to set standards for simulated research
- YouGov created 1,008 “E-twins” and tested them against 693 real, government-ID verified people they were built from – asking both the same new questions
- Twins’ results were validated through a recontact survey of the panellists they were based on within 48 hours
- Twins trained on panellists’ ratings of 4,000+ brands and entities matched their human to within one point on a five-point scale 90% of the time, up from 78% on profiling data alone
- Twins hit the same 90% on a topic they were never trained on
- YouGov calls on the industry to agree on validation standards – and challenges every simulator to predict real-world outcomes in public
YouGov, the international research and data analytics group, today published a white paper on one of the most transparent tests yet of one-to-one human-to-digital-twin simulation.
Rather than comparing aggregate results against public datasets, the groundbreaking research checked each AI twin individually against its own human.
Working with OriginalVoices, an AI platform specialising in simulation, YouGov built digital twins – LLM-based simulations of real people’s attitudes, which it calls “E-twins” – of 1,008 government ID-verified members of its proprietary panel. It then put the same new questions to both the E-twin and the real human respondent. Fully 69% of the panellists responded within 48 hours, giving a direct check on the accuracy of their E-twins. In day-to-day operations, this validation can be done in as little as an hour.
The key findings from YouGov’s AI twins testing:
- Rich data drives accuracy. Adding panellists' ratings of 4,000+ entities lifted person-level accuracy (within one point on a five-point scale) from 78% to 90%.
- YouGov’s E-twins answered what they were never taught. Fiction books were not part of the training data, but when asked about them, the E-twins still came within one point of their human 90% of the time.
- Twin confidence tracks with accuracy. Each open-ended answer a digital twin gives is assigned a confidence level (High, Medium or Low) by OriginalVoices. Panellists said the answer described their own views “very” or “somewhat” well for 81% of High-confidence answers, 72% of Medium and 67% of Low – giving researchers a practical guide to which individual twin responses to trust.
- YouGov's real panelists regularly identify with their twins' opinions. 75% of the time, panellists judged their E-twin's open-ended answers as describing their own views.
- Less stereotyping. More contextual training data reduced twins’ reliance on demographics, with no pattern of biased performance by age, gender, race or education.
Current simulations are built on shaky foundations
YouGov says much of the evidence offered to date by simulators across the industry rests on publicly available datasets. Those questions may already sit in the training data of the underlying language models, so a twin can appear to know an answer it has simply absorbed – and respondents cannot be recontacted to verify the results. Aggregate comparisons can also mask twins that flatten towards the average or stereotype.
Simulated data providers need to show their workings
YouGov is calling on the industry to develop shared standards of verification and validation for defined use cases. It challenges all simulators to follow its lead in validating twins against their individual human, test them on unseen topics, publish their methods, show their misses and predict real-world outcomes in public.
Stephan Shakespeare, CEO and co-founder of YouGov: “The research industry is being transformed by AI in many ways but especially when it comes to simulation. Whether this is a dangerous development or a beneficial advance depends on whether clients can know if its outputs are reliable. I challenge everybody in the industry to work together to establish standards for verifying and validating simulated data. This white paper shows what we think these standards should involve - including the ability to validate simultaneously. If the industry wants simulation to be trusted by clients and the public, it needs to earn it and this can only be achieved through widely agreed standards.”
YouGov’s white paper also publishes the gaps future analysis will address. The current study did not measure a ceiling – how often a person repeats their own answer to the same question. A twin can't reasonably beat the person it copies: if panellists repeated their own rating 70% of the time, the twins' 58% exact-match rate would represent 83% of that ceiling. YouGov will test this in the next iteration of its research.
Read the full white paper: An evaluation of digital twins based on YouGov panel data
About YouGov
YouGov is a global research data and analytics group. Our mission is to offer unparalleled insight into what the world really thinks and does. With operations in the Americas, Mainland Europe, UK and Asia Pacific, we have one of the world's largest research networks.
Above all, YouGov is powered by reality. That stems from a unique panel of millions of registered members across 64 markets, encapsulating some 21 million shopping trips and millions of interconnected data points. Our unique approach to recruiting and engaging with our panel, combined with our state-of-the-art technology platforms, enables us to deliver real-world, real-time insights that lead to better decision-making and a competitive advantage for our clients.
As innovators and pioneers of online market research, we have a strong reputation as a trusted source of accurate data and insights. Testament to this, YouGov data is regularly referenced by the global press, and we are consistently one of the most quoted market research sources in the world.
YouGov /Research Reality
For further information, visit yougov.com
Enquiries
|
YouGov plc Stephan Shakespeare, CEO James Davies, CFO Hannah Jethwani, Head of Corporate Strategy & Investor Relations |
020 7012 6000 |
|
FTI Consulting Charles Palmer / Valerija Cymbal / Jemima Gurney |
020 3727 1000 |
RNS may use your IP address to confirm compliance with the terms and conditions, to analyse how you engage with the information contained in this communication, and to share such analysis on an anonymised basis with others as part of our commercial services. For further information about how RNS and the London Stock Exchange use the personal data you provide us, please see our Privacy Policy.
Latest directors dealings
- 3 hours ago Tertiary Minerals
- 3 hours ago Hargreaves Services
- 16 hours ago ASOS
- 16 hours ago MJ Gleeson
- 16 hours ago Rolls-Royce Holdings