Report supporting Australia's teen social media ban appears to contain AI hallucinations, Senate hears
A $3.48 million report underpinning Australia's under-16 social media ban contains multiple fabricated or erroneous academic citations, raising serious questions about the reliability of AI-assisted government research The UK-based Age Check Certification Scheme (ACCS) initially denied using AI in the report, later conceding ChatGPT was used for editing after Guardian analysis found metadata in links revealing ChatGPT as the source Guardian Australia identified at least six citation errors inclu
Analysis
TL;DR
- A $3.48 million report underpinning Australia's under-16 social media ban contains multiple fabricated or erroneous academic citations, raising serious questions about the reliability of AI-assisted government research
- The UK-based Age Check Certification Scheme (ACCS) initially denied using AI in the report, later conceding ChatGPT was used for editing after Guardian analysis found metadata in links revealing ChatGPT as the source
- Guardian Australia identified at least six citation errors including non-existent DOIs, incorrect author names, mismatched publication years, and papers that do not support the claims made about them
- One cited article was reportedly accessed in March 2025 but was not publicly available until June 2025, and its lead author confirmed a cited co-author named "Jamil" never existed on the paper
- The Australian government has previously penalized contractors for AI-generated errors, with Deloitte refunding part of a $440,000 contract after similar issues surfaced
Why It Matters
This incident highlights a critical vulnerability in AI-assisted policy research: hallucinated citations can lend false credibility to government decisions with real-world consequences, such as regulating an entire generation's access to social media. It also underscores the importance of transparent AI disclosure and rigorous human verification in reports that inform legislation, as unreliable sources can undermine public trust and policy effectiveness.
Technical Details
- The report in question is a 1,000-page document produced by the UK-based Age Check Certification Scheme (ACCS), with a 26-page citation section covering "emerging technologies" for age verification
- Citation errors include: DOIs linking to non-existent papers, DOIs pointing to incorrect articles, author/journal/year combinations that do not match any known references, and DOIs linking to papers that do not support the claims made
- Metadata embedded in at least four hyperlinks within parts E and K of the report explicitly identified ChatGPT as the source, which was the key evidence that forced ACCS to concede AI use
- ACCS's subsequent clarification response also contained errors, including mismatched years, authors, and journal titles when attempting to correct the original citations
- One lead author of a cited study confirmed the paper was not publicly accessible until June 2025, yet the report claimed it was accessed in March 2025, and the cited lead author "Jamil" was entirely fabricated
Industry Insight
- Organizations using AI for research or policy documentation must implement mandatory human verification of all citations and sources, as AI hallucination remains a persistent and well-documented risk that can compromise the integrity of official reports
- The incident sets a precedent for increased scrutiny of AI-assisted government contracts, suggesting that future procurement may require explicit AI disclosure clauses and independent audit trails for cited sources
- Policymakers and regulators should treat AI-generated content as inherently untrustworthy until independently verified, and consider establishing standardized verification protocols before relying on AI-augmented research for legislative decisions
Disclaimer: The above content is generated by AI and is for reference only.