Training Data Transparency
Right now, someone is asking ChatGPT for medical advice. The AI responds confidently, but neither the person asking nor OpenAI will tell you where that answer came from. Was it a peer-reviewed medical journal? A Reddit thread? A discredited blog post from 2015?
You’re eating at a restaurant that refuses to tell you what’s in the food.
The Hidden Kitchen: What AI Companies Won’t Tell You
Training data is the foundation of artificial intelligence. It’s what AI systems like ChatGPT use to provide answers to your prompts. It’s what generative image systems like Midjourney and DALL-E use to conjure AI-created art.
Here’s how it works: AI companies gather information from across the internet (Reddit posts, Wikipedia entries, personal social media pages, both free and paid media outlets) and feed it into their systems. We’re talking billions of pages each month for a given generative AI model. This training data becomes the raw material the AI uses to generate every response, every image, every piece of content.
But here’s the problem: The companies building these systems refuse to tell us what data they used.
OpenAI, the maker of ChatGPT (effectively the world’s most influential AI chatbot) has refused to disclose any information about the datasets used to train their model. When pressed, they call it proprietary information. A trade secret.
Many people believe OpenAI won’t disclose their training data because transparency would open them to legal troubles. Their datasets may or may not include copyrighted material and other legally protected intellectual property. They might have scraped content without permission, without licensing, without compensation to the creators.
The New York Times sued OpenAI in December 2023, alleging the company scraped millions of articles without permission. Authors and artists followed with their own lawsuits. These aren’t fringe complaints; they’re mainstream creators asking a simple question: “Did you steal from me to build this?”
The Hidden Kitchen: What AI Companies Won’t Tell You
AI companies are training their systems on billions of pages scraped from the internet; but they refuse to tell you what data they used or whether it was legally obtained.
Why “Trade Secret” Is a Lie
Training data isn’t a trade secret any more than Coca-Cola’s ingredients are; it’s the raw material, not the recipe, and companies hide it to avoid lawsuits, not to protect innovation.
What Transparency Actually Looks Like
A simple data declaration (the digital equivalent of a food label) would expose bad actors, reward ethical companies, and let you finally ask: “Where did this AI learn that?”
Two Immediate Harms
This secrecy creates two harmful effects that should concern everyone:
First, it hides the unauthorized and possibly illegal use of legally protected intellectual property. That’s a long-winded way of describing theft. If an AI company trained their system on stolen content, they can hide that fact behind the “trade secret” curtain.
Second, it gives us no insight into the quality of the data used to train the AI model; and therefore no information about the quality of the AI model itself. As the saying goes: “Garbage in, garbage out.” If an AI system learned from low-quality, biased, or factually incorrect datasets, it will generate low-quality, biased, or factually incorrect outputs. But you’ll never know, because the kitchen door is locked.
Without transparency in how AI systems source their information, the public becomes increasingly reliant on systems that have no accountability. Beyond the immediate risk of misinformation, the mere awareness that these systems might be unreliable erodes public trust in accurate information itself.
Why “Trade Secret” Is a Convenient Lie
As the push for AI training data transparency gains momentum, some of the world’s most powerful AI companies are fighting back. They argue that revealing even basic information about their training data would force them to divulge trade secrets.
Let’s be clear: Training data is not a trade secret. It’s a raw ingredient.
Think about Coca-Cola. The company famously guards its recipe as one of the world’s best-kept trade secrets. But walk into any store and pick up a can of Coke, and you’ll find a complete list of ingredients printed on the side. High fructose corn syrup. Caramel color. Phosphoric acid. Natural flavors.
Food safety laws strike an appropriate balance. Companies must list their ingredients truthfully, but they get to keep the art of mixing, heating, cooling, proportioning, and timing as proprietary secrets. The ingredients are public; the recipe is protected.
Digital transparency should work the same way.
Where the Real Innovation Lives
Here’s what many people don’t understand: The ingenuity in AI doesn’t lie in the datasets.
Many of today’s most advanced AI systems were trained on the same problematic datasets; Common Crawl, WebText2, Books1, Wikipedia. Everyone’s using the same flour, eggs, and water. So where does the actual innovation happen?
It’s in the construction of the model itself. Developing a competitive AI system requires months or years of work: envisioning the architecture, lining up massive compute power, creating novel algorithms, training the model, carefully weighting the data, fine-tuning the outputs, and building an intuitive user interface.
The datasets are the raw ingredients. The trade secrets lie in the preparation, the process, the heat, the time, and the chef’s skill.
If It’s Not a Trade Secret, Why Hide It?
So if training data isn’t actually a trade secret, why would companies balk at disclosing basic information about it?
The answer is uncomfortable but straightforward: Many of today’s generative AI systems were trained on raw, unlicensed data that may or may not have been legally obtained. Some of the largest AI developers are now facing lawsuits accusing them of mass copyright infringement.
And here’s the irony: Some of these same companies have recently announced high-profile licensing deals with major outlets like the Associated Press, NewsCorp, and the Financial Times. They’re paying for access to quality data now; which proves two things:
1. High-quality training data has significant value
2. That value doesn’t make it a trade secret
When companies announce and publicize their data licensing deals, they’re essentially admitting that training data sources are not confidential business secrets. They’re ingredients that can and should be disclosed.
Why This Matters for Competition
Without transparency requirements, bad actors can hide toxic training data behind the “trade secret” curtain. This undermines the good-faith efforts, innovation, and investments of ethical AI companies who want to compete on quality and legal compliance.
Imagine you’re an AI developer who paid for proper licensing, carefully curated your datasets, and invested in quality assurance. You’re competing against companies that scraped everything they could find for free, legal or not. Without transparency requirements, consumers have no way to reward your ethical approach. The market can’t distinguish between the chef who sources quality ingredients and the one who’s dumpster diving.
Transparency levels the playing field. It lets ethical innovation win.
What Real AI Accountability Looks Like
So what should AI companies be required to disclose? The answer is simpler than you might think.
Developers of AI systems should provide documentation for all training data used to build their models. Not the actual datasets themselves; just a clear description of what they used, where it came from, and under what terms.
This type of auditable information gives transparency and assurance to businesses deploying AI, consumers using it, and regulators overseeing it. It’s similar to the SOC 2 reports that became standard in the cybersecurity industry; third-party audits that assess the risks associated with software and technology services.
A Data Declaration shouldn’t necessarily require government enforcement to be effective. Ideally, it becomes a standard component of every AI model; expected and demanded by anyone deploying these systems as a transparent mark of quality and legal assurance.
The Data Declaration Template
Here’s what a comprehensive AI training data disclosure should include:
| Field Name | Possible Values |
|---|---|
| Data Set Name | Text |
| Data Set Owner | Text |
| Data Set Description | Text |
| Data Set Size | Numerical |
| Data Set Category | Web text, images, music, video, books |
| Data Set License Type | Commercial License, Proprietary, Public Domain, Fair Use claim |
| Data Set License Name | e.g. GPL, Apache, Creative Commons |
| Data Set Collection Period | Start date, End date (or Present) |
| Data Set Usage Period | Start date, End date (or Present) |
| Data Set contains personal or personally identifiable information | Yes or No |
| Personal Information Opt-in obtained | Yes or No |
| Personal Information License mechanism | EULA, Terms of Service, Privacy Policy, Click Through |
| Personal Information anonymized prior to training | Yes or No |
| Data Set contains Copyrighted Information | Yes or No |
| License governing Copyrighted Information | Fair use, Commercial license |
| Synthetic Training Data use | Yes or No |
This isn’t complex. It’s not burdensome. It’s the digital equivalent of an ingredients label; information that should be standard practice for any company putting a product in front of the public.
Two Possible Futures
Right now, we’re standing at a fork in the road.
Future A: AI companies continue concealing their training data. Bad actors flourish because there’s no way to distinguish them from ethical innovators. Public trust in AI erodes as people realize they’re relying on black boxes. Misinformation spreads. Lawsuits multiply. Innovation slows as uncertainty about legal liability chills investment.
Future B: Data declarations become industry standard. Ethical companies compete openly on data quality and legal compliance. Consumers make informed choices about which AI systems to trust. The marketplace rewards honesty and transparency. Third-party auditors verify claims. Bad actors face consequences for deception. Innovation accelerates because everyone knows the rules.
The difference between these futures is legislation.
Why Legislation Closes the Gap
Publishing a high-level data description is easy; but it must be truthful. That’s where legal requirements matter.
California’s Training Data Transparency Act (AB 2013), passed and signed in September 2024, requires developers of generative AI systems to publicly post a high-level summary of the datasets used to train their systems. Those summaries become legally required beginning January 1, 2026.
Here’s why that matters: A developer who lies on their AI training data description would be breaking the law. That gives consumers and competitors an avenue for enforcement. It keeps the data clean by penalizing cheaters.
As it stands today, there are no negative consequences for developers who use unlicensed, inaccurate, or biased data. Once one state creates legal accountability, the entire industry shifts. Because hiding your ingredients only works when everyone else is hiding theirs too.
By requiring the digital version of an ingredient label for AI systems, policymakers encourage innovation by ethical AI companies, expose bad actors, and create a marketplace where quality and legality actually matter.
What Happens Next
The next time an AI gives you an answer, you should be able to ask: “Where did you learn that?”
Soon, that question will have an answer. Not because AI companies suddenly developed a conscience, but because transparency requirements force them to show their work. Once that happens, you’ll be able to make informed decisions about which AI systems deserve your trust; and which ones are just hoping you won’t look in the kitchen.
The restaurant can’t refuse to post a menu forever. The question is whether we demand one now, or wait until someone gets sick.