Consent in Crisis: The Rapid Decline of the AI Data Commons

Shayne Longpre' et al.

Audio by Paper2Audio.

Abstract

General-purpose artificial intelligence (AI) systems are built on massive swathes of public web data, assembled into corpora such as C4, RefinedWeb, and Dolma. To our knowledge, we conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora. Our audit of 14, 000 web domains provides an expansive view of crawlable web data and how codified data use preferences are changing over time. We observe a proliferation of AIspecific clauses to limit use, acute differences in restrictions on AI developers, as well as general inconsistencies between websites' expressed intentions in their Terms of Service and their robots.txt. We diagnose these as symptoms of ineffective web protocols, not designed to cope with the widespread re-purposing of the internet for AI. Our longitudinal analyses show that in a single year (2023-2024) there has been a rapid crescendo of data restrictions from web sources, rendering ~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4, fully restricted from use. For Terms of Service crawling restrictions, a full 45% of C4 is now restricted. If respected or enforced, these restrictions are rapidly biasing the diversity, freshness, and scaling laws for general-purpose AI systems. We hope to illustrate the emerging crises in data consent, for both developers and creators. The foreclosure of much of the open web will impact not only commercial AI, but also non-commercial AI and academic research.

1 Introduction

The web has become the primary communal source of data, or "data commons", for general-purpose and multi-modal AI systems. The scale and heterogeneity of web-sourced training datasets provide the foundation for both open and closed AI systems, such as OLMo, GPT-4o, and Gemini. However, the use of web content for AI poses ethical and legal challenges to data consent, attribution, copyright, and the potential impact on creative industries. This has spurred new initiatives to better verify data quality and provenance, isolate public domain and permissively licensed data, and integrate new infrastructure to signal, detect, and even evade the use of data for AI training.

The focus of this work is to understand the evolving role of the internet as a primary ingredient to AI, and how AI has collided with the limited protocols that govern data use. Web data is traditionally collected using web crawlers —automatic bots that systematically explore the internet and record what they see. However, the mechanisms for indicating restrictions to web crawlers, such as the Robots Exclusion Protocol (REP), were not designed with AI in mind. The REP is referred to as robots.txt in practice. As such, we examine their (in)ability to communicate the nuances in how content creators wish their work to be used, if at all, for AI. And more broadly, we analyze how AI is already re-shaping the culture of web consent, and how this is shifting the landscape for AI training data. Our results foretell significant changes not only to AI data collection practices and data scaling laws, but also the structure of consent on the open web, which will impact more than AI developers.

To this end, we present a large-scale audit of the web sources underlying three open AI training corpora: C4, RefinedWeb, and Dolma. In contrast to prior audits that assess datasets curated snapshots of data—this work looks beneath the datasets at the web domains they were derived from, and traces the temporal evolution of these sources. We are, to our knowledge, the first to systematically measure detailed provenance, crawler consent mechanisms, and content monetization factors, all relevant to the responsible downstream use of this data. These analyses enable us to trace fundamental distribution shifts in how preference signals are expressed and the inadequacy of existing tools. Our work has several key findings: 1. A proliferation of restrictions on the AI data commons. We find a rapid proliferation of restrictions on web crawlers associated with AI development in both websites' robots.txt and Terms of Service. We estimate, in on year (2023-04 to 2024-04), ~25%+ of tokens from the most critical domains, and ~5%+ of tokens from the entire corpora of C4, RefinedWeb, and Dolma have since become restricted by robots.txt. Forecasting these trends forward shows a decline in unrestricted, open web data year-over-year. 2. Consent asymmetries & inconsistencies. OpenAI's crawlers are significantly more restricted than those of other AI developers. More broadly, preference signaling mechanisms like robots.txt see errors and omissions in their coverage across AI developers, as well as contradictions with their terms of services—indicating inefficiencies in the tools used to communicate data intentions. 3. A divergence in content characteristics between the head and tail of public web-crawled training corpora. We find the largest web-based sources of public training corpora have significantly higher rates of user content, multi-modal content, and monetized content, but only slightly less sensitive/explicit content. Top web domains comprise news, encyclopedias, and social media sites, as compared to the many organization websites, blogs, and ecommerce websites in the long tail of web sources. 4. A mismatch between web data and common uses of conversational AI. We contrast data web sources with the real-world usage of conversational AI—showing how substantial portions of web-derived training data may be misaligned with the tasks that AI models are actually used for. These results may have implications for model alignment, future data collection practices, and copyright.

2 Methodology

AI models that are highly performant on tasks in language, images, video, and even audio increasingly depend on massive web-sourced training datasets. These datasets are collected using web crawlers—agents that navigate the web, accessing and retrieving web pages without human intervention. While these robots are essential for a variety of applications, including search engines, studying the internet ( ie archiving), and link verification tools; recently they have also become the backbone of AI training data collection.

In our study, we focus on three popular, open-source, and permissively licensed data sources which are derived from Common Crawl, the largest publicly available crawl of the web, which has collected and stored hundreds of billions of web pages since 2008. For each web-based data source, we sample the web domains from which it was created, and extensively human annotate their properties. Our analysis examines a snapshot of the present, as well as longitudinal changes across time, to understand how ecosystem norms have evolved.

Data sources The data sources used for our study are C4, RefinedWeb, and Dolma. These data sources each have 100k-1M+ downloads, are the primary component in most modern foundation models, as well as being widely used to derive other popular datasets. Common Crawl is released on a monthly basis, and, as see in Table 1, each data source is based on a different set of monthly snapshots. Each of these corpora apply various automatic filtering techniques, including removing duplicative pages, low-quality content, and personal identifiable information such as addresses.

Table 1 summary: The table presents statistics on audited data sources, including crawl dates and the number of web domains for each data source. Dolma has the most web domains, followed by RefinedWeb, and then C4. The intersection of the three has fewer web domains than the individual sources.

Head sample and random sample For each data source, we identified and selected the top 2k web domains ranked by their number of tokens. We refer to the resulting 3.95k union of these web domains as HEADAll. This sample represents the largest, most actively maintained, and critical domains for AI training. For certain analyses, we consider only the head of C4, which we will refer to as HEADC4.

We are also interested in how consent preferences have evolved within a wider sample of internet domains. To capture this, we randomly sampled 10K domains (RANDOM10k) from the intersection of the three corpora, totalling 10,136,147 domains. From the 10k sample, we selected a random subset of 2K for human annotation (RANDOM2k). RANDOM10k was sampled from the intersection of domains listed across all three datasets, which means this subset may skew towards more widely-used or high-quality domains.

Human annotations We trained annotators to manually label the websites for their content modalities (e.g. video, text); website purpose(s) (e.g. news, e-commerce); presence of paywalls and embedded advertisements; the text of the terms of service, if any; and other metadata detailed in Table 2. Annotators received individual instructions, frequent quality calibration, and were compensated well above industry standards at $25-$30 per hour. We collected annotations for the entirety of HEADAll as well as from the random sample RANDOM2k. More details on our annotation process are available in Appendix A, and all annotations will be made publicly available for reproducibility and future research. A goals of our audit is to measure website administrators' intentions for how their site can be crawled and its content used—including for training AI models. We used the Wayback Machine, a digital archive of 835 billion web pages, to collect historical versions of each website's homepage, its Robots Exclusion Protocol (REP), commonly referred to as robots.txt file, and its terms of service page. This was collected at monthly intervals, from January 2016 to April 2024.

Table 2 summary: The table presents a list of attributes collected for each web domain, sampled from multiple sources. The attributes are content modalities, user content, sensitive content, paywall, advertisements, purpose and service, robots.txt, terms and policies, crawling and AI policy, content use policy and non-compete policy. The collection method for each attribute is also specified, either automatic or human annotation.

The REP, first introduced in 1995 and codified in 2022, has become the default mechanism for website owners to indicate to web crawlers what parts of their website, if any, they consent to have crawled. While it is not legally enforceable, it is respected by all major search engines, as it prevents website servers from getting overloaded by crawlers, it allows websites to signal pages that are undesirable to crawl (for example, calendar sites that could lead to infinite loops), and by respecting it, crawlers disincentivize adversarial tactics designed to impede crawlers. Website creators are able to set one set of instructions for all web crawlers or a different instructions for each web crawler. For instance, Google Search respects instructions which specifies the user agent "Googlebot" while Common Crawl listens to the user agent "CCBot." In our audit, we record the robots.txt instructions for a range of crawlers, but focus our analysis on five AI developers, Google, OpenAI, Anthropic, Cohere, and Meta, as well as non-profit web archival organizations such as Common Crawl and the Internet Archive, which have seen their data taken for AI training. Collectively, we refer to these as "AI Organizations". We classify robots.txt for each crawler in ascending order of restrictions, from no robots.txt present, to sitemaps which support crawlers without limitations, to basic restrictions on a subset of directories, to full restrictions on any crawling of the website. For each corpus, we measure the percentage of "restricted tokens" as the portion of tokens from web domains that fully restrict one or more of the AI Organizations's crawlers. For Terms of Service analysis, we define restricted tokens to simply mean the portion of unusable tokens due to terms that preclude crawling or AI. See Appendix B.2 for the full list of agents and Appendix B.1 for the robots.txt restriction classification taxonomy.

In addition to robots.txt, we recorded the Terms of Service (ToS) and other content and copyright policies for each website. These documents support more nuanced preferences than the REP, and allow for blanket bans on downstream use cases rather than just specification of what data agents are allowed to collect. We used an automatic annotation pipeline (see Appendix 2 for details) to categorize ToS agreements according to stance towards use of web crawlers and AI training, content use restrictions, and non-compete clauses, in ascending degrees of restrictiveness.

3 Findings

3.1 The Rise of Restrictions on Open Web Data

To understand the web sources underlying foundation models, we analyze the longitudinal changes in robots.txt and Terms of Service restrictions between January 2016 and April 2024. In Figure 1 the plots depict the percent of tokens present in each category of restriction over time, for the AI Organizations in HEADC4—the largest, most actively maintained, and critical domains for AI training. The fine-grained longitudinal analysis of robots and Terms of Service trends allows us to estimate this time series into the future. We apply Seasonal Autoregressive Integrated Moving Average (SARIMA) models to generate forecasts of future trends for both the head sample and random subset, the details of which can be found in Appendix C along with the coefficients and tests.

Figure 1 summary: The figure includes multiple line charts presenting a temporal analysis of web consent signals in a sample of critical web domains. The analysis spans from 2016 to April 2024, with forecasts extending a year into the future. The first chart focuses on ascending categories of robots.txt restrictions for major AI organizations. The second chart illustrates ascending categories of Terms of Service restrictions. The third chart provides a breakdown of robots.txt restrictions by organization. The proportion of web domains lacking robots.txt and Terms of Service has decreased significantly. Robots.txt crawling restrictions have increased rapidly since mid-2023. Restrictions have increased substantially for both the full corpus and critical domains.

In Figure 2 we measure the restricted tokens, or how many tokens fall into the most restrictive settings for each of robots.txt and Terms of Service, as a portion of the Full Corpus, or HEADAll. The intermittent lack of smoothness for Figures 2c and 2d is mainly due to temporal gaps in the Wayback Machine; however the main trends remain visible. In all analyses we exclude web domains which could not be retrieved from the Wayback Machine, and all proportions are based on the set of web domains which existed in that time period. These analyses show a clear and systematic rise in restrictions to crawl and train on data, from across the webs. We make no assertion regarding whether the prior omission of a robots.txt or restrictions implies consent to use data. To the degree these restrictions are respected, it also foretells a decline in open data, which may impact more than commercial AI developers, or even AI organizations in general. We break down and discuss the findings of this temporal analysis below.

Figure 2 summary: The figure consists of temporal line charts that show the percentage of restricted tokens across different corpora and domain types. The top row shows the percentage of robots.txt restricted tokens, while the bottom row shows the percentage of Terms of Service restricted tokens. The left column shows the data broken down by corpora, including C4, RefinedWeb, and Dolma, distinguishing between the full corpus and a head sample. The right column shows the data broken down by domain type, including academic, e-commerce, encyclopedia, government, news, organization, and social/forum websites. The percentage of restricted tokens has increased considerably since 2023. News websites exhibit a notably high percentage of restricted tokens compared to other domain types. Terms of Service restrictions are more prevalent than robots.txt restrictions across all corpora and domain types.

Web domains are adopting robots.txt and Terms of Service pages to signal preferences. Figure 1( Top & Middle) shows from 2016, the portion of web domains in HEADC4 without a robots.txt and Terms of Service has gone from 20% and 80% respectively, to near zero. 2 This reflects an emerging adoption of these practices to signal and protect data intentions.

Robots.txt crawling restrictions have risen precipitously since mid-2023. Figure 1( Top) shows the rapid re-distribution of robots.txt restrictions, directly after the introduction of GPTBot and Google-Extended crawler agents. This re-distribution to full restrictions mainly comes from websites with previously moderate restrictions, such as disallowed directories, pattern-based or search page restrictions, and partly from websites with no prior restrictions in their robots.txt.

Across the entire corpora, approximately 1 percent of C4, RefinedWeb, and Dolma tokens were restricted in mid 2023, as compared to 5 to 7 percent of tokens in April 2024. Among the most critical domains (HEADAll), 20 to 33 percent of all tokens are restricted, as compared to less than 3 percent one year prior (Figure 2a). From a relative perspective, from 2023 to 4 to 2024 to 4 these restrictions have risen 500 percent plus for both C4 and RefinedWeb's full corpus, and 1000%+ for both C4 and RefinedWeb's head distributions. Note that these measurements only capture full restricted domains, and the numbers are higher for partially restricted domains.

AI developers are restricted at widely varying degrees. Figure 1( Lower) breaks down the restrictions by AI developers and non-profit organizations. OpenAI crawlers are restricted for 25.9% of tokens in HEADC4, followed by Anthropic and Common Crawl (13.3%), Google's AI crawler (9.8%), and more distantly Cohere (4.9%), Meta (4.1%), the Internet Archive (3.2%), and lastly Google Search's crawler (1.0%). These asymmetries in restrictions have significant differences, and tend to advantage less widely known AI developers. In Subsection 3.2 we discuss these asymmetries and their consequences in more depth.

Terms of Service pages have imposed more anti-crawling and now anti-AI restrictions. Figure 1( Middle) illustrates this gradual re-composition of Terms pages—with web domains shifting from no terms pages, to those with restrictions on crawling, commercial-use, using the data for competing services, or re-distribution. Only in 2024 do we see the wider emergence of Terms which specifically mention and restrict the use of their data for generative AI. In the last year, we've seen a 26-53% relative increase in Terms of Service crawling restrictions across C4, RefinedWeb, and Dolma. Figure 2c shows 45-55% of all tokens in these three corpora have a form of data use restriction in their Terms pages. In practice, most automatic crawlers do not heed these Terms, though they may provide some avenue of legal enforcement.

AI restrictions are driven primarily by news, forums, and social media websites. For robots.txt, Figure 2b shows nearly 45% of all News website tokens are fully restricted in HEADAll, as compared to 3% in 2023. For Terms of Service, Figure 2d shows News website tokens have had a 6% rise in the restricted portion since 2023. Paired with the findings in Table 2, this suggests that the composition of tokens in crawls respecting robots.txt may shift away from news, social media, and forums, and towards organization and e-commerce websites.

Forecasting trends in the future suggest a continued and significant decline in open and consenting web data sources. SARIMA forecasts suggest that for just the next year (by April 2025) an additional absolute 2-4% of C4, RefinedWeb, and Dolma tokens will be fully restricted by robots.txt.

Equivalently, an additional 7-11% of the highest quality tokens in the head distribution will become restricted. The forecasts for Terms of Service are even starker, with the restricted tokens in the full corpus expected to rise an absolute 6-10% by April 2025. These trends illustrate a systematic rise in restrictions on data sources, which, where enforced or respected, will severely hamper the data scaling practices in the coming years—which have thus forth been responsible for the remarkable capability improvements.

3.2 Inconsistent and Ineffective Communication on AI Consent

In many cases, data holders fail to effectively communicate their preferences on how their data is used by AI systems. We observe robots.txt instructions which allow some AI organizations to crawl while restricting others, references to non-existent crawlers, and contradictions between the robots.txt and Terms of Service. Together, these issues point to the need for better preference signaling protocols.

Some AI crawlers are allowed, while others are not. We find not all AI agents are disallowed equally. In Table 3 we estimate the conditional probabilities of each organization's crawler being restricted, conditioned on whether any other AI organization is restricted. Whereas OpenAI and Common Crawl agents are frequently disallowed (in 91.5% and 83.4% of cases where any of the organizations are disallowed), the agents of other AI companies, such as Google, Cohere, and Meta are often omitted from robots.txt. The omissions of Cohere, Meta, and other small AI organizations are likely because website administrators are unaware or unable to update their robots.txt to reflect the full list of AI developers. On the other hand, the particularly high omission rates of Internet Archive and Google Search suggest web administrators may be open to more traditional crawler uses like archiving and search engines, even as they seek to restrict AI usage. A full confusion matrix showing the correlation between restrictions for each user agent is provided in Appendix Figure 5.

Table 3 summary: The table shows the percentage that each organization's crawler agents are restricted if at least one other organization in the pool is restricted. OpenAI and Common Crawl are restricted at a higher rate compared to other organizations. Internet Archive and Google Search have lower restriction percentages.

Figure 5 summary: The table shows the percentage that organization B is restricted by a web domain's robots.txt, given organization A's agents have been restricted. OpenAI web agents are nearly always disallowed if any AI organizations are disallowed, but the reciprocal is less frequent. Search agents are the least restricted.

Unrecognized crawler agents cause incorrect specifications. We find several instances where robots.txt refer to user agents that the companies do not recognize. For instance, 4.5% of websites disallowed the unrecognized user agents ANTHROPIC-AI or CLAUDE-WEB (documented as FALSE ANTHROPIC), but not the documented agent for Anthropic's crawler, CLAUDEBOT. The origin and reason for these unrecognized agents remains unclear—Anthropic reports no ownership of these. These inconsistencies and omissions across AI agents suggest that a significant burden is placed on the domain creator to understand evolving agent specifications across developers. AI crawler standardization could address these challenges in consent/preference signaling.

Contradictions exist between robots.txt and ToS. The Robots Exclusion Protocol (REP) is a guideline for web crawlers, while a website's Terms of Service is a legal agreement between the

Consent in Crisis: The Rapid Decline of the AI Data Commons

website and users of the site. The benefit of the REP is its machine-readability. However, its rigid structure, created in 1995, limits what signals it can convey. In contrast, a ToS can communicate rich and nuanced policies in natural language. Without a robots.txt, a ToS lacks practical deterrence of unwanted crawling. Inversely, without a ToS, a robots.txt may lack any plausible enforcement. We found that in many cases, websites' robots.txt implementations fails to capture the intentions specified in their Terms of Service.

In Figure 3, we illustrate the distribution of Terms and REP use criteria (the taxonomy is defined in Table 2 and broken down in detail in Appendix B) . Common use criteria expressed in modern ToS pages include prohibitions specifically on commercial use, conditional use limiting actions such as third-party re-posting, non-compete criteria, or specific prohibitions only against "AI", but not against crawling for search engines. We also see many websites write anti-crawling terms but have no robots.txt file (35.1%), or have no ToS but a restrictive robots.txt (20.3%) that disallows at least some crawlers. Terms specifying only non-commercial uses are also often paired with fully or partially restrictive robots.txt files, which may unintentionally limit academic web crawlers, as a side effect of deterring corporate use. Another formidable challenge is that websites currently have to list every search engine or AI user agent they want to restrict. Empirical evidence from both Figure 5 and Figure 3 suggests the absence of REP expressivity and standardization for AI is leading to inconsistent or unintended signals that fail to reflect intended preferences.

Figure 3 summary: This figure shows a cross tabulation of Terms of Service policies and robots.txt restrictions. The values are measured in percentage of tokens for HEADC4. The figure indicates the relationship between robots.txt restrictions and Terms of Service policies. There is disagreement between the two methods of expressing restrictions on data use for AI. The "No Crawling" category in Terms of Service policies has a relatively large percentage of tokens with "None" in robots.txt restrictions, compared to other Terms of Service policies.

3.3 Correlating Features of Web Data

What does web data actually look like? Prior work has measured the characteristics of web-derived datasets, for the presence of artifacts, undesirable text and images, demographic biases, and quality discrepancies across languages. We expand upon these analyses by measuring what web data sources look like before they have been neatly processed into AI training datasets. We measure the presence of multi-modal content, user-derived content, website monetization schemes, and sensitive content on the most well-represented web domains on the internet (HEADAll) and on a random sample of domains (RANDOM2k). We also annotate the services provided and purpose of each web domain.

Most of the web is comprised of organizational/personal websites, and blogs, however the head distribution is disproportionately news, forums, and encyclopedias. Table 4 shows several notable and statistically significant differences between head distribution (HEADAll) and tail distribution (RANDOM2k) of web domains. HEADAll is comprised mostly of news, and social media/forums, and encyclopedias (72.9%), in contrast to the long tail data in RANDOM2k, which is dominated by personal or organization websites, blogs and E-commerce sites (97%). Academic and government content is also proportionately higher in the head distribution. Note however that though they are all derived from Common Crawl snapshots, C4, RefinedWeb, and Dolma, all show variations in their source compositions—highlighting the importance of curation choices. The head distribution of domains is more multimodal, and heavily monetized. We observe that HEADAll web domains are much more heavily monetized through ads (+47.5%) and paywalls (+24.1%). Accordingly, they also have significantly greater restrictions from both robots.txt (+22.5%) and Terms of Service (+35.3%). This monetization and restrictions likely correspond to the higher quality and heterogeneity of content usually produced by news, periodicals, forums, and databases which are more common in HEADAll. This is reflected by the higher proportions of image (+4.4%), video (+39.8%), and audio content (+38.4%) than the rest of the web. Interestingly, the fraction of user-generated content and sensitive content between the head and tail distributions is less pronounced. Crawlers that respect the restrictions that occur far more frequently in HEADAll will increasingly lose access to the most multi-modal, highly curated, and up-to-date content sources.

Table 4 summary: The table shows the mean incidence rates of web source features across different corpora. Restrictive robots.txt, restrictive terms, paywall, ads, modality video and audio have a relatively large positive difference between the top URL groups and the random sample, while org/personal websites have a large negative difference. Sensitive content has a small difference. The modality image has a very high percentage of tokens in the corpus. Blogs and e-commerce have a negative difference.

3.4 Misalignment between Real-world AI Usage and Web Data

In this section, we measure the degree of alignment between real world uses of ChatGPT and the content in the webcrawls that form the bulk of AI training. For each web domain in HEADAll, we had annotators label the services provided by the website, as well as the presence of some monetization, such as a paywall or automatic ads. We compare these services against the services that real-world users solicit in their interactions with conversational AI systems. We use WildChat, a recent set of 1 million user conversations with ChatGPT, collected through a HuggingFace Space wrapper around OpenAI services. We randomly sampled 100 conversation logs from WildChat, which the paper authors manually clustered by the type of tasks or goals conveyed by each conversation, with the goal of relating the core function of these conversations with the services provided by the websites crawled in training. Subsequently, we used GPT-4o to label 1k randomly selected conversations from the WildChat dataset; these conversations were labelled using the taxonomy we developed to categorize websites. Further details on the taxonomy and labelling procedure can be found in Appendix B.6.

Apparent uses of ChatGPT are misaligned with the popular web domains language models are trained on. Figure 4( a) shows the distribution of services provided by the web domains, broken down by whether those domains are monetized. In contrast, Figure 4( b) shows how ChatGPT is used in the real world. The way that users interact with ChatGPT is different in important ways from the types of content that is most frequently represented in publicly available web-based training datasets. For instance, in over 30% of conversations, users request creative compositions such as fictional story writing or continuation, role-playing, or poetry. However, creative writing is poorly represented among the web data used for model training. These results may provide evidence for where models trained exclusively on unstructured internet data are most "unaligned" with how real users want to use generative AI. Language models trained only on web data are known to struggle to understand the structure of discourse and underperform models trained with instruction finetuning and preference training on highly curated data. The misalignment between real use cases and web crawled data may suggest the key areas of model distributional misalignment, as well as inform future data collection efforts based on real-world uses.

Figure 4 summary: The figure includes two horizontal bar charts comparing web domain services and real ChatGPT uses. Web domains are categorized by service type and monetization method. Real ChatGPT uses are categorized by query type. News websites are a large portion of web domain tokens, but a small portion of ChatGPT queries. Creative composition and sexual content are frequent ChatGPT uses, but are not well-represented in web domain content.

Sexual role-play appears to be a prevalent use of ChatGPT, despite being mostly removed from common public datasets. Whereas sensitive (e.g. sexual) content represents < 1% of the web domains in HEADC4 (see Table 4) , sexual role-play represents 12% of all recorded user interactions in WildChat. All the public datasets we consider—C4, RefinedWeb, and Dolma—have undergone some form of filtering to remove illegal or sexually explicit content, as training on such content introduces potential liability concerns; the web, in general, is known to have high portions of sexually explicit content. OpenAI states in the GPT-4 technical report that it also filtered its training data for harmful content. In addition to filtering web-derived training data, OpenAI's models are further trained to refuse requests that violate OpenAI's Usage Policies. 4. OpenAI's Usage Policies prohibit "sexually explicit or suggestive content" with respect to minors, or re-distribution that may harm others; however, there is ambiguity as to whether this would cover all user requests for sexual roleplay. For instance, the GPT-4 technical report makes a distinction in model refusal instructions between erotic and non-erotic sexual content, "(e.g. literary or artistic value) and contextualized sexual content (e.g. medical)".

Sexual-related uses of AI are a topic of ongoing debate within the scientific community, and rules differ by company, service, and jurisdiction. In a review of 30 generative AI developers' acceptable use policies, Klyman Citation 52 finds that OpenAI's policies are not among the most restrictive with respect to sexual content; while OpenAI has a blanket ban on "sexually explicit or suggestive content," other companies' acceptable use policies also explicitly prohibit "erotic content," "adult content," "pornography," "nudity," and "sexual fetishes". However, harsher restrictions on sexual content come with tradeoffs, as more heavily safety-tuned language models may then be less able to direct users to resources about sex education or generate fictional stories with PG-13 type content.

Common ChatGPT uses appear distinct from the uses of commercialized web sources. Figure 4 shows that a significant portion of tokens in HEADC4 are from web domains with ads, paywalls, or both—in other words they are the most commercialized. However, while news websites (the mostly highly commercialized category) comprise nearly 40% of all tokens in HEADC4, fewer than 1% of ChatGPT queries appear to be related to news or current affairs. It also shows that news websites have the highest instance of ads, paywalls, or both—in other words, they are the most commercialized. Our observations suggest that real-world use cases of ChatGPT are not necessarily directly related to the most prevalent, commercialized content on the web. This finding has interesting implications for the use of AI in industries with web-based services, such as journalism, or for US copyright analysis, which evaluates how the secondary use of a protected work (training AI models) affects the potential market for the original use of the work. We believe our observations provide strong empirical evidence for the (mis)alignment between AI uses and web-derived training data. However, our observations come with significant caveats. The WildChat dataset may not include a representative sample of how people interact with language models. Not only does it solely include conversations with a specific instance ChatGPT, but the WildChat proxy service is hosted on a technical website, HuggingFace Spaces, which could suggest a more technical user base, or one more likely to audit ChatGPT for inappropriate uses. Model uses also change both by time and product; our analysis is specific to the model interactions collected in WildChat between April 9, 2023, at 12 AM to May 1, 2024, using the GPT3.5-Turbo and GPT-4 APIs. Different AI products are likely to have different use distributions, and usage patterns will inevitably change over time. Finally, the use taxonomy, both for web domains and WildChat uses, were developed based on a manual, iterative process that is limited in its granularity. It is possible that data/information from News web domains could be used in responses for non-News classifications in WildChat, e.g. General Information. This would be exceedingly difficult to measure, and merits analysis in future work.

4 Discussion

The web-sourced AI data commons is rapidly becoming more restricted. The web has acted as the primary "data commons" for general-purpose AI. It's scale and heterogeneity have become fundamental to advances in capabilities. However, our results show web domains are rapidly restricting crawling and use of their content for AI. In less than a year, ~5% of the tokens in C4 and other major corpora have recently become restricted by robots.txt. And nearly 45% of these tokens now carry some form of restrictions from the domain's Terms of Service. If these rising restrictions are respected by model developers (as many claim to) or is legally enforced, the availability of high-quality pretraining sources will rapidly diminish.

The rise in restrictions will skew data representativity, freshness, and scaling laws. Prior work has forefronted scaling data as essential to frontier model capabilities. While the declining trend in consent will protect content creators' intentions, it would also challenge these data scaling laws. Not only would these restrictions reduce the scale of available data, but also the composition (away from news and forums), diversity, and representativeness of training data—biasing this data toward older content and less fresh content.

Recently, multiple AI developers have been accused of bypassing robots.txt opt-outs to scrape publisher websites. While it is not possible to confirm, in each case it appears AI systems may be distinguishing between crawling data for training, and crawling data to retrieve information for user questions at inference time. One of the few, OpenAI has two crawler agents, GPTBot for training, and ChatGPT-User for live browsing plugins (see Table 5). Other companies may simply not be registering their inference time crawlers for opt-outs. This circumvention may allow developers to directly attribute the retrieved web pages, as well as better achieve data representativity, freshness, and approximate the scaling laws had they trained on it. However, creators may feel this violates the spirit of the opt-outs, especially if the opportunity to attribute sources is not taken.

Table 5 summary: The table presents a list of organizations and their web crawler user-agents, including details and documentation links. The purpose of crawlers includes training AI models, retrieving information for AI systems, web search, and creating archives.

The web needs better protocols to express intentions and consent. The REP places an immense burden on website owners to correctly anticipate all agents who may crawl their domain for undesired downstream use cases. We consistently find this leads to protocol implementations that don't reflect intended preferences. An alternative scheme might give website owners control over how their webpages are used rather than who can use them. This would involve standardizing a taxonomy that better represents downstream use cases, e.g. allowing domain owners to specify that web crawling only be used for search engines, or only for non-commercial AI, or only for AI that attributes outputs to their source data. New commands could also set extended restriction periods given dynamic sites may want to block crawlers for extended periods of time, e.g. for journalists to protect their data freshness. Ultimately, a new protocol should lead to website owners having greater capacity to self-sort consensual from non-consensual uses, implementing machine-readable instructions that approximate the natural language instructions in their Terms of Service.

Rising expressions of non-consent will affect non-profits, archives, and academic researchers. A new wave of robots.txt and Terms of Service pages have not, or cannot, distinguish the various uses of their data. For instance, having to individually prohibit a plethora of AI crawlers has motivated many domains to simply blanket prohibit any crawling with the wildcard "*" marker. Or domains have also limited crawlers from non-profit archives such as the Common Crawl Foundation or Internet Archive, in order to prevent other organizations from downloaded their data for training. However, these archives are also used for non-commercial uses of AI, as well as academic research, knowledge, and accountability, well beyond the scope of AI. For instance, the Common Crawl is reported to be cited in 10,000+ research articles from varying fields. This tension between data creators and, predominantly, commercial AI developers has left academic and non-commercial interests as secondary victims. As web consent continues to evolve, we believe it is essential that these often essential facilities not be marginalized or severely hampered.

Economic fears of AI systems may change how internet data is created and protected. The content on the internet was not created to be used for training AI models. Its use for this purpose is already resulting in changing incentives around content creation, especially in cases where generative AI competes with the original sources of content. As we show in Figure 4, large portions of today's internet are owned by commercial interests, with sites that are locked behind paywalls or financed by advertisements. We expect small-scale content providers, who are less resourced to protect themselves from undesired crawling, may opt out of the web entirely, or move to posting on walled, content websites. If we don't develop better mechanisms to give website owners control over how their data is used, we should expect to see further decreases in the open web, with more websites locking their data behind login or paywalls to prevent it being trained on.

Real-world AI uses may have implications for copyright and fair use. Analysis of copyright infringement, including fair use, includes a four factor analysis. One analysis evaluates how the use of a protected work (e.g., to train AI models) affects the potential market for the original work (see 17 U.S.C §107). To investigate this question broadly, we document the major use settings of primary training domains, and compare those to the real use cases found in WildChat (Section 3.4). We find that while News domains dominate as a source of data, ChatGPT is not currently used often for news—instead uses like creative compositions (such as role-play or fiction writing), sexual role-play, brainstorming, or general information requests are most common. While there exist several limitations to this analysis, outlined in Subsection 3.4, the mismatch in use cases between training data and popular chatbots might suggest that AI chatbots are not directly competing with many of their training sources. We caution against over-interpreting these results to suggest a stronger case for fair use, as we believe future work is necessary to substantiate these findings and their relation to nuanced legal discussions.

5 Related Work

Prior work has conducted large scale audits of the provenance, quality, biases, and characteristics of AI training data, for pretraining text, finetuning text, as well as multimodal datasets, and challenges in data development. Recent work have looked at collecting non-copyrighted data, interpreting the legal implications of fair use for AI data, and forecasting future data constraints. However, there is little work inspecting the evolution of consent signals on AI data. Prior research have attempted to understand the link decay on the web, the collection process for Common Crawl, or evolving behavior and implications of web crawlers. Initial news reports have begun to investigate the rate of blocking AI web crawlers for general websites and news publishers, laying the foundation for our more rigorous analysis. The dearth of data documentation on AI datasets has been highlighted as a challenge for understanding AI model behavior, reproducibility, consent, and authenticity.

6 Conclusion

In this work, we presented the first, large-scale audit of the web sources underlying the massive training corpora for modern, general-purpose AI. Our audit of 14, 000 web domains provides a view the changing nature of crawlable content, consent norms, and points to daunting trends for the future openness of the highest quality data used to train AI. The inconsistencies and omissions between robots.txt and terms of service pages suggest a data ecosystem ill-equipped to signal or enforce preferences. Lastly, we uncover distributional mismatches in the documented real uses of AI systems and their underlying data. We release all our collected annotations and analysis, with the hope that future work will further investigate the provenance, consent, and composition of the fundamental ingredients to AI systems.

Impact & Ethics Statement

Consent to copy, use and train on data is a complex issue. First, the robots.txt and Terms of Service that communicate these intentions are owned by the web administrators, which are often imperfect proxies for the actual copyright holders. For instance, social media websites or forums often host content that was originally created or belongs to others. This is pervasive across the web. And there are insufficient tools to attribute all content to their copyright holders, or disentangle consenting from non-consenting use content—indeed that is partly demonstrated by this work. As such, it is important to recognize that robots.txt and Terms of Service have become the status quo out of practicality, though they suffer from limitations in ownership, and effective communication of intentions.

Additionally, while many data preference signals exist, which ones should be enforceable and how they should be enforced both remain open questions, legally and ethically. Data crawling restrictions can be motivated by intentions to protect copyright holders, privacy, or a desire to monetize the data themselves. Some of these motivations may not override the competing right for humans to collect public web material, for study, or non-commercial purposes. And, some have argued that humans, and by extension machines, have the "right to read and learn" from open web data. The laws, ethics, and best practices that emerge around these conflicting goals will impact the future efficacy of AI technologies, the types of organizations that are able to acquire sufficient data to compete in frontier model development, as well as the economy of creators from which these datasets are sourced. In this work, we do not prescribe legal or ethical answers, but describe the precise and evolving nature of preference signals on the web. While we advocate for more protocols and mechanisms that enable more effective communication of these intentions, we leave the adherence to these intentions as a broader question for readers, developers, and legislators.

Contributions

Here we break down contributions to this work. Contributors are listed alphabetically, except for team leads who are placed first.

Annotation Process Design for Web Domain Services Shayne Longpre (lead), Robert Mahari (lead), Hanlin Li (lead),Ahmad Mustafa Anis, Deividas Mataciunas, Diganta Misra, Emad Alghamdi, Enrico Shippole, Hamidah Oderinwale, Jianguo Zhang, Joanna Materzynska, Kevin Klyman, Kun Qian, Kush Tiwary, Lester Miranda, Manan Dey, Manuel Cherep, Minnie Liang, Mohammed Hamdy, Nayan Saxena, Nikhil Singh, Niklas Muennighoff, Naana Obeng-Marnu, Robert Mahari, Seonghyeon Ye, Seungone Kim, Shayne Longpre, Shrestha Mohanty, Tobin South, Vipul Gupta, Vivek Sharma, Vu Minh Chien, William Brannon, Xuhui Zhou, Yizhi Li, An Dinh, Ariel Lee, Campbell Lund, Caroline Chitongo, Christopher Klamm, Cole Hunter, Da Yin, Damien Sileo, Hailey Schoelkopf Annotation Process Design for Web Domain Characteristics Robert Mahari (lead), Shayne Longpre (lead)

Annotation Process Design for Terms of Service Robert Mahari (lead); Hamidah Oderinwale (lead), Campbell Lund (lead), Shayne Longpre Annotations & Annotation Quality Review Robert Mahari (lead), Shayne Longpre (lead), Jad Kabbara (lead), Ahmad Mustafa Anis, William Brannon, Caroline Chitongo, Vu Minh Chien, Manan Dey, An Dinh, Da Yin, Vipul Gupta, Mohammed Hamdy, Cole Hunter, Daphne Ippolito, Jad Kabbara, Christopher Klamm, Kevin Klyman, Ariel Lee, Minnie Liang, Hanlin Li, Lester Miranda, Shrestha Mohanty, Niklas Muennighoff, Seungone Kim, Damien Sileo, Hailey Schoelkopf, Enrico Shippole, Tobin South, Nayan Saxena, Xuhui Zhou Data Corpus Collection Tobin South (lead)

Wayback Machine Data Collection Ariel Lee (lead)

Robots.txt Longitudinal Analysis Ariel Lee (lead), Shayne Longpre (lead), Nikhil Singh (lead), Nayan Saxena, Tobin South, Terms of Service Longitudinal Analysis Ariel Lee (lead), Shayne Longpre (lead)

Trend Forecasting Ariel Lee (lead)

Robots.txt and ToS Comparisons Shayne Longpre (lead), William Brannon (lead), Campbell Lund, Ariel Lee Web Domain Characteristics Analysis William Brannon (lead), Shayne Longpre (lead)

Annotation Process Design for WildChat Shayne Longpre (lead), Nayan Saxena (lead)

WildChat vs Web Domain Analysis Shayne Longpre (lead), Manuel Cherep (lead), Campbell Lund, Ariel Lee, Nayan Saxena Writing Shayne Longpre (lead), Jad Kabbara (lead), Robert Mahari (lead), Daphne Ippolito (lead), Sara Hooker (lead)

Legal Analysis Robert Mahari (lead), Luis Villa Visualizations & Visual Data Analysis Naana Obeng-Marnu (lead), Nikhil Singh (lead), Shayne Longpre (lead), William Brannon (lead)

You have reached the end of the main paper. Additional summarized content follows

Table 6 summary: The table provides a breakdown of the top agents mentioned across robots.txt files. It shows the number of times each agent is observed and the percentage of times it is fully, partially, or not at all disallowed. For most agents, a relatively large proportion of observations have some disallowed content.

Table 7 summary: Precision/Recall values are shown for competing services, license type, and crawling and AI policy. Precision/Recall is highest for competing services and lowest for license type.

Table 8 summary: The table describes the SARIMA parameter interpretation. The non-seasonal order is (2, 1, 2), with an AutoRegressive order of 2, an Integrated order of 1, and a Moving Average order of 2. The seasonal order is (1, 1, 1, 6), with a Seasonal AutoRegressive order of 1, a Seasonal Integrated order of 1, a Seasonal Moving Average order of 1, and a Seasonal periodicity of 6.

Table 9 summary: The table shows coefficients and associated hypothesis tests for SARIMA models on the share of tokens restricted, over all three corpora. For Dolma, ar.L1 and ma.S.L6 have relatively larger negative coefficient values, while ma.L1 has a relatively larger positive coefficient. For RefinedWeb, ma.L1 and ma.S.L16 have relatively larger negative coefficient values.