Best Practices for Preventing AI Exploitation of Open Content and Open Data DPGs

Table of Contents
1.0 Introduction and Purpose
2.0 Context: The Shared Challenges
3.0 Recommendations
3.1. Transparency and Traceability
3.2. Reciprocity and Value Sharing
3.3. Licensing, Governance, and Legal Tools
3.4 Technical Mechanisms for DPGs
4.0 Conclusion
5.0 Acknowledgement
6.0 References

1.0 Introduction and Purpose

This concept note presents a consolidated set of recommendations, mechanisms, and a foundational position developed by digital public goods (DPG) product owners across various Open Content and Open Data DPGs in response to shared challenges and concerns described in the subsequent sections below. Its primary purpose is to address the emergent practical challenges associated with the ingestion, processing, and use of open content and open data from DPGs by commercial Large Language Model (LLM) companies, artificial intelligence entities, and other non-commercial research organisations ¹.

The recommendations outlined here are born out of a shared necessity to ensure that the open, public-good nature of DPGs’ data and content is preserved, that the original intent and licensing are respected, and that the value generated by these public-good resources is reinvested and fairly acknowledged, in a manner that benefits the communities and stakeholders who created it ².

2.0 Context: The Shared Challenges

The rapid proliferation and increasing commercialisation of generative AI, particularly large language models, present a novel set of challenges for DPGs, which, by definition, operate on principles of openness, public benefit, and non-exclusivity ³. LLMs derive significant commercial value from ingesting, training on, and repackaging content created for the public good. This dynamic creates a fundamental power imbalance in the ability to use data and derive value from it. The public-good sector provides the foundational “raw material” for proprietary commercial and non-commercial products, yet lacks a mechanism for reciprocal value-sharing, acknowledgement, or reinvestment into the DPG ecosystem ⁴. The core challenges shared by DPG product owners include:

Infrastructure & Technical Challenges: Bot scraping and crawler abuse overwhelm APIs, consume significant resources, increase operational costs, and make platforms less usable for legitimate contributors and users.
AI and Generative Technology Impact: Users are increasingly accessing information through AI chatbots rather than visiting original projects, while open content is being scraped, repackaged, and relicensed without adequate attribution, compensation, or transparency, raising growing concerns among contributors and creators.
Financial Sustainability: Traditional grant funding is becoming less reliable, and fundraising is harder as funders route their programs to new AI technologies. Lower traffic and visitor metrics also weaken the argument for supporting a consistent user base. More organisations are becoming cautious about sharing data openly or sceptical about investing in developing open content due to concerns about value extraction.
Lack of Transparency and Opt-Out Mechanisms: There is currently no established standard or culture for disclosing the use of data in AI, whether knowingly or inadvertently. DPGs often have no clear visibility into which commercial or non-commercial LLM entities are utilising their content, how it is being used, or a straightforward mechanism to engage in dialogue, collaborate on shared goals, request adherence to specific ethical guidelines, or effectively ‘opt out’ of certain high-risk uses. Limited disclosure of how and why major AI companies use data also makes it difficult for organisations to measure impact, protect their assets, and secure long-term funding.
Contribution Model Threats: Community contribution models are at risk as corporate actors extract volunteer-created data and content without clear reciprocal investment in project sustainability, contributor incentives, or respect for open governance. Likewise, when LLMs step between the DPG and its end user, the contact and the goodwill/motivation that produced and sustains the DPG begin to fade. If contributors no longer feel that their work is respected, attributed, or reciprocated, they may not perceive benefit in engaging with and supporting the development of DPGs.
Negative Impact on the DPG Community: The increasing use of LLMs without consent is requiring DPG communities to develop new policies and guardrails, diverting the limited capacity of DPG product owners and volunteers towards solving new AI-related challenges, which is taking away the time needed for maintaining and developing open data and content for shared use. Possible steps to prohibit commercial entities from using open data and open content run counter to the open-source ethos and the licenses broadly applied by DPGs, and negatively impact legitimate users who intend to fairly credit and support DPGs. Finally, if support for the DPG community declines or engagement with DPG for users and potential maintainers becomes complicated, the entire human infrastructure of DPG may collapse.

3.0 Recommendations

Based on the shared challenges described above, the following non-exhaustive recommendations are put forth for engagement with commercial large language model companies, non-commercial research organisations, and the broader AI ecosystem, alongside mechanisms DPGs are exploring:

3.1. Transparency and Traceability

Mandatory Content Index Disclosure: LLM companies should be encouraged to publish, at a high level, the major sources of scraped data used in their training corpora, enabling DPGs to confirm whether their content was used.
Attribution Mechanism: The work should continue on standards that mandate a behavioural loop in which AI outputs must link back to and cite the original source datasets that informed them. Drawing on attribution frameworks and best practice guidance—such as those developed by Creative Commons ⁵. This ensures that machine-generated content maintains clear traceability to its human-authored origins, reinforcing the value of the digital commons.

Financial Reinvestment: Commercial LLM companies that rely heavily on scraped DPG content should establish a mutually sustainable fund or mechanism to provide grants, unrestricted funding, or technical resources back to the DPG organisations whose content contributes to their commercial success.
Ethical Data Access for Research: Commercial LLM companies should commit to providing free, dedicated, and high-fidelity access to their advanced models for non-commercial research and development within the DPG and public-good communities.

3.3. Licensing, Governance, and Legal Tools

Respecting Robot Policies (robots.txt): DPGs should maintain up-to-date robots.txt files that clearly specify disallowances for automated crawlers and AI bots, specify usage parameters, and protect server resources from indiscriminate scraping. Likewise, commercial LLM companies and non-commercial research organisations should commit to developing technical and policy safeguards to ensure compliance with these rules.
Terms of Service (ToS) Anti-Scraping Clauses: Organisations should implement clear ToS that explicitly prohibit unauthorised large-scale data harvesting and automated scraping, establishing a legal baseline for managing commercial interactions and protecting proprietary data assets.

3.4 Technical Mechanisms for DPGs

Tiered API Scheme: Implement a multi-tier access model in which public/non-commercial/research users retain free, unrestricted access, while high-volume commercial entities are routed to a paid API tier that provides dedicated endpoints ⁶. Under this model, commercial entities can pay for other specialised features such as real-time data streams and structured metadata. This mechanism effectively transforms the extraction of open resources into a sustainable funding loop that reinvests in the maintenance and human infrastructure of the DPG ecosystem.
CAPTCHA Implementation: Deploy CAPTCHA or similar challenge-response tests for suspicious traffic to verify human interaction, effectively mitigating automated bot access while maintaining accessibility for genuine users.
Rate-Limiting for Requests with High Bot Score: Configure dynamic API rate limits based on traffic patterns to prevent server degradation from aggressive data extraction, ensuring stability for legitimate human users and open-science researchers.
Web Application Firewalls (WAF): Deploy WAF solutions to monitor and filter traffic, automatically blocking known automated scraper bots and malicious traffic sources that exceed defined safety thresholds.

4.0 Conclusion

These recommendations serve as a starting point for dialogue and a policy framework to govern a more equitable and ethical relationship between the stewards of digital public goods and the developers of transformative commercial AI technologies. DPG product owners desire intentional support to reiterate the value of the human infrastructure behind DPGs and to build a safer mechanism that doesn’t restrict legitimate users from continuing to innovate with our public goods while still motivating community contributors to support maintenance. We do need an open dialogue that doesn’t require community members to pick sides on whether to use AI, but rather to engage with commercial and non-commercial entities to identify an appropriate path forward.

The DPGA Secretariat has likewise developed a DPG Defence Playbook, an overview of available mechanisms to protect against AI exploitation and how they map against the current open definition, and, by extension, the DPG Standard. The playbook’s aim is to deepen understanding and conceptual clarity around the tension between maintaining open access and ensuring the survival of an open resource. We will continue to work with all relevant stakeholders and DPG product owners to advance this work further.

If you’re an expert on any of the recommendation/challenge outlined above and would like to share resources or insights with Open Content & Open Data DPG product owners via their community calls (specifically on the topics below), please send an email.

Measuring Impact for Open Content and Data DPGs.

Ethical Use of AI for Open Content and Data DPGs.

Financial Sustainability for Open Content and Data DPGs.

5.0 Acknowledgement

The following Open Data and Open Content DPGs were involved in data collection and insights that informed this concept note, with specific in-depth reviews from product owners such as Malvika Sharan, Lucia Ixtacuy, Sergio Bogazzi, Miguelángel Verde, Alicia Seidle, and Marc McGowan.