The Future of Open Human Feedback

Shachar Don-Yehiya et al.

Audio by Paper2Audio.

Abstract

Human feedback on conversations with language language models (LLMs) is central to how these systems learn about the world, improve their capabilities, and are steered toward desirable and safe behaviors. However, this feedback is mostly collected by frontier AI labs and kept behind closed doors. In this work, we bring together interdisciplinary experts to assess the opportunities and challenges to realizing an open ecosystem of human feedback for AI. We first look for successful practices in peer production, open source, and citizen science communities. We then characterize the main challenges for open human feedback. For each, we survey current approaches and offer recommendations. We end by envisioning the components needed to underpin a sustainable and open human feedback ecosystem. In the center of this ecosystem are mutually beneficial feedback loops, between users and specialized models, incentivizing a diverse stakeholders community of model trainers and feedback providers to support a general open feedback pool.

1 Introduction

Natural language conversations (i.e., chats) have become a primary mode of human-AI interaction. Since the debut of large language models (LLMs) as general-purpose, multi-task reasoners, the volume and variety of use cases have grown, from providing information and enabling tool use to assisting with writing text and code. The steerability and usability of modern LLMs are underpinned by fine-tuning alignment through human feedback (e.g., preferences), which guides LLM outputs toward desirable properties such as helpfulness, informativeness, and safety (e.g., Citation 2021; Citation 2022; Citation 2024; Citation 2022; Citation 2022; Citation 2021; Citation 2024; Citation 2024). Despite being a cornerstone of modern AI research and development (R&D), mechanisms for sustainably sourcing and sharing human feedback data are still underdeveloped for at least three reasons.

Firstly, many high-quality human preference datasets are proprietary and hence not available for others to study, reuse, or modify. This makes it difficult to infer best practices in the structure and type of preference data that leads to the largest gains in performance. Moreover, often the methodologies are not released, preventing even replication and further study (see Table § 1).

Table 1 summary: The table presents a categorization of human feedback datasets based on several dimensions of openness. These dimensions include methodology, access, model participation, human participation and timeline. Several datasets and models are fully open across all dimensions, while others have more limited openness, particularly regarding human and model participation, and timeline. Some models have no open dimensions.

Secondly, many barriers prevent sharing new datasets, in particular the cost and effort needed for extensive human annotation. For example, feedback collection often requires sophisticated user interface (UI) design and lengthy ethics reviews. The costs further compound when specialist expertise is required. The combination of demonstrable value and expensive collection results in dearth of open feedback datasets.

3rd, unlike many ongoing feedback collection efforts of frontier AI labs, open feedback datasets are often treated as static collections of uniform preferences rather than living artifacts that are dynamically maintained and updated. There are growing efforts to circumvent the costs of dedicated feedback collection. Some widely-used datasets instead collate implicit feedback from existing platform interactions such as on Reddit and Stack Overflow; or an increasingly popular loophole is to simulate feedback using powerful closed models. However, distillation of one model's outputs to improve another carries legal, reproducibility, and transparency issues and adds a dependency on closed-model ecosystems. Comparatively, efforts that directly interface with real human-model interactions are rare. These efforts often base their success on hosting LLMs that users can freely use in exchange for chat logs or preference data; though other mechanisms of exchange exist, including paid participation in large-scale human studies, or mediator-enabled donation schemes where users retain ownership. These contributions are already transforming the R&D of open models, but have known obstacles to sustainability such as depending on a small group of self-selected annotators and being skewed towards bulk contributors. Additionally, efforts that attempt to fairly remunerate contributors depend on funding.

Given these challenges, an unresolved question remains: How can we develop and maintain sustainable systems of open human feedback for current and future eras of AI R&D? In this work, we seek to encourage progress by presenting the key components required for a sustainable and open human feedback ecosystem. We solicit inputs from interdisciplinary experts across academic, industry, and open source communities. Our work consists of three main contributions: We draw from best practices of peer production and open source software (OSS) communities to identify shared requirements conducive to LLM development (§ 2). 2. We present seven themes that we view as central to the future of open human feedback (§ 3 & Summary) . For each theme, we summarize why it is an important challenge, potential existing solutions, and recommendations for building a more sustainable model of open human feedback. 3. We conclude by examining what components underpin a successful ecosystem to support further progress (section 4) . We acknowledge that operationalizing sustainability in practice will require trade-offs and difficult prioritization between these themes and substantial buy-in from various stakeholders. To address these challenges, we propose forming an ecosystem based on effective platforms, aligned privatecommunal incentives, and feedback loops for specialized communities (section 1) .

1.1 Defining Open Human Feedback

We begin by defining our scope, clarifying what we mean by open, human, and feedback. By human, we require some degree of human intervention and judgment to the extent that the feedback is not fully automatically generated, such as models simulating human feedback. This does not exclude aided or seeded feedback (e.g., see platforms § 4 and effort § 3.2).

By feedback, we mean human responses to model outputs that are ascribed with some positive or negative valence, i.e., a value judgement. Moreover, human feedback must comment on model outputs. For example, traditional labeled datasets are not a form of feedback, as they are annotated by humans over human generated text and hence contain no reaction element. Another edge case is "wizard-of-oz" datasets, where humans pretend to be models, though these are rare.

Clear terminology surrounding "open" versus "closed" feedback is also required, especially given growing concerns about open-washing in "open source AI", despite substantial differences in transparency, accessibility, and governance structures. We consider openness along five non-binary axes (see current datasets in Table § 1): 1. Open methodology: Whether sufficient information is given to reproduce the dataset, at least partially (e.g., annotators details, guidelines and interfaces). 2. Open access: Whether the final dataset is released for public access and use under a permissive license. 3. Open model participation: Whether all feedback is collected from one predetermined LLM or if third-parties can upload and include their models. 4. Open human participation: Whether feedback is provided by diverse annotators representing the views of LLM users, in contrast to crowdworkers or hired contractors who may enforce narrow views of ideal model behavior. 5. Open timeline: Whether feedback is dynamic, covering new models, new topics and capabilities, or is a one-time effort with a short or single time frame.

Combining these elements, we consider open human feedback as human-generated, valenced responses to AI model outputs characterized by open data accessibility and permissive terms of use, with ongoing and inclusive participation by humans and AI models.

2 Key Lessons from Peer Production and Open Source

Open feedback for LLMs can draw inspiration from the practices of peer production and OSS communities in promoting open, sustained, and diverse contributions. We provide an overview of these communities and highlight key takeaways from their success.

2.1 Peer Production

Peer production is a (knowledge) production model in which numerous individuals collaborate, often remotely, without relying on traditional hierarchical organization or financial compensation. Peer production communities have effectively brought together diverse contributors (see § 3.4) to pursue their shared missions. For example, Wikipedians make 290k edits per day to create and maintain a free online encyclopedia and OpenStreetMap enthusiastic mappers maintain open digital maps all over the world.

The success of peer production projects relies on both intrinsically and extrinsically motivated contributors. For example, volunteers contribute to Wikipedia because they believe in the shared mission and fundamental value of free online encyclopedias for humanity, but they may also contribute for other reasons such as intellectual stimulation and enjoyment. Wikipedia's Barnstars and Stack Overflow's reputation scores are examples that have fostered external motivations. Sometimes, extrinsic motivators can backfire; for example, psychological research notes that when extrinsic motivations are introduced existing intrinsic motivations may dissipate.

Another factor contributing to the success and sustainability of peer production is the availability of platforms that provide the infrastructure, processes, and tools that support community-driven governance. For example, Meta Stack Overflow allows the Stack Overflow communities to discuss policies and suggest new features. Wikipedia's infrastructure enables the creation of policies, guidelines, and essays for community governance. Flexible governance is especially critical, as each community has unique governance needs; even within Wikipedia, different language editions have unique rules. Community-driven governance is critical for sustained contribution and long-term community health. Open human feedback should follow suit in supporting bottom-up, community-driven governance for sustainably obtaining and moderating high-quality data.

Forms of peer production in AI span a wide range of organization structures and focus areas. Some organizations are structured primarily around the production and exchange of knowledge on AI systems or data in specific linguistic and geographical contexts, category of model architectures, a specific model, or broader focus on values, such as accessibility and replicability. These efforts have unique perspectives on collaboration, governance, code, data, and more. In most cases, however, explicit statements of values, governance processes for the project and artefacts, and codes of conduct play an important role in ensuring that the work benefits the targeted communities and, thus, motivates contributions.

2.2 Open source software

OSS concerns the development and distribution of software under open source licenses that allow anyone to inspect, use, modify, or redistribute the source code. This modus operandi has enabled countless research and innovation achievements, including in AI.

OSS projects are developed by diverse contributors, driven by varied social, economic, and technological incentives. For individuals, intrinsic incentives, such as personal values, needs, or enjoyment, as well as extrinsic incentives, such as reputation, learning, and career benefits, are significant. Individuals' incentives vary based on their volunteer vs. paid status, project governance, and their cultural norms.

Companies are chiefly driven by economic and technological incentives, such as reducing R&D costs, shaping standards, practicing reciprocity and recruiting talent, among others.

A key lesson from successful OSS projects is the importance of open governance and vendor-neutral hosting (e.g., by an independent consortium or foundation), which facilitate contributors with diverse incentives to build consensus and collaborate on non-differentiating base-technologies. Similarly to peer production communities, OSS communities have created tools that enable open governance and collaboration, such as standard licenses, codes of conduct, and tools and metrics that can support monitoring and improving project and ecosystem health over time. These practices can similarly ensure broad participation and sustained innovation in the open human feedback ecosystem.

3 Challenges and Opportunities of Realizing Open Human Feedback

We present seven themes that are central to the future of open human feedback. For each theme, we summarize why it is an important challenge, potential existing solutions, and recommendations for building a sustainable ecosystem for open human feedback.

3.1 Incentives

Aligned incentives are needed to motivate individuals and companies to participate in sharing or annotating open human feedback data. As per § 2, we must account and design for the diverse incentives of contributors, from individuals to companies.

With regards to individuals, the open human feedback ecosystem ought to accommodate both intrinsic and extrinsic incentives of diverse individuals, from volunteers to sponsored contributors. Intrinsic incentives may include learning and skill development, hobbyism, and community kinship, among others. For example, contributors from underrepresented socio-demographic and/or socio-linguistic groups might participate in open human feedback projects to advance AI resources and alignment for the interests and needs of their community. It is advised that projects establish governance structures (including a mission statement) that resonate with their community members' values. However, the "burden" of participation needs careful co-design to mitigate exploitation, benefiting solely those who build and commercialize AI systems. For extrinsic incentives, projects could encourage participation via leaderboards, gamification points, or monetary compensation.

With regards to companies, we must distinguish between frontier AI companies that would share human feedback data and companies that would contribute to open human feedback data projects. First, frontier AI companies can be incentivized to share human feedback collected via their proprietary chatbots by aligning data sharing with their strategic interests, such as reducing R&D costs (e.g., via crowdsourcing annotations) and building recognition in the open source AI community (e.g., via leaderboards). Examples of successfully spun-out OSS and open data projects may help temper concerns and illustrate the benefits of sharing data for open collaboration. In order to facilitate contributions by and collaboration among diverse companies to human feedback data projects, we call for such projects to be hosted by a non-profit, vendor-neutral organization (such as the Human Feedback Foundation), rather than by a single company, as a tried-and-tested enabler of collaboration between "unexpected allies". Foundations with relevant expertise and missions include the Human Feedback Foundation, GenAI Commons, and the LF AI & Data Foundation.

3.2 Effort and Involvement

The success of the open human feedback ecosystem will rely on substantial human effort involving a range of tasks, such as gathering samples, annotating data, and writing alternative answers. Given the range, platforms intended for feedback collection should provide simple interfaces that reduce cognitive load and aid task completion, which is especially important to sustain volunteer and decentralized workflows. At the same time, feedback collection mechanisms should be be implemented at a centralized level in a dedicated platform. Most current feedback platforms use pairwise comparisons for ranking LLMs, but these comparisons typically lack fine-grained assessment (§ 3.4, 3.3). Moreover, feedback on these existing platforms tends to be of limited diversity and coverage. The paired back functionality of these hosted platforms attracts short conversations and relatively low topical, usecase and user diversity. The simplest way to reduce unnecessary effort is to make intuitive user interfaces for feedback platforms. Where possible, prompted feedback, such as users' rating or ranking of LLM responses, should be supplemented with feedback from the naturally-occurring cues in the chat, such as when the user verbally thanks the model or edits their original prompt.

3.3 Expert Contributions

LLMs often perform poorly at tasks that require expert knowledge, and specialized feedback from domains such as healthcare, legal, or finance is more challenging to acquire than general feedback. High-quality specialized feedback is unlikely to naturally occur through experts' use of LLMs, and hiring domain experts is costly. Nonetheless, expert feedback is necessary for evaluating model capabilities and guiding improvements, especially in complex or technical domains where layperson knowledge is not sufficient. Current efforts to collect expert feedback are limited in size and diversity, and/or gate-kept within private organizations. For example, GPQA collects static expert contributions, while Bloomberg GPT relies on internal company data. Companies like OpenAI or DeepMind employ specialized AI trainers and keep the proprietary feedback 'in-house'. Replicating these efforts in open source communities faces challenges including high costs (GPQA costs ~$100/hr), scalability issues, and quality control. Moreover, these efforts risk domain bias; for example, AlpacaEval overrepresents coding while neglecting theoretical mathematics. Successful initiatives for driving expert participation like Wikipedia editing and OSS development (see § 2) allow contributors to choose their tasks based on their interests and skill. To attract experts, platforms should provide tools for identifying topics requiring specialized input, including fact-checking and deliberation, and for routing these tasks to well-suited contributors. Different incentive mechanisms may appeal to experts, including contributing to the development of specialized models or public attribution and recognition for their contributions.

3.4 Linguistic and Cultural Diversity

LLMs struggle to serve diverse human demographics due to limited coverage in their general and feedback training data, which predominately relies on English speakers from narrow communities, and few annotators contribute the majority of data, even in multilingual efforts. Accordingly, culturally-specific queries and minority perspectives tend to be overlooked. To build technologies inclusive of diverse socio-demographic and socio-linguistic groups, it is crucial to go beyond "convenience samples" that rely solely on LLM developers, enthusiasts, or company-employed annotators, and to gather feedback from underrepresented communities. However, reaching different communities can be challenging due to factors such as lack of technology access or different device uses, such as the common use of mobile in the Global South.

There have been few recent attempts at covering demographic and geographic depth, or multilingualism and dialect diversity. Communityled engagement can return more power and control over data to the communities themselves, but must be handled with care to avoid exploitative practices. Opening human feedback data to peer production provides the single most promising step for diverse participation. However, diversity is not guaranteed (§ 2) and to facilitate it, platforms should be accessible, available in different languages, and be complemented by outreach to underrepresented groups to uncover their unfulfilled needs. Moreover, clear guidelines and regular audits might reduce opinion biases and uncover representation gaps.

3.5 Adaptable and Dynamic Feedback

Human feedback differs from traditional dataset annotations: while resolving grammatical sentences or solving linear equations have a single, static ground truth; human preferences, especially those surrounding cultural issues, moral debates, and social conventions vary across individuals and groups (§ 3.4) as well as over time. As human preferences and views differ and evolve, model outputs will necessitate ongoing feedback collection and models trained on existing feedback may require feedback on new types of errors. This bears similarity to existing platforms for ongoing annotation. For example, Dynabench allows iterative model benchmarking Citation 2021, LiveBench mines new examples Citation 2024, ChatBot Arena publicises new model releases, and the ShareLM browser plug-in mediates ongoing user conversations on many platforms.

Acquiring dynamic data is challenging and costly, rendering traditional payment for annotation, as used in static datasets, unsuitable. Instead, to achieve continuous participation from the community, longitudinal incentives need to be designed and built into the platform (§ 3.1) and new methods to be invented to flag and update outdated feedback. In time, the dated feedback itself might become useful, for example in comparison with the updated one.

3.6 Privacy and Data Protection

Sharing feedback can have wide reaching benefits, including for contributors, but equally could harm the contributors if their privacy is not adequately protected. Feedback platforms might also motivate more contributions by upholding data protection legislation and adopting best practices that enable contributors to have a say over how their data is shared. Clear mechanisms of opt-out can support these aims, for example, the " Am I in The Stack? " initiative associated with the BigCode Dataset, which allows code developers to see whether their work is included in the open source software training dataset, and provides a mechanism for them to request to opt out their data from future versions of the dataset. Feedback platforms must comply with existing privacy regulations that standardize and enforce this social responsibility, such as GDPR, CCPA, and regulations for specific applications such as the Health (HIPPA, Act).

In addition to proper security protocols, anonymization and deidentification are central considerations prior to collective data sharing. This includes removing personally identifiable information (PII), such as names, emails, and IP addresses, which can be especially challenging when interleaved in a chat and because PII definitions vary by application. Furthermore, in feedback discussing specialized topics or rare dialects author identities might also be unavoidably compromised by the topic itself.

Beyond anonymization, other privacy-enhancing techniques are also recommended by standard practices (e.g., NIST). Moreover, the text itself inherently shares information that for example could be crossed with other sources to deanonymize. Thus, unless privacy leakage is theoretically restricted — e.g. by noisy aggregation of the data — then research should at least estimate the privacy leakage.

Giving contributors control and autonomy over their personal information is vital for building trust and ensuring legal compliance. For example, allowing users to retract or edit their contributions, or providing a time window for complete removal of their data before feedback is made public. Privacy must be balanced with providing a sense of ownership (section 3.7) as the two might be at odds. As heuristics like hashing names to match users to their content are susceptible to misuse, logins are often preferable. Regardless, any opt-out guarantees presents logistical challenges as to how these requests would propagate to any derivative models trained on the data, which would then need to remove specific details, known as 'machine unlearning'.

3.7 Legal and Ownership

To ensure all contributions are appropriately licensed and available for public use, while mitigating the risk of legal challenges, robust data and legal governance frameworks must be established and concisely communicated to contributors. Particularly, we propose following best licensing practices from successful open-source data collection efforts where contributors retain ownership of the data they share. Notably, both humans and models are involved in data creation, but commonly the creative act is legally attributed to the human, we hence consider the data solely owned by the human contributor. This stance has precedents, though the legal debate remains unsettled. We suggest contributors are asked to give informed revocable consent to share their data under a permissive content license, such as Creative Commons or Open Data License, by signing an agreement (e.g., checking a box). This sets a transparent standard and reduces the fear of legal retribution for both the data collection platform and consumers of the data. Moreover, we suggest that those collecting feedback (e.g., platform providers) submit data card documentation, explaining context, methodology, intended applications, and limitations.

4 Visions of Successful and Sustainable Open Feedback Ecosystems

Building on these challenges (see Summary) , we conclude by considering the tangible components that are necessary for building an ecosystem of open human feedback under the mission: Humanity guiding open AI for humanity.

Any solution requires building an open feedback platform—sites or software that allow the community to dynamically contribute in a standardized and efficient way. Platform designs often shape standards for the community, and careful consideration is required to ensure they can facilitate efficient distribution of work (§ 3.2), are accessible and easy to use by diverse contributors (§ 3.4), enable governance and moderation of feedback (§ 2.1), and encourage participation (§ 3.1). Beyond our current suggestions, such platforms should be open source and will evolve in line with changing feedback requirements, such as feedback types, biases, and privacy issues. While current platforms, especially Hugging Face's Spaces, have lowered the entry barrier for machine learning model experimentation, they lack detailed systematic feedback mechanisms. Industry-centered platforms, like ScaleAI, offer feedback platforms that support internal workforces to continuously refine models on datasets but often suffer from biases due to limited diversity and specificity of crowdworker knowledge (§ 3.4). Argilla represents the closest open source solution for feedback collection, yet it still lacks support for participants to discuss feedback, share incentives, or collectively govern projects. Another important component is a shared resource of the chats and feedback. Chats and feedback should be pooled together to simplify use. Ideally, this pool would combine sustainable data from the platforms but also one-time efforts from the community. The mechanism of contributing to the pool might vary. At one extreme, chat UI platforms connect directly and share. Alternatively, sharing is done by dedicated mediators, like browser plug-ins (Don-Yehiya et al., 2023). Mediators allow users to interact freely with models on any platform (even closed) and ensure the feedback is propagated to the pool.

The last component is participation. To thrive, the ecosystem needs not only to improve the types and ways of collecting feedback, but to have a community. This requires aligning incentives of both individuals and organizational contributors of public resources, which underpins successful citizen science and peer production models.

To achieve all the requirements above (see Summary) , we call for the open human feedback ecosystem to interoperate with the LLM training ecosystem (see Figure 1) . Specifically, self-sustaining feedback loops—systems of mutual benefit for all the stakeholders involved. Consider, for example, an open platform developed by the different contributors in the ecosystem. This platform spins up specialized models equipped with chat UI, model hosting, feedback collection, and automatic training on the acquired feedback, which specialize the model further. To further support the research community and its improvement, such a platform should provide functionalities for experimentation, adaptation, and adoption by new contributors, who may adapt feedback collection in custom ways that address their target questions. The platfom should also provide flexibility, allowing contributors to iteratively refine the system itself, observe the effect on feedback, and repeat, drawing on the success of the MAMA cycle (model, annotate, model, annotate).

Figure 1 summary: The figure illustrates a sustainable open feedback ecosystem. Users with various backgrounds interact and provide feedback to open Large Language Models within the feedback ecosystem. These conversations and feedback are then shared through an open pool of data. Subsequently, AI builders in the building ecosystem utilize this data to develop and share the next generation of Large Language Models within the chat and feedback ecosystem. The diagram shows a continuous loop of interaction, feedback, and model development.

Critically, the ecosystem will benefit contributors with specialized needs. For instance, consider a law student requiring detailed and accurate knowledge from an LLM for exam preparation. In a one-to-one relationship with a model, it would require substantial effort from the student alone to improve it for their needs. However, specialization can come from a communal feedback pool, and with time clustering or other unsupervised learning methods might offer personalization by identifying similar users in the overall community. This process is analogous to how recommendation systems improve as more diverse users engage with the platform.

Consequently, a marketplace of models—specialized by topic, culture, or language—could be sustained if specialist contributors benefit from participating in these communities in exchange for their feedback contributions. This approach has the potential to create a continuously improving ecosystem, with specialized language models tailored to diverse user needs. Meanwhile, a standardized and central pool of feedback can serve the entire ecosystem. For example, the overall pooled data might be used to align more universal and generic models, to identify common patterns across domains, or to understand where irreconcilable differences between communities lie.

5 Conclusion

Human feedback has emerged as a central component for improving the capabilities and safety of AI models. However, human feedback data is concentrated within a few frontier AI companies, and there is currently no ecosystem for sharing and collaborating on such data. We have outlined our vision and key components required for a dynamic and sustainable open human feedback ecosystem: well-designed feedback platforms, aligned incentives for contributors, and feedback loops that can support specialized and general-purpose models. However, significant challenges remain: building and hosting feedback loops for specialized models, incentivizing diverse and quality contributions, protecting privacy, and managing governance and data ownership. Despite these hurdles, we call for recognition of open human feedback data as a critical, yet underdeveloped, ingredient for open and transparent AI R&D, and propose a pathway forward.

Actions for Building the Open Human Feedback Ecosystem

Incentivise participation (§ 3.1)

State a clear mission that resonates with community values Proposed: Humanity guiding open AI for humanity Incentivize companies to share open human feedback data Host projects in a vendor-neutral organization to facilitate participation Design for intrinsic and extrinsic incentives of diverse contributors

Reduce contribution barriers (§ 3.2)

Design interfaces that reduce contribution barriers (e.g., time, effort, etc.) Develop efficient approaches to use, sample, and augment existing data Extract feedback from natural text in chats Facilitate expert annotation (§ 3.3)

Allow the community to define feedback requirements independently Allow experts to choose what and how to contribute independently

Diversify participation (§ 3.4)

Facilitate peer-produced contributions to human feedback data projects Encourage participation from diverse socio-demographic & -linguistic groups Mitigate exploitation through responsible use and contribution policies Provide accessible and multilingual platform interface and tools Audit collected data for potential biases and discriminatory content Design for feedback contestation and consensus-building among contributors Collect updated feedback (§ 3.5)

Pool data into one location Make dedicated efforts for specialized data annotations Create platforms, mediators, and tools for chat & feedback data sharing Encourage and enable the sharing of updated feedback data

Privacy (§ 3.6)

Follow security best practices and comply with relevant regulations Map users to their contributions privately (e.g., via logins)

Develop methods to give users control over their personal information Anonymize data and remove private information

Legal (§ 3.7)

Share datasets with appropriate documentation (e.g., data card)

Ensure informed revocable consent for data sharing (e.g., via box-checking)

Users share data with permissive licenses (e.g., Creative Commons license)

Legend

Feasible with existing knowledge and tools Knowledge and tools partially exist, but some R&D still required R&D required

You have reached the end of the paper.