Please describe your proposed solution.
<u>Introduction</u>
Due to the advancements in large language models (LLMs), the entire world has undergone significant changes. Any governance process based on proposals is now subject to the open creation possibilities by LLMs as well as the useful review and evaluations from them.
Wolfram: AI Revolution & Project Catalyst Video Presentation
<u>The Problem and Our Solution</u>
At present, understanding the full Catalyst process is a significant undertaking. From problem sensing to challenge interpretation, from proposal writing to feedback sharing and ultimately submission; proposers have extensive guidelines and community standards to follow for accurate completions. Writing proposals is a significant undertaking, and thus a strong deterrent against less serious proposals. However, a global community of participants potentially struggle to comprehend, articulate and convey how their proposal will deliver impact in a highly feasible and cost effective way. And while these strict guidelines are necessary, they may simultaneously increase the difficulty in articulating solutions to real problems and ultimately decrease participation. It would be encouraging to this community if supportive and generative "railings" existed to help facilitate the process. A grassroots system of innovation may benefit from a supportive boost in the area of proposal writing.
Conversely, the field of community reviewers face a similar (yet opposite) undertaking when evaluating proposals in each new funding round. With financial incentives and tools like LLMs, what support are we providing these reviewers to combat and enforce the auditability standards? With over 1,000 submissions in each of the last three rounds, what standardized procedures can reviewers unequivocally rely upon when facing the full spectrum of community submitted proposals? Even the most veteran reviewers might bias depending on native language or recognition of "friendly" proposals. We know there's a wide variety of reasons why community reviewers need support in the functions they provide the Catalyst community. So they may support the idea of an assistant that boosts their ability to efficiently and effectively draw conclusions about impact, feasibility and cost effectiveness of each proposal in a standardized way.
So, our solution is to research what is needed to create an LLM-based assistant that acts as a guiding companion in the Catalyst process. The objective of this research is twofold: to explore the feasibility of providing users with support throughout the stages of proposal submission, evaluation, and voting, and to understand the preferred implementation approach according to the community's perspective. It's important to note that this proposal does not encompass the actual implementation of the LLM assistant.
With the support of Wolfram Research's extensive work on building LLM tools over the past six months, with the successful Wolfram plug-in to ChatGPT and more, we are well-equipped to research the requirements of a tool that addresses the proposal process's pain points. The potential of LLMs is vast, opening up numerous possibilities for assistance on all stages of the proposal process – from researching, to writing, to reviewing and, finally, to voting. Our team acknowledges both the extreme benefits and drawbacks of LLMs inside a community process based primarily on written communication. However, it would be prudent to acknowledge that LLMs are now a part of society and we can likely find a balance so that LLMs can be used effectively. In fact, this is how open systems naturally evolve and grow.
<u>Researching LLM with Guided Participation</u>
Our research will center around what is needed to provide step-by-step guidance throughout the Catalyst process through a chatbot type interface. Critically, this is not the creation of an AI chatbot or AI Assistant for Catalyst. That is an enormous undertaking at this point. Rather, this project is significant research and exploration to see if the useful creation of an assistant is possible—by investigating novel approaches and proofs of concept—and if so, how to keep the operational costs low enough to provide a significant return on investment for the Catalyst community. Because this project is fully open source, we believe leading this research effort would establish a strong precedent for community actors to get up to speed and build from there.
<u>Researching the Catalyst Assistant</u>
To build a robust and cost-effective LLM-based Catalyst Assistant, extensive research is required.
Literature Review & Community Surveys
The initial phase of the work will involve conducting thorough literature reviews and community surveys. Wolfram has already dedicated significant time to studying and engaging in discussions about the utilization of LLMs for developing expansive virtual assistants. However, it is essential to delve into detailed discussions and provide comprehensive summaries of various approaches. In addition to researching existing literature, conducting surveys among Cardano Catalyst participants is necessary to gain a deeper insight into the execution of different roles within Catalyst and understand the expectations users would have for a potential assistant.
Content Collection & Curation
There have been 10 Cardano Catalyst funds, and Wolfram Blockchain Labs (WBL) has already constructed curated collections of past proposals, including analyses for ranking proposals based on community voting and reviews in our existing dashboard solution. However, for this specific use case, we will need to further curate these proposals. Additionally, we must gather all instructional content related to participating in different roles within Catalyst. Furthermore, it is likely that we will need to develop a significant amount of new material on how to engage in Catalyst. Initial assessments of the existing material reveal that many instructional resources assume a significant amount of prior knowledge—unfortunately we can’t assume that instructional content would be available in different LLMs, nor would it be validated or trustworthy. These tasks would provide curated content and a deep understanding of the unique aspects of Catalyst proposals that would be the key inputs into LLM-based assistance in detail aspects of proposal creation, refining, and evaluation.
LLM Evaluation
Drawing from our extensive experience collaborating with various LLM providers, such as OpenAI, we are aware of the wide range of performance capabilities and inference costs associated with these models. To develop a highly efficient system, striking a delicate balance between LLM performance and inference expenses is crucial. This entails establishing a test suite and evaluating different LLMs specifically for the use case of an LLM enhanced Catalyst Assistant. Furthermore, during the LLM exploration process, we will have the opportunity to delve into other aspects of the training pipeline to create and experiment with use cases, and design and refine proofs of concept and rank them for the possible benefits they may provide to the proposal pipeline.
Final Output
The culmination of our research and development efforts will result in a detailed research report. This report will document the exploration of LLMs that meet the necessary licensing requirements and ROI, presenting the findings to the Cardano community. Insights gathered from surveys conducted within the Cardano community will also be included in the research report, Computational essays will provide a step-by-step account of the potential development process, highlighting the critical aspects of building a potential assistant. Specifications and architecture will be detailed to provide transparency and clarity on potential functionality and implementation.
Project Objectives Summary:
- Conduct a comprehensive literature review
- Implement community survey
- Collect all instructional content related to Catalyst
- Evaluate LLMs for suitability for explored assistant use cases
- Research specification architectures for LLM-based catalysis of proposal pipeline
Overall Benefits:
- New instructional content for potential training material
- Research report of potential LLM assistant architectures
- Specification architectures of an assistant and beneficial model
- Community discussion on application of new technology
<u>Conclusion</u>
The introduction of an LLM-based assistant for the Cardano Catalyst proposal process holds immense potential for transformation. Harnessing the capabilities of large language models, we can equip participants with seamless navigation through Cardano Catalyst, offering real-time assistance and guidance. Together, we can revolutionize the proposal process, foster innovation, and propel the Cardano Catalyst community to unprecedented achievements.
How does your proposed solution address the challenge and what benefits will this bring to the Cardano ecosystem?
The "Catalyst System improvements" challenge asks "what research and development is required to <u>advance the state of the art</u><u> </u><u>of Catalyst</u> and allow Catalyst to serve the community's needs better?'
Our proposed solution addresses this question by researching LLMs and trying to understand through a community supported research process how it can be applied to Project Catalyst.
We believe this proposal directly addresses the Catalyst Systems Improvements because this research has a high likelihood of advancing the state of the art for the innovation platform for Cardano. LLMs are at their infancy of development and we're just beginning to see the wide-ranging effects. A decentralized governance and proposal-based innovation fund would greatly benefit from understanding the upsides and downsides of LLM assistance. This is a direction that our community will inevitably face more in the future, so adoption now may prevent catastrophic failure in the future.
The challenge goes on to detail the category of Academic research. "Clearly defines a known Catalyst-specific problem-space where the intention is to identify facts and/or clearly stated opinions that will likely assist in solving Catalyst-specific problems, or a detailed study of a Catalyst-specific subject, especially in order to discover (new) information or reach a (new) understanding."
LLMs are state of the art technology and Wolfram Research has been exploring many ways to use it.
One of the critical issues is how many text-based processes will change with the introduction of such revolutionary technologies. We believe it's important to explore how to integrate LLM services in ways that help all members of the Catalyst community participate in all roles of the Catalyst process. Participants should not be hampered, nor should the intrinsic value of their proposals be lessened by, for instance, writing in a second language or having limited experience in proposal writing. In a way, establishing the use case for individuals ensures that power is shared more democratically in a process that is based on decentralization as core tenet.
How do you intend to measure the success of your project?
A general community agreement that this line of research is interesting and worth exploring further. Clear success would be if a this research enables other participants success using LLMs and/or implementations into Catalyst.
Please describe your plans to share the outputs and results of your project?
We have deliverables for public repository of curated Catalyst training data, a research report, breakout rooms to share findings and receive community feedback, and twitter spaces to engage in discussion.