Please describe your proposed solution.
Cardano nodes can assume one of two roles:
- block producer: in charge of minting blocks
- relay node: in charge relaying blocks from / to peers.
Each of these roles has concrete responsibilities and runtime requirements. Criteria such as network topology, resource allocation, backup procedures, etc vary by role.
We argue that there’s a 3rd role that should be treated independently with the goal of optimizing its workload: nodes that are used with the sole purpose of resolving local state queries or serving as data source for downstream tools that require ledger data.
There are many potential optimizations for nodes performing this type of workload that are not currently possible with the Cardano node:
- drastically limiting the amount of memory required to execute the node
- switching to storage solutions with different trade-offs (eg: S3, NFS, etc)
- providing alternative wire protocols more friendly for data queries (eg: REST, gRPC)
- providing an auth layer in front of the API endpoints
The goal of this project is to provide a very limited and focused version of the Cardano node that can be used by DevOps as a cost-effective, performant option to deploy data nodes side-by-side with the producer / relay nodes.
This new role would be useful in the following scenarios:
- As data source for well-known tools such as DB-sync, Ogmios, CARP, Oura, etc.
- As a fast, low resource node for syncing other producer / relay nodes.
- As a ledger data source that scales dynamically according to query load.
- As a node that leverages network / cloud storage technology instead of mounted drives.
- As a node that scales horizontally, allowing high-availability topologies.
- As a low resource local node for resolving local state queries.
Data nodes will share some of the features with the mainstream Cardano node:
- Node-to-Node and Node-to-Client Chain-Sync mini-protocol
- Node-to-Node Block-Fetch mini-protocol
- Node-to-Client Local-State-Query mini-protocol
This new type of node will also provide features not currently available in the mainstream Cardano node:
- HTTP/JSON endpoint for common local state queries
- gRPC endpoint for local state queries and chain-sync procedure
- Different storage options including NFS, S3 & GCP Buckets
- Low memory consumption (allowed by the trade-offs in scope)
Dolos will be developed as an open-source project using Rust as its main development language. The code will do heavy usage of the Pallas library developed by this team and already available as version 0.11.
Please describe how your proposed solution will address the Challenge that you have submitted it in.
Having an efficient way to access ledger data without incurring high infrastructure costs will accelerate development for both solo developers and small to medium sized teams. A lower entry barrier will drive more developers to the ecosystem.
New wire protocols such as HTTP/JSON and gRPC will also widen the spectrum of options for developers to integrate ledger data without the requirement of integrating low-level mini-protocols directly into their projects.
What are the main risks that could prevent you from delivering the project successfully and please explain how you will mitigate each risk?
Our experience developing Oura and Pallas allowed us to gain knowledge and implement libraries that will serve as foundational components for this project. Regardless of this advantage, there are some known-unknowns that could present a risk:
- Performance gains and resource allocation optimizations are theoretical, these were extrapolated from our experience implementing Cardano data processing pipelines using components written in Rust. We won’t have a strict, quantifiable measurement until we develop a PoC of this project. To mitigate this issue, our development process will include performance benchmarks execution at each development milestone. Reports will be included as part of each release.
- There’s some documentation lacking regarding local state queries wire-format which will need some reverse engineering from the mainstream Cardano node. We have experience with this approach but the level-of-effort associated with the task is hard to anticipate. To try mitigate this issue, we'll reach out to IOG for advise and documentation in case it's available.
Is important to highlight that we consider this project feasible because the complexity of a data node is orders of magnitude lower than the complexity of a full-node. Please note that we are NOT proposing an alternative to the mainstream Cardano node written in Rust, that would be imposible to achieve with our current development bandwidth.