Please describe your proposed solution.
Executive Summary
- We'll build a streaming, real-time API for Cardano, providing a more natural mechanism for interacting with the blockchain.
- The API will use gRPC, a very performant and robust protocol that is used extensibly across the industry.
- It is powered by Dolos, another project by TxPipe, serving as an efficient replacement of the Cardano node for data integration scenarios.
- It will be integrated into the Demeter platform to provide a zero-configuration experience. A free-tier will be available for developers.
Introduction
Connecting to the Cardano node is extremely hard due to a set of design choices. Most of the other blockchains ecosystems have a much more streamlined experience. This is an important barrier for developer adoption and a performance penalty for existing Cardano projects.
gRPC is an open-source project that originated at Google. It is used extensibly across the industry. It's slowly becoming the defacto protocol for backend APIs due to it's performance and extensive toolchain.
Dolos is a Cardano "data-node" developed by TxPipe that has a different set of design-choices that make it suitable for providing a gRPC API. In fact, it already provides gRPC API with limited functionality.
By extending Dolos' functionality to other type of gRPC interactions, we can provide an efficient and developer-friendly alternative to existing integration mechanisms.
Why gRPC instead of Ouroboros mini-protocols
Ouroboros mini-protocols is a custom binary protocol which isn't well documented (there're gaps in the spec). Connecting through this protocol requires building custom client libraries in each of programming language. This is a huge effort and prone to errors. On top of that, the restriction of having to connect through unix socket is a blocker for many use-cases.
gRPC uses a binary encoding called "protobuf" which has similar (or better under specific scenarios) performance and efficiency as CBOR, both in terms of space optimization and encoding/decoding speed.
gRPC uses a strict schema, both client and server needs to know in advance the structure of the messages that will be exchanged. What may seem like a burden, is actually a very powerful mechanism to ensure forward and backward compatibility between servers and clients.
gRPC has an extensive toolchain that provides automatic code generation for all of the main stream programming languages (Python, C++, Golang, NodeJS, Javascript, Rust, etc). By using the protocol schema, client and server libraries can be created for any of the supported languages with a strict guarantee of compatibility between them.
Why gRPC instead of traditional HTTP/JSON APIs
gRPC uses HTTP/2, a version of the HTTP protocol that support multi-plexing. This means that many client-server interactions can be bundled into the same HTTP connection, greatly improving the latency and throughput of the communication.
gRPC uses "protobuf", a much more compact binary encoding that provides between 2x - 10x reduction in payload size compared to JSON.
gRPC has streaming support. Both client-side, server-side or bidirectional streaming. Streaming provides an efficient mechanism to implement event-driven scenarios, where one party needs to wait for the other to notify about new events.
gRPC also provides an out-of-the-box bridge to websockets, allowing javascript clients to connect directly from the browser.
Why a streaming API
The blockchain is a live sequence of events (new blocks, new transactions, rollbacks). dApps need to constantly monitor these events to create a consistent view of the state and make apply corresponding changes.
Most popular APIs in Cardano rely on request / reply semantics where each request responds with a fixed view of the state of the chain (eg: give me the utxos of this address). Monitoring changes requires continuous polling of the data just in case anything has changed. This is extremely inefficient.
Streaming is a mechanism where two parties (eg: client / server) exchange a sequence messages efficiently without relying on poling. In our case, a single gRPC connection is used where the server will push a new event (message) every time a relevant change is detected.
For example, just to mention a few, a dApp can connect to our API and ask for notifications every time a change is detected on a particular Cardano address, or get a notifications when a transaction has reached the chain, or when a particular asset has moved between addresses.
About UTxO RPC
UTxO RPC is a spec built on top of gRPC, specifically designed to interact with blockchains that use the UTxO model. It provides well-defined methods for the different type of common interactions that are required (eg: chain sync, state queries, transaction submission, etc).
Having a spec which is implementation-agnostic promotes interoperability. It allows different teams and tool providers to build their own concrete, independent implementations of clients and servers while maintaining compatibility among each other.
This proposal will use UTxO RPC as the interface for the gRPC endpoints. There're other teams / projects in the Cardano ecosystem which are planning on using the same spec. This will foster interoperability.