Skip to main content
Version: Old ArrowSquid docs

Subsquid Overview

What is Subsquid

Subsquid is a full-stack blockchain indexing solution which includes an open-source SDK, specialized data lakes for on-chain data (Archives) and a hosted service (Subsquid Cloud). Subsquid solves the data access problem for a wide range of online and analytical use cases, including:

  • A flexible and performant backend for decentralized applications. In most cases, Subsquid can completely replace client RPC read requests with a tailored GraphQL API, significantly reducing the infrastructure costs and shortening the frontend development cycles.
  • A data pipeline for preparing, transforming and loading large volumes of on-chain data for data analysis and forensics.
  • A highly customizable data source for dashboards and on-chain activity monitoring.

Subsquid is designed ground-up around batch processing in contrast to other block-based and event-based indexers. The batch-based programming model embraced by the Squid SDK boosts its indexing performance to 50k+ blocks per second.

Architecture

The Subsquid indexing stack partially separates on-chain data ingestion (Archives) from data transformation and presentation (squids).

Squid SDK indexing projects (or simply squids) are Extract-Transform-Load-Query (ETLQ) projects built using the open-source Squid SDK. Squids ingest historical on-chain data from Archive in batches and transform it with a user-defined data processor. After reaching the highest block available in the archive, a squid switches to ingesting and processing the new blocks from an RPC endpoint in near real-time. Any data from orphaned blocks is detected and replaced with consensus data. Subsquid SDK offers a built-in server to present the transformed data with a GraphQL API as well as customizable adapters for transactional databases (e.g. Postgres) and data lakes (e.g. s3).

Archives are specialized data lakes optimized for extracting and filtering large volumes of raw on-chain data in batches. Until fully decentralized, Subsquid Labs maintains public Archive endpoints and offers batch access via the Squid SDK free of charge. A full list of Archive endpoints for the supported EVM and Substrate networks is available in this repo and is published as a package @subsquid/archive-registry for easy access.

Squids can be run locally, on-premises or deployed to Subsquid Cloud.

Subsquid ecosystem

Squid SDK

Squid SDK is a set of tools and libraries to efficiently query the Archive data, transform, enrich and persist into the target store. Squid SDK projects are called squids.

Squids have a certain structure and are supposed to be developed as regular node.js packages. Use sqd init command to scaffold a new squid project from a suitable template.

A squid project consists of a long-running processor service fetching and transforming the data from an archive and an optional api service presenting the transformed data with a GraphQL API generated from schema.graphql.

The Squid SDK offers an extensive set of tools for developing squids:

  • Core classes for the processor service: EvmBatchProcessor for EVM chains and SubstrateBatchProcessor for Substrate-based chains.
  • The sqd-typeorm-codegen tool for generating TypeORM entities from schema.graphql. See schema file and codegen.
  • Tools for generating type-safe facade classes for decoding on-chain data. See typegen.
  • graphql-server is the backend for the GraphQL API served by the api service. The GraphQL schema is auto-generated from schema.graphql. The resulting API loosely follows the OpenCRUD standard and supports the most common query filters and selectors out-of-the box. See the GraphQL API section for more details and configuration options.

Subsquid Cloud

Squids can be deployed to Subsquid Cloud free of charge. The deployment of the squid services to Cloud is managed by the squid.yaml manifest. Go to the Deploy Squid section for more information.

What's next?