Development flow
Below is a general outline of the squid development steps.
0. Prerequisites
- Get familiar with squids and Archives by reading the Overview
- Follow through the Quickstart and scaffold a new squid project using sqd init and a suitable template.
1. Model the data with a schema file
Start the development by defining the data schema in schema.graphql
in the squid root folder. The schema will be used both for the target database and for the GraphQL API. It consists of regular GraphQL type declarations annotated with custom directives to define:
- relations between the entities
- entity properties, property types and entity relations
- indexes to be created in the database
- the schema of the auto-generated GraphQL API
A full reference of the schema.graphql
dialect is available in the schema file section.
2. Generate TypeORM classes
The squid processor data handlers use TypeORM entities
to interact with the target database during the data processing. All necessary entity classes are
generated by the squid framework from schema.graphql
with
sqd codegen
By convention, the generated model classes are kept in src/model/generated
. Custom user-defined entities can
be added in src/model/index.ts
.
3. Generate the database migrations
For this step you need a clean Postgres database running locally:
# drop the old schema
sqd down
# start a clean Postgres in Docker
sqd up
Database schema changes (including initialization) are applied through migration files located at db/migrations
. Generate migrations as follows:
# compiles the code, removes the old migrations and replaces them with new ones
sqd migration:generate
Consult database migrations for more details.
4. Define the squid processor and the data handlers
A squid processor is a node.js process that fetches historical on-chain data, performs arbitrary transformations and saves the result into the target database schema defined above. By convention, the processor entry point is src/main.ts
and the main processor object is defined at src/processor.ts
.
EvmBatchProcessor
(imported from@subsquid/evm-processor
) is used for EVM chainsSubstrateBatchProcessor
(imported from@subsquid/substrate-processor
) is used for Substrate-based chains
5. Initialize a suitable processor instance
Configure the processor by defining:
- the archive endpoint
- indexing data range
- data to be extracted from the Archive
Example:
export const processor = new EvmBatchProcessor()
.setDataSource({
archive: lookupArchive('eth-mainnet'),
chain: 'https://eth-rpc.gateway.pokt.network'
})
.setFinalityConfirmation(75)
.addTransaction({
address: ['0x0000000000000000000000000000000000000000'],
range: {from: 6_000_000}
})
.setFields({
transaction: {
from: true,
input: true,
to: true
}
})
See EvmBatchProcessor configuration and SubstrateBatchProcessor configuration for details.
6. Generate Typescript facade classes to decode the obtained on-chain data
- For EVM data, use
evm-typegen
- For Substrate data, use
substrate-typegen
- For ink! smart contract data, use
ink-typegen
7. Define the processor batch handler for the processor.run()
call
Squid SDK embraces the batch-based programming model. Within a running processor, the .run()
method repeatedly applies a user-supplied batch handler function to the batches of data retrieved from an Archive. The method takes two arguments: a store adaptor for connecting to the database of choice and an async
batch handler function. The only argument of the batch handler is a context object that contains the batch data, some useful metadata and a store adapter reference. Batch data comes as an array of BlockData
objects whose interface depends on the processor flavor:
- For
EvmBatchProcessor
, see Block data for EVM - For
SubstrateBatchProcessor
, see the Block data for Substrate
Example:
processor.run(new TypeormDatabase(), async (ctx) => {
const entities: Map<string, FooEntity> = new Map()
// process a batch of data
for (let block of ctx.blocks) {
// The data is packed into blocks.
// Each block contains the requested items.
// It may also contain some items that were not requested,
// so the data must be filtered in the batch handler.
for (let log of block.logs) {
if (log.address === CONTRACT_ADDRESS && log.topic0 === fooEventTopic) {
// decode, extract and transform the evm log data
const { baz, bar, id } = extractLogData(log)
entities.set(id, new FooEntity({
id,
baz,
bar
}))
}
}
for (let txn of block.transactions) {
// filter and decode txn data if requested
}
}
// upsert data to the target db in single batch
await ctx.store.save([...entities.values()])
})
For an end-to-end walkthrough, see
8. Run the squid services
Run the processor with
sqd process
Run the GraphQL server in a separate terminal window with
sqd serve
The GraphQL playground will be available at http://localhost:4350/graphql
.
9. Deploy the squid
Follow the Deploy Squid section.
What's next?
- Learn from the Squids examples
- Get familiar with the typegen tools for EVM or Substrate