Migrate to FireSquid

Overview

This guide goes through the necessary steps to migrate v5 squids to FireSquid and provide a summary of the new features. For the sake of simplicity, this guide is built upon SubstrateProcessor as it has the same execution model as the previous versions. Once the guide is complete, we recommend refactoring the squid to a much more performant SubstrateBatchProcessor as the next step.

1 Package version updates for FireSquid

The FireSquid release requires version bumps for most of the Subsquid packages. Consult squid-template for a reference setup.

The following versions should be updated to latest:

npm i @subsquid/cli@latest \
@subsquid/substrate-processor@latest \
@subsquid/typeorm-store@latest \
@subsquid/substrate-typegen@latest \
@subsquid/substrate-metadata-explorer@latest \
@subsquid/typeorm-migration@latest \
@subsquid/typeorm-codegen@latest 

The following new packages will be installed:

• @subsquid/typeorm-migration:0.1.x. Separate package to replace sqd db commands with squid-typeorm-migration
• @subsquid/typeorm-codegen:0.0.x. Separate package to replace sqd codegen with squid-typeorm-codegen
• @subsquid/typeorm-store:0.1.x. Separate package for the TypeORM implementation of the processor store

CLI commands updates

• Use npx squid-typeorm-migration instead of npx sqd db
• Use npx squid-typeorm-codegen instead of npx sqd codegen
• The command npx sqd squid tail is deprecated. Use npx sqd squid logs

2 Pluggable ctx.store

The store interface provided by the mapping context is now configured by the SubstrateProcessor constructor argument. The type of ctx.store is inferred from it.

The new package @subsquid/typeorm-store provides two implementations of the Database interface accepted by SubstrateProcessor.

FullTypeormDatabase

FullTypeormDatabase works exactly as in v5 providing a plain EntityManager as a store, albeit without .get() method

Usage:

import { FullTypeormDatabase } from '@subsquid/typeorm-store'
import { EntityManager } from 'typeorm'

// in a handler:
processor.addEventHandler('Some.Event', ctx => {  
  ctx.store.save(new FooEntity({ id: 1}))
})

In the snippet above, ctx.store passed to the handlers will be of type EntityManager.

TypeormDatabase

TypeormDatabase (recommended) provides ctx.store, which is

• Lazy (no transaction is opened if no data is read or written to the store). This is useful e.g. when one want to subscribe for some frequent events but only interested in the cases when the events were emitted under specific conditions (e.g. by a specific contract)
• Looks like a stripped down version of EntityManager (no .query(), no cascading saves)
• Much faster for data updates
• The schema name and the transaction isolation level can be passed as an optional constructor argument

The performance improvements come from the fact that Store implements .save() via upsert and all the data manipulation methods are translated into a single SQL statement (compared to multiple SQL statements generated by EntityManager)

Usage:

import { Store, TypeormDatabase } from '@subsquid/typeorm-store'

// in a handler:
processor.addEventHandler('Some.Event', ctx => {  
  ctx.store.save(new FooEntity({ id: 1}))
})

In the snippet above, ctx.store passed to the handlers will be of type Store.

3 Change the processor initialization

Initialize the processor

For SubstrateProcessor one MUST specify a Database implementation (i.e. choose one of the above) and change dataSource to a FireSquid archive with a new version of lookupArchive. The config statements can now be chained. Other processor's parameters haven't changed.

const processor = new SubstrateProcessor(new TypeormDatabase())
    .setBatchSize(500)
    .setDataSource({
        // Lookup archive by the network name in the Subsquid registry
        archive: lookupArchive("kusama", { release: "FireSquid" })

        // Use archive created by archive/docker-compose.yml
        // archive: 'http://localhost:8888/graphql'
    })

Rename the handlers

All the pallets' names now start from uppercase and must follow the same naming convention as defined by the substrate runtime. As a rule of thumb, pallet and event names are upper-cased, while call names are lowercase. Instead of ExtrinsicHandler one now should use CallHandler which is triggered even if the call was wrapped in a sudo, proxy or a batch.util extrinsic:

processor.addEventHandler('Balances.Transfer', ... 
processor.addCallHandler('Balances.transfer_keep_alive', ...

Change the handlers context

One SHOULD specify a data projection for the handler context. Only the selected data field will be fetched from the Archive. If it is skipped, the context will contain the default set of data fields.

One can see all the supported projection options here and rely on auto-complete assist from an IDE.

Further, one MUST explicitly specify a Store interface for handlers defined as explicit external functions. Arrow-function handlers will infer it automatically.

Example: an arrow-function handler

processor.addEventHandler('Balances.Transfer', {
  // he we specify which data will be fetched and passed
  // to the handler context
    data: {
        event: {args: true}
    }
  }, async ctx => {
    // Handler's business logic, only ctx.event.args is available
    // The type of ctx is inferred
})

Example: an explicitly defined handler

processor.addEventHandler('Balances.Transfer', handleTransferEvent)
// An explicit typing for the handler context
async function handleTransferEvent(ctx: EventHandlerContext<Store, {event: {args: true}}>){
    // Handler's business logic, only ctx.event.args is available
}

Typegen

The chainVersions config field has been renamed to specVersions. It now accepts either

• an Archive endpoint
• a jsonl file generated by squid-substrate-metadata-explorer(1)

With an archive:

typegen.json

{
  "outDir": "src/types",
  "specVersions": "https://kusama.archive.subsquid.io/graphql",
  "events": [
    "Balances.Transfer",
    "Balances.Deposit"
  ],
  "calls": [],
  "storage": []
}

With squid-substrate-metadata-explorer, first generate the metadata versions file:

npx squid-substrate-metadata-explorer \
  --chain wss://kusama-rpc.polkadot.io \
  --out kusamaVersions.jsonl

and then provide in the typegen config

typegen.json

{
  "outDir": "src/types",
  "specVersions": "kusamaVersions.jsonl",
  "events": [
    "Balances.Transfer",
    "Balances.Deposit"
  ],
  "calls": [],
  "storage": []
}

Once the config is ready, (re)generate the typings for the requested events, calls and storage items:

make typegen

What's next?

Get familiar with SubstrateBatchProcessor and refactor the squid to get up to a 100x speed up of the squid synchronization time.