Skip to main content

Development flow

Below is a general outline of the squid development steps.

0. Prerequisites​

  • Get familiar with squids and Archives by reading the Overview
  • Follow through the Quickstart and scaffold a new squid project using sqd init and a suitable template.

1. Model the data with a schema file​

Start the development by defining the data schema in schema.graphql in the squid root folder. The schema will be used both for the target database and for the GraphQL API. It consists of regular GraphQL type declarations annotated with custom directives to define:

  • relations between the entities
  • entity properties, property types and entity relations
  • indexes to be created in the database
  • the schema of the auto-generated GraphQL API

A full reference of the schema.graphql dialect is available in the schema file section.

2. Generate TypeORM classes​

The squid processor data handlers use TypeORM entities to interact with the target database during the data processing. All necessary entity classes are generated by the squid framework from schema.graphql with 

sqd codegen

By convention, the generated model classes are kept in src/model/generated. Custom user-defined entities can be added in src/model/index.ts.

3. Generate the database migrations​

For this step you need a clean Postgres database running locally:

# drop the old schema
sqd down
# start a clean Postgres in Docker
sqd up

Database schema changes (including initialization) are applied through migration files located at db/migrations. Generate migrations as follows:

# remove the old migrations
sqd migration:clean
# compile the squid code, notably the TypeORM model classes
sqd build
# generate the new schema migrations at db/migrations
sqd migration:generate

Consult database migrations for more details.

4. Define the squid processor and the data handlers​

A squid processor is a node.js process that fetches historical on-chain data, performs arbitrary transformations and saves the result into the target database schema defined above. By convention, the processor entry point is src/processor.ts.

5. Initialize a suitable processor instance​

Configure the processor by defining:

  • the archive endpoint
  • indexing data range
  • data to be extracted from the Archive

Example:

const processor = new EvmBatchProcessor()
  .setDataSource({
    archive: 'https://eth.archive.subsquid.io',
  })
  .addTransaction([
    '0x0000000000000000000000000000000000000000'
  ], {
    range: {
      from: 6_000_000
    },
    data: {
      transaction: {
        from: true,
        input: true,
        to: true
      }
    }
  });

See EvmBatchProcessor configuration and SubstrateBatchProcessor configuration for details.

6. Generate Typescript facade classes to decode the obtained on-chain data​

7. Define the processor batch handler for the processor.run() call​

Squid SDK embraces the batch-based programming model. Within a running processor, the .run() method repeatedly applies a user-supplied batch handler function to the batches of data retrieved from an Archive. The method takes two arguments: a store adaptor for connecting to the database of choice and an async batch handler function. The only argument of the batch handler is a context object that contains the batch data, some useful metadata and a store adapter reference. Its interface slightly varies depending on the processor flavor:

Example:

processor.run(new TypeormDatabase(), async (ctx) => {
  const entities: Map<string, FooEntity> = new Map();
  // process a batch of data 
  for (const c of ctx.blocks) {
    // the data is packed into blocks, 
    // each block contains only the requested items
    for (const irem of c.items) {
      if(e.kind === 'evmLog') {
        // decode, extract and transform the evm log data
        const { baz, bar, id } = extractLogData(e.evmLog)
        entities.set(id, new FooEntity({
          id,
          baz,
          bar
        })) 
      }
      if (e.kind === 'transaction') {
        // decode tx data if requested
      }
    }
  }
  // upsert data to the target db in single batch
  await ctx.store.save([...entities.values()])
});

For an end-to-end walkthrough, see

8. Run the squid services​

Run the processor with

sqd process

Run the GraphQL server in a separate terminal window with

sqd serve

The GraphQL playground will be available at http://localhost:4350/graphql.

9. Deploy the squid​

Follow the Deploy Squid section.

What's next?​