Skip to main content
Version: ArrowSquid

Development flow

Below is a general outline of the squid development steps.

0. Prerequisites​

  • Get familiar with squids and Archives by reading the Overview
  • Follow through the Quickstart and scaffold a new squid project using sqd init and a suitable template.

1. Model the data with a schema file​

Start the development by defining the data schema in schema.graphql in the squid root folder. The schema will be used both for the target database and for the GraphQL API. It consists of regular GraphQL type declarations annotated with custom directives to define:

  • relations between the entities
  • entity properties, property types and entity relations
  • indexes to be created in the database
  • the schema of the auto-generated GraphQL API

A full reference of the schema.graphql dialect is available in the schema file section.

2. Generate TypeORM classes​

The squid processor data handlers use TypeORM entities to interact with the target database during the data processing. All necessary entity classes are generated by the squid framework from schema.graphql with 

sqd codegen

By convention, the generated model classes are kept in src/model/generated. Custom user-defined entities can be added in src/model/index.ts.

3. Generate the database migrations​

For this step you need a clean Postgres database running locally:

# drop the old schema
sqd down
# start a clean Postgres in Docker
sqd up

Database schema changes (including initialization) are applied through migration files located at db/migrations. Generate migrations as follows:

# compiles the code, removes the old migrations and replaces them with new ones
sqd migration:generate

Consult database migrations for more details.

4. Define the squid processor and the data handlers​

A squid processor is a node.js process that fetches historical on-chain data, performs arbitrary transformations and saves the result into the target database schema defined above. By convention, the processor entry point is src/main.ts and the main processor object is defined at src/processor.ts.

5. Initialize a suitable processor instance​

Configure the processor by defining:

  • the archive endpoint
  • indexing data range
  • data to be extracted from the Archive

Example:

src/processor.ts
export const processor = new EvmBatchProcessor()
  .setDataSource({
    archive: lookupArchive('eth-mainnet'),
    chain: 'https://eth-rpc.gateway.pokt.network'
  })
  .setFinalityConfirmation(75)
  .addTransaction({
    address: ['0x0000000000000000000000000000000000000000'],
    range: {from: 6_000_000}
  })
  .setFields({
    transaction: {
      from: true,
      input: true,
      to: true
    }
  })

See EvmBatchProcessor configuration and SubstrateBatchProcessor configuration for details.

6. Generate Typescript facade classes to decode the obtained on-chain data​

7. Define the processor batch handler for the processor.run() call​

Squid SDK embraces the batch-based programming model. Within a running processor, the .run() method repeatedly applies a user-supplied batch handler function to the batches of data retrieved from an Archive. The method takes two arguments: a store adaptor for connecting to the database of choice and an async batch handler function. The only argument of the batch handler is a context object that contains the batch data, some useful metadata and a store adapter reference. Batch data comes as an array of BlockData objects whose interface depends on the processor flavor:

Example:

src/main.ts
processor.run(new TypeormDatabase(), async (ctx) => {
  const entities: Map<string, FooEntity> = new Map()
  // process a batch of data 
  for (let block of ctx.blocks) {
    // The data is packed into blocks.
    // Each block contains the requested items.
    // It may also contain some items that were not requested,
    // so the data must be filtered in the batch handler.
    for (let log of block.logs) {
      if (log.address === CONTRACT_ADDRESS && log.topic0 === fooEventTopic) {
        // decode, extract and transform the evm log data
        const { baz, bar, id } = extractLogData(log)
        entities.set(id, new FooEntity({
          id,
          baz,
          bar
        }))
      }
    }
    for (let txn of block.transactions) {
      // filter and decode txn data if requested
    }
  }
  // upsert data to the target db in single batch
  await ctx.store.save([...entities.values()])
})

For an end-to-end walkthrough, see

8. Run the squid services​

Run the processor with

sqd process

Run the GraphQL server in a separate terminal window with

sqd serve

The GraphQL playground will be available at http://localhost:4350/graphql.

9. Deploy the squid​

Follow the Deploy Squid section.

What's next?​