Skip to main content

Indexer from scratch

Here's an example of how SDK packages can be combined into a working indexer (called squid).

This page goes through all the technical details to make the squid architecture easier to understand. If you would like to get to a working indexer ASAP, bootstrap from a template.

USDT transfers API‚Äč

Pre-requisites: NodeJS 16.x or newer, Docker.

Suppose the task is to track transfers of USDT on Ethereum, then save the resulting data to PostgreSQL and serve it as a GraphQL API. From this description we can immediately put together a list of packages:

  • @subsquid/evm-processor - for retrieving Ethereum data
  • the triad of @subsquid/typeorm-store, @subsquid/typeorm-codegen and @subsquid/typeorm-migration - for saving data to PostgreSQL
  • @subsquid/graphql-server

We also assume the following choice of optional packages:

  • @subsquid/evm-typegen - for decoding Ethereum data and useful constants such as event topic0 values

To make the indexer, follow these steps:

  1. Create a new folder and initialise a new project

    • create package.json

      npm init
    • add .gitignore

      touch .gitignore
      .gitignore
      /node_modules

      package-lock.json

      .env
  2. install the packages:

    npm i dotenv typeorm @subsquid/evm-processor @subsquid/typeorm-store @subsquid/typeorm-migration @subsquid/graphql-server
    npm i typescript @subsquid/typeorm-codegen @subsquid/evm-typegen --save-dev
  3. Add a minimal tsconfig.json:

    tsconfig.json
    {
    "compilerOptions": {
    "rootDir": "src",
    "outDir": "lib",
    "module": "commonjs",
    "target": "es2020",
    "esModuleInterop": true,
    "skipLibCheck": true,
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true
    }
    }
  4. Define the schema for both the database and the core GraphQL API in schema.graphql:

    schema.graphql
    type Transfer @entity {
    id: ID!
    from: String! @index
    to: String! @index
    value: BigInt!
    }
  5. Generate TypeORM classes based on the schema:

    npx squid-typeorm-codegen

    The TypeORM classes are now available at src/model/index.ts.

  6. Prepare the database:

    • create .env and docker-compose.yaml files
      .env
      DB_NAME=squid
      DB_PORT=23798
      docker-compose.yaml
      version: "3"
      services:
      db:
      image: postgres:15
      environment:
      POSTGRES_DB: "${DB_NAME}"
      POSTGRES_PASSWORD: postgres
      ports:
      - "${DB_PORT}:5432"
    • start the database container
      docker compose up -d
    • compile the TypeORM classes
      npx tsc
    • generate the migration file
      npx squid-typeorm-migration generate
    • apply the migration with
      npx squid-typeorm-migration apply
  7. Generate utility classes for decoding USDT contract data based on its ABI downloaded from Etherscan:

    npx \
    squid-evm-typegen \
    src/abi \
    0xdAC17F958D2ee523a2206206994597C13D831ec7#usdt

    The utility classes are now available at src/abi/usdt.ts

  8. Tie all the generated code together with a src/main.ts executable with the following code blocks:

    • Imports
      import { EvmBatchProcessor } from '@subsquid/evm-processor'
      import { TypeormDatabase } from '@subsquid/typeorm-store'
      import * as usdtAbi from './abi/usdt'
      import { Transfer } from './model'
    • EvmBatchProcessor object definition
      const processor = new EvmBatchProcessor()
      .setGateway('https://v2.archive.subsquid.io/network/ethereum-mainnet')
      .setRpcEndpoint({
      // set RPC endpoint in .env
      url: process.env.RPC_ETH_HTTP,
      rateLimit: 10
      })
      .setFinalityConfirmation(75) // 15 mins to finality
      .addLog({
      address: [ '0xdAC17F958D2ee523a2206206994597C13D831ec7' ],
      topic0: [ usdtAbi.events.Transfer.topic ]
      })
    • TypeormDatabase object definition
      const db = new TypeormDatabase()
    • A call to processor.run() with an inline definition of the batch handler
      processor.run(db, async ctx => {
      const transfers: Transfer[] = []
      for (let block of ctx.blocks) {
      for (let log of block.logs) {
      let {from, to, value} = usdtAbi.events.Transfer.decode(log)
      transfers.push(new Transfer({
      id: log.id,
      from, to, value
      }))
      }
      }
      await ctx.store.insert(transfers)
      })
      Note how supplying a TypeormDatabase to the function caused ctx.store to be a PostgreSQL-compatible Store object.
  9. Compile the project and start the processor process

    npx tsc
    node -r dotenv/config lib/main.js
  10. In a separate terminal, configure the GraphQL port and start the GraphQL server:

.env
 DB_NAME=squid
DB_PORT=23798
+GQL_PORT=4350
npx squid-graphql-server

The finished GraphQL API with GraphiQL is available at localhost:4350/graphql.