Create a WASM-indexing Squid

Objective

This tutorial will start off the squid template and go through all the necessary changes to index the events of a WASM contract developed with Ink!.

The Subsquid SDK natively supports only WASM contracts executed by the Contracts pallet. In particular, it's enabled by the following network runtimes:

• Shibuya (Astar testnet)
• Shiden (Kusama-cousin of Astar)
• Astar (a Polkadot parachain)

For this tutorial we will use a simple test ERC20-type token contract deployed to Shibuya at the address 0x5207202c27b646ceeb294ce516d4334edafbd771f869215cb070ba51dd7e2c72. Our squid will track all the token holders and account balances, together with the historical token transfers.

The final result of this tutorial is available in this repo. It can be used as a template for WASM-indexing squids.

Pre-requisites

Same as for the Quickstart

Run the template

Clone

https://github.com/subsquid/squid-template.git

and run the template:

npm ci
make build
make up
make process
# open a separate terminal for this next command
make serve

WASM tools

Subsquid SDK offers additional tooling for dealing with Ink contracts:

• @subsquid/ink-abi -- A performant library to decode the binary contract data using the contract ABI
• @subsquid/ink-typegen -- A tool to generate type-safe TypeScript classes and interfaces for the contract event and call data from the contract metadata

npm i @subsquid/ink-abi && npm i @subsquid/ink-typegen --save-dev

Since @subsquid/ink-typegen is only used to generate source files, we install it as a dev dependency.

Define the data schema

This part is not specific to WASM and is standard for all squids. To index ERC-20 token transfers, we will need to track:

• Ownership of tokens (a wallet and the current balance)
• Token transfers (with from, to and amount fields)

The Owner-Transfer relationship is one-to-many. We want our squid API to support filtering by the holder balance and the transfer amounts, so we throw in a bunch of indexes.

The schema.graphql file modelling the data above is straightforward:

type Owner @entity {
  id: ID!
  balance: BigInt! @index
}
 
type Transfer @entity {
  id: ID!
  from: Owner
  to: Owner
  amount: BigInt! @index
  timestamp: DateTime! @index
  block: Int!
}
 

Next, we generate TypeORM entity classes from the schema with the squid-typeorm-codegen tool:

make codegen

The generated entity classes can be found under src/model/generated.

To generate the database migrations matching the schema, we first drop the existing database and the existing migrations:

make down
rm -rf db/migrations/*.js

Next, we start a clean db, build the code and generate the new migrations matching the entities generated with squid-typeorm-codegen:

make up
make build
npx squid-typeorm-migration generate

ABI Definition and Wrapper

The Contracts pallet stores the contract execution logs (calls and events) in a binary format. The decoding of this data is contract-specific and is done with the help of an ABI file typically published by the contract developer. For our contract the data can be found here

The ink-typegen tool provided by Subsquid SDK generates the necessary boilerplate to decode the contract data. The generated classes will be later be used by the squid processor event handlers.

To follow the convention, we recommend keeping the ABI JSON file in the src/abi subfolder. To automatically generate TypeScript interfaces from an ABI definition, and decode event data, simply run this command from the project's root folder

npx squid-ink-typegen --abi src/abi/erc20.json --output src/abi/erc20.ts

The abi parameter points at the JSON file previously created, and the output parameter is the name of the file that will be generated by the command itself.

This command will automatically generate a TypeScript file named erc20.ts, under the src/abi subfolder, that defines data interfaces to represent output of the WASM events defined in the ABI, as well as functions necessary to decode these events (for example, see the decodeEvent function in the aforementioned file).

Define and Bind Event Handler(s)

The Subsquid SDK provides users with a processor class, named SubstrateProcessor or, in this specific case SubstrateBatchProcessor. The processor connects to the Shibuya Archive to get chain data.

The SubstrateBatchProcessor class exposes functions to configure it to request the Archive for specific on-chain data such as Substrate events, extrinsics, storage items etc. The Contracts pallet emits ContractEmitted events wrapping the logs emitted by the WASM contracts. The processor allows one to subscribe for such events emitted by a specific contract using one or multiple WASM handlers.

Configure Processor and Attach Handler

The src/processor.ts file is where the template project instantiates the SubstrateBatchProcessor class, configures it for execution, and attaches the handler functions. We need to make fundamental changes to the logic expressed in this code, starting from the configuration of the processor:

• we need to change the archive used to shibuya
• we need to remove the addEvent function call, and add addContractsContractEmitted instead, specifying the address of the contract we are interested in
• the logic defined in the processor.run() and below it, including the interfaces has to be replaced, as we no longer deal with Kusama balances transfers

Recall that we use with the contract deployed at 0x5207202c27b646ceeb294ce516d4334edafbd771f869215cb070ba51dd7e2c72 on Shibuya.

Look at this code snippet for the end result:

// src/processor.ts
import { lookupArchive } from "@subsquid/archive-registry"
import * as ss58 from "@subsquid/ss58"
import {BatchContext, BatchProcessorItem, SubstrateBatchProcessor} from "@subsquid/substrate-processor"
import {Store, TypeormDatabase} from "@subsquid/typeorm-store"
import {In} from "typeorm"
import * as erc20 from "./erc20"
import {Owner, Transfer} from "./model"
 
 
const CONTRACT_ADDRESS = '0x5207202c27b646ceeb294ce516d4334edafbd771f869215cb070ba51dd7e2c72'
 
 
const processor = new SubstrateBatchProcessor()
    .setDataSource({
        archive: lookupArchive("shibuya", { release: "FireSquid" })
    })
    .addContractsContractEmitted(CONTRACT_ADDRESS, {
        data: {
            event: {args: true}
        }
    } as const)
 
 
type Item = BatchProcessorItem<typeof processor>
type Ctx = BatchContext<Store, Item>
 
 
processor.run(new TypeormDatabase(), async ctx => {
    let txs = extractTransferRecords(ctx)
 
    let ownerIds = new Set<string>()
    txs.forEach(tx => {
        if (tx.from) {
            ownerIds.add(tx.from)
        }
        if (tx.to) {
            ownerIds.add(tx.to)
        }
    })
 
    let owners = await ctx.store.findBy(Owner, {
        id: In([...ownerIds])
    }).then(owners => {
        return new Map(owners.map(o => [o.id, o]))
    })
 
    let transfers = txs.map(tx => {
        let transfer = new Transfer({
            id: tx.id,
            amount: tx.amount,
            block: tx.block,
            timestamp: tx.timestamp
        })
 
        if (tx.from) {
            transfer.from = owners.get(tx.from)
            if (transfer.from == null) {
                transfer.from = new Owner({id: tx.from, balance: 0n})
                owners.set(tx.from, transfer.from)
            }
            transfer.from.balance -= tx.amount
        }
 
        if (tx.to) {
            transfer.to = owners.get(tx.to)
            if (transfer.to == null) {
                transfer.to = new Owner({id: tx.to, balance: 0n})
                owners.set(tx.to, transfer.to)
            }
            transfer.to.balance += tx.amount
        }
 
        return transfer
    })
 
    await ctx.store.save([...owners.values()])
    await ctx.store.insert(transfers)
})
 
 
interface TransferRecord {
    id: string
    from?: string
    to?: string
    amount: bigint
    block: number
    timestamp: Date
}
 
 
function extractTransferRecords(ctx: Ctx): TransferRecord[] {
    let records: TransferRecord[] = []
    for (let block of ctx.blocks) {
        for (let item of block.items) {
            if (item.name == 'Contracts.ContractEmitted' && item.event.args.contract == CONTRACT_ADDRESS) {
                let event = erc20.decodeEvent(item.event.args.data)
                if (event.__kind == 'Transfer') {
                    records.push({
                        id: item.event.id,
                        from: event.from && ss58.codec(5).encode(event.from),
                        to: event.to && ss58.codec(5).encode(event.to),
                        amount: event.value,
                        block: block.header.height,
                        timestamp: new Date(block.header.timestamp)
                    })
                }
            }
        }
    }
    return records
}

The extractTransferRecords function generates a list of TransferRecord interfaces, containing the data we need to fill the models we have defined with our schema. This data is extracted from the events found in the BatchContext. It is then used in the main body of the arrow function used as an argument of the .run() function call to fetch or create the Owners on the database and create a Transfer instance for every event found in the context.

All of this data is then saved on the database at the very end of the function, all in one go. This is to increase the performance, by reducing the I/O towards the database.

info

As you can see in the extractTransferRecords function, we loop over the blocks we have been given in the BatchContext and loop over the items contained in them. The if checks are redundant when there's a single handler but will be needed when the processor has multiple handlers and so block.items will contain a mix of different event and extrinsic data.

Launch the Project

To launch the processor (this will block the current terminal), you can run the following command:

make process

Finally, in a separate terminal window, launch the GraphQL server:

make serve

Visit localhost:4350/graphql to access the GraphiQl console. From this window, you can perform queries such as this one, to find out the account owners with the biggest balances:

query MyQuery {
  transfersConnection(orderBy: id_ASC) {
    totalCount
  }
}

query MyQuery {
  owners(limit: 10, where: {}, orderBy: balance_DESC) {
    balance
    id
  }
}

Or this other one, looking up the tokens owned by a given owner:

query MyQuery {
  transfers(limit: 10, orderBy: amount_DESC) {
    amount
    block
    id
    timestamp
    to {
      balance
      id
    }
    from {
      balance
      id
    }
  }
}

Have some fun playing around with queries, after all, it's a playground!