Create a WASM-indexing Squid
Objective
This tutorial will start off the squid template and go through all the necessary changes to index the events of a WASM contract developed with Ink!.
The Subsquid SDK natively supports only WASM contracts executed by the Contracts pallet. In particular, it's enabled by the following network runtimes:
Shibuya
(Astar
testnet)Shiden
(Kusama
-cousin ofAstar
)Astar
(aPolkadot
parachain)
For this tutorial we will use a simple test ERC20-type token contract deployed to Shibuya at the address 0x5207202c27b646ceeb294ce516d4334edafbd771f869215cb070ba51dd7e2c72
. Our squid will track all the token holders and account balances, together with the historical token transfers.
The final result of this tutorial is available in this repo. It can be used as a template for WASM-indexing squids.
Pre-requisites
Same as for the Quickstart
Run the template
Clone
https://github.com/subsquid/squid-template.git
and run the template:
npm ci
make build
make up
make process
# open a separate terminal for this next command
make serve
WASM tools
Subsquid SDK offers additional tooling for dealing with Ink contracts:
@subsquid/ink-abi
-- A performant library to decode the binary contract data using the contract ABI@subsquid/ink-typegen
-- A tool to generate type-safe TypeScript classes and interfaces for the contract event and call data from the contract metadata
npm i @subsquid/ink-abi && npm i @subsquid/ink-typegen --save-dev
Since @subsquid/ink-typegen
is only used to generate source files, we install it as a dev dependency.
Define the data schema
This part is not specific to WASM and is standard for all squids. To index ERC-20 token transfers, we will need to track:
- Ownership of tokens (a wallet and the current balance)
- Token transfers (with
from
,to
andamount
fields)
The Owner-Transfer relationship is one-to-many. We want our squid API to support filtering by the holder balance and the transfer amounts, so we throw in a bunch of indexes.
The schema.graphql
file modelling the data above is straightforward:
type Owner @entity {
id: ID!
balance: BigInt! @index
}
type Transfer @entity {
id: ID!
from: Owner
to: Owner
amount: BigInt! @index
timestamp: DateTime! @index
block: Int!
}
Next, we generate TypeORM
entity classes from the schema with the squid-typeorm-codegen
tool:
make codegen
The generated entity classes can be found under src/model/generated
.
To generate the database migrations matching the schema, we first drop the existing database and the existing migrations:
make down
rm -rf db/migrations/*.js
Next, we start a clean db, build the code and generate the new migrations matching the entities generated with squid-typeorm-codegen
:
make up
make build
npx squid-typeorm-migration generate
ABI Definition and Wrapper
The Contracts
pallet stores the contract execution logs (calls and events) in a binary format. The decoding of this data is
contract-specific and is done with the help of an ABI file typically published by the contract developer. For our contract the data can be found here
The ink-typegen
tool provided by Subsquid SDK generates the necessary boilerplate to decode the contract data. The generated classes will be later be used by the squid processor event handlers.
To follow the convention, we recommend keeping the ABI JSON file in the src/abi
subfolder. To automatically generate TypeScript interfaces from an ABI definition, and decode event data, simply run this command from the project's root folder
npx squid-ink-typegen --abi src/abi/erc20.json --output src/abi/erc20.ts
The abi
parameter points at the JSON file previously created, and the output
parameter is the name of the file that will be generated by the command itself.
This command will automatically generate a TypeScript file named erc20.ts
, under the src/abi
subfolder, that defines data interfaces to represent output of the WASM events defined in the ABI, as well as functions necessary to decode these events (for example, see the decodeEvent
function in the aforementioned file).
Define and Bind Event Handler(s)
The Subsquid SDK provides users with a processor class, named SubstrateProcessor
or, in this specific case SubstrateBatchProcessor
. The processor connects to the Shibuya Archive to get chain data.
The SubstrateBatchProcessor
class exposes functions to configure it to request the Archive for specific on-chain data such as Substrate events, extrinsics, storage items etc. The Contracts
pallet emits ContractEmitted
events wrapping the logs emitted by the WASM contracts. The processor allows one to subscribe for such events emitted by a specific contract using one or multiple WASM handlers.
Configure Processor and Attach Handler
The src/processor.ts
file is where the template project instantiates the SubstrateBatchProcessor
class, configures it for execution, and attaches the handler functions. We need to make fundamental changes to the logic expressed in this code, starting from the configuration of the processor:
- we need to change the archive used to
shibuya
- we need to remove the
addEvent
function call, and addaddContractsContractEmitted
instead, specifying the address of the contract we are interested in - the logic defined in the
processor.run()
and below it, including the interfaces has to be replaced, as we no longer deal with Kusama balances transfers
Recall that we use with the contract deployed at 0x5207202c27b646ceeb294ce516d4334edafbd771f869215cb070ba51dd7e2c72
on Shibuya
.
Look at this code snippet for the end result:
// src/processor.ts
import { lookupArchive } from "@subsquid/archive-registry"
import * as ss58 from "@subsquid/ss58"
import {BatchContext, BatchProcessorItem, SubstrateBatchProcessor} from "@subsquid/substrate-processor"
import {Store, TypeormDatabase} from "@subsquid/typeorm-store"
import {In} from "typeorm"
import * as erc20 from "./erc20"
import {Owner, Transfer} from "./model"
const CONTRACT_ADDRESS = '0x5207202c27b646ceeb294ce516d4334edafbd771f869215cb070ba51dd7e2c72'
const processor = new SubstrateBatchProcessor()
.setDataSource({
archive: lookupArchive("shibuya", { release: "FireSquid" })
})
.addContractsContractEmitted(CONTRACT_ADDRESS, {
data: {
event: {args: true}
}
} as const)
type Item = BatchProcessorItem<typeof processor>
type Ctx = BatchContext<Store, Item>
processor.run(new TypeormDatabase(), async ctx => {
let txs = extractTransferRecords(ctx)
let ownerIds = new Set<string>()
txs.forEach(tx => {
if (tx.from) {
ownerIds.add(tx.from)
}
if (tx.to) {
ownerIds.add(tx.to)
}
})
let owners = await ctx.store.findBy(Owner, {
id: In([...ownerIds])
}).then(owners => {
return new Map(owners.map(o => [o.id, o]))
})
let transfers = txs.map(tx => {
let transfer = new Transfer({
id: tx.id,
amount: tx.amount,
block: tx.block,
timestamp: tx.timestamp
})
if (tx.from) {
transfer.from = owners.get(tx.from)
if (transfer.from == null) {
transfer.from = new Owner({id: tx.from, balance: 0n})
owners.set(tx.from, transfer.from)
}
transfer.from.balance -= tx.amount
}
if (tx.to) {
transfer.to = owners.get(tx.to)
if (transfer.to == null) {
transfer.to = new Owner({id: tx.to, balance: 0n})
owners.set(tx.to, transfer.to)
}
transfer.to.balance += tx.amount
}
return transfer
})
await ctx.store.save([...owners.values()])
await ctx.store.insert(transfers)
})
interface TransferRecord {
id: string
from?: string
to?: string
amount: bigint
block: number
timestamp: Date
}
function extractTransferRecords(ctx: Ctx): TransferRecord[] {
let records: TransferRecord[] = []
for (let block of ctx.blocks) {
for (let item of block.items) {
if (item.name == 'Contracts.ContractEmitted' && item.event.args.contract == CONTRACT_ADDRESS) {
let event = erc20.decodeEvent(item.event.args.data)
if (event.__kind == 'Transfer') {
records.push({
id: item.event.id,
from: event.from && ss58.codec(5).encode(event.from),
to: event.to && ss58.codec(5).encode(event.to),
amount: event.value,
block: block.header.height,
timestamp: new Date(block.header.timestamp)
})
}
}
}
}
return records
}
The extractTransferRecords
function generates a list of TransferRecord
interfaces, containing the data we need to fill the models we have defined with our schema. This data is extracted from the events found in the BatchContext
. It is then used in the main body of the arrow function used as an argument of the .run()
function call to fetch or create the Owner
s on the database and create a Transfer
instance for every event found in the context.
All of this data is then saved on the database at the very end of the function, all in one go. This is to increase the performance, by reducing the I/O towards the database.
info
As you can see in the extractTransferRecords
function, we loop over the blocks we have been given in the BatchContext
and loop over the items contained in them. The if
checks are redundant when there's a single handler but will be needed when the processor has multiple handlers and so block.items
will contain a mix of different event and extrinsic data.
Launch the Project
To launch the processor (this will block the current terminal), you can run the following command:
make process
Finally, in a separate terminal window, launch the GraphQL server:
make serve
Visit localhost:4350/graphql
to access the GraphiQl console. From this window, you can perform queries such as this one, to find out the account owners with the biggest balances:
query MyQuery {
transfersConnection(orderBy: id_ASC) {
totalCount
}
}
query MyQuery {
owners(limit: 10, where: {}, orderBy: balance_DESC) {
balance
id
}
}
Or this other one, looking up the tokens owned by a given owner:
query MyQuery {
transfers(limit: 10, orderBy: amount_DESC) {
amount
block
id
timestamp
to {
balance
id
}
from {
balance
id
}
}
}
Have some fun playing around with queries, after all, it's a playground!