Squid generation tools
Subsquid provides tools for generating ready-to-use squids that index events and function calls of smart contracts. EVM/Solidity and WASM/ink! smart contracts are supported. The tools can be configured to make squids that save data to a PostgreSQL database or to a file-based dataset. All that is required is NodeJS, Subsquid CLI and, if your squid will be using a database, Docker.
Squid generation procedure is very similar for both contract types. Here are the steps:
Create a new blank squid with
sqd initusing a suitable template:
# for EVM/Solidity contracts
sqd init my-squid -t abi
# for WASM/ink! contracts
sqd init my-squid -t https://github.com/subsquid-labs/squid-ink-abi-template
Enter the squid folder and install the dependencies:
Write the configuration of the future squid to
squidgen.yaml. Retrieve any necessary contract ABIs and store them at
./abi. For simple use cases that only involve one contract and a database consider using CLI instead (documented in
npx squid-gen abi --help).
Generate the squid code:
npx squid-gen config squidgen.yaml
Prepare your squid for launching. If it is using a database, start a PostgreSQL container and generate migations:
If it is storing its data to a dataset, strip the project folder of database-related facilities that are no longer needed.
Test the complete squid by running it locally. Start a processor with
sqd process. If your squid will be serving GraphQL also run
sqd servein a separate terminal. Make sure that the squid saves the requested data to its target dataset:
- if it is serving GraphQL, visit the local GraphiQL playground;
- for PostgreSQL-based squids you can also connect to the database with
PGPASSWORD=postgres psql -U postgres -p 23798 -h localhost squidand take a look at the contents;
- if it is storing data to a file-based dataset, wait for the first filesystem sync then verify that all the expected files are present and contain the expected data.
At this point your squid is ready. You can run it on your own infrastructure or deploy it to Aquarium.
A valid config for the
squid-gen config is a YAML file with the following sections:
archive is an alias or an endpoint URL of an Archive. Find an appropriate public archive at the Supported networks page or in Archive Registry. Alternatively, self-host a private Archive (EVM/WASM).
target section describes how the scraped data should be stored. Set
to use a PostgreSQL database that can be presented to users as a GraphQL API or used as-is. Another option is to store the data to a file-based dataset.
contracts is a list of contracts to be indexed. Define the following fields for each contract:
- range (optional): block range to be indexed. An object with
toproperties, each of which can be omitted. Defaults to indexing the whole chain.
- abi (optional on EVM): path to the contract JSON ABI. If omitted for an EVM contract, the tool will attempt to fetch the ABI by address from the Etherscan API or a compatible alternative set by the
- proxy (EVM-only): when indexing a proxy contract for events or calls defined in the implementation, set this option to its address and the
addressoption to the address of the implementation contract. That way the tool will retrieve the ABI of the implementation and use it to index the output of the proxy.
- events (optional): a list of events to be indexed or a boolean value. Set to
trueto index all events defined in the ABI. Defaults to
false, meaning that no events are to be indexed.
- functions (EVM-only, optional): a list of functions the calls of which are to be indexed or a boolean value. Set to
trueto index calls of all functions defined in the ABI. Defaults to
false, meaning that no function calls are to be indexed.
etherscanApi (EVM-only, optional): Etherscan API-compatible endpoint to fetch contract ABI by a known address. Default: https://api.etherscan.io/.
Currently the only file-based data target type supported by
squid-gen packages is
parquet. When used, it requires that the
path field is also set alongside
path can be a local path or an URL pointing to a folder within a bucket on an S3-compatible cloud service.
file-store is in alpha stage. Known caveats:
If a S3 URL is used, then the S3 region, endpoint and user credentials will be taken from the default environment variables. Fill your
.envfile and/or set your Aquarium secrets accordingly.
Unlike their PostgreSQL-powered equivalents, the squids that use
file-storemay not write their data often. You may have to configure the
syncIntervalBlocksparameters of the
Databaseclass manually to strike an acceptable balance between the lag of the indexed data and the number of files in the resulting dataset. See Filesystem store overview for details.
Decimal(38)column type is used by the code generator to represent
uint256. This is done for compatibility reasons: very few tools seem to support reading wider decimals from Parquet files. If you're getting a lot of errors containing
value ... does not fit into Decimal(38, 0), consider replacing the
Decimal(38)column type with
At the moment, squids generated with file-based data targets contain a lot of facilities for managing the database and have to be stripped of them before use.
Strip the squid folder for
Steps to convert a squid made with a database-enabled template for use with
- Remove unneeded files and packages.
npm uninstall @subsquid/graphql-server @subsquid/typeorm-migration @subsquid/typeorm-store @subsquid/util-internal-json pg typeorm @subsquid/typeorm-codegen
commands.jsonwith the one from the file-store-parquet-example repo.
curl -o commands.json https://raw.githubusercontent.com/subsquid-labs/file-store-parquet-example/main/commands.json
squid.yaml, remove the
deploy.addonssection and replace the
cmd: [ "sleep", "3600" ]
- Install any required
# if target.type was `parquet`
npm install @subsquid/file-store-parquet
# if target.path was an S3 URL
npm install @subsquid/file-store-s3
LiquidationCallevents of the AAVE V2 Lending Pool contract, starting from block 11362579 when it was deployed. Save the results to PostgreSQL. Use the ABI located at
- name: aave-v2-pool
Index events and function calls by the DPX contract (a proxy) on Arbitrum, based on the ABI of the implementation contract retrieved from Arbiscan API. Save the results to Parquet files at './data'.
- name: dpx
Note: this example is known to run into the integer length issue described in the
file-storetargets section. One way to make it work is to widen all
Decimalcolumn types from 38 to 78 symbols:
sed -i -e 's/38/78/g' src/table.ts
Index all events and function calls of the Positions NFT and Factory contracts of Uniswap, send the results to the
uniswap-datafolder of the
- name: positions
- name: factory
- This example is also susceptible to the integer length issue and will drop at least two events if used as-is, without widening the column types.
- The generated squid requires some variables to be set to connect to S3. Here's an example of what
.envmay look like:
Transferevents emitted by a ERC20 contract on Shibuya, save results to PostgreSQL. Do not forget to use the ink-abi template!Note: you can get the ABI from the
- name: testToken
curl -o abi/erc20.json https://raw.githubusercontent.com/subsquid/squid-gen/master/tests/ink-erc20/abi/erc20.json