How to launch an Archive
Quick how-to on running an Archive for a specific blockchain.
Overview
This quick Recipe covers how to launch an Archive for a blockchain that is already covered by Subsquid and how to run one for a new blockchain.
Requirements
In order to run an Archive, it is required to have Docker installed. For a quick installation guide, head over to the dedicated page.
Squid Archive repository
Launching an Archive is as easy as visiting our dedicated Archive setup repository and checking if the blockchain you want to synchronize is listed. Let's say Equilibrium, in the context of this example.
Archives list in the repository
If that is the case, simply clone the repository
1
git clone [email protected]:subsquid/squid-archive-setup.git
Copied!
Launch the Archive locally
Navigate to the corresponding folder in a command line console window and run
1
docker-compose up -d
Copied!
The -d parameter is advised to avoid locking the console window and being flooded by log messages.
This is what launching an Archive locally should look like
To check that everything went well and verify the ports, launch the following command
1
docker container ls
Copied!
The result should look like this (you could have more entries, if you launched other containers in a separate context):
List of running containers, their names, and ports
Now the last check to perform is visit the console, by typing this URL
http://localhost:4010/console in a browser.Use local Archive in a Squid API
In order for a Squid API to use the Archive we just launched locally, it is necessary to change the values in the
setDataSource function call, typically in the processor.ts file (if the project was not differently customized). The archive value should be set to http://localhost:4010/v1/graphql:processor.ts
1
// ...
2
3
processor.setDataSource({
4
archive: "http://localhost:4010/v1/graphql",
5
chain: "<insert chain WSS URL here>",
6
});
7
8
// ...
Copied!
Running in production environment
The provided docker compose setup is a minimal configuration best suitable for dev and testing environments. For a stable production deployment we recommend the following.
- Use a private gRPC endpoint (
WS_PROVIDER_ENDPOINT_URIenv variable in the docker file) - Use managed Postgres database with non-root access (
DB_*env variables in the docker file) - Increase
WORKERS_NUMBERenvironment variable to speed up the syncing. Usually somewhere between 5-50 workers is a sweet spot depending on the gRPC endpoint capacity.
To reliably run an Archive we recommend 16GB RAM and modern CPU. Database storage requirements depend on the size of the network. A rule of thumb is to reserve around 100 kb per block, so e.g. for Kusama with ~10M blocks one needs about 1Tb for Postgres storage.
Caveats for 🍏 M1 Macs
- 2.checkout v5 branch
- 3.build the gateway image with:
1
./scripts/docker-build.sh --target indexer-gateway -t subsquid/hydra-indexer-gateway:5
Copied!
After that, you can run docker-compose as usual.
Launch an Archive for a new blockchain
If a specific blockchain is not listed in the repository, it is possible to add it by contributing to the repository itself, following these steps:
- 1.Fork the repository
- 2.Create a folder dedicated to the new blockchain
- 3.Copy one of the
docker-compose.ymlfile from another folder- Edit the
WS_PROVIDER_ENDPOINT_URIenvironment variable to the correct endpoint for the given chain - Make sure to create a types bundle file named as the one mentioned in the
BUNDLE_TYPESenvironment variable (see next point)
- 4.The Archive needs to know which types the blockchain Runtime is using, and to instruct it, a types bundle JSON file needs to be passed in.
- 5.Create a pull-request towards the main repository
Launch Archives for EVM-compatible blockchain
In the case of an EVM-compatible blockchain, you'd want to fully leverage the native support for EVM logs to process contracts, by making sure that the Archive is extracting that data from the blockchain itself.
To do so, simply follow the procedure above, but make sure to check that the images for the
indexer and indexer-gateway are, respectively:subsquid/hydra-evm-indexer:5subsquid/hydra-evm-indexer-gateway:5
Like so:
docker-compose.yml
1
version: "3.4"
2
services:
3
db:
4
image: postgres:12
5
restart: always
6
volumes:
7
- /var/lib/postgresql/data
8
environment:
9
POSTGRES_USER: postgres
10
POSTGRES_PASSWORD: postgres
11
indexer:
12
image: subsquid/hydra-evm-indexer:5
13
restart: unless-stopped
14
environment:
15
- WORKERS_NUMBER=5
16
- DB_NAME=indexer
17
- DB_HOST=db
18
- DB_USER=postgres
19
- DB_PASS=postgres
20
- DB_PORT=5432
21
- REDIS_URI=redis://redis:6379/0
22
- BLOCK_HEIGHT=0 # starting block height
23
- FORCE_HEIGHT=true
24
- WS_PROVIDER_ENDPOINT_URI= # chain WSS
25
depends_on:
26
- db
27
- redis
28
command: >
29
sh -c "yarn db:bootstrap && yarn start:prod"
30
ports:
31
- 9090:9090
32
indexer-gateway:
33
image: subsquid/hydra-evm-indexer-gateway:5
34
restart: unless-stopped
35
depends_on:
36
- redis
37
- db
38
- indexer-status-service
39
- indexer
40
ports:
41
- "4010:8080"
42
environment:
43
- DEV_MODE=true
44
- DB_NAME=indexer
45
- DB_HOST=db
46
- DB_USER=postgres
47
- DB_PASS=postgres
48
- DB_PORT=5432
49
- HYDRA_INDEXER_STATUS_SERVICE=http://indexer-status-service:8081/status
50
indexer-status-service:
51
image: subsquid/hydra-indexer-status-service:5
52
restart: unless-stopped
53
depends_on:
54
- redis
55
environment:
56
REDIS_URI: redis://redis:6379/0
57
PORT: 8081
58
redis:
59
image: redis:6.0-alpine
60
restart: always
61
ports:
62
- "6379"
Copied!
Last modified 4d ago