Relayer
This section contains instructions on how to setup the rust relayer "hermes" and relay IBC packets between bitsong-2b and other IBC-enabled networks.
Relayer Tutorial
Hardware specs:
16+ vCPUs or Intel or AMD 16 core CPU
at least 64GB RAM
4TB+ nVME drives
To assist operators in setting up relayers, Bitsong provides tutorials for the following IBC relayers:
Hermes (rust)
https://hermes.informal.systems/
Pre-requisites:
latest go-version https://golang.org/doc/install
Fresh Rust installation: For instructions on how to install Rust on your machine please follow the official Notes about Rust installation at https://www.rust-lang.org/tools/install
build-essential, git
openssl for rust. The OpenSSL library with its headers is required. Refer to https://docs.rs/openssl/0.10.38/openssl/
It is recommended to always build binaries on dedicated machine (dev-box), as dev dependencies (rust & go) shouldn't be on your production machine
sudo apt install librust-openssl-dev build-essential gitSetup full nodes & configure seeds, peers and endpoints
To successfully relay IBC packets you need to run private full nodes (custom pruning or archive node) on all networks you want to support. Since relaying-success highly depends on latency and disk-IO-rate it is currently recommended to service these full/archive nodes on the same machine as the relayer process.
Because the relaying process needs to be able to query the chain back in height for at least 2/3 of the unstaking period ("trusting period") it is recommended to use pruning settings that will keep the full chain-state for a longer period of time than the unstaking period:
edit app.toml - note: at an average block time of 6.5sec pruning-keep-recent=400000 will result in a retained chainstate of ~30d. This will suffice for most cosmos-sdk chains with an unstaking period < 30d
pruning="custom"
pruning-keep-recent=400000
pruning-keep-every=0
pruning-interval=100hermes needs to be able to query the RPC- and gRPC-endpoints of your nodes, you will need to maintain a well-organized port-setup.
edit app.toml & config.toml, choose a unique port-config for each chain, write down your port-config
app.toml
[grpc]
# Enable defines if the gRPC server should be enabled.
enable = true
# Address defines the gRPC server address to bind to.
address = "0.0.0.0:7012"config.toml - choose unique pprof_laddr port
# pprof listen address (https://golang.org/pkg/net/http/pprof)
pprof_laddr = "localhost:7019"[rpc]
# TCP or UNIX socket address for the RPC server to listen on
laddr = "tcp://127.0.0.1:7013"[p2p]
# Address to listen for incoming connections
laddr = "tcp://0.0.0.0:7010"config.toml - set persistent-peers & seeds for each chain
bitsong-2b seeds:
[email protected]:26656,[email protected]:26656bitsong-2b persistent-peers:
[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:15631,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:36656osmosis-1 seeds:
83adaa38d1c15450056050fd4c9763fcc7e02e2c@ec2-44-234-84-104.us-west-2.compute.amazonaws.com:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656osmosis-1 persistent-peers:
[email protected]:26656, [email protected]:26656, [email protected]:26656, [email protected]:26656, [email protected]:26656, [email protected]:26656, [email protected]:26656, [email protected]:26656, [email protected]:36657, [email protected]:26656, [email protected]:26656, [email protected]:26656cosmoshub-4 seeds:
bf8328b66dceb4987e5cd94430af66045e59899f@public-seed.cosmos.vitwit.com:26656,[email protected]:26656,[email protected]:26656,ba3bacc714817218562f743178228f23678b2873@public-seed-node.cosmoshub.certus.one:26656,[email protected]:26656cosmoshub-4 persistent-peers:
[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656please reference https://github.com/cosmos/chain-registry for a maintained list of peers & seeds.
To simplify the config process you can use Environment-Variables in the systemd file: sudo vim /etc/systemd/system/bitsongd.service
[Unit]
Description=Bitsong Daemon
[Service]
User=relay
Environment=BITSONGD_P2P_LADDR=tcp://0.0.0.0:7010
Environment=BITSONGD_RPC_LADDR=tcp://0.0.0.0:7011
Environment=BITSONGD_GRPC_ADDRESS=127.0.0.1:7012
Environment=BITSONGD_PPROF_LADDR=localhost:7019
Environment=BITSONGD_P2P_PERSISTENT_PEERS="[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:15631,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:36656"
Environment=BITSONGD_P2P_SEEDS="[email protected]:26656,[email protected]:26656"
Environment=BITSONGD_SNAPSHOT_INTERVAL=1000
Environment=BITSONGD_P2P_MAX_NUM_INBOUND_PEERS=100
Environment=BITSONGD_P2P_MAX_NUM_OUTBOUND_PEERS=100
LimitNOFILE=500000
ExecStart=/usr/local/bin/bitsongd start --pruning custom --pruning-keep-recent 400000 --pruning-keep-every=0 --pruning-interval 100 --home --x-crisis-skip-assert-invariants
Environment=BITSONGD_LOG_LEVEL=info
Restart=always
RestartSec=3
[Install]
WantedBy=multi-user.targetBuild & setup Hermes
Beware that for security reasons this step should be done 'on some remote pc'
Make the directory where you'll place the source, clone the hermes source repository and build it using the latest release. Optional: copy binary to /usr/bin (or preferred directory for systemd execution)
mkdir -p $HOME/hermes
git clone https://github.com/informalsystems/ibc-rs.git hermes
cd hermes
git checkout v0.13.0
cargo install ibc-relayer-cli --bin hermes --locked
sudo cp ~/.cargo/bin/hermes /usr/binIf you have built your binary on a remote machine, move the binary to your producion environment
Make hermes config & keys directory, copy config-template to config directory:
mkdir -p $HOME/.hermes
mkdir -p $HOME/.hermes/keys
cp config.toml $HOME/.hermesCheck hermes version & config dir setup
hermes version
hermes 0.13.0Edit hermes config (use ports according to your port config, set filter=true to filter channels you don't relay) vim ~/.hermes/config.toml
# The global section has parameters that apply globally to the relayer operation.
[global]
# Specify the strategy to be used by the relayer. Default: 'packets'
# Two options are currently supported:
# - 'all': Relay packets and perform channel and connection handshakes.
# - 'packets': Relay packets only.
strategy = 'packets'
# Enable or disable the filtering mechanism. Default: 'false'
# Valid options are 'true', 'false'.
# Currently Hermes supports two filters:
# 1. Packet filtering on a per-chain basis; see the chain-specific
# filter specification below in [chains.packet_filter].
# 2. Filter for all activities based on client state trust threshold; this filter
# is parametrized with (numerator = 1, denominator = 3), so that clients with
# thresholds different than this will be ignored.
# If set to 'true', both of the above filters will be enabled.
#filter = true # breaking in v0.11.0 where filter = true by default
# Specify the verbosity for the relayer logging output. Default: 'info'
# Valid options are 'error', 'warn', 'info', 'debug', 'trace'.
log_level = 'info'
# Parametrize the periodic packet clearing feature.
# Interval (in number of blocks) at which pending packets
# should be eagerly cleared. A value of '0' will disable
# periodic packet clearing. Default: 100
clear_packets_interval = 25
# Toggle the transaction confirmation mechanism.
# The tx confirmation mechanism periodically queries the `/tx_search` RPC
# endpoint to check that previously-submitted transactions
# (to any chain in this config file) have delivered successfully.
# Experimental feature. Affects telemetry if set to false.
# Default: true.
tx_confirmation = true
# The REST section defines parameters for Hermes' built-in RESTful API.
# https://hermes.informal.systems/rest.html
[rest]
# Whether or not to enable the REST service. Default: false
enabled = true
# Specify the IPv4/6 host over which the built-in HTTP server will serve the RESTful
# API requests. Default: 127.0.0.1
host = '127.0.0.1'
# Specify the port over which the built-in HTTP server will serve the restful API
# requests. Default: 3000
port = 3000
# The telemetry section defines parameters for Hermes' built-in telemetry capabilities.
# https://hermes.informal.systems/telemetry.html
[telemetry]
# Whether or not to enable the telemetry service. Default: false
enabled = true
# Specify the IPv4/6 host over which the built-in HTTP server will serve the metrics
# gathered by the telemetry service. Default: 127.0.0.1
host = '127.0.0.1'
# Specify the port over which the built-in HTTP server will serve the metrics gathered
# by the telemetry service. Default: 3001
port = 3001
[[chains]]
id = 'osmosis-1'
rpc_addr = 'http://localhost:7001'
grpc_addr = 'http://localhost:7002'
websocket_addr = 'ws://localhost:7001/websocket'
rpc_timeout = '10s'
account_prefix = 'osmo'
key_name = 'osmosis'
address_type = { derivation = 'osmosis' }
store_prefix = 'ibc'
default_gas = 5000000
max_gas = 15000000
gas_price = { price = 0.000, denom = 'uosmo' }
gas_adjustment = 0.1
max_msg_num = 20
max_tx_size = 2097152
clock_drift = '20s'
max_block_time = '10s'
trusting_period = '10days'
memo_prefix = ''
trust_threshold = { numerator = '1', denominator = '3' }
[chains.packet_filter]
policy = 'allow'
list = [
['transfer', 'channel-73']
]
[[chains]]
id = 'bitsong-2b'
rpc_addr = 'http://127.0.0.1:7011'
grpc_addr = 'http://127.0.0.1:7012'
websocket_addr = 'ws://127.0.0.1:7011/websocket'
rpc_timeout = '10s'
account_prefix = 'bitsong'
key_name = 'bitsong'
address_type = { derivation = 'bitsong' }
store_prefix = 'ibc'
default_gas = 2000000
max_gas = 4000000
gas_price = { price = 0.026, denom = 'ubtsg' }
gas_adjustment = 0.1
max_msg_num = 25
max_tx_size = 1800000
clock_drift = '10s'
max_block_time = '10s'
trusting_period = '14d'
memo_prefix = ''
trust_threshold = { numerator = '1', denominator = '3' }
[chains.packet_filter]
policy = 'allow'
list = [
['transfer', 'channel-0'],
['transfer', 'channel-1']
]
[[chains]]
id = 'cosmoshub-4'
rpc_addr = 'http://127.0.0.1:7021'
grpc_addr = 'http://127.0.0.1:7022'
websocket_addr = 'ws://127.0.0.1:7021/websocket'
rpc_timeout = '10s'
account_prefix = 'cosmos'
key_name = 'cosmos'
address_type = { derivation = 'cosmos' }
store_prefix = 'ibc'
default_gas = 2000000
max_gas = 3000000
gas_price = { price = 0.001, denom = 'uatom' }
gas_adjustment = 0.1
max_msg_num = 25
max_tx_size = 180000
clock_drift = '10s'
max_block_time = '10s'
trusting_period = '14days'
memo_prefix = ''
trust_threshold = { numerator = '1', denominator = '3' }
[chains.packet_filter]
policy = 'allow'
list = [
['transfer', 'channel-229']
]Add your relaying-wallets to hermes' keyring (located in $HOME/.hermes/keys)
Best practice is to use the same mnemonic over all networks, do not use your relaying-addresses for anything else because it might lead to mismatched account sequence errors.
hermes keys restore bitsong-2b -m "24-word mnemonic seed" --hd-path m/44'/639'/0'/0/0
hermes keys restore osmosis-1 -m "24-word mnemonic seed"
hermes keys restore cosmoshub-4 -m "24-word mnemonic seed"You can validate your hermes configuration file:
hermes config validate
INFO ThreadId(01) using default configuration from '/home/relay/.hermes/config.toml'
Success: "validation passed successfully"Create hermes service file:
[Unit]
Description=hermes
[Service]
User=relay
ExecStart=/usr/bin/hermes start
LimitNOFILE=500000
Restart=always
RestartSec=3
[Install]
WantedBy=multi-user.targetRefresh service files, enable hermes on system-startup, enable all node-daemons on system-startup, start node-daemons, sync, start hermes.
Tipp: use chainlayer quicksync to bootstrap your nodes faster: https://quicksync.io/networks/osmosis.html
sudo systemctl daemon-reload
sudo systemctl enable hermes.service bitsongd.service osmosisd.service gaiad.service
sudo systemctl start bitsongd osmosisd gaiadWatch node-daemon output to check if your nodes are syncing:
journaltctl -u bitsongd -fWhen your nodes are fully synced you can start the hermes daemon:
sudo systemctl start hermes && journalctl -u hermes -fHermes does a chain-health-check at startup. Watch the output to check if all connected nodes are up and synced
INFO ThreadId(01) using default configuration from '/home/relay/.hermes/config.toml'
INFO ThreadId(01) telemetry service running, exposing metrics at http://0.0.0.0:3001/metrics
INFO ThreadId(01) starting REST API server listening at http://127.0.0.1:3000
INFO ThreadId(01) [osmosis-1] chain is healthy
INFO ThreadId(01) [cosmoshub-4] chain is healthy
INFO ThreadId(01) [bitsong-2b] chain is healthy
...
INFO ThreadId(01) Hermes has startedHermes will try & clear any unreceived packets after startup has completed.
Snippets
Query Hermes for unreceived packets & acknowledgements (check if channels are "clear")
hermes query packet unreceived-packets bitsong-2b transfer channel-0
hermes query packet unreceived-acks bitsong-2b transfer channel-0hermes query packet unreceived-packets osmosis-1 transfer channel-73
hermes query packet unreceived-acks bitsong-1 transfer channel-73Query Hermes for packet commitments:
hermes query packet commitments osmosis-1 transfer channel-73
hermes query packet commitments bitsong-2b transfer channel-0Clear channel (only works on hermes v0.12.0 and higher)
hermes clear packets omniflixhub-1 transfer channel-1
hermes clear packets osmosis-1 transfer channel-199Clear unreceived packets manually. Experimental: you'll need to stop your hermes daemon for it not to get confused with account sequences.
hermes tx raw packet-recv osmosis-1 bitsong-2b transfer channel-0
hermes tx raw packet-recv bitsong-2b osmosis-1 transfer channel-73Thanks to @ccclaimens from @crypto_crew****
Last updated