Relayer

This section contains instructions on how to setup the rust relayer "hermes" and relay IBC packets between bitsong-2b and other IBC-enabled networks.

Official BitSong IBC Channels

source chain-id	source channel	source denom	destination chain-id	destinaion channel	IBC token-address on destinaion chain
bitsong-2b	channel-0	ubtsg	osmosis-1	channel-73	IBC/4E5444C35610CC76FC94E7F7886B93121175C28262DDFDDE6F84E82BF2425452
bitsong-2b	channel-1	ubtsg	cosmoshub-4	channel-229	IBC/E7D5E9D0E9BF8B7354929A817DD28D4D017E745F638954764AA88522A7A409EC
osmosis-1	channel-73	uosmo	bitsong-2b	channel-0	IBC/ED07A3391A112B175915CD8FAF43A2DA8E4790EDE12566649D0C2F97716B8518
cosmoshub-4	channel-229	uatom	bitsong-2b	channel-1	IBC/C4CFF46FD6DE35CA4CF4CE031E643C8FDC9BA4B99AE598E9B0ED98FE3A2319F9

Relayer Tutorial

Hardware specs:

• 16+ vCPUs or Intel or AMD 16 core CPU

• at least 64GB RAM

• 4TB+ nVME drives

To assist operators in setting up relayers, Bitsong provides tutorials for the following IBC relayers:

Hermes (rust)

https://hermes.informal.systems/

Pre-requisites:

• latest go-version https://golang.org/doc/install

• Fresh Rust installation: For instructions on how to install Rust on your machine please follow the official Notes about Rust installation at https://www.rust-lang.org/tools/install

• build-essential, git

• openssl for rust. The OpenSSL library with its headers is required. Refer to https://docs.rs/openssl/0.10.38/openssl/

It is recommended to always build binaries on dedicated machine (dev-box), as dev dependencies (rust & go) shouldn't be on your production machine

sudo apt install librust-openssl-dev build-essential git

Setup full nodes & configure seeds, peers and endpoints

To successfully relay IBC packets you need to run private full nodes (custom pruning or archive node) on all networks you want to support. Since relaying-success highly depends on latency and disk-IO-rate it is currently recommended to service these full/archive nodes on the same machine as the relayer process.

Because the relaying process needs to be able to query the chain back in height for at least 2/3 of the unstaking period ("trusting period") it is recommended to use pruning settings that will keep the full chain-state for a longer period of time than the unstaking period:

edit app.toml - note: at an average block time of 6.5sec pruning-keep-recent=400000 will result in a retained chainstate of ~30d. This will suffice for most cosmos-sdk chains with an unstaking period < 30d

pruning="custom"
pruning-keep-recent=400000 
pruning-keep-every=0 
pruning-interval=100

hermes needs to be able to query the RPC- and gRPC-endpoints of your nodes, you will need to maintain a well-organized port-setup.

edit app.toml & config.toml, choose a unique port-config for each chain, write down your port-config

app.toml

[grpc]

# Enable defines if the gRPC server should be enabled.
enable = true

# Address defines the gRPC server address to bind to.
address = "0.0.0.0:7012"

config.toml - choose unique pprof_laddr port

# pprof listen address (https://golang.org/pkg/net/http/pprof)
pprof_laddr = "localhost:7019"

[rpc]

# TCP or UNIX socket address for the RPC server to listen on
laddr = "tcp://127.0.0.1:7013"

[p2p]

# Address to listen for incoming connections
laddr = "tcp://0.0.0.0:7010"

config.toml - set persistent-peers & seeds for each chain

bitsong-2b seeds:

 [email protected]:26656,[email protected]:26656

bitsong-2b persistent-peers:

[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:15631,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:36656

osmosis-1 seeds:

[e-mail]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656

osmosis-1 persistent-peers:

[email protected]:26656, [email protected]:26656, [email protected]:26656, [email protected]:26656, [email protected]:26656, [email protected]:26656, [email protected]:26656, [email protected]:26656, [email protected]:36657, [email protected]:26656, [email protected]:26656, [email protected]:26656

cosmoshub-4 seeds:

[e-mail]:26656,[email protected]:26656,[email protected]:26656,[e-mail]:26656,[email protected]:26656

cosmoshub-4 persistent-peers:

[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656

please reference https://github.com/cosmos/chain-registry for a maintained list of peers & seeds.

To simplify the config process you can use Environment-Variables in the systemd file: sudo vim /etc/systemd/system/bitsongd.service

[Unit]
Description=Bitsong Daemon

[Service]
User=relay
Environment=BITSONGD_P2P_LADDR=tcp://0.0.0.0:7010
Environment=BITSONGD_RPC_LADDR=tcp://0.0.0.0:7011
Environment=BITSONGD_GRPC_ADDRESS=127.0.0.1:7012
Environment=BITSONGD_PPROF_LADDR=localhost:7019
Environment=BITSONGD_P2P_PERSISTENT_PEERS="[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:15631,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:26656,[email protected]:36656"
Environment=BITSONGD_P2P_SEEDS="[email protected]:26656,[email protected]:26656"
Environment=BITSONGD_SNAPSHOT_INTERVAL=1000
Environment=BITSONGD_P2P_MAX_NUM_INBOUND_PEERS=100
Environment=BITSONGD_P2P_MAX_NUM_OUTBOUND_PEERS=100
LimitNOFILE=500000
ExecStart=/usr/local/bin/bitsongd start --pruning custom --pruning-keep-recent 400000 --pruning-keep-every=0 --pruning-interval 100 --home --x-crisis-skip-assert-invariants
Environment=BITSONGD_LOG_LEVEL=info

Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

Build & setup Hermes

Beware that for security reasons this step should be done 'on some remote pc'

Make the directory where you'll place the source, clone the hermes source repository and build it using the latest release. Optional: copy binary to /usr/bin (or preferred directory for systemd execution)

mkdir -p $HOME/hermes
git clone https://github.com/informalsystems/ibc-rs.git hermes
cd hermes
git checkout v0.13.0
cargo install ibc-relayer-cli --bin hermes --locked
sudo cp ~/.cargo/bin/hermes /usr/bin

If you have built your binary on a remote machine, move the binary to your producion environment

Make hermes config & keys directory, copy config-template to config directory:

mkdir -p $HOME/.hermes
mkdir -p $HOME/.hermes/keys
cp config.toml $HOME/.hermes

Check hermes version & config dir setup

hermes version
hermes 0.13.0

Edit hermes config (use ports according to your port config, set filter=true to filter channels you don't relay) vim ~/.hermes/config.toml

# The global section has parameters that apply globally to the relayer operation.
[global]

# Specify the strategy to be used by the relayer. Default: 'packets'
# Two options are currently supported:
#   - 'all': Relay packets and perform channel and connection handshakes.
#   - 'packets': Relay packets only.
strategy = 'packets'

# Enable or disable the filtering mechanism. Default: 'false'
# Valid options are 'true', 'false'.
# Currently Hermes supports two filters:
# 1. Packet filtering on a per-chain basis; see the chain-specific
#   filter specification below in [chains.packet_filter].
# 2. Filter for all activities based on client state trust threshold; this filter
#   is parametrized with (numerator = 1, denominator = 3), so that clients with
#   thresholds different than this will be ignored.
# If set to 'true', both of the above filters will be enabled.
#filter = true # breaking in v0.11.0 where filter = true by default

# Specify the verbosity for the relayer logging output. Default: 'info'
# Valid options are 'error', 'warn', 'info', 'debug', 'trace'.
log_level = 'info'

# Parametrize the periodic packet clearing feature.
# Interval (in number of blocks) at which pending packets
# should be eagerly cleared. A value of '0' will disable
# periodic packet clearing. Default: 100
clear_packets_interval = 25

# Toggle the transaction confirmation mechanism.
# The tx confirmation mechanism periodically queries the `/tx_search` RPC
# endpoint to check that previously-submitted transactions
# (to any chain in this config file) have delivered successfully.
# Experimental feature. Affects telemetry if set to false.
# Default: true.
tx_confirmation = true


# The REST section defines parameters for Hermes' built-in RESTful API.
# https://hermes.informal.systems/rest.html
[rest]

# Whether or not to enable the REST service. Default: false
enabled = true

# Specify the IPv4/6 host over which the built-in HTTP server will serve the RESTful
# API requests. Default: 127.0.0.1
host = '127.0.0.1'

# Specify the port over which the built-in HTTP server will serve the restful API
# requests. Default: 3000
port = 3000


# The telemetry section defines parameters for Hermes' built-in telemetry capabilities.
# https://hermes.informal.systems/telemetry.html
[telemetry]

# Whether or not to enable the telemetry service. Default: false
enabled = true

# Specify the IPv4/6 host over which the built-in HTTP server will serve the metrics
# gathered by the telemetry service. Default: 127.0.0.1
host = '127.0.0.1'

# Specify the port over which the built-in HTTP server will serve the metrics gathered
# by the telemetry service. Default: 3001
port = 3001

[[chains]]
id = 'osmosis-1'
rpc_addr = 'http://localhost:7001'
grpc_addr = 'http://localhost:7002'
websocket_addr = 'ws://localhost:7001/websocket'
rpc_timeout = '10s'
account_prefix = 'osmo'
key_name = 'osmosis'
address_type = { derivation = 'osmosis' }
store_prefix = 'ibc'
default_gas = 5000000
max_gas = 15000000
gas_price = { price = 0.000, denom = 'uosmo' }
gas_adjustment = 0.1
max_msg_num = 20
max_tx_size = 2097152
clock_drift = '20s'
max_block_time = '10s'
trusting_period = '10days'
memo_prefix = ''
trust_threshold = { numerator = '1', denominator = '3' }
[chains.packet_filter]
policy = 'allow'
list = [
  ['transfer', 'channel-73']
]

[[chains]]
id = 'bitsong-2b'
rpc_addr = 'http://127.0.0.1:7011'
grpc_addr = 'http://127.0.0.1:7012'
websocket_addr = 'ws://127.0.0.1:7011/websocket'
rpc_timeout = '10s'
account_prefix = 'bitsong'
key_name = 'bitsong'
address_type = { derivation = 'bitsong' }
store_prefix = 'ibc'
default_gas = 2000000
max_gas = 4000000
gas_price = { price = 0.026, denom = 'ubtsg' }
gas_adjustment = 0.1
max_msg_num = 25
max_tx_size = 1800000
clock_drift = '10s'
max_block_time = '10s'
trusting_period = '14d'
memo_prefix = ''
trust_threshold = { numerator = '1', denominator = '3' }
[chains.packet_filter]
policy = 'allow'
list = [
 ['transfer', 'channel-0'],
 ['transfer', 'channel-1']
 ]

[[chains]]
id = 'cosmoshub-4'
rpc_addr = 'http://127.0.0.1:7021'
grpc_addr = 'http://127.0.0.1:7022'
websocket_addr = 'ws://127.0.0.1:7021/websocket'
rpc_timeout = '10s'
account_prefix = 'cosmos'
key_name = 'cosmos'
address_type = { derivation = 'cosmos' }
store_prefix = 'ibc'
default_gas = 2000000
max_gas = 3000000
gas_price = { price = 0.001, denom = 'uatom' }
gas_adjustment = 0.1
max_msg_num = 25
max_tx_size = 180000
clock_drift = '10s'
max_block_time = '10s'
trusting_period = '14days'
memo_prefix = ''
trust_threshold = { numerator = '1', denominator = '3' }
[chains.packet_filter]
policy = 'allow'
list = [
   ['transfer', 'channel-229']
 ]

Add your relaying-wallets to hermes' keyring (located in $HOME/.hermes/keys)

Best practice is to use the same mnemonic over all networks, do not use your relaying-addresses for anything else because it might lead to mismatched account sequence errors.

hermes keys restore bitsong-2b -m "24-word mnemonic seed" --hd-path m/44'/639'/0'/0/0 
hermes keys restore osmosis-1 -m "24-word mnemonic seed"
hermes keys restore cosmoshub-4 -m "24-word mnemonic seed"

You can validate your hermes configuration file:

hermes config validate
INFO ThreadId(01) using default configuration from '/home/relay/.hermes/config.toml'
Success: "validation passed successfully"

Create hermes service file:

[Unit]
Description=hermes

[Service]
User=relay
ExecStart=/usr/bin/hermes start
LimitNOFILE=500000

Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

Refresh service files, enable hermes on system-startup, enable all node-daemons on system-startup, start node-daemons, sync, start hermes.

Tipp: use chainlayer quicksync to bootstrap your nodes faster: https://quicksync.io/networks/osmosis.html

sudo systemctl daemon-reload
sudo systemctl enable hermes.service bitsongd.service osmosisd.service gaiad.service
sudo systemctl start bitsongd osmosisd gaiad

Watch node-daemon output to check if your nodes are syncing:

journaltctl -u bitsongd -f

When your nodes are fully synced you can start the hermes daemon:

sudo systemctl start hermes && journalctl -u hermes -f

Hermes does a chain-health-check at startup. Watch the output to check if all connected nodes are up and synced

INFO ThreadId(01) using default configuration from '/home/relay/.hermes/config.toml'
INFO ThreadId(01) telemetry service running, exposing metrics at http://0.0.0.0:3001/metrics
INFO ThreadId(01) starting REST API server listening at http://127.0.0.1:3000
INFO ThreadId(01) [osmosis-1] chain is healthy
INFO ThreadId(01) [cosmoshub-4] chain is healthy
INFO ThreadId(01) [bitsong-2b] chain is healthy
...
INFO ThreadId(01) Hermes has started

Hermes will try & clear any unreceived packets after startup has completed.

Snippets

Query Hermes for unreceived packets & acknowledgements (check if channels are "clear")

hermes query packet unreceived-packets bitsong-2b transfer channel-0
hermes query packet unreceived-acks bitsong-2b transfer channel-0

hermes query packet unreceived-packets osmosis-1 transfer channel-73
hermes query packet unreceived-acks bitsong-1 transfer channel-73

Query Hermes for packet commitments:

hermes query packet commitments osmosis-1 transfer channel-73
hermes query packet commitments bitsong-2b transfer channel-0

Clear channel (only works on hermes v0.12.0 and higher)

hermes clear packets omniflixhub-1 transfer channel-1
hermes clear packets osmosis-1 transfer channel-199

Clear unreceived packets manually. Experimental: you'll need to stop your hermes daemon for it not to get confused with account sequences.

hermes tx raw packet-recv osmosis-1 bitsong-2b transfer channel-0
hermes tx raw packet-recv bitsong-2b osmosis-1 transfer channel-73

Thanks to @ccclaimens from @crypto_crew****