How do I fix indexer timeouts?

BlockScout utilizes two separate indexers in order to index the network history as well as keep up with new incoming blocks. Due to this process, a node may become overloaded and not respond to BlockScout's RPC requests within the designated timeout period.

The indexer umbrella application can be tweaked to meet your node's size and responsiveness. Outlined below are the locations and adjustments that can be made to each fetcher.

Fetchers are available at https://github.com/blockscout/blockscout/tree/master/apps/indexer/lib/indexer/fetcher

Fetcher
Description
Batch size
Concurrency

This fetcher indexes blocks, transactions and receipts starting from the tip of the chain working backward to the Genesis block.

10 INDEXER_CATCHUP_BLOCKS_BATCH_SIZE

10 INDEXER_CATCHUP_BLOCKS_CONCURRENCY

Transaction receipts fetcher.

250 INDEXER_RECEIPTS_BATCH_SIZE

10 INDEXER_RECEIPTS_CONCURRENCY

This fetcher indexes internal transactions. For the real-time fetcher this process is done synchronously and for the catchup fetcher, this process is done asynchronously.

10 INDEXER_INTERNAL_TRANSACTIONS_BATCH_SIZE

4 INDEXER_INTERNAL_TRANSACTIONS_CONCURRENCY

This fetcher indexes each coin balance at the block height the address was interacted with.

100 INDEXER_COIN_BALANCES_BATCH_SIZE

4 INDEXER_COIN_BALANCES_CONCURRENCY

This fetcher indexes non-consensus blocks, transactions, and receipts.

10

10

This fetcher indexes token balances. Note you may experience some token balances that cannot be fetched due to a malformed smart contract or other functions that do not allow a balance to be fetched.

100 INDEXER_TOKEN_BALANCES_BATCH_SIZE

10 INDEXER_TOKEN_BALANCES_CONCURRENCY

This indexer fetches the metadata for a token contract

1

10

Block reward fetcher.

10 INDEXER_BLOCK_REWARD_BATCH_SIZE

10 INDEXER_BLOCK_REWARD_CONCURRENCY

Token instance realtime fetcher.

1 INDEXER_TOKEN_INSTANCE_REALTIME_BATCH_SIZE

10 INDEXER_TOKEN_INSTANCE_REALTIME_CONCURRENCY

Token instance retry fetcher.

10 INDEXER_TOKEN_INSTANCE_RETRY_CONCURRENCY

10 INDEXER_TOKEN_INSTANCE_RETRY_BATCH_SIZE

Token instance sanitize fetcher.

10 INDEXER_TOKEN_INSTANCE_SANITIZE_BATCH_SIZE

10 INDEXER_TOKEN_INSTANCE_SANITIZE_CONCURRENCY

Understanding Errors and Timeouts

application=indexer fetcher=coin_balance count=500 error_count=500 [error] failed to fetch: :timeout

In the error provided above, we can tell that the indexer application failed to fetch 500 coin balances due to a timeout error. Usually, when an indexer starts to receive timeouts, many more timeouts from other fetchers will occur as well.

Resolving Timeout Issues

The best action to take when fetchers start to receive timeouts is to lower the batch size and concurrency of a few fetchers which will put less pressure on the node.

The two indexers that cause the most strain to the node are the block fetcher and the internal transactions fetcher. It would be advised to cut these fetchers values in half and restart the application. A restart of the node may also be required.

Other Actions

BlockScout comes equipped with functionality to automatically stop sending requests for n seconds if timeouts are detected. You may adjust these settings by setting a different value for wait_per_timeout.

https://github.com/poanetwork/blockscout/blob/1475e3bfd002d9397efd0f0cc29c20f39a70d023/apps/ethereum_jsonrpc/config/config.exs#L3-L9

Last updated

Logo

Copyright © Blockscout Limited 2023-2024