Problem

StackGraph backup fails with the following timeout:

backup-stackgraph 2021-03-04 05:39:20,421 ERROR com.stackvista.graph.migration.Backup - Backup couldn't be completed                                                                 │
│ backup-stackgraph com.stackvista.graph.transaction.StackTransactionException: Error while accessing HBase StackTephraTransaction{ id: Optional[TransactionId(id=1614826937312000000) │
....                                             
backup-stackgraph Caused by: org.apache.tephra.TransactionNotInProgressException: Transaction timed out                                      

Cause

StackGraph is attempting to back up a large amount of data.

Solution

Follow the instructions below to increase the read only timeout and give the backup more time to complete.

Be aware: changing this setting has performance consequences.

Kubernetes installations

• Increase components.all.extraEnv.open.HBASE_CONF_data_tx_max_readonly_timeout in the values.yaml helm chart

Linux installations

• Increase data.tx.max.readonly.timeout in /opt/stackstate/etc/stackgraph/hbase-site.xml