Historical metric and event data is missing after upgrading to 1.15.0 or higher

Problem

Historical data in StackState has disappeared for metrics, events and stackstate events (not topology) after upgrading from StackState version < 1.15 to stackstate version >= 1.15.0.

Cause

The Elasticsearch data store used for these data types got upgraded, yet data migration is not enabled by default since it takes a lot of time and resources and is not necessary in all use cases.

Solution

If it is no problem for your use-case that this historical data is missing (new history will build up over time from after the upgrade) the old data can be removed to free up diskspace. Do this from the machine that runs StackState with this command: sudo rm -rf /opt/stackstate/var/lib/elasticsearch.

If the data is important it can be migrated via a manual action. For that purpose a small tool is included in the StackState distribution. Note that this process can take quite some time, depending on the amount of stored data it can take multiple hours, it requires extra memory and it may slow down StackState performance due to the additional load of the migration. Before running it make sure that:

• Make sure StackState is running, for example with sudo systemctl status stackstate
• There is sufficient disk space available, with the command du -h -d0 /opt/stackstate/var/lib/elasticsearch the total disk space used by the old data can be determined. Free disk space of at least 30% of this is required for the migration. After migration less diskspace will be used due to improved storage.
• There is sufficient memory available, at least 2.5 to 3 GB of memory is required for the migration to run

Note that the migration as described here will automatically delete the data that it has migrated. If that is not enabled more diskspace will be needed. Since the migration may take a long time it is advised to run it inside a screen or with nohup so that a disconnect of an ssh session won't interrupt. However if the migration is interrupted it is safe to run it again, it will find where it left of and resume migration from there on.

First open a screen session and then start the migration from the /opt/stackstatedirectory with the following command:

sudo -u stackstate kafka-to-es/bin/migrate_es5_to_es7 --delete-data

Alternatively run it with nohup:

nohup sudo -u stackstate kafka-to-es/bin/migrate_es5_to_es7 --delete-data &

After starting it it is wise to, at least for the start, keep an eye on the output. Make sure there are no errors in the log, after a short time progress log messages will start appearing indicating migration progress per index that is migrated. When migration is finished it will be made clear with a Migration finished! log line and the process will terminate (any errors after the migration finished log line can be safely ignored).