The portal has a number of good system monitoring reports, but a hole we had to dig through was if Indexing itself is behind. If indexing starts running behind, then searches start returning no results for users, even if the data is written to disk, and the event data was picked up for reporting just fine.

The data is actually all recorded in the IndexStats log files on each canister. There are two main columns of interest, that would make a great monitor:

* FilesToIndexAvg: This is how much work it has in queue on Avg per hour (since a moving number). It should stay low.

** If it starts growing Hour over Hour as a system total, then you've got an issue maybe in data coming in, or what you are indexing. If a single canister starts growing out of trend, then that canister may be having disk IO or other system issues with disk.

* IndexTotal: Amount of data, records its indexing per hour. Not sure what these equate to (can't be hits), but it seems accurate. Changes to your Session Indexer configuration will reflect here, based on the number of records per hour it is able to run through. If you increase efficiency of your disk, indexing settings or amount of data to index, then this number should get higher per hour, knowing you are making progress.