I’ve noticed that our invalidation log is high. It is 59 GB in size, almost the size of our largest hypertable at 69 GB after compression.
I think this might have to do with how we ingest data. We’re receiving data from clients and shipping it through an AWS SQS standard (no FIFO guarantees), and a single worker writing to Timescale. Either there’s a large variability in ordering, or that the timestamps we are sending out lag enough to trigger writes into the invalidation log.
So I got some questions:
- Can the invalidation log be pruned? Are there log entries that can be deleted after being processed for materialization? I’m attempting a migration and would like to reduce the data we have to dump and restore.
- How can I find out if we’re writing excessively to the invalidation log too high because we have too many things out of order, and/or the materialization threshold is set too low?
- Can materialization threshold be modified and tuned?
