Ingesting compressed parquet files into Timescale

TimescaleDB and PostgreSQL
1

I understand that hypertables or chunks within them are decompressed for constraint validation whenever data is inserted into a compressed hypertable. Is it also the case that a compressed file (for example, parquet) is decompressed before insertion into a hypertable?

For instance, if I have a 1GB snappy compressed parquet file that I want to directly insert into a hypertable (using something like this) and it actually contains 50GB of data, will that lead to ingestion time indicative of ingestion of 1GB data or 50GB data? Put another way, are compressed files treated as and ingested as compressed, or are they uncompressed before or during ingestion?

2

It will depend on the way you want to store it. If you’re able to split your binary, you may be able to make it. If you have binaries, than, check if you’ll not fall into TOAST.

1 Like