Performance comparison: LogsDB + LZ4 vs. DEFLATE
To evaluate the query performance impact of enabling LogsDB, we ran a series of tests using LogsDB under different compression settings in Elasticsearch. We measured query latency under three scenarios:
- LogsDB with DEFLATE (
best_compression) - Standard index mode with DEFLATE (
best_compression) - LogsDB with LZ4
The query used for testing involved filtering network traffic data by source and destination IPs and ports, followed by several aggregations: summing the total packets, and calculating percentiles and averages for packet sizes.
Test results
Query latency across the three configurations:
- LogsDB with
best_compression(DEFLATE): 1.544 s average latency - Standard mode with
best_compression(DEFLATE): 1.448 s average latency - LogsDB with LZ4: 1.179 s average latency
LogsDB with LZ4 compression produced the lowest query latency at 1.179 seconds. That's a 24% improvement over LogsDB with DEFLATE, and it beats standard index mode with DEFLATE by a significant margin. By accepting a slight reduction in compression efficiency, you still get fast queries with LogsDB — making LZ4 an excellent choice when query performance is paramount but you still want to capture the storage savings of enabling LogsDB.
Storage impact of LZ4 on LogsDB
Just how much of a reduction in compression efficiency does LZ4 introduce?
Across 60 million network flow logs, we compared three configurations:
LogsDB with DEFLATE, LogsDB with LZ4, and the standard index mode with
best_compression enabled.
The difference is smaller than most people expect.
The power of LogsDB and LZ4
Using default as your index.codec can unlock faster query response times
with a marginal reduction in compression efficiency — about ~1% less
compression.
This trade-off is minimal compared to the advantages. With LZ4, you reduce query latency while retaining most of the storage benefits LogsDB offers. Because the data is already sorted during ingestion, switching to LZ4 buys you up to 30% better query performance for less than 1% loss in compression efficiency.
TL;DR
Consider using index.codec: default when enabling Elasticsearch's LogsDB.
It typically offers query performance advantages while still achieving 99%
of the storage savings versus a non-LogsDB index.
In a future post we'll explore how to get back even more than the 1% compression lost by switching to LZ4 — while still retaining the query performance benefits of LZ4 with LogsDB. Stay tuned.