What Is Data Compression? The Complete Guide for Engineers

Data compression is the process of encoding information using fewer bits than its original representation.

In desktop computing and multimedia, it is a tool to save storage space and speed up data transmission across networks. In IoT, the problem is different: payloads are measured in tens of bytes, MCUs run on kilobytes of RAM, radios dominate the energy budget, and links are billed per message or per byte. Data compression plays a central role in determining whether a deployment's connectivity budget survives contact with production traffic. This article covers where bytes actually go in a telemetry message, which compression techniques and data compression algorithms matter for constrained devices, and when compression is not worth the trade-off.

What data compression actually does to an IoT payload

Understanding how data compression works starts with redundancy. Compression algorithms exploit predictable structure, repeated patterns, and data redundancy in the input data. High-entropy sensor noise compresses poorly; verbose JSON keys and text-encoded floats compress well. The compression ratio, which measures file size reduction through compression, depends entirely on how much removable structure exists in the payload. A compression ratio of 50% means a file is halved in size, but on a 60-byte telemetry packet that ratio may only save 30 bytes, which may or may not cross a billing or fragmentation threshold.

Consider a 120-byte JSON message carrying temperature, humidity, device ID, and a timestamp. Converting to binary serialisation with fixed-point encoding and applying lossless data compression might reduce that to roughly 60 bytes. On desktop systems, data compression reduces file sizes by up to 95% for large, repetitive data files. On a Cortex-M MCU with 32 KB of RAM, the flash and CPU cost of the compression library itself matters. Compressed files require less bandwidth for transmission, but the energy and memory spent compressing them must justify the radio savings. Data compression improves system performance by speeding up data access on servers, but on an embedded device the calculation is more nuanced.

This article focuses on payload optimization in flight, over links like LoRaWAN, NB-IoT, LTE-M, cellular, and satellite IoT. Compression at rest for logs and firmware updates is a related but separate concern. ZNano operates in this space as a specialized IoT data efficiency stack; the rest of the discussion addresses general engineering trade-offs applicable to smart metering, asset tracking, industrial IoT, and similar deployments.

Where the bytes go: anatomy of a telemetry message

Take a concrete example: a smart meter publishes a JSON payload over MQTT containing a device ID, ISO-8601 timestamp, voltage, current, and power factor. The uncompressed files for that single reading might total around 150 bytes. Of those, actual sensor readings may account for only 30 bytes. The rest is field names, quotes, colons, braces, and a 20-character timestamp string. For small IoT messages, framing and metadata can rival or exceed the measurement data. Compression can save storage space by an average factor of 4.5:1 on large datasets, but on a 150-byte message the gains depend on attacking the right layer. Compression improves storage efficiency by allowing more data to fit in less space, yet the real IoT win is fewer bytes on air.

Field names, timestamps and float encoding

Verbose JSON keys like "temperature_celsius" or "device_serial_number" cost 15 to 20 bytes each, including quotes and delimiters. Mapping these to short identifiers or numeric field IDs is one of the simplest data compression techniques available, often cutting overhead before any compression algorithm runs. An ISO-8601 string like "2026-08-03T12:34:56Z" consumes over 20 ASCII characters; a 32-bit Unix epoch uses 4 bytes. For devices sampling at regular intervals, delta encoding on timestamps can reduce that to 1 or 2 bytes per reading.

Floats serialised as text ("23.78125") take 6 to 10 bytes. A signed 16-bit fixed-point integer representing tenths of a degree uses 2 bytes with no data loss at the precision most sensors actually provide. Before and after: a 150-byte JSON message with verbose keys, ISO-8601 timestamps, and ASCII floats can drop to 70 or 80 bytes just by shortening keys, switching to epoch timestamps, and using fixed-point encoding. These simple techniques are often more effective on small IoT payloads than running a generic compressor on the original verbose payload.

Protocol, transport and security overhead

Below the application payload sits a stack of headers: MQTT topic strings and QoS flags, TCP or UDP headers, IPv4 or IPv6 headers, and TLS or DTLS framing. For a 30 to 50 byte sensor payload, these layers can add another 40 to 80 bytes. Encryption is a particular concern: TLS and DTLS turn the payload into high-entropy ciphertext that no compressor can shrink. Payload compression must happen before encryption, on the device or at a trusted gateway.

On LoRaWAN, the maximum application payload varies by data rate and regional plan. In EU863-870, DR0 (SF12) allows only 51 bytes, rising to roughly 222 bytes at higher data rates. Crossing a fragmentation boundary doubles or triples airtime, so a payload that is a few bytes over a limit costs far more than the marginal bytes suggest. Header compression schemes like SCHC and 6LoWPAN attack this overhead separately from payload compression; they are covered in a dedicated section below.

Reduce, encode, then compress: the right order of operations

The core rule is straightforward: first reduce what you send, then encode it efficiently, then compress what remains. Compression techniques are broadly divided into lossless and lossy compression, but neither category helps much if the payload is full of redundant data that should never have been transmitted.

Reduce means lowering the sampling rate where the application tolerates it, applying deadbands so values are sent only when they change beyond a threshold, aggregating samples on the device, and dropping fields that do not drive decisions or analytics. For a predictive maintenance sensor on a motor, sending vibration RMS every 10 seconds instead of raw waveforms at 10 kHz can cut volume by orders of magnitude before any compression process begins.

Encode means switching from human-readable formats to binary serialisation: CBOR, Protobuf, MessagePack, or FlatBuffers. Fixed-width or bit-packed representations for booleans, enums, and small integers eliminate the overhead of text delimiters. Compressing verbose JSON without first cleaning up fields is fixing the wrong layer and typically yields disappointing results on 50 to 150 byte payloads. Compression techniques require careful selection based on data recovery needs and the structure of what is being sent.

Compress is the final layer. Lightweight lossless compressors like heatshrink, LZ4, DEFLATE (the basis for zlib and gzip), or Zstandard can be applied to the binary payload. But Barr and Asanović found that transmitting a single bit wirelessly can cost over 1,000 times the energy of a single 32-bit computation, yet with several standard compressors there was a net energy increase when compression was applied before transmission. The CPU and RAM overhead of the compression process is not free. For teams that lack the firmware budget to build and maintain a custom stack on constrained devices, evaluating a product like ZNano against a home-grown implementation is a reasonable step in the build-vs-buy decision.

Lossless vs lossy compression in IoT

Compression can lead to faster data transmission across networks, but the choice between lossless and lossy compression in IoT is a data governance decision, not just an algorithm selection. Lossless compression retains all original data without loss. Lossy compression reduces file size by discarding data deemed non-essential. Most production telemetry pipelines use a mix: lossless compression for metadata and critical fields, lossy methods or aggressive downsampling for high-rate sensor channels with defined tolerances.

When lossless is non-negotiable

Device firmware versions, configuration changes, billing and metering counters, safety-critical limits, and regulated telemetry in medical devices or utilities must use lossless compression or no compression at all. Lossless compression ensures that the decompressed output matches the original data bit for bit, which is essential for audit trails, forensic analysis, and warranty disputes. Lossless compression is ideal for text documents, software files, and any computer file where data integrity cannot be compromised.

Lossless techniques include Huffman coding and run-length encoding. Huffman coding assigns shorter codes to more frequent symbols, making it effective on payloads with skewed byte distributions. Run-length encoding compresses data by replacing sequences of identical values with counts, which works well for status bit streams or sparse event logs. Common lossless formats include ZIP, PNG, and FLAC. FLAC and PNG are examples of lossless compression formats used in audio and image contexts respectively, while in IoT the same lossless algorithms apply to telemetry and text files. Among lossless audio compression formats, FLAC is perhaps the most widely known, though in IoT the focus is on sensor data rather than audio files.

Engineering teams should document which fields in each payload schema are covered by strict lossless policies and verify that every stage of the pipeline, from edge to cloud to analytics, maintains that guarantee. Any data loss in these fields is a compliance failure, not a compression trade-off.

When controlled loss is the better engineering decision

High-rate data sources are where lossy compression earns its place: vibration streams for predictive maintenance, audio snippets for anomaly detection, dense environmental sampling, and high-frequency current waveforms. Lossy compression reduces file size by discarding unimportant data that falls below the resolution of the sensor or the tolerance of the downstream model. Lossy compression achieves higher compression ratios than lossless compression, and lossy compression permanently eliminates non-essential data for smaller files. To achieve higher compression ratios, you accept controlled fidelity loss.

In the broader world of computer science, lossy techniques exploit human sensory perception to remove unnoticeable data. Human perception limits allow lossy compression to discard imperceptible details in audio and video. Perceptual coding removes imperceptible data for audio and video compression, and psychovisual modeling is used in audio compression like MP3 and AAC. Common lossy formats include JPEG, MP3, and MP4. MP3 and JPEG are examples of lossy compression formats that produce significantly smaller file sizes. Lossy compression is suitable for multimedia files where quality loss is acceptable, and data compression is essential for efficient multimedia applications. In image and video compression, transform coding is a common method for lossy image compression; the Discrete Cosine Transform is widely used in image compression, and DCT based transform coding underpins jpeg compression specifically. Video compression algorithms like H.264 can produce video compression ratios ranging from 20 to 200. Compression ratios indicate the efficiency of a compression algorithm, and these lossy compression algorithms for video data, video files, audio files, digital images, and multimedia formats achieve results that lossless algorithms and lossless compression algorithm approaches cannot match in file size alone, though with lower compression ratios lossless methods preserve everything. Lossy compression sacrifices exact reconstruction for smaller payloads. Audio codecs and audio compression techniques using frequency analysis and audio formats tuned for perceptual coding deliver similarly aggressive savings on multimedia files. Even fractal compression has been explored for digital images, though it sees little use in IoT. Lossy formats and lossy algorithms trade image quality or audio fidelity for bandwidth, while lossy data compression in IoT trades sensor precision or temporal resolution.

For IoT specifically, the relevant lossy compression methods are not JPEG or MP3 but quantisation to fixed precision, deadband encoding, downsampling, piecewise-linear approximation, and windowed feature extraction. An industrial motor monitoring system might sample vibration at 10 kHz but transmit only RMS, peak, and crest factor every second, keeping the error well within sensor accuracy. The acceptable data loss must be tied to a defined requirement: RMS error below a limit, fidelity within sensor accuracy, or downstream model performance above a threshold. Lossy techniques are often applied on the device or edge gateway, with the cloud never seeing discarded samples, so data science and operations teams must be informed of these choices.

Compression approaches that fit constrained devices

Popular desktop compressors like full gzip or Zstandard with large window sizes are too heavy for small Cortex-M MCUs. Embedded IoT firmware typically combines simple, data-specific compression methods with lightweight libraries. The main technique families relevant to IoT are delta encoding, run length encoding, bit packing, fixed-point numerical representations, dictionary-based LZ methods, and entropy coding such as huffman coding. Each carries trade-offs in code size, RAM footprint for sliding windows or dictionaries, worst-case latency, compression speed, decompression speed, and deterministic behaviour for real-time firmware.

Delta encoding and bit packing

Delta encoding stores the difference between successive values rather than absolute values. For monotonic or slowly varying series, such as meter counters, temperatures, or tank levels, deltas are small and can be stored in fewer bits. Delta-of-delta encoding works particularly well for regular timestamps: if a device samples every 60 seconds, the delta is nearly constant and the delta-of-delta is often zero or one, requiring just a few bits per reading.

Bit packing groups booleans, small enums, and low-range integers into compact bitfields. Eight digital input flags fit in a single byte instead of eight separate JSON booleans at roughly 5 bytes each. Combined with fixed-point encoding, bit packing avoids IEEE 754 floats on the wire entirely, improving both compression ratio and decoding cost on small MCUs and gateways. The warning: packed formats require careful schema versioning and documentation. Without good tooling, inspecting a packed payload during field diagnostics is painful.

Dictionary and entropy methods

LZ-family dictionary compressors, including LZ4, DEFLATE (the core of zlib), and Zstandard, use sliding windows to reference repeated substrings. On microcontrollers, the RAM cost of the sliding window matters: a 4 KB window is feasible on a Cortex-M4, a 32 KB window may not be. These compression methods shine on repetitive payloads like configuration templates or verbose JSON logs, and across batched messages, but offer diminishing returns on tiny, independent telemetry packets.

Entropy coding assigns variable-length codes to symbols based on their frequency. Huffman coding assigns shorter codes to frequent data symbols, producing a lossless compression algorithm that is lightweight to implement. Arithmetic coding offers finer-grained results but is rarely practical on tiny MCUs. Lempel-Ziv-Welch (LZW) is used in the GIF image format and has historical significance, but patent-free LZ77 and LZ78 variants dominate IoT use. For embedded work, implementations like heatshrink (an LZSS variant designed for under 1 KB of RAM) are worth profiling against actual device constraints and input data before committing flash space.

Dictionary-trained compression for small payloads

Dictionary-trained compression, as supported by Zstandard's dictionary mode (RFC 8878), builds a static dictionary from representative IoT payloads and shares it between encoder and decoder. For small, repetitive payloads typical in telemetry, such as similar JSON schemas with the same key names, this approach can deliver strong results on compressed data that would otherwise be too short for conventional LZ methods to find repeated substrings.

Operational considerations are significant: dictionaries must be distributed and versioned across the fleet, gateways and cloud services must use the matching dictionary revision, and silent decode failures from version skew must be handled. A smart metering fleet using a shared dictionary for tariff and meter field names is a good fit: the payload structure is stable, the compressed format is consistent, and the management overhead is justified by the volume of devices. Dictionary-trained methods work best when the deployment has stable payload formats and the team can tolerate the coordination cost.

Serialisation is not compression (and you probably need both)

Serialisation formats control how fields map to bytes. Compression removes redundancy from whatever serialisation produces. These solve overlapping but distinct problems, and engineers frequently conflate them. Switching from JSON to a binary format can yield large size reductions for IoT messages even without a compression algorithm, especially for data files containing repetitive field structures. In many IoT deployments, the biggest win comes from moving to compact binary serialisation, with optional lossless compression on large or batched messages. Different file formats suit different constraints.

Common serialisation formats for IoT include:

JSON: Largest message size and serves as the baseline. It requires no schema and has low decode complexity through text parsing, but is poorly suited to small MCUs. Its ecosystem is very mature.
CBOR: Typically around 40–60% of JSON size. It is self-describing and does not require a separate schema, has low-to-medium decode complexity, is well suited to MCUs, and has a mature ecosystem under RFC 8949.
Protobuf: Typically around 30–50% of JSON size. It requires .proto schema files, has medium decode complexity, moderate MCU suitability, and a very mature ecosystem.
MessagePack: Typically around 40–55% of JSON size. It requires no schema, has low-to-medium decode complexity, works well on MCUs, and has a mature ecosystem.
FlatBuffers: Typically around 30–50% of JSON size. It requires a schema and offers very low, zero-copy decode complexity. It is well suited to read-heavy MCU workloads, with a moderately mature ecosystem.

The comparison above is directional, not a benchmark.

Header compression: the lever most teams forget

Header compression targets the IP, UDP, and protocol headers that are repeated almost identically across messages. On LPWAN links with very small MTUs, saving bytes in headers can matter more to airtime and duty cycle than squeezing a few more bytes out of the payload itself.

SCHC (Static Context Header Compression, RFC 8724) defines a framework where the device and network share a static context describing expected header field values. In most cases, a small rule identifier stands in for the full IPv6/UDP header. Profiles exist for LoRaWAN (RFC 9011) and Sigfox (RFC 9442). 6LoWPAN provides similar header compression for IEEE 802.15.4 networks and is often built into stack implementations.

Consider a LoRaWAN deployment at DR0, where the application payload maximum is 51 bytes. If 20 of those bytes are consumed by IPv6 and CoAP headers, only 31 remain for sensor readings. SCHC can replace those 20 bytes with a 1- or 2-byte rule ID, nearly doubling available payload capacity. In this scenario, header compression makes a payload schema feasible that would otherwise require a higher data rate, more airtime, or fragmentation. Engineers should check whether their connectivity provider and network server already support SCHC before adding more complex payload compression on the device.

Where to compress: device, gateway, edge or cloud

Compression can happen at four points: on-device in firmware, at an IoT gateway, in edge computing nodes, or in the cloud ingestion pipeline. Only compression performed before a costly hop, such as the device-to-gateway radio link or a satellite uplink, reduces bytes on air and thus saves IoT bandwidth or battery. Gateway-side compression cannot shrink airtime on the device's uplink. It can help reduce storage footprint and volume on the gateway-to-cloud backhaul.

Decode-side symmetry matters: whoever receives compressed data must decompress it. Rolling out a new compression algorithm or dictionary to thousands of devices in the field is a firmware update, with all the risks that entails for executable files running on constrained hardware. Version skew between encoder and decoder can cause silent failures. A fallback path for diagnostics, where the device can send uncompressed payloads on demand, is worth building in.

The trade-offs at each compression point are:

Device (firmware): Saves airtime, cellular volume, and cloud ingest cost. The trade-off is high MCU resource usage across flash, RAM, and CPU, plus firmware update risk.
Gateway: Does not save device airtime, but does save cellular volume and cloud ingest cost. MCU resource cost is low because compression can run on Linux, while the main operational consideration is gateway management.
Edge node: Does not save device airtime, but can save cellular volume and cloud ingest cost. Resource cost is low, with edge infrastructure cost as the primary operational consideration.
Cloud pipeline: Does not save airtime or cellular volume. Its benefit is primarily storage, with no MCU resource cost and the lowest operational risk.

Prototype at both device and gateway on real data. Measure energy per delivered message, not just CPU time, to understand whether the compression overhead is justified.

What compression does to your connectivity bill

Different IoT connectivity models bill differently, and the optimisation that pays on one link may be irrelevant on another. Cost savings from compression are real but depend on where the billing threshold falls.

AWS IoT Core accepts messages up to 128 KB and meters them in 5 KB increments: an 8 KB message is billed as two messages, but a 400-byte payload and a 150-byte payload both count as one message. Shrinking below the 5 KB block changes nothing on this meter. Azure IoT Hub meters in 4 KB blocks on paid tiers. Microsoft's own documentation gives the batching example: 40 sensor reads of 100 bytes each, batched into one 4 KB message, consume one message of quota instead of 40. On satellite links, billing is near per-byte, so every byte saved translates directly to cost savings and efficient storage of transmission credits.

The optimisation that pays depends on the link type and billing model:

LoRaWAN: The primary constraints are payload size, duty cycle, and airtime. Billing is typically per message or airtime, so payload compression, header compression, and reduced sampling usually deliver the most value.
NB-IoT: Volume, latency, and retries are the main constraints, with billing based on data volume. Batching and binary serialisation usually provide the strongest optimisation.
LTE-M: Volume and power are the primary constraints, with billing based on data volume. Batching and sampling reduction are usually the most effective approaches.
Cellular (4G/5G): Data volume is the main constraint, with costs determined by data plan tiers. Batching and binary serialisation usually provide the most value.
Satellite IoT: Per-byte cost is the dominant constraint, with billing typically per byte or per message. Every byte matters, so a full optimisation stack is appropriate.
Cloud broker (AWS/Azure): Message count in KB blocks is the main constraint, with billing typically based on 4–5 KB blocks. Batching to fill those blocks usually matters more than payload compression alone, unless compression is also saving file storage at scale.

Model costs using actual tariffs and traffic patterns. Be wary of optimising below a billing threshold that does not exist for your link. On LoRaWAN, where maximum payload size constrains what you can transmit data at a given spreading factor, compression can enable richer payloads or allow operation at a lower data rate. On LTE-M or NB-IoT, retransmission costs from poor coverage may dominate, making reliable delivery more valuable than payload compression. Storage capacity savings in the cloud are a separate calculation from connectivity cost.

When compression is not worth it

There are many IoT cases where compression does not help or actively harms energy, latency, or debuggability. Skipping compression is often the correct engineering choice.

Red flags that compression will not pay off:

The payload is already smaller than the billing block. A 200-byte message on AWS IoT Core is one message whether compressed or not.
The data is already compressed or encrypted. Trying to compress JPEG images, MP4 video, or TLS-encrypted blobs produces no gain because the data already appears high-entropy. Zip files and other compressed files cannot be meaningfully recompressed.
Entropy is too high. Random sensor noise or already-dense binary data leaves no redundant data for a compressor to exploit, and the output may actually be larger due to compression metadata.
The compressor expands tiny payloads. Dictionary-based schemes initialised per message can add header bytes that exceed any savings on sub-50-byte input data, producing a smaller compression ratio than 1:1, meaning expansion.

Barr and Asanović's research demonstrated that for several standard lossless compressors, the added CPU work increased total energy even though wireless transmission is orders of magnitude more expensive per bit than computation. Implementation details, including wake time, compression speed, decompression speed, and memory allocation patterns, determine whether the net effect is positive. On a Cortex-M0 or M3, the code size and RAM footprint of a compression library can crowd out other firmware features and degrade system performance.

Operational cost matters too. Compressed payloads are harder to inspect with simple tools. Field debugging and support teams may resist opaque binary blobs without good tooling and logging, and data integrity verification becomes more complex. If gains are marginal or negative after testing on representative data, stop. Compression settings and algorithms are not a checkbox; they are an engineering trade-off that must improve system performance overall or be discarded.

How to measure the benefit honestly

Capture representative field data, not synthetic sequences. Simulate or replay it through your compression pipeline and compare bytes on air, not just bytes in a buffer. The framed, encrypted, and transmitted size is what determines cost and airtime, so test with your production MQTT broker, LPWAN network server, and TLS configuration.

Key metrics to record side by side:

Compression ratio on representative and worst-case data patterns
Code size in flash (bytes)
Peak RAM usage during compression
Added CPU time and its impact on wake duration
Changes in message retry rates or packet loss
Energy per delivered message (measured via current profiling on the device, not estimated)

Measure worst-case as well as typical payloads. Many compression algorithms have data-dependent performance and can expand certain patterns. If you only benchmark on friendly data, production will surprise you. Compare against uncompressed data baselines under the same compression settings and network conditions. Query performance and analytics latency on the cloud side may also change if payloads need decompression before indexing, and the original data must remain reconstructable for any lossless fields. The goal is to improve system performance end to end, not to optimise a single metric in isolation.

A decision checklist for your deployment

Identify your binding constraints: battery life, airtime, duty cycle, cloud message cost, or storage.
Map where the bytes go in your current payload: field names, timestamps, floats, headers, protocol overhead, redundant data.
Reduce: lower sampling rate, apply deadbands, drop fields that do not drive decisions. Eliminate data redundancy at the source.
Encode: switch from JSON to binary serialisation (CBOR, Protobuf, MessagePack). Use fixed-point and bit packing.
Compress: test a lightweight lossless compressor on the encoded payload. Measure whether the gain survives contact with your billing model and MTU.
Ask: Is my payload below or above the billing block? Which fields must remain lossless? Can I change the schema without breaking existing consumers? Do we have the firmware budget for this library? Does file compression work well enough on our actual payloads to justify the complexity?
Prototype one or two techniques on real data before assembling a complex stack. Start with binary serialisation plus delta encoding.