Regressive JPEGs: Making One Image Behave Like Video

A JPEG is supposed to settle down as it loads. The first glimpse may be fuzzy, but every new byte should make the same photograph more accurate.

That expectation is built into a conforming progressive scan sequence, but decoders also have to cope with imperfect files. Progressive JPEG gives every scan a header describing which parts of the image it carries. Some permissive decoders will accept a later scan that improperly revisits coefficients already supplied and replace the pixels they previously rendered. Feed such a decoder scans taken from different photographs and the image does not progress toward clarity. It changes its mind.

Maurycy’s “Regressive JPEGs” experiment pushes that decoder tolerance until a plain <img> element appears to play a short, low-resolution video. There is no GIF container, video element, canvas, CSS animation, or JavaScript. The apparent frames are scan segments in one JPEG-shaped stream arriving over the network.

The result is a delightful file-format trick. It is also a compact lesson in transform coding, incremental decoding, defensive limits, and the distance between strict conformance and real-world parser behavior.

Why Progressive JPEG Exists

A baseline JPEG is usually decoded in spatial order. As a slow download arrives, the browser can reveal a sharp strip at the top while the rest remains empty. That was tolerable on fast connections, but it made large images feel painfully slow on the early web.

Progressive JPEG rearranges the compressed information into multiple full-image scans. An early scan contains enough low-frequency information to display a coarse version of the complete picture. Later scans add detail and precision. The image fills its box quickly, then sharpens in place.

This is possible because JPEG does not store a photograph as a grid of ready-to-display RGB pixels. The encoder typically converts it to YCbCr: one luminance component, Y, and two color-difference components, Cb and Cr. Because human vision notices fine brightness detail more readily than fine color detail, chroma can often be sampled less densely without an obvious loss.

Each component is divided into 8-by-8 blocks and transformed into 64 discrete cosine transform coefficients. The first coefficient, called DC, represents the block’s average level. The remaining 63 AC coefficients describe increasingly fine changes within it. Quantization discards precision that is less likely to be noticed, and entropy coding packs the resulting values efficiently.

A progressive encoder can distribute that material in two ways:

  • Spectral selection sends groups of coefficient positions in separate scans: DC first, then bands of AC coefficients.
  • Successive approximation sends rough bits first and refines their precision in later scans.

A typical scan plan mixes both. It may send rough DC values for all three components, several bands of luminance and chrominance detail, then final refinement bits. The exact plan is an encoder decision rather than a universal sequence.

The Scan Header Is The Doorway

Every scan begins with a Start of Scan marker, FF DA, followed by a header. Among other things, the header names the included image components, selects entropy tables, gives the starting and ending DCT coefficient positions, and specifies the successive-approximation bits.

Those fields are normally instructions for improving one image. A decoder may first learn the average value of every block, then learn its broad edges, and finally receive fine texture. Crucially, however, every scan declares its own coefficient range. A later scan is not inherently bound to the visual meaning of an earlier one.

That creates the opening: construct a JPEG stream whose later scans improperly supply the same coefficient positions again, but take their values from a different source image. This is not a conforming progressive sequence, and libjpeg-family decoders may warn about a bogus progression. A tolerant incremental decoder can nevertheless render the data it has now. When the replacement scan arrives, the canvas changes.

The simplest proof uses several progressive JPEGs with identical dimensions. Keep one shared set of structural markers and tables, remove the redundant start-of-image, start-of-frame, and end-of-image markers from the additional files, and append their scan segments. Maurycy used a small C program, but the operation is conceptually just bitstream splicing.

When delivered slowly, the combined file first shows one photograph and then another. The browser still believes it is decoding a single image. The “animation” is a side effect of partial rendering.

Why The Obvious Version Breaks Down

Joining complete progressive images is enough for a demonstration, but it is a poor route to video. A normal progressive image may need roughly ten scans. If each visible frame consumes all of them, even a generous decoder limit buys only a few frames.

It also produces ghosting. Some AC scans introduce new spectral bands, while others refine the precision of values sent earlier. During a transition between spliced images, only part of the coefficient set may have been replaced. The reconstructed block then contains a mixture of both pictures, so the images smear together instead of switching cleanly.

Starting with baseline JPEG does not solve the problem. A baseline scan can contain DC and AC coefficients together, which sounds perfect for one scan per frame, but a baseline decoder considers the image complete after that scan. Progressive mode has a stricter rule: DC and AC data cannot occupy the same scan.

The smallest useful progressive frame is therefore a single DC-only scan.

That scan carries the average value for each transform block, with no higher-frequency texture. It produces a coarse mosaic rather than a flat color. With common 4:2:0 chroma subsampling, the coded data is organized in larger minimum coded units for color, but the luminance transform is still based on 8-by-8 blocks. The precise visual scaling depends on the component sampling and decoder smoothing; the durable point is that each frame is recognizable but intentionally blocky.

Removing AC data fixes both earlier problems. Each picture needs only one scan, and every scan replaces the complete set of DC values instead of combining new detail with an old frame.

Building A One-Scan Frame

The standard libjpeg toolchain already exposes the required control. jpegtran accepts a scan script, so a source image can be transformed into a progressive JPEG containing only one full-precision DC scan:

# frame.scans
0,1,2: 0-0, 0, 0;
jpegtran -scans frame.scans -outfile frame-01.jpg input-01.jpg

The component list 0,1,2 selects the three image components. The 0-0 range selects only the DC coefficient, and the final values request full precision rather than a later refinement pass.

Repeat that conversion for same-sized source frames. A merger can copy the common JPEG structure, then append the entropy-coded scan from each DC-only file, and finally write one end-of-image marker. The output still identifies itself as a progressive JPEG. Its unusual behavior emerges only while a decoder consumes it incrementally.

There are important engineering constraints:

  • All frames need compatible dimensions, component layout, sampling factors, quantization tables, and coding assumptions.
  • Byte stuffing and marker parsing must be handled correctly; blindly searching compressed data for marker-looking bytes is unsafe.
  • The server must stream the response instead of buffering the complete object.
  • Intermediaries may buffer, recompress, optimize, or reject the file.
  • Browser behavior is not a stable playback API.

Maurycy found that a roughly 90-scan image worked across the browsers tested, with Chrome stopping around that point and Firefox accepting more. That number is an observation, not a guarantee. Versions, libraries, and platform image pipelines can differ.

Scan Limits Are A Security Boundary

Why would a decoder continue far enough to render a few repeated scans, then refuse more of them? Work amplification.

Progressive decoding can require a full-image coefficient buffer and repeated output passes. A compact file with an unreasonable number of scans may force disproportionately large allocations or repeated computation. libjpeg-turbo added scan-limit controls specifically so applications can defend against abuse of progressive JPEG. Its command-line tools expose -maxscans, while its API lets callers choose a limit.

The animation trick is not remote code execution. Its repeated initial scans are nonconforming input that tolerant decoders may process, and it inhabits the same territory that defensive limits protect. A decoder designed around normal photographic scan plans has good reason to distrust thousands of redundant passes.

For application developers, the lesson is broader than JPEG: validate decoded cost, not just compressed file size. Pixel dimensions, frame counts, nesting depth, scan counts, expansion ratios, and total work can all matter more than the upload’s byte length.

For forensic tools and content pipelines, “JPEG” is also too coarse a classification. It can be useful to record whether a file is baseline or progressive, how many scans it contains, which spectral ranges recur, and whether later scans improperly redefine earlier data. A file that a browser accepts as a still image can have highly unusual temporal behavior.

Network Delivery Becomes The Clock

The file contains no frame durations. Its playback rate is whatever rate the decoder receives, processes, and paints new scans.

On a local disk the changes may happen too quickly to see. On a slow connection they become visible, but congestion and buffering produce irregular timing. A CDN may deliver the whole object in a burst. A browser may wait before repainting. A proxy may transform the JPEG. The same file can look animated in one path and static in another.

The Hacker News discussion pointed out one partial workaround: if the server writes one small scan, flushes it, waits, and then writes the next, server timing can dominate when the network is fast enough. That makes the demonstration more predictable, but it still does not turn JPEG into a reliable media container. Buffering anywhere in the chain can erase the cadence.

This is why the technique is interesting precisely where it is impractical. It treats transport progress as presentation state. The image is not merely incomplete while downloading; incompleteness becomes its animation mechanism.

Existing Formats Already Solve Animation

If the goal is production animation, use a format designed for it.

Animated GIF, APNG, animated WebP, AVIF sequences, and video containers encode frame order and timing intentionally. Motion JPEG can send a sequence of independent JPEG images, commonly as a multipart response, which avoids progressive-scan dependencies and permits long-running streams. A <video> element adds seeking, buffering behavior, audio, controls, efficient inter-frame compression, and accessibility hooks.

Those alternatives miss the point of this experiment. The achievement is not a better codec. It is making an old still-image path do something its surrounding software does not expect, while staying close enough to familiar JPEG structure that mainstream decoders play along.

The related examples are a long tradition: Adam7-interlaced PNG used for changing imagery, endless GIF streams, multipart camera feeds, polyglot files, and server responses whose meaning evolves as bytes arrive. Each one exposes a layer that developers usually treat as invisible.

The Real Lesson Is About Specifications

Software rarely implements only the happy-path story we tell about a format. It implements a grammar, state transitions, resource limits, and years of compatibility behavior.

Progressive JPEG was designed so partial data could be useful. That required decoders to preserve state between scans, understand spectral ranges, accept successive refinements, and repaint an image before the end marker arrived. Once those capabilities exist, replacing earlier data is not a huge conceptual leap. It is merely an unfashionable scan plan.

That makes regressive JPEGs valuable beyond the joke. They are a small, visual example of several durable engineering ideas:

  • Incremental parsers expose states that complete-file parsers hide.
  • Tolerant parsers can turn nonconforming input into surprising behavior.
  • Resource ceilings are part of interoperability, even when the standard does not prescribe them.
  • Streaming behavior depends on the entire delivery path, not just file bytes.
  • “Supported” means more than “eventually decodes to the correct final image.”

Most progressive JPEGs still do the sensible thing: reveal a hazy photograph and refine it. One carefully assembled file can instead show that the first picture was only provisional. The browser was never promised a still moment. It was promised a sequence of scans.

Sources