Skip to content

rw2: allow optional gzip and zstd compression (rc.7) #3173

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kgeckhart wants to merge 1 commit into
base: main
Choose a base branch
from
Open

rw2: allow optional gzip and zstd compression (rc.7) #3173

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 13 additions & 4 deletions docs/specs/prw/remote_write_spec_2_0.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ nav_title: "2.0"
sort_rank: 2
---

* Version: 2.0-rc.6
* Version: 2.0-rc.7
* Status: **Experimental**
* Date: October 2026

Expand Down Expand Up @@ -65,7 +65,7 @@ The protobuf serialization MUST use either of the following Protobuf Messages:
* The `prometheus.WriteRequest` introduced in [the Remote-Write 1.0 specification](./remote_write_spec.md#protocol). As of 2.0, this message is deprecated. It SHOULD be used only for compatibility reasons. Senders and Receivers MAY NOT support the `prometheus.WriteRequest`.
* The `io.prometheus.write.v2.Request` introduced in this specification and defined [below](#protobuf-message). Senders and Receivers SHOULD use this message when possible. Senders and Receivers MUST support the `io.prometheus.write.v2.Request`.

Protobuf Message MUST use binary Wire Format. Then, MUST be compressed with [Google’s Snappy](https://github.com/google/snappy). Snappy's [block format](https://github.com/google/snappy/blob/2c94e11145f0b7b184b831577c93e5a41c4c0346/format_description.txt) MUST be used -- [the framed format](https://github.com/google/snappy/blob/2c94e11145f0b7b184b831577c93e5a41c4c0346/framing_format.txt) MUST NOT be used.
Protobuf Message MUST use binary Wire Format. Then, MUST be compressed with one of the compressions listed in [Content-Encoding](#content-encoding).

Senders MUST send a serialized and compressed Protobuf Message in the body of an HTTP POST request and send it to the Receiver via HTTP at the provided URL path. Receivers MAY specify any HTTP URL path to receive metrics.

Expand All @@ -89,8 +89,17 @@ Content-Encoding: <compression>

<!---
Rationales: https://github.com/prometheus/proposals/blob/alexg/remote-write-20-proposal/proposals/2024-04-09_remote-write-20.md#no-new-compression-added--yet-
Relaxed in 2.0-rc.7: `snappy` stays mandatory, `gzip` and `zstd` are optional for Receivers.
-->
Content encoding request header MUST follow [the RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-content-encoding). Senders MUST use the `snappy` value. Receivers MUST support `snappy` compression. New, optional compression algorithms might come in 2.x or beyond.
Content encoding request header MUST follow [the RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-content-encoding). The allowed values are:

@bwplotka bwplotka Nov 2, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Content encoding request header MUST follow [the RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-content-encoding). The allowed values are:
Content encoding request header MUST follow [the RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-content-encoding). Senders are allowed to use following values:

I'd use the specific form as much as possible to be precise


* `snappy`: [Google’s Snappy](https://github.com/google/snappy). The [block format](https://github.com/google/snappy/blob/2c94e11145f0b7b184b831577c93e5a41c4c0346/format_description.txt) MUST be used -- [the framed format](https://github.com/google/snappy/blob/2c94e11145f0b7b184b831577c93e5a41c4c0346/framing_format.txt) MUST NOT be used.
* `gzip`: MUST follow [RFC 1952](https://www.rfc-editor.org/rfc/rfc1952.html).
* `zstd`: MUST follow [RFC 8878](https://www.rfc-editor.org/rfc/rfc8878.html) and [RFC 9659](https://www.rfc-editor.org/rfc/rfc9659.html).

Senders and Receivers MUST support `snappy`. Receivers SHOULD support `gzip` and `zstd`. `gzip` and `zstd` MUST only be used with `io.prometheus.write.v2.Request`.

@bwplotka bwplotka Nov 2, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since gzip and zstd are SHOULD for Receivers and 415 retry is MAY for Senders, Senders SHOULD still default to snappy unless configured otherwise or advertised via Accept-Encoding so 2.x Senders and snappy-only Receivers interoperate out of the box. Also, MUST only be used leaves it ambiguous whether HTTP decompression middlewares running before Content-Type parsing must reject prometheus.WriteRequest compressed with gzip or zstd:

Suggested change
Senders and Receivers MUST support `snappy`. Receivers SHOULD support `gzip` and `zstd`. `gzip` and `zstd` MUST only be used with `io.prometheus.write.v2.Request`.
Senders and Receivers MUST support `snappy`. Receivers SHOULD support `gzip` and `zstd`. Senders MAY support `gzip` and `zstd`, and SHOULD use `snappy` by default unless configured otherwise or advertised by the Receiver via `Accept-Encoding`. Senders MUST NOT use `gzip` or `zstd` with `prometheus.WriteRequest`, and Receivers MAY reject such requests with `415 HTTP Unsupported Media Type`.


Receivers MAY include the `Accept-Encoding` response header on 2xx responses to advertise supported `Content-Encoding` values, as defined in [RFC 7694](https://www.rfc-editor.org/rfc/rfc7694.html). Senders MAY use it to choose a supported encoding for subsequent requests.

@bwplotka bwplotka Nov 2, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a Receiver pool sits behind an L7 load balancer during a rolling update or rollback, upgrading from a 2xx Accept-Encoding header without 415 retry support will drop batches when subsequent requests hit snappy-only replicas, or flip-flop across replicas. Also, RFC 9110 obsoleted RFC 7694 and folded response Accept-Encoding directly into Section 12.5.3:

Suggested change
Receivers MAY include the `Accept-Encoding` response header on 2xx responses to advertise supported `Content-Encoding` values, as defined in [RFC 7694](https://www.rfc-editor.org/rfc/rfc7694.html). Senders MAY use it to choose a supported encoding for subsequent requests.
Receivers MAY include the `Accept-Encoding` response header on 2xx responses to advertise supported `Content-Encoding` values for that request, as defined in [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-accept-encoding). Senders using `io.prometheus.write.v2.Request` MAY use it to choose an allowed encoding for subsequent requests, in which case they MUST fall back to `snappy` and retry on `415 HTTP Unsupported Media Type` and SHOULD avoid immediately re-upgrading across load-balanced Receiver replicas.


#### Content-Type

Expand Down Expand Up @@ -189,7 +198,7 @@ Senders SHOULD use Remote-Write to send samples for multiple series in a single

#### Unsupported Request Content

Receivers MUST return [415 HTTP Unsupported Media Type](https://www.rfc-editor.org/rfc/rfc9110.html#name-415-unsupported-media-type) status code if they don't support a given content type or encoding provided by Senders. When returning `415 HTTP Unsupported Media Type`, Receivers SHOULD include the `Accept` (listing supported `Content-Type` values, including supported `;proto=` and `;mode=` parameters) and `Accept-Encoding` (listing supported `Content-Encoding` values) response headers as defined in [RFC 7694](https://www.rfc-editor.org/rfc/rfc7694.html) so Senders can determine what the Receiver supports without guessing.
Receivers MUST return [415 HTTP Unsupported Media Type](https://www.rfc-editor.org/rfc/rfc9110.html#name-415-unsupported-media-type) status code if they don't support a given content type or encoding provided by Senders. When returning `415 HTTP Unsupported Media Type`, Receivers SHOULD include the `Accept` response header (listing supported `Content-Type` values, including supported `;proto=` and `;mode=` parameters) so Senders can determine what the Receiver supports without guessing. If the 415 is due to an unsupported `Content-Encoding`, Receivers SHOULD also include the `Accept-Encoding` response header listing supported `Content-Encoding` values. Otherwise, in a 415 response, Receivers MUST NOT include `Accept-Encoding`, as defined in [RFC 7694](https://www.rfc-editor.org/rfc/rfc7694.html).

@bwplotka bwplotka Nov 2, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping Accept as an unconditional SHOULD on all 415 responses means decompression middlewares like SnappyDecodeMiddleware or reverse proxies violate SHOULD when rejecting Content-Encoding before the inner handler knows accepted ;proto= and ;mode= values. Maybe scope Accept to Content-Type rejections per RFC 9110 Section 15.5.16 and link RFC 9110 Section 12.5.3 instead of obsoleted RFC 7694:

Suggested change
Receivers MUST return [415 HTTP Unsupported Media Type](https://www.rfc-editor.org/rfc/rfc9110.html#name-415-unsupported-media-type) status code if they don't support a given content type or encoding provided by Senders. When returning `415 HTTP Unsupported Media Type`, Receivers SHOULD include the `Accept` response header (listing supported `Content-Type` values, including supported `;proto=` and `;mode=` parameters) so Senders can determine what the Receiver supports without guessing. If the 415 is due to an unsupported `Content-Encoding`, Receivers SHOULD also include the `Accept-Encoding` response header listing supported `Content-Encoding` values. Otherwise, in a 415 response, Receivers MUST NOT include `Accept-Encoding`, as defined in [RFC 7694](https://www.rfc-editor.org/rfc/rfc7694.html).
Receivers MUST return [415 HTTP Unsupported Media Type](https://www.rfc-editor.org/rfc/rfc9110.html#name-415-unsupported-media-type) status code if they don't support a given content type or encoding provided by Senders. If the 415 is due to an unsupported `Content-Type`, Receivers SHOULD include the [`Accept`](https://www.rfc-editor.org/rfc/rfc9110.html#name-accept) response header listing supported `Content-Type` values, including supported `;proto=` and `;mode=` parameters, so Senders can determine what the Receiver supports without guessing. If the 415 is due to an unsupported `Content-Encoding`, Receivers SHOULD include the [`Accept-Encoding`](https://www.rfc-editor.org/rfc/rfc9110.html#name-accept-encoding) response header listing supported `Content-Encoding` values. Receivers that return 415 for reasons unrelated to `Content-Encoding` MUST NOT include `Accept-Encoding`, as defined in [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-accept-encoding).


Senders SHOULD expect [400 HTTP Bad Request](https://www.rfc-editor.org/rfc/rfc9110.html#name-400-bad-request) for the above reasons from 1.x Receivers, for backwards compatibility.

Expand Down
Loading