Skip to content

Move every TLS path onto rustls (eMudhra distrust) and clear cargo audit - #236

Merged
bennyhodl merged 3 commits into
masterfrom
fix/esplora-rustls
Oct 21, 2026
Merged

bennyhodl merged 3 commits into
masterfrom
fix/esplora-rustls

Conversation

@bennyhodl

@bennyhodl bennyhodl commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Why

eMudhra: we have a problem (2026-10-20): the private keys of eight eMudhra intermediate CAs (16 certificates, counting their Sectigo cross-signs) are recoverable. Anyone can mint a TLS certificate for any host that chains to "AAA Certificate Services" (Comodo) or "emSign ECC Root CA - G3".

ddk's esplora client verified TLS with native-tls, which on Linux means OpenSSL with the system root store. Debian 12 based images, including gcr.io/distroless/cc-debian12, still ship both roots. OpenSSL has no distrust list for these intermediates. A MITM on esplora traffic can serve a forged certificate for the esplora host and feed a DLC node fake chain state: heights, confirmations, transactions, and fee estimates.

Fixed versions: rustls-webpki 0.103.16 rejects any path through the distrusted SPKIs, whatever the root store, and webpki-roots / webpki-root-certs 1.0.10 drop the affected trust.

Why downstream crates can't fix this themselves

  • bdk_esplora/async-https → esplora-client/async-https → reqwest 0.12/default-tls → native-tls.
  • Cargo features are additive, so a consumer can't turn default-tls back off.
  • reqwest 0.12 picks native-tls whenever default-tls is on, even if rustls-tls is also on (TlsBackend::default() in reqwest/src/tls.rs).
  • EsploraClient::new builds its own reqwest::Client, so there's no way to inject a different client either.

Change

  1. ddk/Cargo.toml: bdk_esplora feature async-https → async-https-rustls. reqwest 0.12 now uses rustls 0.23 with the bundled webpki-roots.
  2. Cargo.lock: rustls-webpki 0.103.15 → 0.103.16, webpki-roots 1.0.8 → 1.0.10, webpki-root-certs 1.0.8 → 1.0.10, and rustls 0.23.45 → 0.23.46. The rustls bump fixes RUSTSEC-2026-0346/0347/0348/0349, which are now on the esplora path.
  3. testenv/Cargo.toml: postgresql_embedded (still 0.19, because of the MSRV pin) uses its rustls feature instead of the default native-tls. This was the last path to OpenSSL in the workspace. The lockfile no longer contains native-tls, openssl, openssl-sys, hyper-tls or tokio-native-tls.
  4. To get cargo audit passing:
    • bitcoind / electrsd 0.41 → 0.43.2. bitcoind's download build script no longer uses zip 0.6.6 (RUSTSEC-2026-0367/0368/0369). testenv's use of their API is unchanged.
    • negentropy 0.5.0 → 0.5.2 (RUSTSEC-2026-0365), via nostr.

The feature switch alone is not enough. With rustls but the old lockfile (rustls-webpki 0.103.15, webpki-roots 1.0.8), the G3B host is still accepted, because webpki-roots 1.0.8 bundles "emSign ECC Root CA - G3":

https://dv-g3b-emudhra.we-have-a-problem.com: ERR Parsing(ParseIntError { kind: InvalidDigit })
    raw GET .../blocks/tip/height: TLS OK, HTTP 200 OK, body "<h1>hello</h1>\n"
https://dv-g3b-aaa-emudhra.we-have-a-problem.com: ERR ... InvalidCertificate(UnknownIssuer) ...

Important

A library's Cargo.lock doesn't bind its consumers. Every downstream binary must also run cargo update -p rustls-webpki -p webpki-roots -p webpki-root-certs -p rustls in its own lockfile.

Verification

Each run uses ddk's own EsploraClient::new, the production config, from a scratch crate outside this workspace that depends on ddk by path. Running it outside the workspace keeps ddk's dev-dependencies out of feature unification, so the build matches what a consumer gets. When get_height() fails, the check repeats the GET with the same reqwest::Client to show whether the TLS handshake succeeded.

The Linux runs use a Debian bookworm container (OpenSSL 3.0.22) with the CA bundle copied out of gcr.io/distroless/cc-debian12. That bundle contains both affected roots. The current rust:bookworm image (ca-certificates 20250419~deb12u2) has already dropped them, so it would not be representative.

Before: master @ 1ee82da (native-tls)

Linux, OpenSSL, distroless debian12 roots. Both forged certificates are accepted:

https://dv-g3b-emudhra.we-have-a-problem.com: ERR Parsing(ParseIntError { kind: InvalidDigit })
    raw GET https://dv-g3b-emudhra.we-have-a-problem.com/blocks/tip/height: TLS OK, HTTP 200 OK, body "<h1>hello</h1>\n"
https://dv-g3b-aaa-emudhra.we-have-a-problem.com: ERR Parsing(ParseIntError { kind: InvalidDigit })
    raw GET https://dv-g3b-aaa-emudhra.we-have-a-problem.com/blocks/tip/height: TLS OK, HTTP 200 OK, body "<h1>hello</h1>\n"
https://mempool.space/api: OK get_height() = 972514
native-tls: native-tls v0.2.18  openssl: openssl v0.10.81

The macOS run (native-tls → Security.framework) gives the same result: both handshakes succeed and return HTTP 200.

After: this branch (rustls)

Linux, distroless debian12 roots. Both are rejected as distrusted, and mempool.space still works:

https://dv-g3b-emudhra.we-have-a-problem.com: ERR Reqwest(reqwest::Error { kind: Request, url: "https://dv-g3b-emudhra.we-have-a-problem.com/blocks/tip/height", source: hyper_util::client::legacy::Error(Connect, Custom { kind: Other, error: Custom { kind: InvalidData, error: InvalidCertificate(Other(OtherError(CertDistrusted))) } }) })
https://dv-g3b-aaa-emudhra.we-have-a-problem.com: ERR Reqwest(reqwest::Error { kind: Request, url: "https://dv-g3b-aaa-emudhra.we-have-a-problem.com/blocks/tip/height", source: hyper_util::client::legacy::Error(Connect, Custom { kind: Other, error: Custom { kind: InvalidData, error: InvalidCertificate(Other(OtherError(CertDistrusted))) } }) })
https://mempool.space/api: OK get_height() = 972514
native-tls: error: package ID specification `native-tls` did not match any packages
openssl: error: package ID specification `openssl` did not match any packages

macOS gives identical output.

Dependency graph

  • native-tls, openssl, openssl-sys, hyper-tls and tokio-native-tls are gone from Cargo.lock. cargo tree -i for each, with --all-features --target all -e all, reports no match.
  • The one remaining crate with "openssl" in its name is openssl-probe. It doesn't use OpenSSL: it's a pure-Rust crate that rustls-native-certs uses to find the system CA bundle path on Linux. It comes in through rustls-platform-verifier, which reqwest 0.13 always uses in rustls mode (the kormir/p2pderivatives oracle clients), and through postgresql_archive's rustls native roots. Paths verified that way still go through rustls-webpki 0.103.16, so the distrusted intermediates are rejected even with the system root store.
  • cargo tree -i rustls-webpki: a single version, 0.103.16. The other rustls users (reqwest 0.13 for the kormir/p2pderivatives oracle clients, nostr's tungstenite, rustls-platform-verifier) all resolve to it. webpki-roots 0.26.11 is still in the lockfile (via nostr's tokio-tungstenite), but it's the compat shim that re-exports webpki-roots 1.0.10.

Longer term

esplora-client 0.13 moved to bitreq with rustls by default. No bdk_esplora release uses it yet (0.22.3 is still on esplora-client 0.12). Once one does, moving to it removes reqwest 0.12 and this feature trap from the esplora path entirely.

Tests

  • cargo fmt --check, cargo clippy -- -D warnings, and cargo +1.88 check --all-features (MSRV): pass.
  • cargo audit --ignore RUSTSEC-2023-0071: passes, exit 0 (master currently fails with 16 advisories). Only the existing "unmaintained"/unsound warnings remain (fxhash, instant, nostr-relay-builder, nostr-relay-pool, lru), and they don't fail the job.
  • The esplora before/after check above was rerun on the final lockfile: both eMudhra hosts still fail with CertDistrusted, and mempool.space still succeeds.
  • I ran the CI commands on the final tree in a linux/amd64 Debian container (the same OS family as the CI runners):
    • cargo test --all-features --no-fail-fast: every target passes except ddk --lib, where 17 postgres storage tests fail. master fails the same 17, test for test. The current theseus postgres 18.6.1 Linux build ships a partial share/timezone with no UTC, so the server rejects sqlx's TimeZone=UTC. CI is shielded by its ~/.theseus cache. The embedded postgres download itself now runs over rustls, and the server installs and starts. The tests that run against bitcoind/electrs 0.43 (wallet::tests, balance, chain, nostr) pass.
    • The CI integration matrix (--ignored, features ddk-manager/parallel,ddk-manager/use-serde): manager_execution_tests 86/86 and stateless_execution 47/47 pass against bitcoind/electrs 0.43. manager_tests (7) and contract_updater (1) have no ignored tests; they ran and passed in the unit step.

bdk_esplora's `async-https` feature turns on reqwest 0.12 `default-tls`,
which is native-tls (OpenSSL with the system root store on Linux). Eight
eMudhra intermediate CA keys are compromised and chain to roots that
system stores still ship ("AAA Certificate Services", "emSign ECC Root
CA - G3"), so anyone can mint a certificate the esplora client accepts
and feed a node fake chain state.

Switch to `async-https-rustls` and bump the verification crates:

- rustls-webpki 0.103.16 rejects paths through the distrusted SPKIs
  regardless of root store.
- webpki-roots / webpki-root-certs 1.0.10 drop the affected trust.
- rustls 0.23.46 fixes RUSTSEC-2026-0346..0349, now on the esplora path.

Downstream crates cannot undo this themselves: features are additive,
and reqwest 0.12 picks native-tls whenever `default-tls` is on, even
with `rustls-tls` also enabled.
postgresql_embedded's default features use native-tls, which was the
last path to OpenSSL in the workspace (reqwest 0.12 via
postgresql_archive). Use its `rustls` feature instead. The lockfile no
longer contains native-tls, openssl, openssl-sys or hyper-tls.
- bitcoind / electrsd 0.41 -> 0.43.2: bitcoind's `download` build
  script no longer depends on zip 0.6.6 (RUSTSEC-2026-0367, -0368,
  -0369).
- negentropy 0.5.0 -> 0.5.2 (RUSTSEC-2026-0365), via nostr.
@bennyhodl bennyhodl changed the title ddk: verify esplora TLS with rustls instead of native-tls Move every TLS path onto rustls (eMudhra distrust) and clear cargo audit Oct 21, 2026
@bennyhodl
bennyhodl merged commit 38d67be into master Oct 21, 2026
188 of 294 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant