From 1e8eb94e7e71c0e6653334b306a687347a4ea91e Mon Sep 17 00:00:00 2001 From: grumbach Date: Wed, 21 Oct 2026 12:09:21 +0900 Subject: [PATCH 1/5] feat(payment)!: enforce the close-group price floor on every node The ADR-0006 floor has run in shadow mode since v0.17.1. A week of production telemetry (2026-10-14..20, 699 node instances on 72 hosts) showed that its 65% default would have refused 2.5% of admissions, and 2.1% of chunks at every receiver that saw them. Neighbours gossiping inflated commitments could refuse a chunk everywhere. One accepting node could still replicate a below-floor write through PaidNotify and paid-list repair. And a peer that refused a write on price would be penalised for it by possession checks, responsible audits and prune audits. This makes enforcement safe and turns it on for every node, with nothing to configure. - Cap the reference at the receiver's own price: min(group median, calculate_price(own committed count)). A node never demands more than the tolerance of what it would itself quote, so lying neighbours cannot make it refuse a payment that clears its own price. - Lower the tolerance from 65% to 30%, a deliberate starting point. No measured admission is refused at 30% or at 40%, but the worst clears 30% by 1.78x and 40% by only 1.34x, on a week of 48 upload bursts. A baseline quote is still refused by 91% of receivers (98% at 40%). - Floor paid-list admission too. PaidNotify only announces a client's write, with the proof the client presented, so this reprices no historical receipt. It stops one accepting node from giving a below-floor write the paid-list votes that repair needs. - Judge no peer on a contested write. A payment is contested when the node cannot price it, or its paid price is below 60% of the uncapped group median (twice the floor). The verifier remembers each contested chunk payment it admits, in a memory of its own, and every record path (PUT, fresh offer, PaidNotify) marks the key's paid-list entry from that decision, never from the unchecked proof a cache hit carries; the first paid-list record of a key decides. A marked key goes out in paid hints only, so no repair proof is recorded, no responsible audit samples it and no peer penalises this node as its sole hinter; the paid list keeps its marked keys in memory too, so building hints scans nothing on disk for them. A prune audit still needs its proofs but scores no absent peer; paid-list pruning keeps the marked entry while the record is held; and a mark that cannot be read counts as set, and is logged. A chunk fan-out schedules no possession check when the write is marked (by the verified decision or by the record that won, read after its own is written) or the proof it forwards is contested, and a pointer fan-out none when its proof is contested. Releases before this one only test paid-list presence, so they read a marked key as paid. The 2x margin covers the largest spread measured between receivers' views (1.91x); it is a margin, not a guarantee. - One floor for every node. PriceFloorConfig and PaymentVerifierConfig::price_floor are removed: the tolerance is the constant PRICE_FLOOR_TOLERANCE_PERCENT and the contest threshold a private one. ANT_PRICE_FLOOR_ENFORCE and ANT_PRICE_FLOOR_TOLERANCE_PERCENT are no longer read, and backing the floor out takes a release. - The telemetry line adds target, group_median_price, own_price, paid_quote_age_secs and contested, and keeps enforce=true constant. - A comment on the group sample says why it needs no self-filter: the routing scan includes this node, but the gossip cache never holds it, since ingest accepts only senders in the routing table, so its own commitment joins the sample exactly once. - Repair tests/poc_price_floor_live.rs, which no longer exercised the close-group floor and failed on main. Pin the chunk and pointer possession schedulers, the mark and hints, the prune audit and all three record paths with e2e tests, each with a control. Add an env-gated A/B workload driver, tests/e2e/price_floor_ab.rs. ADR-0006 gains Amendment 2, recording the measurement, these decisions and the residuals. ADR-0008's status row is updated. BREAKING CHANGE: PriceFloorConfig (with from_env, enforce and tolerance_percent), PaymentVerifierConfig::price_floor, PRICE_FLOOR_ENFORCE_ENV and PRICE_FLOOR_TOLERANCE_ENV are removed, and PRICE_FLOOR_TOLERANCE_PERCENT is added. The floor is enforced at 30% on every node. neighbor_sync::build_replica_hints_for_peer and audit::audit_tick_with_repair_proofs take the paid list, and ReplicationEngine::replicate_fresh is only built with test-utils. --- .../ADR-0006-receiver-side-revenue-floor.md | 486 +++++- ...-storage-economics-and-payment-protocol.md | 6 +- src/devnet.rs | 3 +- src/node.rs | 13 +- src/payment/mod.rs | 5 +- src/payment/verifier.rs | 1509 +++++++++++------ src/pointer/service.rs | 3 +- src/replication/audit.rs | 55 +- src/replication/fresh.rs | 134 +- src/replication/mod.rs | 42 +- src/replication/neighbor_sync.rs | 80 +- src/replication/paid_list.rs | 531 +++++- src/replication/pointer.rs | 15 + src/replication/pruning.rs | 141 +- src/storage/handler.rs | 7 +- tests/e2e/data_types/chunk.rs | 5 +- tests/e2e/harness.rs | 35 +- tests/e2e/mod.rs | 3 + tests/e2e/pointer_replication.rs | 95 ++ tests/e2e/price_floor_ab.rs | 743 ++++++++ tests/e2e/replication.rs | 433 ++++- tests/e2e/testnet.rs | 1 - tests/poc_price_floor_live.rs | 235 +-- 23 files changed, 3796 insertions(+), 784 deletions(-) create mode 100644 tests/e2e/price_floor_ab.rs diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index 4dc93035..c7194743 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -1,7 +1,8 @@ # ADR-0006: Close-group pricing for the receiver-side revenue floor - **Status:** Proposed -- **Date:** 2026-07-14 (amended 2026-08-05 after canary measurement) +- **Date:** 2026-07-14 (amended 2026-08-05 after canary measurement; Amendment 2 + 2026-10-20: enforced on every node) - **Decision owners:** Anselme (@grumbach) - **Reviewers:** - **Supersedes:** none (amends the economic scope of ADR-0004; see Decision) @@ -129,6 +130,13 @@ prospective rather than observed. ## Decision +> **Amended.** [Amendment 2](#amendment-2-2026-10-20-enforced-on-every-node) +> changes four things below: the tolerance (65% → 30%), the reference (now +> capped at the receiver's own price), the scope (paid-list admission is +> floored too) and the rollout (enforced on every node, with nothing to +> configure). Where the two disagree, the amendment wins. The text here is kept +> as decided, so the reasons for each step stay readable. + On single-node store admissions (direct client PUT and immediate fresh replication), the receiver additionally requires, alongside the existing `settled >= 3 x median` check: @@ -260,6 +268,482 @@ payment when it settles below its close group's median-priced floor. revisiting this ADR. Changing the default tolerance or extending the floor to any currently exempt path also requires revisiting it. +## Amendment 2 (2026-10-20): enforced on every node + +This amendment turns enforcement on for every node, with nothing to +configure. To make that safe it also adds a cap to the reference, lowers the +tolerance, floors paid-list admission, and keeps a refusal under the floor, +within a stated margin and once nodes run this release, from costing the +refuser trust. It answers the review +trigger in Validation above. The manipulation result now has a fix for the +dangerous direction and an accepted bound for the other. One-quote liveness +is accepted as a residual that could not be measured. + +### What the shadow period and a review of the code found + +Shadow telemetry from ant-prod-01, 2026-10-14T00:33Z to 2026-10-20T00:07Z, on +node 0.22.0 and 0.23.0: 2,117 floor lines from 699 node instances on 72 hosts. +54 of them (2.6%) skipped, all `thin_sample` with 14 known neighbours; across +all lines the median receiver knew 18. Replaying the logged formula +reproduced all 53 of the nodes' own `below_floor=true` lines (47 distinct +admissions) exactly. Taking each chunk and receiver once leaves 1,898 +admissions over 536 chunks (944 first seen as a client PUT, 954 as fresh +replication), arriving in 48 upload bursts. Of those, 1,801 admissions over +521 chunks were matched to the receiver's own committed count, from the +`Storage commitment rotated` line written when `current()` changes. + +- **65% was not safe.** Against the group median it would have refused 47 of + the 1,898 admissions (2.5%), all among the 1,801 matched, and 2.1% of + chunks would have been refused by every receiver that evaluated them. The + window 2026-09-10 to 09-16 showed the same shape: 3.4% of admissions at + 65%. +- **Honest payments sit further below the group median than the simulation + assumed.** The lowest observed ratio of the settled amount to `3 x` group + median was 0.473, and 1% of admissions settled below 0.57x. Receivers of the + same chunk also disagree about the median: by 1.08x at the median, 1.36x at + p95 (1.37x among receivers matched to their own commitment) and 1.91x at + worst. A node's own committed count swings hour to hour as neighbours come + and go (3,132 to 5,921 keys within nine hours on one node), while the fleet + median held between about 3,700 and 4,100 keys all week. +- **Griefing takes the whole chunk, not one replica.** Every receiver of a + chunk prices against nearly the same median, so neighbours that push it up + make every receiver refuse, after the money has settled. That was the first + enforcement blocker above. +- **One accepting node was enough to replicate a below-floor write.** The node + that stores a write announces it with `PaidNotify` to its paid close group + before it sends fresh offers. `PaidNotify` was verified as + `PaidListAdmission`, which the floor exempted, so peers recorded the key as + paid and paid-list repair later copied the bytes from that one node. The + floor refused the fresh offers and then let repair deliver the same data. +- **A refusal became a trust penalty.** Shadow mode never refused anything, + so none of this has fired, but four existing checks judge a peer on data it + does not hold: + - the storing node probes every fresh-offer target 5 to 15 minutes later + and penalises a peer that does not hold the data: at weight 5 for a chunk + (about a 48% cut in trust) and weight 1 for a pointer (about 12%); + - neighbour sync records a repair proof for every key a node hints to a + peer as a replica, and from an hour later the responsible audit + challenges that peer on the key, at weight 5; + - a node that has held an out-of-range record for three days challenges + the current close group before deleting it, and penalises an absent peer + at weight 5; + - a peer whose quorum check fails for a key only one node hinted penalises + that node, at weight 1, up to three times per verification round. + + A peer that refused on price would be penalised by the first three for + following policy, and the node that accepted the write by the last. + +### Decision + +1. **The reference is capped at the receiver's own price:** + `reference = min(group median price, calculate_price(own committed count))`. + A receiver never demands more than the tolerance of what it would itself + quote right now. So `settled >= 3 x tolerance x own price` always clears + it, however many neighbours lie. While gossip is honest, a receiver above + the group median prices exactly as before. The withdrawn own-price rule's + failure, honest rejections on the fullest nodes, therefore does not return. +2. **Tolerance 30%, a deliberate starting point.** 40% also refuses no + measured admission, but the week measured is light and 40% leaves less + margin (see below). Starting lower gives up some detection for that + margin, and the trade is accepted on purpose. A follow-up reads enforced + traffic, and only a later amendment raises the tolerance toward 40% (see + Validation of this amendment). +3. **Paid-list admission is floored too.** `PaidNotify` is only sent when a + node stores a client's write, and it carries the proof the client + presented for it, within the same reuse window as the PUT itself. Repair + and fetch never call the verifier, and an already verified key is answered + from the cache. So this floors the one remaining fresh economic event and + still reprices no historical receipt. (A client PUT that hits the payment + cache forwards whatever proof the client attached, unverified, exactly as + before this change; its receivers verify it.) +4. **No peer is judged on a contested write.** When a node's verifier first + admits the payment for a write (as the node a client PUT it to, on a fresh + offer, or on a `PaidNotify`), it prices the proof against its current + view; a later admission of the same write from the payment cache reuses + that decision. The write is *contested* in two cases: + - the node cannot price it (no own commitment, or too thin a view); + - its paid price is below 60% of the **uncapped** group median, twice the + 30% floor. + + An admitted contested write, one the node cannot price or one paid between + the floor and that threshold, is still stored and its offers still sent, + but the node does not judge its close-group peers on it: + - a fan-out (chunk or pointer) schedules no possession check. This is + decided from the proof as it stands, even one a cached PUT forwards + unverified, since its peers verify that proof and may refuse it; + - for a chunk, the key's paid-list entry carries a one-byte mark, which + survives restarts. Releases before this one only test presence, so they + read it as an ordinary paid key; + - a marked key goes out in paid hints only, never as a replica hint. No + repair proof is recorded for it, so no responsible audit challenges a + peer on it, and no peer can penalise this node as its sole hinter. A + paid hint is never punished, and repair still works wherever the group + accepted the key. The node also keeps its marked keys in memory, so + leaving them out of the hints it builds for a peer reads nothing on + disk. Each paid-list write updates that copy in commit order, a new + mark just before its commit, so the copy never lacks a committed mark. + If the marks cannot be read when the node starts, it still starts, as + before, and hints no key as a replica until a restart can read them; + - the responsible audit never samples a marked key, which also covers one + hinted in the moment before its mark was written; + - a prune audit still needs its proofs before deleting a marked key, but + scores no absent peer for it, and paid-list pruning keeps a marked entry + while the record is held; + - a mark that cannot be read counts as set: when in doubt, judge no one. + While reads fail the node judges no peer on any key they hit, so it + warns at the first failure of a run and, when a read succeeds again, + logs how many failed. + + The mark follows the decision the node took when it verified the payment: + the verifier remembers which admitted chunk payments the floor contested, + in a memory of its own sized for 100,000 of them, so a small payment cache + cannot make it forget one before the key is recorded. A write admitted + later from the payment cache is recorded by that decision. The proof + attached to it was never checked, and anyone can attach one to a + `PaidNotify` or a fresh offer, so it can neither set the mark nor veto it. + The first paid-list record of a key decides its mark, so a later payment + can neither add nor remove it. A fan-out skips its possession check + whenever the write is marked, by this node's verified decision or by the + record of the key that won, read after the write records its own, as + well as when the proof it forwards is contested. Pointers need no mark: + the possession check is the only check that penalises a peer for a + missing pointer. + + The paid quote's price stands in for a third of the settled amount; + settlement is at least that, so an overpaid write is only ever judged more + leniently. Deciding from the proof itself means a write stored after + prices moved is judged like any other. + + The 2x margin covers the largest spread measured between receivers' views + of the same chunk (1.91x; 1.37x at p95 among matched receivers). **It is a + margin, not a guarantee**, because only the refusing peer knows its own + view. Replayed over all 5,536 ordered pairs of measured receivers of the + same chunk, at payment levels from 0.2x to 1x the median in 0.02x steps, + no write one would judge was refused by the other. A 1.5x margin left 4 + such pairs, at payments near 0.3x the median. The cost is coverage: on the + measured week these checks are given up for 1.4% of the 1,801 admissions + matched to their receiver's own commitment, plus the 2.6% of evaluations + the floor could not price. A mark stays while the node holds the key, so a + node keeps unchecked every key it recorded while it could not price + (its first hour before a commitment, or while its gossip view refills + after a restart); the other holders of those keys still check them, + unless the mark was front-run (see the consequences below). + Penalising a peer for following the floor is the worse failure, so that + coverage is given up. A real guarantee needs the + refusing peer to tell the sender. A peer that can say "refused on price" + can also dodge ADR-0003's full-node detection that way, so that is a + separate decision. + + The threshold is its own constant, so a raise of the tolerance needs care. + A node still at 30% judges every write paid above 0.6x of its median, and + a peer already at 40% refuses below 0.4x of its own view. On the measured + pairs that leaves 3 of 5,536 where a refusal would be judged. A raise + therefore takes two releases: the first moves this threshold to 80%, and + the tolerance moves only once nearly every node runs it. Nodes that never + upgrade past the first stay a residual. + + The contest reads the **uncapped** median, which the cap does not + protect. Neighbours that overstate their commitments raise it, more of + their group's writes then count as contested, and the peers holding those + writes go unchecked on them. A minority can lift the median only as far as + the dearest honest members. Overstating also needs fake leaves, which fail + the commitment byte spot-checks (ADR-0004). Neighbours that understate at + the storer narrow the margin instead: on the test fixture, seven of + fifteen pull its median down to the cheapest honest member, which takes + the threshold to about 0.33x of the honest median, so a peer whose own + view and price both sit at least 1.11x above the honest median may refuse + a write the storer judges. That needs the same minority that disarms the + floor (see Griefing below). +5. **Enforced on every node, with nothing to configure.** There is no + per-node switch and no tolerance setting: `ANT_PRICE_FLOOR_ENFORCE` and + `ANT_PRICE_FLOOR_TOLERANCE_PERCENT` are no longer read. A node whose floor + sat above its peers' would refuse writes they store and then fail their + possession checks for it, and one below would just admit underpayment, so + the tolerance is the same everywhere. A node that still sets either + variable gets no warning; the variable is ignored. Backing the floor out + takes a release, so the release train's canary is where it is stopped. A + node whose replication engine fails to start runs without one, as before, + and its floor then has nothing to price from and skips every evaluation + (`skip_reason=no_commitment_cache`). So does a local `ant-devnet`, which + runs no replication engine. + +The telemetry line keeps every earlier field. `enforce` is now always +`true`, which tells these nodes apart from shadow-mode releases while the +fleet upgrades (as does the presence of `target`). `reference_price` is +still the value `required` is computed from. It adds `target` (chunk or +pointer), `group_median_price`, `own_price`, `paid_quote_age_secs` and +`contested`. + +### Why the cap, and why 30% + +The measured admissions replayed through both rules, as admissions refused +out of 1,801: + +| tolerance | median only | capped | +|---|---|---| +| 30% (chosen) | 0 | 0 | +| 40% | 0 | 0 | +| 45% | 0 | 0 | +| 50% | 1 | 0 | +| 55% | 11 | 1 | +| 60% | 26 | 5 | +| 65% (old default) | 47 | 11 | + +- **Margin, not the zero count, picks the tolerance.** Zero refusals out of + 1,801 only bounds the true rate below 0.17% per admission, 0.58% per chunk, + or 6.4% per upload burst at 95% confidence, because admissions in one burst + are not independent. The tolerance therefore rests on how far the worst + observed payment sits above it. With the cap, the lowest ratio is 0.534, so + 30% leaves 1.78x of headroom. 40% would leave 1.34x, 45% 1.19x and 50% only + 1.07x. The week measured is light traffic, 48 upload bursts, so the floor + starts where the margin is widest and moves up as enforced traffic adds to + it. +- **Proof reuse.** A client may present the same proof for up to about a + day. A receiver's capped reference can rise with its own count (one node's + price can double within a day) up to its group median, so a reused proof is + safe only against the uncapped median. There, the worst measured admission + sat at 0.473x, 1.58x above 30% (1.18x above 40%). Fleet-wide the median + reference moved by up to about 1.2x in price from one day to the next, + which 1.58x covers. Neither per-group drift over a day nor refusals of aged + proofs were measured. The new `paid_quote_age_secs` field makes them + measurable. +- **What it catches.** The realistic underpayment pays a near-empty, fresh + or sybil quoter. A baseline quote is about a sixth of the group median. + Share of measured receivers that refuse a 3x payment to a quoter, with the + cap: + + | quoter's count, as a share of the group median | 30% (chosen) | 40% | + |---|---|---| + | 0 (a baseline quote) | 91.4% | 98.0% | + | a quarter | 75.4% | 94.6% | + | 40% | 27.9% | 75.2% | + | half | 0% | 39.1% | + + Every receiver observed for a chunk refuses a baseline quote on 84.5% of + chunks at 30%, and 95.2% at 40%. The median-only rule at 40% refuses the + first three rows at 100%, 99.6% and 95.8%. That detection is what the cap + and the lower tolerance give up. +- **What it does not catch.** A quoter 40% as full as its group is admitted + by 72% of receivers, and one half as full by all of them. Shopping for the + cheapest member of the close group is not caught either: members measured + in August spanned about 1.4x in price, so the cheapest sits no lower than + about 0.7x the median. No tolerance that is safe for honest traffic catches + that, with or without the cap. +- **Griefing.** In the worst case, every neighbour gossips the protocol + maximum. The median-only rule then refuses every honest payment. With the + cap, all 1,801 measured admissions still clear at 30% (1,799 at 40%). On + the test fixture, seven of fifteen overstating neighbours flip no verdict + and eight do. Understating is where 30% costs something. Six neighbours + claiming almost nothing leave the floor standing against a baseline quote, + and seven disarm it, because the median then falls to the cheapest honest + member (3,000 keys), and 30% of that member's price is below a baseline + quote. At 40% it took eight. Disarming admits an underpayment and costs no + one an honest payment, so this minority bound is an accepted residual of + starting at 30%. For overstating, the close group's honest majority stays + the trust boundary the rest of the protocol already rests on. + +### Consequences of this amendment + +- **Liveness is a quorum question.** A client needs 4 stores out of up to 20 + targets (a pointer needs 5 of 7). It moves to the next target, with the same + proof, when one refuses. Under the cap, every receiver whose own price is at + most `settled / (3 x 30%)` accepts unconditionally. That is nearly every + receiver for a payment at the group median. The measurement could not + observe the quorum itself: the median chunk shows only two receivers' client + PUT lines. +- **One-quote liveness is an accepted, unmeasured residual.** An honest client + whose only reachable quoter is a fresh node pays 3x baseline, and nearly + every mature receiver refuses it. This cannot be told apart from the attack. + None of the measured payments was at baseline, and the lowest paid issuer + committed 2,147 keys, but that does not bound how often it happens. A client + retries a refused proof + for up to about a day, then re-quotes and pays again. A client-side change, + such as not settling a lone baseline quote when a committed quote is + available, is the remaining fix and belongs to the client. +- **A write below the floor stays where it was accepted.** A near-empty quoter + accepts its own payment, because its cap is its own price. A node with no + commitment or a thin view skips the floor. Mature peers now refuse both the + fresh offer and the `PaidNotify`. Unless the accepting nodes make up a + majority of the group, neither paid-list repair nor presence repair has the + votes to authorise copying the data. The accepting node finds such a write + contested and marks it (item 4), so it judges none of the peers that + refused it while it holds the key. When the accepting nodes are a majority + of the paid group, the refusers repair the key from those votes like any + other, unmarked, since repair carries no proof to price; a refuser whose own + quorum check fails while its neighbours' passes can then be judged on it. +- **Narrow gaps remain on nodes running this release.** + - On the PUT path the record is stored a moment before its paid-list + entry is written. A sync that scans the store in that moment can hint + the key once; the audit filter keeps that from becoming an audit, but a + refusing peer can charge the storer weight 1 once. A crash in that + moment leaves the stored key unrecorded, so unmarked unless a later + verified `PaidNotify` records it. + - A PUT served from the payment cache with no proof attached stores the + record without writing a paid-list entry at all, as before this change, + so it is not marked either. + - The verifier's memory of contested admissions is bounded: a decision is + lost before its key is recorded only if 100,000 newer contested + admissions reach the node in the moments between its verification and + its record. + - A paid-list insert that fails before its commit leaves the key out of + the paid list, so unmarked. When a marked insert's commit is what + failed, the in-memory copy keeps the mark, since the commit may still + have reached the disk, so this node does not hint the key as a replica + while the mark stays; a restart rebuilds the copy from the database. + - Paid-list pruning checks for the record before and after its network + round, but a record admitted between that check and the delete would + lose its mark. Both store paths require this node inside the same width + the prune requires it outside, so that needs its routing view to change + at that moment. +- **The first paid-list record decides a key's mark, which a quoter can + front-run.** A quoter in the 20 closest to an honest key can pay itself + for the key, for gas only, and announce it before the client's write + lands. At a price between the floor and the contest threshold, or at any + price on nodes that cannot price, the nodes it reaches first mark the key + and leave it unchecked, though they still store and repair it; those among + the client's storers skip their possession check too, while storers it did + not reach first still run theirs. Paying above the + threshold first does the opposite: those nodes record the key unmarked and + can judge peers that later refuse the client's cheaper write, until those + peers repair it from the votes the front-run created. +- **Rollout skew is the main rollout risk.** Until the fleet upgrades, a node + on an older release admits every payment and still probes every + fresh-offer target 5 to 15 minutes later. Each upgraded peer that refused a + below-floor chunk it fanned out is penalised at weight 5. In that node's + view the peer falls from a neutral 0.5 to about 0.26, below the 0.35 swap + threshold, for about 9.5 hours (a pointer costs weight 1: 0.5 to 0.44). The + score is local to that node (for peers in its persisted close group it + survives a restart), and falling below 0.35 only lets a better candidate + replace the peer in that node's routing + table when its bucket is full; nothing is blocked. Honest writes are not + refused, so this needs below-floor writes, which an attacker can buy for + little more than gas by paying its own near-empty node within the 20 + closest to the key: one write stored by `s` older nodes yields up to `s` + penalties on each upgraded member of the close group. Such a key also + stays on the older node unmarked, before and after it upgrades: its replica + hints and responsible and prune audits keep judging the refusers on it, + repeatedly, while it holds the key, and the refusers penalise it (weight 1) + as the sole hinter of a key their quorum rejects. On the v0.17.0 rollout the + managed fleet went from 24 of 73 hosts in the first hour to 65 in six and + 71 within about 17; community nodes follow on their own schedule. Enabling + enforcement one release after item 4 ships would remove this, at the cost + of a release that does not enforce. That was not chosen. +- **Retry load is new.** A proof that was accepted before can now be refused. + Refusals are not cached, so every peer a client tries re-verifies it, the + on-chain read included: up to 20 targets over 4 rounds. The load falls on + below-floor proofs, including the honest residuals above (a lone baseline + quote, an aged proof). +- Floor refusals stay economic. The refusal itself feeds no trust or + misbehaviour scoring, and on nodes running this release item 4 keeps the + checks that judge peers on held data from turning it into a penalty, within + the margin stated there and apart from the gaps above. Merkle batches still + bypass the floor entirely. + +### Validation of this amendment + +- Unit tests pin each piece: + - the cap binds below the median, and no number of overstating neighbours + lifts a receiver above its own price; + - seven of fifteen overstating neighbours leave the verdict unchanged and + eight flip it, and six understating neighbours leave the floor standing + while seven disarm it (pinned as known limits); + - the cheapest member of a realistic group is admitted, and a near-empty + quoter is refused; + - paid-list admission and pointer payments are floored; + - a write below the contest threshold, or one the node cannot price, is + contested, and one at the median is not. The threshold is 60% of the + uncapped median to the atto, it stays on the median when the cap binds, + and it follows the view when the write is recorded; + - the tolerance is 30%: exactly `3 x 30%` of the group median clears it + and one atto less does not; + - a node with no floor wired contests nothing; + - the verifier remembers a contested payment it admitted, in a memory a + small payment cache cannot shrink, a later cache-hit admission reads that + decision, and a refused payment is never remembered; the paid-list mark + is written only with a new entry, survives a restart and a later plain + insert, and goes with the entry; + - a marked key is left out of replica hints and of responsible-audit + sampling, and its paid entry outlives paid pruning while its record is + held; + - the in-memory copy of the marks is filled on opening and follows every + write and removal, and a marked insert and a batch removal change it + only in the order writes take; building + replica hints works while the paid list cannot be read at all, and a + paid list whose marks cannot be read on opening still opens and hints no + replica; + - a mark that cannot be read counts as set in the responsible audit, paid + pruning and a write's fan-out, and a run of such reads logs one + warning. The tests make the read fail by holding the thread's one LMDB + reader slot; + - a record reports the winning record's mark, or its own when that is + set, so a plain decision that loses to a marked record still leaves the + write unjudged. +- Four e2e tests run the real schedulers on a local testnet, each with a + control. They settle nothing on chain, so where a test needs a verified + contested admission it stands one in; unit tests pin that a real + verification records it: + - a chunk PUT through the handler schedules no possession check when the + storer's verified decision was contested (the key is marked and hinted + to a close peer as paid only), when only the proof it forwards is + contested, or when a marked record of the key won before its own plain + one, while the control write is penalised; + - a fresh pointer state the storer cannot price is not used to judge its + peers, while the control is; + - a prune audit of a marked record keeps it but scores none of the absent + peers, while the control scores them; + - a PUT, a fresh offer and a `PaidNotify`, each admitted from the payment + cache with a forged proof the floor would contest, mark the key exactly + when the recording node's own verified decision was contested, and never + from that proof. +- 48 mutations of the floor, the contest decision, the verifier's memory, + the mark, its in-memory copy and every check that reads the mark were run + against these unit and e2e tests on this amendment's code, and 47 fail at + least one of them. The one that survives makes prune-audit scoring count + a mark that cannot be read as unset. Only the e2e prune test reaches that + check, and it cannot make a node's paid-list read fail; the same read + counts as set, under test, in the audit, paid pruning and the fan-out. +- `tests/e2e/price_floor_ab.rs` drives the same honest and near-empty-quoter + uploads through a production-wired local testnet. Nodes are seeded to + production-sized commitments, payments are real (on Anvil), and possession + checks are shortened. It reports who stores each write, whose paid list + holds it, and whether any peer is penalised, so two builds can be compared. + It asserts what the floor promises: + - every honest upload reaches a quorum with no floor refusal; + - no peer that refused a near-empty write, or that the quoter fanned it + out to, loses penalty-sized trust in the quoter's view; + - every near-empty write stays off a quorum, with every refusal the + floor's. A mature storer whose own price is low enough may accept it, as + the cap allows. + + It refuses to report when the floor could not evaluate at all. Its tail is + minutes, too short for the responsible audit (an hour) or a prune audit + (three days). The prune e2e test covers the prune audit; the responsible + audit is covered by keeping marked keys out of replica hints, so no repair + proof exists, and by the sampling filter's unit test. It returns at once + unless `PRICE_FLOOR_AB=1`, so CI only compiles it. +- `tests/poc_price_floor_live.rs` runs the floor against a real Anvil vault + and real signed neighbour commitments. It shows that a 3x-baseline + settlement is refused, and that an at-floor overpayment clears while one + atto less does not. +- After release, watch `ant_node::payment::price_floor` for `below_floor=true` + with `enforce=true`, split by `context` and `target`, along with the + `contested=true` and `skipped=true` rates. Any refusal of a payment whose + paid issuer holds a mature commitment is a reason to revisit this amendment. + So is a refusal rate above 0.1% of evaluations. +- **Raising to 40%.** Each evaluated line carries `settled` and + `reference_price`, so it answers whether 40% would have refused it: + `settled < 3 x 40% x reference_price`. The case for 40% is enforced traffic + where that never happens, counted per upload burst: chunks less than 120 s + apart, and one event if any admission in the burst would be refused. Zero + events in 300 bursts bound the rate below 1% at 95% confidence, against + 6.4% from the 47 bursts measured so far that hold a matched admission. + Bursts from one client are not independent, so that bound is the least the + evidence must show, not proof. Production logs are kept for seven days, so + each week's counts must be saved before they expire. Before any raise, + aged proofs (`paid_quote_age_secs`) and + payments to near-empty quoters are read the same way. The raise then + follows the two steps in item 4. + ## Notes for AI-assisted work AI tools may help draft this ADR, but **must not mark it Accepted without diff --git a/docs/adr/ADR-0008-storage-economics-and-payment-protocol.md b/docs/adr/ADR-0008-storage-economics-and-payment-protocol.md index 509f7cc5..aa7bc08e 100644 --- a/docs/adr/ADR-0008-storage-economics-and-payment-protocol.md +++ b/docs/adr/ADR-0008-storage-economics-and-payment-protocol.md @@ -359,7 +359,7 @@ and a node that was actually paid is nominated for a first audit. | Node re-verification of **non-paid** quotes in the bundle | not done (bad signature on an unpaid quote is accepted) | | Node rejection of off-curve quotes (`QUOTE_ARITHMETIC_RECHECK_ENABLED`) | **enforced**, on every quote in the bundle | | Quote/commitment mismatch reported to trust | off (telemetry only) | -| Receiver-side price floor (ADR-0006) | shadow (`ANT_PRICE_FLOOR_ENFORCE`, 65% tolerance), priced against the **close group's median** committed count, not the receiver's own; the own-price basis was withdrawn after canary measurement | +| Receiver-side price floor (ADR-0006) | **enforced** on every node at 30% of a reference price: the **close group's median** committed-count price, capped at the receiver's own; covers client PUT, fresh replication and paid-list admission; no per-node switch (ADR-0006 Amendment 2) | | Payee eligibility gate (ADR-0005) | **not merged**; observe-only on its branch (`ADR5_ENFORCE`) | | Client pays the 3× multiplier on the merkle path | **enforced** by the published client, immediately and unconditionally (no flag, no date) | | Storer requires it (`MERKLE_PARITY_ENFORCED_FROM_UNIX`) | **enforced** by upgraded nodes, for receipts stamped from 2026-08-04 15:00 UTC onward | @@ -452,7 +452,9 @@ indistinguishable from a single receipt. mechanism to reprice. - **Median-of-K is only as honest as the quote set the client presents.** A modified client can shop the neighbourhood and settle the cheapest valid - quote; the receiver-side floor is the counterweight and is not yet enforcing. + quote. The receiver-side floor, enforced on every node (ADR-0006 Amendment + 2), is the counterweight: it refuses near-empty quotes, but not the + cheapest member of the close group. ### Neutral / Operational diff --git a/src/devnet.rs b/src/devnet.rs index 18e459bb..ddf5170c 100644 --- a/src/devnet.rs +++ b/src/devnet.rs @@ -7,7 +7,7 @@ use crate::ant_protocol::CHUNK_PROTOCOL_ID; use crate::config::{default_root_dir, NODES_SUBDIR, NODE_IDENTITY_FILENAME}; use crate::logging::{debug, info, warn}; use crate::payment::{ - EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, PriceFloorConfig, QuoteGenerator, + EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, QuoteGenerator, QuotingMetricsTracker, }; use crate::replication::config::ReplicationConfig; @@ -960,7 +960,6 @@ impl Devnet { cache_capacity: DEVNET_PAYMENT_CACHE_CAPACITY, close_group_size: replication_config.close_group_size, local_rewards_address: rewards_address, - price_floor: PriceFloorConfig::default(), }; let payment_verifier = PaymentVerifier::new(payment_config); let metrics_tracker = QuotingMetricsTracker::new(DEVNET_INITIAL_RECORDS); diff --git a/src/node.rs b/src/node.rs index ac1f3588..63580116 100644 --- a/src/node.rs +++ b/src/node.rs @@ -9,9 +9,7 @@ use crate::event::{create_event_channel, NodeEvent, NodeEventsChannel, NodeEvent use crate::logging::{debug, error, info, warn}; use crate::payment::metrics::QuotingMetricsTracker; use crate::payment::wallet::parse_rewards_address; -use crate::payment::{ - EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, PriceFloorConfig, QuoteGenerator, -}; +use crate::payment::{EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, QuoteGenerator}; use crate::replication::config::ReplicationConfig; use crate::replication::fresh::FreshWriteEvent; use crate::replication::ReplicationEngine; @@ -504,16 +502,7 @@ impl NodeBuilder { cache_capacity: config.payment.cache_capacity, close_group_size, local_rewards_address: rewards_address, - // Shadow mode by default; enforcement is a per-node canary opt-in - // via ANT_PRICE_FLOOR_ENFORCE (see PriceFloorConfig). - price_floor: PriceFloorConfig::from_env(), }; - if payment_config.price_floor.enforce { - info!( - "Price floor ENFORCEMENT enabled (tolerance {}%)", - payment_config.price_floor.tolerance_percent - ); - } let payment_verifier = PaymentVerifier::new(payment_config); let metrics_tracker = QuotingMetricsTracker::new(0); let mut quote_generator = QuoteGenerator::new(rewards_address, metrics_tracker); diff --git a/src/payment/mod.rs b/src/payment/mod.rs index c9b2a32d..107f16f6 100644 --- a/src/payment/mod.rs +++ b/src/payment/mod.rs @@ -63,8 +63,7 @@ pub use ant_protocol::payment::verify::{ }; pub use single_node::SingleNodePayment; pub use verifier::{ - EvmVerifierConfig, PaymentStatus, PaymentVerifier, PaymentVerifierConfig, PriceFloorConfig, - VerificationContext, MAX_PAYMENT_PROOF_SIZE_BYTES, MIN_PAYMENT_PROOF_SIZE_BYTES, - PRICE_FLOOR_ENFORCE_ENV, PRICE_FLOOR_TOLERANCE_ENV, + EvmVerifierConfig, PaymentStatus, PaymentVerifier, PaymentVerifierConfig, VerificationContext, + MAX_PAYMENT_PROOF_SIZE_BYTES, MIN_PAYMENT_PROOF_SIZE_BYTES, PRICE_FLOOR_TOLERANCE_PERCENT, }; pub use wallet::{is_valid_address, parse_rewards_address, WalletConfig}; diff --git a/src/payment/verifier.rs b/src/payment/verifier.rs index 0d9a126d..a8f57c84 100644 --- a/src/payment/verifier.rs +++ b/src/payment/verifier.rs @@ -12,6 +12,7 @@ use crate::payment::proof::{ deserialize_merkle_proof, deserialize_single_node_proof, detect_proof_type, ProofType, }; use crate::replication::commitment::MAX_COMMITMENT_KEY_COUNT; +use crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL; use crate::replication::config::K_BUCKET_SIZE; use crate::storage::ChunkStore; use ant_protocol::payment::verify::{verify_quote_content, verify_quote_signature}; @@ -30,7 +31,7 @@ use saorsa_core::P2PNode; use std::collections::HashMap; use std::num::NonZeroUsize; use std::sync::Arc; -use std::time::Instant; +use std::time::{Instant, SystemTime}; /// Minimum allowed size for a payment proof in bytes. /// @@ -61,29 +62,33 @@ const PAYMENT_VERIFY_SLOW_LOG_MS: u128 = 500; /// This is the Kademlia K width, intentionally wider than `CLOSE_GROUP_SIZE`. const PAID_QUOTE_ISSUER_CLOSENESS_WIDTH: usize = K_BUCKET_SIZE; -/// Default tolerance for the receiver-side price floor, as a percentage of the -/// close group's MEDIAN commitment-bound price (see -/// [`PaymentVerifier::group_reference_price`]). +/// The receiver-side price floor, as a percentage of the reference price. /// -/// Derived from production, not chosen: on ant-prod-01 over 2026-07-29..08-04, -/// pricing each receiver against its OWN price rejected 0.377% of honest stores -/// while still admitting the cheapest-of-K underpayment at 62.9% of receivers, -/// because node prices span ~4x across the fleet and an honest client pays the -/// group median. Against the group median instead, honest payments land near the -/// reference and the underpayment lands well below it, so a tolerance separates -/// them. +/// The reference is the close group's median commitment-bound price, capped +/// at the receiver's own; a single-node payment must settle at least `3x` +/// this share of it. One network value, the same on every node and never +/// configured: a node whose floor sat above its peers' would refuse writes +/// they store and fail their checks for it. Chosen from production telemetry +/// as a deliberate starting point below what the measurements allow; the +/// measurements, and the rule for raising it, live in ADR-0006 Amendment 2. +pub const PRICE_FLOOR_TOLERANCE_PERCENT: u64 = 30; + +/// A write whose paid price is below this percentage of the UNCAPPED group +/// median is contested: a close-group peer whose view of the group sits higher +/// than this node's may have refused it under its floor, so this node does not +/// judge its peers on it (see [`PaymentVerifier::price_contested`]). /// -/// At the minimum permitted view, 65% rejects 0.140% of honest payments while -/// catching ~90% of underpayment, against the own-price floor's measured 0.377% -/// and 37.1%. Both axes improve, by roughly 2.7x on rejections rather than the -/// order of magnitude an earlier revision claimed. 60% is the conservative -/// alternative: 5x safer on honest traffic, at 66% detection. -/// -/// The full sweep, the sample-size table and the simulation's assumptions live -/// in ADR-0006 — deliberately in one place, because the same tables -/// duplicated here and in the ADR had already drifted apart once. Tighten or -/// loosen only from observed shadow telemetry, never speculatively. -const PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT: u64 = 65; +/// Twice the floor of this node's view, so a peer needs a view twice as high +/// before it can refuse a write this node judges. A margin, not a guarantee: +/// ADR-0006 Amendment 2 carries the measured spread between receivers' views +/// and the replay this was checked against. It is its own constant because a +/// raise of the tolerance moves it first, one release ahead. +const PRICE_FLOOR_CONTESTED_PERCENT: u64 = 60; + +/// Contested chunk admissions the verifier remembers, the most recent first, +/// so each at least until its paid-list record is written (see +/// [`PaymentVerifier::verified_payment_contested`]). +const PRICE_CONTESTED_MEMORY_CAPACITY: usize = 100_000; /// Minimum number of fresh NEIGHBOUR commitments (this node's own is counted /// separately) required before the price floor will evaluate at all. @@ -103,13 +108,12 @@ const PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT: u64 = 65; /// [`PAID_QUOTE_ISSUER_CLOSENESS_WIDTH`] closest, with the full table in /// ADR-0006. /// -/// Availability cost is expected to be small — the gossip TTL is hours against a -/// much shorter sweep, so a settled node should normally know most of its -/// neighbours — but that is an assumption, not a measurement, and the sample is -/// drawn from peers close to the CHUNK while the cache is filled from peers -/// close to THIS NODE, so edge-of-range chunks will skip more often. The -/// `skipped=true` / `skip_reason` telemetry exists to measure that in shadow -/// mode before anyone enforces. +/// Availability cost is small, and measured: on ant-prod-01 over +/// 2026-10-14..20, 54 of 2,117 floor lines (2.6%) skipped, all `thin_sample` +/// with 14 known neighbours; across all lines the median receiver knew 18. +/// The sample is drawn from peers close to the CHUNK while the cache is filled +/// from peers close to THIS NODE, so edge-of-range chunks skip more often. The +/// `skipped=true` / `skip_reason` telemetry keeps measuring it. /// /// A node that could suppress its neighbours' gossip can therefore disable this /// receiver's floor, but cannot make it reject honest traffic. That asymmetry is @@ -118,9 +122,9 @@ const PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS: usize = 15; /// Why a price-floor evaluation did not produce a reference price. /// -/// Shadow mode exists to calibrate -/// [`PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS`] from real deployments, and these -/// three causes call for completely different responses: a mid-rotation node is +/// The telemetry calibrates [`PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS`] from +/// real deployments, and these three causes call for completely different +/// responses: a mid-rotation node is /// expected and self-correcting, an unwired cache is a startup-order bug, and a /// persistently thin sample means the gate is set above what the network can /// supply. Collapsing them into one counter would make the telemetry unable to @@ -151,6 +155,17 @@ impl SkipReason { } } +/// What the price floor priced one admission against. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct FloorReference { + /// The amount the tolerance applies to: the lower of the two below. + price: Amount, + /// The close group's lower-median commitment-bound price. + group_median: Amount, + /// The price this node would itself quote right now. + own: Amount, +} + /// What the price floor could see of the close group on one admission. /// /// `neighbours` is the quantity the gate actually tests; `total` includes this @@ -188,107 +203,6 @@ impl GroupSample { } } -/// Environment variable enabling price-floor ENFORCEMENT (`1`/`true`). -/// -/// Absent or any other value means shadow mode: the floor is computed and -/// logged, never enforced. Per-node so enforcement can roll out canary-first; -/// unsetting it (and restarting) is the kill switch. -pub const PRICE_FLOOR_ENFORCE_ENV: &str = "ANT_PRICE_FLOOR_ENFORCE"; - -/// Environment variable overriding the price-floor tolerance percentage -/// (`0..=100`). Invalid or absent values use the default. -pub const PRICE_FLOOR_TOLERANCE_ENV: &str = "ANT_PRICE_FLOOR_TOLERANCE_PERCENT"; - -/// Receiver-side price floor for single-node store admissions. -/// -/// ADR-0004 gives every quote a price CEILING (`price == -/// calculate_price(committed_key_count)`), but a ceiling is not a revenue -/// floor: a modified client can fetch quotes from the whole neighbourhood and -/// settle only the cheapest valid one in a 1-quote proof. This policy is the -/// floor half: the settled amount must also clear the close group's MEDIAN -/// commitment-bound price, scaled by a tolerance. -/// -/// The reference is the group's median rather than this receiver's own price. -/// A client pays `3x` the median of the quotes it collected, so comparing that -/// to a single node's price compares a median to one draw from a distribution -/// spanning ~4x — which in production rejected honest stores on the fullest -/// nodes while admitting underpayment at the emptiest. The reference is built -/// only from this node's own commitment and its neighbours' TTL-fresh gossiped -/// ones, never from the payment bundle. -/// -/// Rollout is shadow-first: with `enforce == false` (the default) the floor -/// is evaluated and logged on every single-node store admission but never -/// rejects. Enforcement is enabled per node via [`PRICE_FLOOR_ENFORCE_ENV`] -/// for canary rollout. A floor rejection is an economic admission decision -/// only — it must never feed trust/misbehaviour scoring. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub struct PriceFloorConfig { - /// When true, a below-floor settled payment is rejected. When false - /// (default), the verifier only logs would-reject telemetry. - pub enforce: bool, - /// Floor as a percentage of the close group's median commitment-bound - /// price (`0..=100`). - pub tolerance_percent: u64, -} - -impl Default for PriceFloorConfig { - fn default() -> Self { - Self { - enforce: false, - tolerance_percent: PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT, - } - } -} - -impl PriceFloorConfig { - /// Build from the environment (see [`PRICE_FLOOR_ENFORCE_ENV`] and - /// [`PRICE_FLOOR_TOLERANCE_ENV`]). Never panics. - /// - /// An unset tolerance uses the default. A tolerance that is *present but - /// invalid* (unparseable or `> 100`) is an operator error: rather than - /// silently enforce at the default, this **fails closed to shadow mode** - /// (`enforce = false`) and logs a prominent error, so a fat-fingered - /// tolerance can never enforce an unintended floor against real payments. - #[must_use] - pub fn from_env() -> Self { - let enforce_requested = std::env::var(PRICE_FLOOR_ENFORCE_ENV) - .is_ok_and(|v| matches!(v.trim(), "1" | "true" | "TRUE" | "True")); - - let tolerance_raw = std::env::var(PRICE_FLOOR_TOLERANCE_ENV).ok(); - let valid_tolerance = tolerance_raw - .as_deref() - .map(str::trim) - .and_then(|s| s.parse::().ok()) - .filter(|percent| *percent <= 100); - // A tolerance that is set but does not parse to `0..=100` is an operator - // error. Fail CLOSED: never enforce a tolerance the operator did not - // actually specify. - let tolerance_present_but_invalid = tolerance_raw.is_some() && valid_tolerance.is_none(); - - if tolerance_present_but_invalid { - if enforce_requested { - crate::logging::error!( - "{PRICE_FLOOR_TOLERANCE_ENV}={tolerance_raw:?} is not an integer in 0..=100; \ - refusing to enforce the price floor with an unspecified tolerance. \ - Price-floor enforcement DISABLED (shadow mode). Fix the value and restart \ - to enable enforcement." - ); - } else { - crate::logging::warn!( - "{PRICE_FLOOR_TOLERANCE_ENV}={tolerance_raw:?} is not an integer in 0..=100; \ - using the default {PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT}% for shadow-mode \ - telemetry." - ); - } - } - - Self { - enforce: enforce_requested && !tolerance_present_but_invalid, - tolerance_percent: valid_tolerance.unwrap_or(PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT), - } - } -} - #[derive(Clone, Copy)] struct LegacyMedianCandidate<'a> { encoded_peer_id: &'a evmlib::EncodedPeerId, @@ -444,8 +358,6 @@ pub struct PaymentVerifierConfig { /// Kept in the verifier config for payment policies that bind receipts to /// this node's payout address. pub local_rewards_address: RewardsAddress, - /// Receiver-side price floor policy (shadow mode by default). - pub price_floor: PriceFloorConfig, } /// What a payment authorizes, and where it routes. @@ -523,8 +435,12 @@ impl PaymentTarget { /// touching direct-PUT behaviour. /// - **`PaidListAdmission`** — the node is admitting fresh paid-list metadata. /// It runs the same live payment checks, but writes a weaker cache entry -/// that does not authorize future chunk stores. The price floor never -/// applies here: paid-list records reprice no fresh economic decision. +/// that does not authorize future chunk stores. The price floor applies +/// here too: a `PaidNotify` announces a write a client has just made, with +/// the proof the client presented (on a payment-cache hit, whatever proof it +/// attached, as before), and a paid-list vote is what later lets repair copy +/// the data, so exempting it would let one accepting node replicate a +/// below-floor write to its whole group. /// /// The caller must check local receiver/admission membership before invoking /// the verifier for replication admission: fresh chunk replication requires @@ -740,11 +656,6 @@ pub struct PaymentVerifier { /// entirely rather than pricing against anything, so a node in that state /// never rejects. local_commitment_source: RwLock>>, - /// Live price-floor policy, initialized from - /// [`PaymentVerifierConfig::price_floor`]. Behind a lock so tests (and any - /// future ops surface) can flip enforcement without rebuilding the - /// verifier. - price_floor: RwLock, /// ADR-0008: the instant from which a merkle receipt must settle at the /// single-node 3x multiplier, defaulting to /// [`crate::replication::config::MERKLE_PARITY_ENFORCED_FROM_UNIX`]. @@ -769,6 +680,17 @@ pub struct PaymentVerifier { /// Incremented only on first emission for a pool, so it measures /// settlements sampled rather than chunks admitted. merkle_parity_emissions: std::sync::atomic::AtomicUsize, + /// ADR-0006: chunk addresses whose payment this verifier admitted while the + /// price floor contested it (see [`Self::verified_payment_contested`]). + /// Written during a full verification, before the payment cache learns + /// the chunk, so every later record of it, a cache hit included, is judged + /// by the proof that was verified and never by whatever proof came with + /// it. Only contested admissions enter, and a chunk is recorded within + /// moments of its verification, so the fixed + /// [`PRICE_CONTESTED_MEMORY_CAPACITY`], whatever the payment cache is sized + /// to, loses a decision before it is read only if that many newer + /// contested admissions arrive in between. + verified_price_contested: Mutex>, /// Configuration. config: PaymentVerifierConfig, } @@ -858,6 +780,9 @@ impl PaymentVerifier { "pool cache capacity must be > 0" ); let cache = VerifiedCache::with_capacity(config.cache_capacity); + let verified_price_contested = Mutex::new(LruCache::new( + NonZeroUsize::new(PRICE_CONTESTED_MEMORY_CAPACITY).unwrap_or(NonZeroUsize::MIN), + )); let pool_cache_size = NonZeroUsize::new(DEFAULT_POOL_CACHE_CAPACITY).unwrap_or(NonZeroUsize::MIN); let pool_cache = Mutex::new(LruCache::new(pool_cache_size)); @@ -897,7 +822,6 @@ impl PaymentVerifier { ))), monetized_pin_tx: RwLock::new(None), local_commitment_source: RwLock::new(None), - price_floor: RwLock::new(config.price_floor), merkle_parity_from: RwLock::new( crate::replication::config::MERKLE_PARITY_ENFORCED_FROM_UNIX, ), @@ -906,6 +830,7 @@ impl PaymentVerifier { .unwrap_or(NonZeroUsize::MIN), )), merkle_parity_emissions: std::sync::atomic::AtomicUsize::new(0), + verified_price_contested, config, } } @@ -986,17 +911,82 @@ impl PaymentVerifier { debug!("PaymentVerifier: local commitment source attached for price-floor checks"); } - /// The live price-floor policy this verifier applies. - #[must_use] - pub fn price_floor_config(&self) -> PriceFloorConfig { - *self.price_floor.read() + /// Whether this node must not judge its close-group peers on a write routed + /// at `routing` and paid by `proof`: a peer may have refused it under its + /// own floor, and refusing a below-floor payment is network policy, never + /// misbehaviour (ADR-0006). + /// + /// True when the floor cannot price the write here (no own commitment, or + /// too thin a view), or when its paid price is below + /// [`PRICE_FLOOR_CONTESTED_PERCENT`] of the UNCAPPED group median: the + /// question is what a peer whose own price sits above this node's may have + /// refused. The paid quote's price stands in for a third of the settled + /// amount: settlement is at least that, so an overpaid write can only be + /// judged more leniently, never less. Evaluated from the proof as it + /// stands, verified or not: this is what a fan-out uses, since the peers + /// it offers the proof to verify it themselves and may refuse it. + /// + /// False for a merkle proof, which the floor never refuses, for a proof + /// that does not parse (its peers refuse it on structure, as before), and + /// where no floor is wired at all (no cache or no routing view). + pub(crate) async fn price_contested(&self, routing: &XorName, proof: &[u8]) -> bool { + let Some(paid_price) = Self::single_node_paid_price(proof) else { + return false; + }; + let (reference, sample) = self.group_reference_price(routing).await; + Self::contested(paid_price, reference, sample) } - /// Test-only setter for the price-floor policy, so enforcement and - /// tolerance can be exercised without environment variables. + /// Whether this node admitted a payment for the chunk at `address` that + /// the price floor contested when it verified it. + /// + /// Unlike [`Self::price_contested`], this only ever reflects a proof whose + /// signature and settlement this node checked: a write admitted later from + /// the payment cache reads the decision taken then, so a proof attached to + /// a cache hit, which nobody checked, can neither set nor veto it. + pub(crate) fn verified_payment_contested(&self, address: &XorName) -> bool { + self.verified_price_contested.lock().contains(address) + } + + /// Test-only stand-in for a verified admission of the chunk at `address` + /// that the price floor contested, for tests that cannot settle payments + /// on chain. #[cfg(any(test, feature = "test-utils"))] - pub fn set_price_floor_for_tests(&self, config: PriceFloorConfig) { - *self.price_floor.write() = config; + pub fn note_contested_admission_for_tests(&self, address: &XorName) { + self.verified_price_contested.lock().put(*address, ()); + } + + /// The contest decision for a write paid at `paid_price`, from what the + /// floor could price it against. + fn contested( + paid_price: Amount, + reference: Option, + sample: GroupSample, + ) -> bool { + reference.map_or( + matches!( + sample.skip_reason, + Some(SkipReason::NoOwnCommitment | SkipReason::ThinSample) + ), + |reference| paid_price < Self::contested_price(reference.group_median), + ) + } + + /// The paid (median) quote's price in a single-node proof, or `None` for a + /// merkle proof or one that does not parse. + fn single_node_paid_price(proof: &[u8]) -> Option { + if !matches!(detect_proof_type(proof), Some(ProofType::SingleNode)) { + return None; + } + let parsed = deserialize_single_node_proof(proof).ok()?; + let candidates = Self::legacy_median_candidates(&parsed.proof_of_payment).ok()?; + candidates.first().map(|candidate| candidate.quote.price) + } + + /// The paid (per-quote) price below which a write is contested. + fn contested_price(group_median: Amount) -> Amount { + group_median.saturating_mul(Amount::from(PRICE_FLOOR_CONTESTED_PERCENT)) + / Amount::from(100u64) } /// Test-only setter for the ADR-0008 merkle parity boundary, so both the @@ -1475,9 +1465,9 @@ impl PaymentVerifier { Self::validate_quote_arithmetic(payment)?; let candidates = Self::legacy_median_candidates(payment)?; let mut failures = Vec::with_capacity(candidates.len()); - // The paid (median) price and settled on-chain amount of the winning - // candidate, kept for the receiver-side price-floor policy below. - let mut verified_paid_quote: Option<(Amount, Amount)> = None; + // The winning candidate's quote and settled on-chain amount, kept for + // the receiver-side price floor below. + let mut verified_paid_quote: Option<(&PaymentQuote, Amount)> = None; // ADR-0004 Amendment 2: remember WHICH candidate's on-chain settlement // verified — only that peer's commitment actually earned money, so only // its pin is nominated for a deterministic first audit below. The @@ -1486,11 +1476,10 @@ impl PaymentVerifier { let mut paid_peer: Option<[u8; 32]> = None; for candidate in candidates { - let paid_price = candidate.quote.price; let candidate_peer = *candidate.encoded_peer_id.as_bytes(); match self.verify_legacy_median_candidate(target, candidate).await { Ok(settled_amount) => { - verified_paid_quote = Some((paid_price, settled_amount)); + verified_paid_quote = Some((candidate.quote, settled_amount)); // First settlement-verified median candidate wins the paid // slot and the sole first-audit nomination. If a client // settled several TIED-median candidates, only this one is @@ -1504,7 +1493,7 @@ impl PaymentVerifier { } } - let Some((paid_price, settled_amount)) = verified_paid_quote else { + let Some((paid_quote, settled_amount)) = verified_paid_quote else { let xorname_hex = hex::encode(xorname); let details = if failures.is_empty() { "no median-priced candidates were available".to_string() @@ -1516,17 +1505,24 @@ impl PaymentVerifier { ))); }; - // Receiver-side price floor (single-node store admissions only). Runs - // AFTER the paid quote's signature and settlement verified above, so - // unauthenticated bundles can never poison floor telemetry. Shadow - // mode logs; enforcement rejects — an economic admission decision - // only, never trust/misbehaviour evidence. - // The floor compares against the median commitment price of the close - // group that is RESPONSIBLE for the data, so it takes the routing - // address. For a chunk that is the same value; for a pointer the paid - // content is its state, which names no close group at all. - self.enforce_price_floor(target.routing(), paid_price, settled_amount, context) + // Receiver-side price floor, in every context. Runs AFTER the paid + // quote's signature and settlement verified above, so unauthenticated + // bundles can never poison floor telemetry. A rejection is an economic + // admission decision only, never trust/misbehaviour evidence. + // The floor prices against the close group that is RESPONSIBLE for the + // data, so it reads the target's routing address. For a chunk that is + // the content address; for a pointer the paid content is its state, + // which names no close group at all. + let contested = self + .enforce_price_floor(target, paid_quote, settled_amount, context) .await?; + // Remembered before the payment cache learns the chunk, so any record + // of it from a cache hit reads this decision (see + // `verified_payment_contested`). Only ever set: a later payment does + // not unsay an earlier one. + if let (true, PaymentTarget::Chunk(address)) = (contested, target) { + self.verified_price_contested.lock().put(*address, ()); + } // ADR-0004 observe-only telemetry: log off-curve quotes only AFTER the // paid (median) quote's ML-DSA-65 signature has verified above, so @@ -1679,70 +1675,52 @@ impl PaymentVerifier { .collect() } - /// The close group's MEDIAN commitment-bound price for `xorname`, and the - /// number of peers that median was taken over. - /// - /// This is the price floor's reference, and choosing the median of the group - /// rather than this node's own price is the whole point of the policy. - /// - /// A client pays `3x` the MEDIAN of the quotes it collected. Pricing the - /// floor against a receiver's OWN price compares that median to a single - /// draw from a distribution that spans ~4x across the fleet, so the fullest - /// receivers reject honest payments while the emptiest admit deep - /// underpayment — measured on ant-prod-01, 0.377% of honest stores rejected - /// with the cheapest-of-K underpayment still admitted by 62.9% of receivers. - /// Both sides of that are the same defect: the reference is a sample, and - /// the payment is a median. Comparing median to median removes it. + /// The price floor's reference for `routing`: the close group's MEDIAN + /// commitment-bound price, capped at this node's own, plus the sample the + /// median was taken over. /// - /// The reference is built ONLY from state this node already holds: + /// **Why the median.** A client pays `3x` the MEDIAN of the quotes it + /// collected. Pricing the floor against a receiver's OWN price compares + /// that median to a single draw from a distribution that spans ~4x across + /// the fleet, so the fullest receivers reject honest payments while the + /// emptiest admit deep underpayment — measured on ant-prod-01, 0.377% of + /// honest stores rejected with the cheapest-of-K underpayment still + /// admitted by 62.9% of receivers. Comparing median to median removes that. /// - /// * the K peers closest to `xorname` in its own routing table, and - /// * those peers' own signed commitments from the gossip cache, TTL-gated - /// by [`GOSSIP_ANSWERABILITY_TTL`], plus this node's own live commitment. + /// **Why the cap.** The median is taken over gossiped commitments, and a + /// signature proves who said a number, never that the number is true. + /// Neighbours that OVERSTATE their counts push the median up and would + /// make this node refuse payments that are already settled and cannot be + /// refunded. Capping the reference at this node's own price bounds it by + /// the one number no neighbour controls: a payment of `3x tolerance x` + /// this node's own price always clears here, however many neighbours lie. + /// While the gossip is honest an above-median receiver prices against the + /// median exactly as before. The cost is detection: a receiver below the + /// median admits payments down to the tolerance of its own, lower, price. + /// Neighbours that UNDERSTATE still pull the median down, which costs a + /// missed underpayment, the accepted direction. ADR-0006 Amendment 2 + /// carries both bounds. /// - /// Nothing in the payment bundle feeds it. A client cannot pad, prune or - /// reorder quotes to move this number, which is what makes the floor - /// resistant to the quote-selection games it exists to stop. + /// The reference is built ONLY from state this node already holds: the K + /// peers closest to `routing` in its own routing table, those peers' signed + /// commitments from the gossip cache, TTL-gated by + /// [`GOSSIP_ANSWERABILITY_TTL`], and this node's own live commitment. + /// Nothing in the payment bundle feeds it, so a client cannot pad, prune or + /// reorder quotes to move it. Counts above `MAX_COMMITMENT_KEY_COUNT` are + /// dropped, since gossip ingest authenticates the sender without bounding + /// the count. /// - /// The remaining lever is gossiping a false commitment, and it cuts BOTH - /// ways, which is why the sample gate is set where it is: - /// - /// * **Understating** drags the median down and weakens the floor. The cost - /// is a missed underpayment. - /// * **Overstating** drags it up and makes this node reject settled, honest - /// payments. That is the dangerous direction, because the money is already - /// spent and cannot be refunded, so a griefer would be spending nothing to - /// destroy someone else's payment. - /// - /// The bound is NOT "a majority of the group", and it is worth stating - /// precisely because it is what an enforcement decision rests on. The floor - /// does not need the median to reach the liars' value — only to move past - /// the tolerance headroom. Each liar shifts the order statistic by one rank, - /// and on the measured fleet spread one rank is worth roughly 15% of price - /// while a 65% tolerance buys 1.54x. So **about a quarter of the sample** - /// suffices in either direction: measured on this crate's own fixture, 4 of - /// 15 overstating neighbours reject an honest payment sitting at the true - /// median, and 4 of 15 understating ones (placed above the median, where an - /// attacker would put them) admit a cheapest-of-K underpayment. - /// - /// Counts above `MAX_COMMITMENT_KEY_COUNT` are dropped outright, since - /// gossip ingest authenticates the sender without bounding the count. That - /// caps how far a single liar can reach but does not change the cardinality - /// bound above. A signature proves who said a number, never that the number - /// is true, and address grinding into a close group is a known adjacent - /// problem. This is why the policy ships shadow-only: the ~25% bound is a - /// reason not to enforce yet, not a property to rely on. - /// - /// Returns the reference price only when at least + /// Returns a reference only when at least /// [`PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS`] neighbour commitments are - /// fresh AND this node has its own live commitment; the `usize` is the - /// sample size either way, so the caller can log how close it came. A `None` - /// price means "do not evaluate". - /// - /// [`GOSSIP_ANSWERABILITY_TTL`]: crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL - async fn group_reference_price(&self, xorname: &XorName) -> (Option, GroupSample) { + /// fresh AND this node has its own live commitment; the sample comes back + /// either way, so the caller can log how close it came. `None` means "do + /// not evaluate". + async fn group_reference_price( + &self, + routing: &XorName, + ) -> (Option, GroupSample) { // This node's own live commitment. It is in the group whenever it is - // being asked to store the chunk, so it belongs in the sample. Bound to + // being asked to store the data, so it belongs in the sample. Bound to // a local first so the source lock is released before the awaits below. // // It is counted separately from the neighbour tally: the gate below asks @@ -1752,13 +1730,13 @@ impl PaymentVerifier { // Wiring faults are reported before state faults, so a startup-order bug // is never mistaken for "this node is mid-rotation" or for "the gate is // above what the network supplies". Those three have three different - // fixes and shadow mode exists to tell them apart. + // fixes and the telemetry exists to tell them apart. let Some(cache) = self.commitment_cache.read().as_ref().map(Arc::clone) else { return (None, GroupSample::none(SkipReason::NoCommitmentCache)); }; // Cache is checked first so a node without one does not pay for a // routing-table scan it will discard. - let closest = self.local_k_closest_peer_ids(xorname).await; + let closest = self.local_k_closest_peer_ids(routing).await; if closest.is_empty() { return (None, GroupSample::none(SkipReason::NoRoutingView)); } @@ -1772,8 +1750,11 @@ impl PaymentVerifier { let Some(own) = own_source.and_then(|source| source.current_binding_snapshot()) else { return (None, GroupSample::none(SkipReason::NoOwnCommitment)); }; - let now = std::time::Instant::now(); - let ttl = crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL; + // The routing scan includes this node, which the gossip cache never + // holds (ingest only accepts senders in the routing table, and the + // routing table never holds self), so its own commitment joins the + // sample exactly once, below. + let now = Instant::now(); let mut key_counts: Vec = { let guard = cache.read().await; closest @@ -1783,7 +1764,8 @@ impl PaymentVerifier { // Stale gossip is treated as unknown rather than as a low // price, so a peer going quiet can never drag the median // down, nor count toward the minimum sample. - if now.saturating_duration_since(record.received_at) >= ttl { + if now.saturating_duration_since(record.received_at) >= GOSSIP_ANSWERABILITY_TTL + { return None; } record @@ -1827,72 +1809,100 @@ impl PaymentVerifier { .get((sample.total.saturating_sub(1)) / 2) .copied() .unwrap_or_default(); - let reference = calculate_price(usize::try_from(median_key_count).unwrap_or_default()); + let group_median = calculate_price(usize::try_from(median_key_count).unwrap_or_default()); + // The price this node quotes right now, from the same commitment its + // `QuoteGenerator` binds quotes to. + let own_price = calculate_price(usize::try_from(own.key_count).unwrap_or_default()); + let reference = FloorReference { + price: group_median.min(own_price), + group_median, + own: own_price, + }; (Some(reference), sample) } - /// Receiver-side price floor for single-node store admissions. + /// Receiver-side price floor for single-node payments, in every context. /// /// Requires the settled on-chain amount to also clear - /// `PAID_QUOTE_PAYMENT_MULTIPLIER × tolerance% ×` the close group's median - /// commitment-bound price (see [`Self::group_reference_price`]). Together - /// with the per-quote check above, the effective rule is - /// `settled >= 3 × max(paid_price, tolerated_floor)`. + /// `PAID_QUOTE_PAYMENT_MULTIPLIER × tolerance% ×` the reference price (see + /// [`Self::group_reference_price`]). Together with the per-quote check + /// above, the effective rule is `settled >= 3 × max(paid_price, + /// tolerated_floor)`. + /// + /// It applies to direct PUTs, fresh replication and paid-list admission + /// alike. Any one of them left open would let a single accepting node + /// spread a below-floor write to its whole group: fresh replication copies + /// it directly, and a paid-list vote lets repair copy it later. /// - /// When the group reference cannot be built — too few fresh commitments, no - /// P2P handle, no gossip cache — the floor SKIPS the admission rather than - /// falling back to this node's own price. Falling back is what the previous - /// revision did, and it is precisely the comparison that rejected honest - /// stores on the fullest nodes. A skipped evaluation is logged with its - /// real `neighbours` / `group_sample` counts and a `skip_reason`, so shadow + /// When the reference cannot be built — too few fresh commitments, no own + /// commitment, no P2P handle, no gossip cache — the floor SKIPS the + /// admission rather than guessing. A skipped evaluation is logged with its + /// real `neighbours` / `group_sample` counts and a `skip_reason`, so the /// telemetry can separate an unwired node from one whose gate simply sits /// above what the network supplies. /// + /// Returns, for an admitted payment, whether it is `contested`: the + /// decision [`Self::price_contested`] makes from the same reference. A + /// node that stores a contested write does not judge its peers on it. The + /// line logs it too. + /// /// Always emits one telemetry line per evaluated admission (target - /// `ant_node::payment::price_floor`) so shadow mode measures the exact - /// rejections enforcement would cause. Rejects only when - /// [`PriceFloorConfig::enforce`] is set. + /// `ant_node::payment::price_floor`), and rejects every admission below + /// the floor. The line keeps a constant `enforce=true`, which separates + /// these nodes from releases that only logged (`enforce=false`) while the + /// fleet upgrades. async fn enforce_price_floor( &self, - xorname: &XorName, - paid_price: Amount, + target: &PaymentTarget, + paid_quote: &PaymentQuote, settled_amount: Amount, context: VerificationContext, - ) -> Result<()> { - if !context.is_store_admission() { - return Ok(()); - } - - let floor = *self.price_floor.read(); - let tolerance_percent = floor.tolerance_percent.min(100); - - let (reference, group_sample) = self.group_reference_price(xorname).await; - let Some(reference_price) = reference else { + ) -> Result { + let routing = target.routing(); + let paid_price = paid_quote.price; + // Only read by the telemetry, which compiles to nothing without the + // `logging` feature. + #[cfg_attr(not(feature = "logging"), allow(unused_variables))] + let target_kind = match target { + PaymentTarget::Chunk(_) => "chunk", + PaymentTarget::Pointer { .. } => "pointer", + }; + #[cfg_attr(not(feature = "logging"), allow(unused_variables))] + let paid_quote_age_secs = SystemTime::now() + .duration_since(paid_quote.timestamp) + .map_or(0, |age| age.as_secs()); + + let (reference, group_sample) = self.group_reference_price(routing).await; + let contested = Self::contested(paid_price, reference, group_sample); + let Some(reference) = reference else { // Not enough of the close group is known to price anything. Skip // rather than guess: see `group_reference_price`. The reason and - // both counts are logged because shadow telemetry has to separate a + // both counts are logged because the telemetry has to separate a // mid-rotation node from an unwired cache from a gate set above what // the network can supply — three causes with three different fixes. info!( target: "ant_node::payment::price_floor", "Price floor: context={context:?}, xorname={}, paid_price={paid_price}, \ settled={settled_amount}, neighbours={}, group_sample={}, reference_price=0, \ - tolerance_percent={tolerance_percent}, required=0, below_floor=false, \ - skipped=true, skip_reason={}, enforce={}", - hex::encode(xorname), + tolerance_percent={PRICE_FLOOR_TOLERANCE_PERCENT}, required=0, below_floor=false, \ + skipped=true, skip_reason={}, enforce=true, target={target_kind}, \ + group_median_price=0, own_price=0, paid_quote_age_secs={paid_quote_age_secs}, \ + contested={contested}", + hex::encode(routing), group_sample.neighbours, group_sample.total, group_sample.skip_reason_str(), - floor.enforce, ); - return Ok(()); + return Ok(contested); }; - // Amount is a U256: reference_price is bounded by the saturating pricing - // curve and the multipliers are <= 300, so this arithmetic cannot + // Amount is a U256: prices are bounded by the saturating pricing curve + // and every multiplier here is at most 100, so this arithmetic cannot // overflow; saturating keeps that a proof instead of an assumption. - let tolerated_price = - reference_price.saturating_mul(Amount::from(tolerance_percent)) / Amount::from(100u64); + let tolerated_price = reference + .price + .saturating_mul(Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT)) + / Amount::from(100u64); let required_amount = tolerated_price.saturating_mul(Amount::from(PAID_QUOTE_PAYMENT_MULTIPLIER)); let below_floor = settled_amount < required_amount; @@ -1901,25 +1911,29 @@ impl PaymentVerifier { target: "ant_node::payment::price_floor", "Price floor: context={context:?}, xorname={}, paid_price={paid_price}, \ settled={settled_amount}, neighbours={}, group_sample={}, \ - reference_price={reference_price}, tolerance_percent={tolerance_percent}, \ + reference_price={}, tolerance_percent={PRICE_FLOOR_TOLERANCE_PERCENT}, \ required={required_amount}, below_floor={below_floor}, skipped=false, \ - skip_reason=none, enforce={}", - hex::encode(xorname), + skip_reason=none, enforce=true, target={target_kind}, group_median_price={}, \ + own_price={}, paid_quote_age_secs={paid_quote_age_secs}, contested={contested}", + hex::encode(routing), group_sample.neighbours, group_sample.total, - floor.enforce, + reference.price, + reference.group_median, + reference.own, ); - if below_floor && floor.enforce { + if below_floor { return Err(Error::Payment(format!( "Settled payment {settled_amount} is below the close-group price floor \ - {required_amount} ({PAID_QUOTE_PAYMENT_MULTIPLIER}x {tolerance_percent}% of \ - the group median price {reference_price}, taken over {} peers)", - group_sample.total + {required_amount} ({PAID_QUOTE_PAYMENT_MULTIPLIER}x {PRICE_FLOOR_TOLERANCE_PERCENT}% of \ + the reference price {}: the lower of the group median {} over {} peers and \ + this node's own price {})", + reference.price, reference.group_median, group_sample.total, reference.own ))); } - Ok(()) + Ok(contested) } fn validate_paid_quote_content( @@ -3770,7 +3784,7 @@ mod tests { use saorsa_core::MlDsa65; use saorsa_pqc::pqc::types::MlDsaSecretKey; use saorsa_pqc::pqc::MlDsaOperations; - use std::time::SystemTime; + use std::time::{Duration, SystemTime}; /// Create a verifier for unit tests. EVM is always on, but tests can /// pre-populate the cache to bypass on-chain verification. @@ -3835,7 +3849,6 @@ mod tests { cache_capacity: 100, close_group_size: CLOSE_GROUP_SIZE, local_rewards_address: RewardsAddress::new([1u8; 20]), - price_floor: PriceFloorConfig::default(), }; PaymentVerifier::new(config) } @@ -4507,36 +4520,32 @@ mod tests { } } - /// Receiver commitment count for floor tests, chosen so the local price is - /// comfortably more than double the baseline (the 50% floor then rejects a - /// baseline-priced settlement). Asserted in each test so a pricing-curve - /// change fails loudly instead of silently weakening the tests. - const FLOOR_TEST_LOCAL_KEY_COUNT: u32 = 1_000_000; + /// Receiver commitment count for floor tests: the protocol maximum, far + /// above every group member, so the own-price cap never binds in the tests + /// that are about the group median. The cap has tests of its own, built on + /// [`floor_test_verifier_with_own_count`]. + const FLOOR_TEST_LOCAL_KEY_COUNT: u32 = MAX_COMMITMENT_KEY_COUNT; - fn floor_test_verifier(enforce: bool) -> PaymentVerifier { + fn floor_test_verifier() -> PaymentVerifier { + floor_test_verifier_with_own_count(FLOOR_TEST_LOCAL_KEY_COUNT) + } + + /// A verifier whose own live commitment holds `own_key_count` keys, at the + /// default tolerance. + fn floor_test_verifier_with_own_count(own_key_count: u32) -> PaymentVerifier { let verifier = create_test_verifier(); verifier.attach_local_commitment_source(Arc::new(FixedFloorSource { - key_count: FLOOR_TEST_LOCAL_KEY_COUNT, + key_count: own_key_count, })); - verifier.set_price_floor_for_tests(PriceFloorConfig { - enforce, - tolerance_percent: PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT, - }); - let local_price = price_at_records(FLOOR_TEST_LOCAL_KEY_COUNT as usize); - assert!( - local_price > price_at_records(0) * Amount::from(2u64), - "floor tests need a local price above twice baseline, got {local_price}" - ); verifier } - /// The settled amount the floor requires: 3x the tolerated fraction of the - /// close group's median commitment-bound price. - fn group_floor_required_amount() -> Amount { - let tolerated = price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize) - * Amount::from(PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT) - / Amount::from(100u64); - tolerated * Amount::from(PAID_QUOTE_PAYMENT_MULTIPLIER) + /// The tolerated fraction of the close group's median commitment-bound + /// price: a quote priced below this fails the floor. + fn tolerated_group_floor_price() -> Amount { + price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize) + * Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT) + / Amount::from(100u64) } /// One cheap (baseline-priced) quote paid `settled`: the cheapest-of-K @@ -4557,23 +4566,34 @@ mod tests { /// /// Exactly [`PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS`] entries, so these tests /// sit on the gate rather than comfortably past it: drop one and the floor - /// must skip. Spread widely enough that the cheapest sits below the tolerated - /// fraction of the median (asserted in the rejection test), and kept in the - /// low thousands because each entry builds a real signed merkle commitment. + /// must skip. Shaped like a real close group: members measured in August + /// spanned about 1.4x in price around a fleet median near 4,000 keys, and + /// this one spans 1.9x in keys (2.9x in price), so the cheapest member is + /// shown to clear the floor with room to spare. Kept in the low thousands because each + /// entry builds a real signed merkle commitment. const GROUP_TEST_KEY_COUNTS: [u32; 15] = [ - 500, 800, 1_100, 1_400, 1_700, 2_000, 2_300, 2_600, 2_900, 3_200, 3_500, 3_800, 4_100, - 4_400, 4_700, + 3_000, 3_200, 3_400, 3_600, 3_800, 4_000, 4_200, 4_400, 4_600, 4_800, 5_000, 5_200, 5_400, + 5_600, 5_800, ]; /// The LOWER median of [`GROUP_TEST_KEY_COUNTS`] plus this node's own /// [`FLOOR_TEST_LOCAL_KEY_COUNT`]. /// - /// Sixteen samples, so the lower median is the eighth smallest: `2_600`. The + /// Sixteen samples, so the lower median is the eighth smallest: `4_400`. The /// receiver's own count is an extreme outlier at the top and does not move /// it, which is exactly the robustness a median buys. An even sample is /// deliberate — it is the case where upper and lower median differ, and the /// lower one is what production safety depends on. - const GROUP_TEST_MEDIAN_KEY_COUNT: u32 = 2_600; + const GROUP_TEST_MEDIAN_KEY_COUNT: u32 = 4_400; + + /// A quoter holding under a quarter of the group median: the near-empty + /// node a modified client pays instead of the median. Its price sits well + /// below the floor, while the cheapest real group member sits well above. + const GROUP_TEST_NEAR_EMPTY_KEY_COUNT: usize = 1_000; + + /// A quote that clears the 30% floor but sits below the 0.6x contest + /// threshold: about half the group median price. + const GROUP_TEST_NEAR_FLOOR_KEY_COUNT: usize = 2_800; /// Build a gossip commitment cache holding one fresh commitment per entry of /// `key_counts`, attach it, and return the peer ids so the caller can put @@ -4688,10 +4708,10 @@ mod tests { /// does not move it. #[tokio::test] async fn group_reference_price_is_the_close_group_median() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xD0u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); verifier.set_paid_quote_k_closest_for_tests(neighbours); let (reference, sample) = verifier.group_reference_price(&xorname).await; @@ -4712,13 +4732,19 @@ mod tests { "a fully populated group must not report a skip reason" ); assert_eq!( - reference, + reference.group_median, price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize), - "the reference must be the group median price, not this node's own" + "the median must be taken over the whole group" ); - assert!( - reference < price_at_records(FLOOR_TEST_LOCAL_KEY_COUNT as usize), - "this node's own price is far above the median and must not be the reference" + assert_eq!( + reference.own, + price_at_records(FLOOR_TEST_LOCAL_KEY_COUNT as usize), + "this node's own price is the price of its own commitment" + ); + assert_eq!( + reference.price, reference.group_median, + "this node's own price is far above the median, so the cap does not bind \ + and the reference is the group median" ); } @@ -4731,10 +4757,10 @@ mod tests { /// of them on the fullest nodes. #[tokio::test] async fn price_floor_prices_against_the_group_median_not_the_receiver() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xD1u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let median_price = price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize); let own_price = price_at_records(FLOOR_TEST_LOCAL_KEY_COUNT as usize); @@ -4763,23 +4789,23 @@ mod tests { ); } - /// The attack the floor exists to stop: collect the neighbourhood's quotes, - /// settle only the cheapest one, submit it as a one-quote proof. + /// What the floor does NOT catch, stated as a test so nobody reads more + /// into it: settling the cheapest real member of the close group. Members + /// measured in August spanned about 1.4x in price, so the cheapest sits no + /// lower than about 0.7x the median, far above any tolerance that is safe + /// for honest traffic. #[tokio::test] - async fn price_floor_rejects_cheapest_of_group_payment() { - let verifier = floor_test_verifier(true); + async fn price_floor_admits_the_cheapest_member_of_a_realistic_group() { + let verifier = floor_test_verifier(); let xorname = [0xD2u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let cheapest = GROUP_TEST_KEY_COUNTS.first().copied().unwrap_or(0) as usize; let cheapest_price = price_at_records(cheapest); - let tolerated = price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize) - * Amount::from(PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT) - / Amount::from(100u64); assert!( - cheapest_price < tolerated, - "the cheapest group price must sit below the tolerated floor for this test to bite" + cheapest_price >= tolerated_group_floor_price(), + "the cheapest real member must clear the floor for this test to mean anything" ); let proof_bytes = group_floor_proof( @@ -4790,10 +4816,43 @@ mod tests { &neighbours, ); + let result = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await; + + assert_eq!( + result.expect("shopping inside the close group is not what the floor stops"), + PaymentStatus::PaymentVerified + ); + } + + /// The underpayment the floor exists to stop: collect the neighbourhood's + /// quotes, settle only a near-empty quoter, submit it as a one-quote proof. + #[tokio::test] + async fn price_floor_rejects_a_near_empty_quoter() { + let verifier = floor_test_verifier(); + let xorname = [0xD7u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + + let near_empty_price = price_at_records(GROUP_TEST_NEAR_EMPTY_KEY_COUNT); + assert!( + near_empty_price < tolerated_group_floor_price(), + "the near-empty quote must sit below the tolerated floor for this test to bite" + ); + + let proof_bytes = group_floor_proof( + &verifier, + xorname, + GROUP_TEST_NEAR_EMPTY_KEY_COUNT, + three_times(near_empty_price), + &neighbours, + ); + let err = verifier .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) .await - .expect_err("settling only the cheapest group quote must not clear the floor"); + .expect_err("settling a near-empty quoter must not clear the floor"); assert!( format!("{err}").contains("below the close-group price floor"), @@ -4801,20 +4860,359 @@ mod tests { ); } + /// The cap: a receiver below the group median prices against its own, + /// lower, price, so it never demands more than the tolerance of what it + /// would itself quote. The settlement here fails the median-only rule and + /// clears the capped one, so the cap is what decides it. + #[tokio::test] + async fn price_floor_caps_the_reference_at_this_nodes_own_price() { + let own_count = GROUP_TEST_KEY_COUNTS.first().copied().unwrap_or(0); + let verifier = floor_test_verifier_with_own_count(own_count); + let xorname = [0xD8u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + verifier.set_paid_quote_k_closest_for_tests(neighbours.clone()); + + let (reference, _) = verifier.group_reference_price(&xorname).await; + let reference = reference.expect("a fully populated group must produce a reference"); + let own_price = price_at_records(own_count as usize); + assert_eq!(reference.own, own_price); + assert!( + reference.group_median > own_price, + "this receiver must sit below its group's median for the cap to bind" + ); + assert_eq!( + reference.price, own_price, + "the reference must be capped at this node's own price" + ); + + // A quote priced between the tolerance of the capped reference and the + // tolerance of the median. + let quote_records = 1_200; + let quote_price = price_at_records(quote_records); + let tolerance = Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT); + assert!( + quote_price >= own_price * tolerance / Amount::from(100u64) + && quote_price < reference.group_median * tolerance / Amount::from(100u64), + "the quote must fail the median-only floor and clear the capped one" + ); + let proof_bytes = group_floor_proof( + &verifier, + xorname, + quote_records, + three_times(quote_price), + &neighbours, + ); + + let result = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await; + + assert_eq!( + result.expect("a receiver must not refuse what clears the tolerance of its own price"), + PaymentStatus::PaymentVerified + ); + } + + /// A write paid close to the floor is contested: it clears the floor here, + /// but not the contest threshold, so a peer with a higher view of the + /// group may have refused it. + #[tokio::test] + async fn a_write_near_the_floor_is_contested() { + let verifier = floor_test_verifier(); + let xorname = [0xD9u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + + let quote_price = price_at_records(GROUP_TEST_NEAR_FLOOR_KEY_COUNT); + let contest_price = PaymentVerifier::contested_price(price_at_records( + GROUP_TEST_MEDIAN_KEY_COUNT as usize, + )); + assert!( + quote_price >= tolerated_group_floor_price() && quote_price < contest_price, + "the quote must clear the floor but not the contest margin" + ); + let proof_bytes = group_floor_proof( + &verifier, + xorname, + GROUP_TEST_NEAR_FLOOR_KEY_COUNT, + three_times(quote_price), + &neighbours, + ); + + let result = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await; + + assert_eq!( + result.expect("a payment above the floor must be admitted"), + PaymentStatus::PaymentVerified + ); + assert!( + verifier.price_contested(&xorname, &proof_bytes).await, + "a write inside the contest margin must not be used to judge peers" + ); + assert!( + verifier.verified_payment_contested(&xorname), + "the verified admission must be remembered as contested" + ); + + // A later admission from the payment cache reads that decision, so a + // proof attached to it, here one the floor would not contest, can + // neither set nor veto it. + let median_proof = group_floor_proof( + &verifier, + xorname, + GROUP_TEST_MEDIAN_KEY_COUNT as usize, + three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)), + &neighbours, + ); + let cached = verifier + .verify_payment( + &xorname, + Some(&median_proof), + VerificationContext::PaidListAdmission, + ) + .await + .expect("a cached payment is admitted"); + assert_eq!(cached, PaymentStatus::CachedAsVerified); + assert!(verifier.verified_payment_contested(&xorname)); + } + + /// A write paid at the group median is not contested: this node keeps + /// judging its peers on it exactly as before. + #[tokio::test] + async fn a_write_at_the_group_median_is_not_contested() { + let verifier = floor_test_verifier(); + let xorname = [0xE3u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + + let proof_bytes = group_floor_proof( + &verifier, + xorname, + GROUP_TEST_MEDIAN_KEY_COUNT as usize, + three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)), + &neighbours, + ); + + let result = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await; + + assert_eq!( + result.expect("an honest median payment must clear the floor"), + PaymentStatus::PaymentVerified + ); + assert!( + !verifier.price_contested(&xorname, &proof_bytes).await, + "a payment at the group median must leave possession checks in place" + ); + assert!(!verifier.verified_payment_contested(&xorname)); + // A merkle or unparseable proof is never refused on price. + assert!(!verifier.price_contested(&xorname, &[0u8; 8]).await); + } + + /// The memory of contested admissions has its own capacity, so a payment + /// cache configured small cannot make it forget a decision before the key + /// is recorded. + #[test] + fn contested_admissions_outlive_a_small_payment_cache() { + let verifier = PaymentVerifier::new(PaymentVerifierConfig { + evm: EvmVerifierConfig::default(), + cache_capacity: 1, + close_group_size: CLOSE_GROUP_SIZE, + local_rewards_address: RewardsAddress::new([1u8; 20]), + }); + let keys = [[0xF1u8; 32], [0xF2u8; 32], [0xF3u8; 32]]; + for key in &keys { + verifier.note_contested_admission_for_tests(key); + } + assert!(keys + .iter() + .all(|key| verifier.verified_payment_contested(key))); + } + + /// A node with no floor wired (no gossip cache, or no routing view) refuses + /// nothing on price, so it judges its peers on every write as before. + #[tokio::test] + async fn a_node_with_no_floor_wired_contests_nothing() { + let xorname = [0xEDu8; 32]; + let (peer_id, quote) = make_signed_quote(xorname, price_at_records(0), 1); + let proof_bytes = serialize_proof(vec![(peer_id, quote)]); + + let verifier = create_test_verifier(); + assert!(!verifier.price_contested(&xorname, &proof_bytes).await); + + verifier.attach_commitment_cache(Arc::new(tokio::sync::RwLock::new(HashMap::new()))); + assert!(!verifier.price_contested(&xorname, &proof_bytes).await); + } + + /// The decision is taken from the proof against the current view when the + /// write is recorded (or, for a pointer, fanned out), so a write stored + /// after the group's prices rose is judged against the view peers will + /// price it against now. + #[tokio::test] + async fn the_contest_follows_the_current_view() { + let verifier = floor_test_verifier(); + let xorname = [0xE7u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + + let proof_bytes = group_floor_proof( + &verifier, + xorname, + GROUP_TEST_MEDIAN_KEY_COUNT as usize, + three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)), + &neighbours, + ); + assert!(!verifier.price_contested(&xorname, &proof_bytes).await); + + // Every neighbour's commitment grows to twice the old median, as after + // heavy churn: the same proof is now close to the floor. + forge_key_counts( + &verifier, + Some(&neighbours), + GROUP_TEST_MEDIAN_KEY_COUNT * 2, + ) + .await; + assert!( + verifier.price_contested(&xorname, &proof_bytes).await, + "a write that peers would now price near the floor must not be used to judge them" + ); + } + + /// The contest threshold is 60% of the UNCAPPED group median, to the atto: + /// a write paid at it is judged and one paid an atto below is not. Written + /// out rather than read from the constant, since moving it is an ADR + /// decision. + #[tokio::test] + async fn contest_threshold_is_sixty_percent_of_the_group_median() { + let verifier = floor_test_verifier(); + let xorname = [0xEBu8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + verifier.set_paid_quote_k_closest_for_tests(neighbours); + let threshold = price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize) + * Amount::from(60u64) + / Amount::from(100u64); + let proof_at = |price: Amount| { + let (peer_id, quote) = make_signed_quote(xorname, price, 1); + serialize_proof(vec![(peer_id, quote)]) + }; + + assert!( + !verifier + .price_contested(&xorname, &proof_at(threshold)) + .await + ); + assert!( + verifier + .price_contested(&xorname, &proof_at(threshold - Amount::from(1u64))) + .await + ); + } + + /// A receiver whose own price caps its floor still contests against the + /// group median: a peer whose own price is higher may refuse what this + /// node's lower floor admitted. + #[tokio::test] + async fn contest_reads_the_uncapped_median_when_the_cap_binds() { + let verifier = floor_test_verifier_with_own_count(3_000); + let xorname = [0xECu8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + verifier.set_paid_quote_k_closest_for_tests(neighbours); + let (reference, _) = verifier.group_reference_price(&xorname).await; + let reference = reference.expect("a fully populated group must produce a reference"); + assert!( + reference.price < reference.group_median, + "the cap must bind for this test to mean anything" + ); + + // Above the contest threshold of the capped reference, below that of + // the median. + let price = (PaymentVerifier::contested_price(reference.price) + + PaymentVerifier::contested_price(reference.group_median)) + / Amount::from(2u64); + let (peer_id, quote) = make_signed_quote(xorname, price, 1); + let proof_bytes = serialize_proof(vec![(peer_id, quote)]); + + assert!( + verifier.price_contested(&xorname, &proof_bytes).await, + "the contest must be judged against the median, not the capped reference" + ); + } + + /// A pointer payment is floored against the close group of the pointer's + /// address, and its fan-out is contested the same way as a chunk's. + #[tokio::test] + async fn price_floor_floors_and_contests_a_pointer_state() { + let verifier = floor_test_verifier(); + let address = [0xE4u8; 32]; + let state_id = [0xE5u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + + let quote_price = price_at_records(GROUP_TEST_NEAR_FLOOR_KEY_COUNT); + let proof_bytes = group_floor_proof( + &verifier, + state_id, + GROUP_TEST_NEAR_FLOOR_KEY_COUNT, + three_times(quote_price), + &neighbours, + ); + + verifier + .verify_pointer_payment_in( + &address, + &state_id, + &proof_bytes, + VerificationContext::ClientPut, + ) + .await + .expect("a pointer payment above the floor must be admitted"); + + assert!( + verifier.price_contested(&address, &proof_bytes).await, + "a pointer write inside the contest margin must not be used to judge peers" + ); + + // And a near-empty quoter is refused for a pointer exactly as for a chunk. + let cheap_state = [0xE6u8; 32]; + let cheap_proof = group_floor_proof( + &verifier, + cheap_state, + GROUP_TEST_NEAR_EMPTY_KEY_COUNT, + three_times(price_at_records(GROUP_TEST_NEAR_EMPTY_KEY_COUNT)), + &neighbours, + ); + let err = verifier + .verify_pointer_payment_in( + &address, + &cheap_state, + &cheap_proof, + VerificationContext::ClientPut, + ) + .await + .expect_err("a near-empty quoter must not pay for a pointer state either"); + assert!(format!("{err}").contains("below the close-group price floor")); + } + /// The anti-manipulation property: the reference price is built from this /// node's own routing view and the neighbours' signed gossip, never from the /// bundle. Padding the bundle with cheap quotes (the median-padding shape) /// must not move the floor. #[tokio::test] async fn price_floor_reference_ignores_client_supplied_quotes() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xD3u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); - // A bundle stuffed with baseline-priced quotes: the client's own median - // is now the floor price, which is exactly what it would want. - let cheapest = GROUP_TEST_KEY_COUNTS.first().copied().unwrap_or(0) as usize; + // A bundle stuffed with near-empty quotes: the client's own median is + // now the cheap price, which is exactly what it would want. + let cheapest = GROUP_TEST_NEAR_EMPTY_KEY_COUNT; let mut peer_quotes = Vec::new(); let mut k_closest: Vec<[u8; 32]> = neighbours.clone(); for seed in 1..=CLOSE_GROUP_SIZE { @@ -4852,7 +5250,7 @@ mod tests { /// counts (the obvious choice) would pass either way and prove nothing. #[tokio::test] async fn price_floor_skips_when_group_view_is_too_thin() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xD4u8; 32]; let thin: Vec = GROUP_TEST_KEY_COUNTS .iter() @@ -4861,7 +5259,7 @@ mod tests { .take(PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS - 1) .collect(); assert_eq!(thin.len(), PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS - 1); - let neighbours = attach_group_commitments(&verifier, &thin, std::time::Duration::ZERO); + let neighbours = attach_group_commitments(&verifier, &thin, Duration::ZERO); // Deep underpayment that this sample, if it were used, would reject. let settled = three_times(price_at_records(0)); @@ -4869,7 +5267,7 @@ mod tests { assert!( settled < price_at_records(would_be_median as usize) - * Amount::from(PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT) + * Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT) / Amount::from(100u64) * Amount::from(PAID_QUOTE_PAYMENT_MULTIPLIER), "the thin sample must price ABOVE the settlement, so deleting the gate fails this test" @@ -4884,13 +5282,21 @@ mod tests { result.expect("a thin group view must skip the floor, not reject"), PaymentStatus::PaymentVerified ); + assert!( + verifier.price_contested(&xorname, &proof_bytes).await, + "a write this node cannot price must not be used to judge its peers" + ); + assert!( + verifier.verified_payment_contested(&xorname), + "a write admitted unpriced is remembered as contested" + ); } /// Exactly ON the gate must evaluate, so the boundary is pinned from both /// sides alongside the one-short test above. #[tokio::test] async fn price_floor_evaluates_at_exactly_the_minimum_sample() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xDAu8; 32]; assert_eq!( GROUP_TEST_KEY_COUNTS.len(), @@ -4898,7 +5304,7 @@ mod tests { "the fixture must sit exactly on the gate for this boundary test" ); let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let proof_bytes = group_floor_proof( &verifier, @@ -4915,7 +5321,8 @@ mod tests { assert!(format!("{err}").contains("below the close-group price floor")); } - /// A receiver with no live commitment of its own must skip. + /// A receiver with no live commitment of its own must skip, and leave its + /// peers unjudged on the write. /// /// Such a node prices its OWN quotes at baseline, so holding an incoming /// payment to its neighbours' higher median would reject settlements it @@ -4924,13 +5331,9 @@ mod tests { #[tokio::test] async fn price_floor_skips_when_this_node_has_no_commitment() { let verifier = create_test_verifier(); - verifier.set_price_floor_for_tests(PriceFloorConfig { - enforce: true, - tolerance_percent: PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT, - }); let xorname = [0xDBu8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let proof_bytes = group_floor_proof( &verifier, @@ -4948,6 +5351,9 @@ mod tests { result.expect("a receiver with no commitment of its own must skip the floor"), PaymentStatus::PaymentVerified ); + // It cannot price the write, so it cannot tell whether its peers + // refused it either, and must not judge them on it. + assert!(verifier.price_contested(&xorname, &proof_bytes).await); } /// A neighbour gossiping a count no honest node could hold must be dropped, @@ -4959,18 +5365,19 @@ mod tests { /// floor becomes a way to reject other people's settled payments. #[tokio::test] async fn price_floor_ignores_commitments_above_the_protocol_cap() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xDCu8; 32]; // The honest builder refuses to construct an over-cap commitment, so // forge one the way a modified node would: build a valid commitment and // overwrite the signed count. Dropping it leaves the sample one short of // the gate, so the floor skips — that cardinality change is what this // test detects if the filter is removed. One over-cap value alone would - // NOT move a 16-sample median; inflating the reference that way needs - // about a quarter of the sample, which is the cardinality bound on - // `group_reference_price` rather than something this filter fixes. + // NOT move a 16-sample median; inflating the reference that way needs a + // majority of the sample at the default tolerance, which is the + // cardinality bound on `group_reference_price` rather than something + // this filter fixes. let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let over_cap = crate::replication::commitment::MAX_COMMITMENT_KEY_COUNT + 1; { let cache = verifier @@ -5012,22 +5419,29 @@ mod tests { ); } - /// Pins the other KNOWN LIMIT: four of fifteen OVERSTATING neighbours push - /// the reference up far enough to reject an honest payment sitting at the - /// true group median. + /// Pins the KNOWN LIMIT for a receiver whose own price sits far above its + /// group, where the cap does not bind: overstating neighbours that make up + /// a MAJORITY of the sample reject an honest payment at the true median. /// /// This is the griefing direction, and the more serious of the two: the /// attacker spends nothing and the victim's money is already settled and - /// unrefundable. Like its understating sibling this test asserts a weakness - /// deliberately. If a future change improves the bound it will fail and must - /// be re-derived rather than deleted. + /// unrefundable. On this realistic fixture 65% falls to seven liars. At 30% + /// the headroom covers every honest member of the group, so liars lift the + /// median only as far as the dearest honest member until they outnumber + /// the rest: seven still clear (pinned by + /// `group_median_resists_seven_overstated_commitments`), eight take the + /// median. A receiver at or below the payment's price is immune at any + /// count (pinned by + /// `capped_reference_cannot_be_griefed_by_any_number_of_overstated_commitments`). + /// This test asserts the weakness deliberately. If a future change improves + /// the bound it will fail and must be re-derived rather than deleted. #[tokio::test] - async fn group_median_is_griefed_by_four_overstated_commitments() { - let verifier = floor_test_verifier(true); + async fn group_median_alone_is_griefed_by_a_majority_of_overstated_commitments() { + let verifier = floor_test_verifier(); let xorname = [0xE0u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); - forge_key_counts(&verifier, neighbours.get(..4), MAX_COMMITMENT_KEY_COUNT).await; + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + forge_key_counts(&verifier, neighbours.get(..8), MAX_COMMITMENT_KEY_COUNT).await; // A completely honest payment, priced at the TRUE group median. let honest = three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)); @@ -5043,34 +5457,38 @@ mod tests { .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) .await .expect_err( - "KNOWN LIMIT: four overstated neighbours reject an honest at-median payment. \ - If this now succeeds, the bound improved — re-derive it and update the docs.", + "KNOWN LIMIT: a majority of overstated neighbours rejects an honest at-median \ + payment at a receiver far above its group. If this now succeeds, the bound \ + improved — re-derive it and update the docs.", ); assert!(format!("{err}").contains("below the close-group price floor")); } - /// Pins the KNOWN LIMIT, so it is a fact in the test suite rather than a - /// claim in a comment: four of fifteen understating neighbours DISARM the - /// floor, admitting the cheapest-of-K underpayment it exists to reject. + /// The griefing direction below a majority: seven of fifteen neighbours + /// gossiping the protocol maximum must not make even a receiver far above + /// its group refuse an honest, already settled payment. /// - /// This test asserts a weakness on purpose. It is the concrete reason - /// enforcement stays off, and if a future change improves the bound this - /// test will fail and must be re-derived rather than quietly deleted. + /// They take the seven LOWEST slots, which is where a liar moves the + /// median most, and lift the lower median from `4_400` to `5_800`, the + /// dearest honest member. The honest payment still clears 30% of that by + /// 2.07x (1.55x at 40%). Forged by overwriting the signed count rather than + /// building real 1,000,000-leaf commitments, which cost about a second and + /// 200 MB each. #[tokio::test] - async fn group_median_is_disarmed_by_four_understated_commitments() { - let verifier = floor_test_verifier(true); - let xorname = [0xDFu8; 32]; - let mut counts: Vec = GROUP_TEST_KEY_COUNTS.to_vec(); - for slot in counts.iter_mut().rev().take(4) { - *slot = 1; - } - let neighbours = attach_group_commitments(&verifier, &counts, std::time::Duration::ZERO); + async fn group_median_resists_seven_overstated_commitments() { + let verifier = floor_test_verifier(); + let xorname = [0xDDu8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + forge_key_counts(&verifier, neighbours.get(..7), MAX_COMMITMENT_KEY_COUNT).await; + // An honest payment at the TRUE group median must still be admitted. + let honest = three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)); let proof_bytes = group_floor_proof( &verifier, xorname, - 0, - three_times(price_at_records(0)), + GROUP_TEST_MEDIAN_KEY_COUNT as usize, + honest, &neighbours, ); @@ -5079,49 +5497,40 @@ mod tests { .await; assert_eq!( - result.expect( - "KNOWN LIMIT: four understated neighbours disarm the floor. If this now \ - rejects, the bound improved — re-derive it and update the docs." - ), + result.expect("a minority of overstated commitments must not reject honest payments"), PaymentStatus::PaymentVerified ); } - /// The griefing direction: a small minority gossiping OVERSTATED - /// commitments must not push the reference up far enough to make this node - /// reject an honest, already settled payment. - /// - /// This is the dangerous direction of a false commitment. Understating only - /// costs a missed underpayment; overstating destroys someone else's money, - /// and the attacker spends nothing to do it. + /// The cap's guarantee: no number of overstating neighbours can make a + /// receiver refuse a payment of `3x tolerance x` its own price. /// - /// The bound is real but partial, and worth stating precisely rather than - /// claiming immunity. Each liar shifts the order statistic up by one rank, - /// so what protects the payment is the tolerance headroom, not the median - /// itself. In this fixture three liars move the reference `2_600` -> - /// `3_500` and an at-median payment still clears, by only 1.9%; four move it - /// to `3_800` - /// and the same payment is REJECTED (pinned by - /// `group_median_is_griefed_by_four_overstated_commitments`). The griefing - /// bound is therefore about a quarter of the sample, the same as the - /// disarming one — NOT "roughly half", which was the incorrect claim this - /// policy originally rested on. - /// - /// Estimator coverage: this case fails under a 25th percentile, and its - /// disarming sibling fails under a mean and under the upper median, so the - /// pair pins the estimator jointly. Neither does so alone. + /// Every neighbour here claims the protocol maximum, so the group median is + /// the liars' value. A receiver that itself holds the group's true median + /// still admits the honest at-median payment, because its reference is its + /// own price. #[tokio::test] - async fn group_median_resists_three_overstated_commitments() { - let verifier = floor_test_verifier(true); - let xorname = [0xDDu8; 32]; - // Three of fifteen neighbours claim the protocol maximum. Forged by - // overwriting the signed count rather than building real 1,000,000-leaf - // commitments, which cost about a second and 200 MB each. + async fn capped_reference_cannot_be_griefed_by_any_number_of_overstated_commitments() { + let verifier = floor_test_verifier_with_own_count(GROUP_TEST_MEDIAN_KEY_COUNT); + let xorname = [0xE1u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); - forge_key_counts(&verifier, neighbours.get(..3), MAX_COMMITMENT_KEY_COUNT).await; + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + forge_key_counts(&verifier, Some(&neighbours), MAX_COMMITMENT_KEY_COUNT).await; + + verifier.set_paid_quote_k_closest_for_tests(neighbours.clone()); + let (reference, _) = verifier.group_reference_price(&xorname).await; + let reference = reference.expect("a fully populated group must produce a reference"); + assert_eq!( + reference.group_median, + price_at_records(MAX_COMMITMENT_KEY_COUNT as usize), + "every neighbour lied, so the median is the liars' value" + ); + assert_eq!( + reference.price, + price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize), + "the reference must be capped at this node's own price" + ); - // An honest payment at the TRUE group median must still be admitted. let honest = three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)); let proof_bytes = group_floor_proof( &verifier, @@ -5136,7 +5545,85 @@ mod tests { .await; assert_eq!( - result.expect("a minority of overstated commitments must not reject honest payments"), + result.expect("lying neighbours must not lift a receiver above its own price"), + PaymentStatus::PaymentVerified + ); + } + + /// Understated commitments pull the reference DOWN and weaken the floor. + /// That costs a missed underpayment rather than anyone's money. + /// + /// Six of fifteen neighbours claiming almost nothing leave the floor + /// standing against a baseline quote. The placement matters and is easy to + /// get wrong: understating a peer that was already below the median + /// changes nothing, so a test that rewrites the CHEAPEST entries passes + /// while testing nothing. These rewrite the DEAREST ones, which is where an + /// attacker would sit. `1`, not `0`: a commitment over an empty key set + /// cannot be built at all, so one key is the least a peer can commit to. + #[tokio::test] + async fn group_median_resists_six_understated_commitments() { + let verifier = floor_test_verifier(); + let xorname = [0xD6u8; 32]; + let mut counts: Vec = GROUP_TEST_KEY_COUNTS.to_vec(); + for slot in counts.iter_mut().rev().take(6) { + *slot = 1; + } + let neighbours = attach_group_commitments(&verifier, &counts, Duration::ZERO); + + let proof_bytes = group_floor_proof( + &verifier, + xorname, + 0, + three_times(price_at_records(0)), + &neighbours, + ); + + let err = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await + .expect_err("a minority of understated commitments must not disarm the floor"); + + assert!( + format!("{err}").contains("below the close-group price floor"), + "Error should name the group price floor: {err}" + ); + } + + /// Pins the KNOWN LIMIT, so it is a fact in the test suite rather than a + /// claim in a comment: seven understating neighbours of fifteen DISARM the + /// floor, admitting the baseline payment it exists to reject. The lower + /// median then falls to the cheapest honest member (3,000 keys), and 30% of + /// that price is below a baseline quote. At 40% it took eight. + /// + /// This test asserts a weakness on purpose. If a future change improves the + /// bound it will fail and must be re-derived rather than quietly deleted. + #[tokio::test] + async fn group_median_is_disarmed_by_seven_understated_commitments() { + let verifier = floor_test_verifier(); + let xorname = [0xDFu8; 32]; + let mut counts: Vec = GROUP_TEST_KEY_COUNTS.to_vec(); + for slot in counts.iter_mut().rev().take(7) { + *slot = 1; + } + let neighbours = attach_group_commitments(&verifier, &counts, Duration::ZERO); + + let proof_bytes = group_floor_proof( + &verifier, + xorname, + 0, + three_times(price_at_records(0)), + &neighbours, + ); + + let result = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await; + + assert_eq!( + result.expect( + "KNOWN LIMIT: seven understated neighbours of fifteen disarm the floor. If \ + this now rejects, the bound improved: re-derive it and update the docs." + ), PaymentStatus::PaymentVerified ); } @@ -5149,7 +5636,7 @@ mod tests { /// floor skips instead of pricing against this node's own (huge) count. #[tokio::test] async fn price_floor_ignores_stale_group_commitments() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xD5u8; 32]; let stale_age = crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL + std::time::Duration::from_mins(1); @@ -5190,14 +5677,13 @@ mod tests { async fn price_floor_excludes_only_the_stale_entries_from_a_mixed_cache() { use crate::replication::commitment_state::PeerCommitmentRecord; - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xDEu8; 32]; let stale_age = crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL + std::time::Duration::from_mins(1); // Fresh: the full fixture, exactly on the gate. - let fresh = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + let fresh = attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); // Stale: four cheap counts, keeping the total within the 19 neighbours a // real routing view can supply. @@ -5236,83 +5722,19 @@ mod tests { "stale entries must not count toward the gate" ); assert_eq!( - reference, + reference.map(|reference| reference.group_median), Some(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)), "the reference must be the fresh-only median; counting the stale cheap \ entries would drag it below" ); } - /// Understated commitments pull the reference DOWN and weaken the floor. - /// - /// Three adversarially-placed liars leave it standing; four disarm it. The - /// placement matters and is easy to get wrong: understating a peer that was - /// already below the median changes nothing, so a test that rewrites the - /// CHEAPEST entries passes while testing nothing. These rewrite the DEAREST - /// ones, which is where an attacker would sit, and the count is set to the - /// largest that still holds rather than to an aspirational majority. - #[tokio::test] - async fn group_median_resists_three_understated_commitments() { - let verifier = floor_test_verifier(true); - let xorname = [0xD6u8; 32]; - // Three of fifteen neighbours claim to store almost nothing. `1`, not - // `0`: a commitment over an empty key set cannot be built at all, so one - // key is the least a peer can honestly (or dishonestly) commit to. - let mut counts: Vec = GROUP_TEST_KEY_COUNTS.to_vec(); - for slot in counts.iter_mut().rev().take(3) { - *slot = 1; - } - let neighbours = attach_group_commitments(&verifier, &counts, std::time::Duration::ZERO); - - let cheapest_price = price_at_records(0); - let proof_bytes = group_floor_proof( - &verifier, - xorname, - 0, - three_times(cheapest_price), - &neighbours, - ); - - let err = verifier - .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) - .await - .expect_err("a minority of understated commitments must not disarm the floor"); - - assert!( - format!("{err}").contains("below the close-group price floor"), - "Error should name the group price floor: {err}" - ); - } - - #[tokio::test] - async fn test_price_floor_shadow_mode_accepts_below_floor_settlement() { - let verifier = floor_test_verifier(false); - let xorname = [0xC1u8; 32]; - let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); - let settled = three_times(price_at_records(0)); - assert!( - settled < group_floor_required_amount(), - "shadow mode is only meaningful on a settlement that WOULD be rejected" - ); - let proof_bytes = group_floor_proof(&verifier, xorname, 0, settled, &neighbours); - - let result = verifier - .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) - .await; - - assert_eq!( - result.expect("shadow mode must never reject on the floor"), - PaymentStatus::PaymentVerified - ); - } - #[tokio::test] async fn test_price_floor_enforced_rejects_cheapest_of_k_settlement() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xC2u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let settled = three_times(price_at_records(0)); let proof_bytes = group_floor_proof(&verifier, xorname, 0, settled, &neighbours); @@ -5327,30 +5749,44 @@ mod tests { ); } + /// An honest client may OVERPAY a cheap quote to clear stricter receivers: + /// the floor compares the settled amount, not the quote price, and + /// exactly-at-floor must pass. The floor is 30% of the reference (ADR-0006 + /// Amendment 2), written out here rather than read from the constant, so + /// moving the tolerance either way, an ADR decision, fails here first. #[tokio::test] async fn test_price_floor_enforced_accepts_exactly_at_floor_overpayment() { - // An honest client may OVERPAY a cheap quote to clear stricter - // receivers: the floor compares the settled amount, not the quote - // price, and exactly-at-floor must pass. - let verifier = floor_test_verifier(true); - let xorname = [0xC3u8; 32]; + assert_eq!(PRICE_FLOOR_TOLERANCE_PERCENT, 30); + let verifier = floor_test_verifier(); let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); - let proof_bytes = group_floor_proof( - &verifier, - xorname, - 0, - group_floor_required_amount(), - &neighbours, - ); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + let median_price = price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize); + let at_thirty_percent = + three_times(median_price * Amount::from(30u64) / Amount::from(100u64)); - let result = verifier + let xorname = [0xC3u8; 32]; + let proof_bytes = group_floor_proof(&verifier, xorname, 0, at_thirty_percent, &neighbours); + let status = verifier .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) - .await; + .await + .expect("exactly 3x 30% of the group median must clear the floor"); + assert_eq!(status, PaymentStatus::PaymentVerified); - assert_eq!( - result.expect("an exactly-at-floor settlement must pass"), - PaymentStatus::PaymentVerified + let xorname = [0xCAu8; 32]; + let one_short = at_thirty_percent - Amount::from(1u64); + let proof_bytes = group_floor_proof(&verifier, xorname, 0, one_short, &neighbours); + let err = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await + .expect_err("one atto under 3x 30% of the group median must not clear the floor"); + assert!( + err.to_string() + .contains("below the close-group price floor"), + "{err}" + ); + assert!( + !verifier.verified_payment_contested(&xorname), + "a refused payment is never remembered as an admitted contested one" ); } @@ -5359,10 +5795,10 @@ mod tests { // Fresh replication is the same fresh economic event as a direct PUT; // exempting it would let one cheap accepting node fan the proof out // around every other storer's floor. - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xC4u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let settled = three_times(price_at_records(0)); let proof_bytes = group_floor_proof(&verifier, xorname, 0, settled, &neighbours); @@ -5381,26 +5817,31 @@ mod tests { ); } + /// Paid-list admission is floored like a store. A `PaidNotify` only ever + /// announces a write that was just paid for, and a paid-list vote is what + /// lets repair copy the data later, so exempting it would let one + /// accepting node replicate a below-floor write to its whole group. #[tokio::test] - async fn test_price_floor_never_reprices_paid_list_admission() { - let verifier = floor_test_verifier(true); + async fn test_price_floor_enforced_applies_to_paid_list_admission() { + let verifier = floor_test_verifier(); let xorname = [0xC5u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let settled = three_times(price_at_records(0)); let proof_bytes = group_floor_proof(&verifier, xorname, 0, settled, &neighbours); - let result = verifier + let err = verifier .verify_payment( &xorname, Some(&proof_bytes), VerificationContext::PaidListAdmission, ) - .await; + .await + .expect_err("a below-floor proof must not earn a paid-list vote"); - assert_eq!( - result.expect("paid-list admission reprices no fresh economic decision"), - PaymentStatus::PaymentVerified + assert!( + format!("{err}").contains("below the close-group price floor"), + "Error should name the price floor: {err}" ); } @@ -5414,13 +5855,13 @@ mod tests { /// nothing. #[tokio::test] async fn test_price_floor_without_any_group_view_skips() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xC6u8; 32]; let settled = three_times(price_at_records(0)); assert!( settled < price_at_records(FLOOR_TEST_LOCAL_KEY_COUNT as usize) - * Amount::from(PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT) + * Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT) / Amount::from(100u64) * Amount::from(PAID_QUOTE_PAYMENT_MULTIPLIER), "the old own-price rule would have rejected this settlement, so the skip is load-bearing" @@ -5435,42 +5876,9 @@ mod tests { result.expect("no group view must skip the floor, never reject"), PaymentStatus::PaymentVerified ); - } - - /// `from_env` must fail CLOSED: a present-but-invalid tolerance can never - /// leave enforcement on with a tolerance the operator did not specify. - /// (This test owns the `PRICE_FLOOR_*` env vars — no other test reads them.) - #[test] - fn price_floor_from_env_fails_closed_on_invalid_tolerance() { - // Defaults with nothing set. - std::env::remove_var(PRICE_FLOOR_ENFORCE_ENV); - std::env::remove_var(PRICE_FLOOR_TOLERANCE_ENV); - let cfg = PriceFloorConfig::from_env(); - assert!(!cfg.enforce); - assert_eq!(cfg.tolerance_percent, PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT); - - // Enforce on, valid tolerance: honoured. - std::env::set_var(PRICE_FLOOR_ENFORCE_ENV, "1"); - std::env::set_var(PRICE_FLOOR_TOLERANCE_ENV, "80"); - let cfg = PriceFloorConfig::from_env(); - assert!(cfg.enforce); - assert_eq!(cfg.tolerance_percent, 80); - - // Enforce on, out-of-range tolerance: enforcement DISABLED (fail closed). - std::env::set_var(PRICE_FLOOR_TOLERANCE_ENV, "150"); - let cfg = PriceFloorConfig::from_env(); - assert!( - !cfg.enforce, - "an out-of-range tolerance must not silently enforce a default" - ); - - // Enforce on, unparseable tolerance: also disabled. - std::env::set_var(PRICE_FLOOR_TOLERANCE_ENV, "loose"); - let cfg = PriceFloorConfig::from_env(); - assert!(!cfg.enforce); - - std::env::remove_var(PRICE_FLOOR_ENFORCE_ENV); - std::env::remove_var(PRICE_FLOOR_TOLERANCE_ENV); + // No cache means no floor is wired here at all, and so nothing to + // contest: possession checks stay as they were. + assert!(!verifier.price_contested(&xorname, &proof_bytes).await); } #[tokio::test] @@ -6988,7 +7396,6 @@ mod tests { cache_capacity: 100, close_group_size: CLOSE_GROUP_SIZE, local_rewards_address: RewardsAddress::new([1u8; 20]), - price_floor: PriceFloorConfig::default(), }; let verifier = PaymentVerifier::new(config); diff --git a/src/pointer/service.rs b/src/pointer/service.rs index 7d4c866f..8af79e91 100644 --- a/src/pointer/service.rs +++ b/src/pointer/service.rs @@ -561,7 +561,7 @@ fn cross_kind_refusal(address: XorName, chunk_present: Result) -> Option

, paid_list: &PaidList) -> Vec { + sampled + .into_iter() + .filter(|key| !paid_list.counts_as_price_contested(key)) + .collect() +} + // --------------------------------------------------------------------------- // Main audit tick // --------------------------------------------------------------------------- @@ -107,7 +120,7 @@ pub(crate) fn responsible_audit_response_timeout( /// /// This is the production path used by the replication engine. Direct /// callers that have not adopted repair proofs remain conservative and do not -/// audit peers for unproven keys. +/// audit peers for unproven keys, nor for price-contested ones (ADR-0006). #[allow( clippy::implicit_hasher, clippy::too_many_arguments, @@ -116,6 +129,7 @@ pub(crate) fn responsible_audit_response_timeout( pub async fn audit_tick_with_repair_proofs( p2p_node: &Arc, storage: &Arc, + paid_list: &PaidList, config: &ReplicationConfig, sync_history: &HashMap, repair_proofs: &Arc>, @@ -172,6 +186,7 @@ pub async fn audit_tick_with_repair_proofs( .copied() .collect() }; + let sampled_keys = keys_open_to_audit(sampled_keys, paid_list); // Step 4: Filter to keys where the chosen peer is in the close group and // this node has proof that it already sent the peer a repair hint for the @@ -901,6 +916,44 @@ mod tests { /// incoming audit limit (`2 * sqrt(N)`) never rejects small test challenges. const TEST_STORED_CHUNKS: usize = 1_000_000; + /// ADR-0006: the responsible audit never samples a key marked + /// price-contested, and samples every other key as before. + #[tokio::test] + async fn price_contested_keys_are_not_audited() { + let dir = TempDir::new().expect("temp dir"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let plain: XorName = [0x71; 32]; + let contested: XorName = [0x72; 32]; + let unpaid: XorName = [0x73; 32]; + paid_list.insert(&plain).await.expect("insert"); + paid_list + .insert_price_contested(&contested) + .await + .expect("mark"); + + assert_eq!( + keys_open_to_audit(vec![plain, contested, unpaid], &paid_list), + vec![plain, unpaid] + ); + } + + /// ADR-0006: while the paid list cannot be read, the responsible audit + /// samples no key, since any of them may be marked. + #[tokio::test] + async fn an_unreadable_mark_keeps_a_key_out_of_the_audit() { + let dir = TempDir::new().expect("temp dir"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let plain: XorName = [0x74; 32]; + let unpaid: XorName = [0x75; 32]; + paid_list.insert(&plain).await.expect("insert"); + + let _reader = paid_list.hold_reader_for_tests().expect("hold reader"); + assert_eq!( + keys_open_to_audit(vec![plain, unpaid], &paid_list), + Vec::::new() + ); + } + #[test] fn first_challenged_key_label_truncates_to_16_hex_chars() { let mut key = [0u8; 32]; diff --git a/src/replication/fresh.rs b/src/replication/fresh.rs index af656cb2..6de7174c 100644 --- a/src/replication/fresh.rs +++ b/src/replication/fresh.rs @@ -11,7 +11,7 @@ use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Arc; -use crate::logging::{debug, warn}; +use crate::logging::{debug, info, warn}; use bytes::Bytes; use rand::Rng; use saorsa_core::identity::PeerId; @@ -19,6 +19,7 @@ use saorsa_core::P2PNode; use tokio::sync::{mpsc, OwnedSemaphorePermit, Semaphore}; use crate::ant_protocol::XorName; +use crate::payment::PaymentVerifier; use crate::replication::config::{ ReplicationConfig, FRESH_REPLICATION_DELIVERY_MAX_RETRIES, REPLICATION_PROTOCOL_ID, }; @@ -48,6 +49,9 @@ pub struct FreshWriteEvent { pub(crate) struct FreshOfferEvent { pub(crate) key: XorName, pub(crate) payment_proof: Vec, + /// The fan-out must not judge peers on this write (see + /// [`announce_paid_write`]). + pub(crate) contested: bool, /// Storage read-backs attempted so far; see `MAX_FRESH_READ_ATTEMPTS`. pub(crate) read_attempts: u32, } @@ -84,8 +88,48 @@ impl AsRef<[u8]> for EncodedOffer { } } -/// Rules 6-8: record the paid key locally and announce it to -/// `PaidCloseGroup(K)`. +/// Rule 6: a node that validates `PoP` adds K to `PaidForList(self)`, marked +/// when `mark`. +/// +/// `mark` is the decision this node took when it verified the payment +/// (`PaymentVerifier::verified_payment_contested`), so a write +/// admitted from the payment cache is recorded by that decision, never by the +/// proof attached to it, which nobody checked. A contested write is marked +/// (ADR-0006) because a close-group peer may have refused it under its own +/// floor, and refusing a below-floor payment is network policy, never +/// misbehaviour. While this node holds the key, the mark keeps it from judging +/// its peers on it: the key goes out in paid hints only, so no repair proof is +/// recorded and no responsible audit samples it, and a prune audit of it +/// scores no one. Every other check that reads the mark points here. +/// +/// The first record of a key decides its mark: an existing entry is never +/// changed, so a later payment can neither add nor remove it. Returns whether +/// `mark` or the key's record is marked once this call is done, whichever +/// record won: ours, or one a concurrent `PaidNotify` or fresh offer wrote +/// first. A mark that cannot be read counts as set (see +/// `PaidList::counts_as_price_contested`). +pub(crate) async fn record_paid_key(key: &XorName, mark: bool, paid_list: &PaidList) -> bool { + let recorded = if mark { + paid_list.insert_price_contested(key).await + } else { + paid_list.insert(key).await + }; + if let Err(e) = recorded { + warn!("Failed to add key {} to PaidForList: {e}", hex::encode(key)); + } + mark || paid_list.counts_as_price_contested(key) +} + +/// Rules 6-8: record the paid key locally (see [`record_paid_key`]) and +/// announce it to `PaidCloseGroup(K)`. Returns whether the write's fan-out +/// must leave its peers unjudged (ADR-0006). +/// +/// The record is marked by the decision this node took when it verified the +/// payment. The fan-out is left unjudged when that decision or the key's +/// record is marked once recorded, whichever record won, so a marked write is +/// never judged, and also when the proof being forwarded is contested: a +/// cached PUT forwards it unverified, and its peers verify it and may refuse +/// it. /// /// This is the evidence peers need to repair the key later, so it runs the /// moment a write is accepted and is never gated by the pending-offer permit @@ -93,20 +137,28 @@ impl AsRef<[u8]> for EncodedOffer { pub(crate) async fn announce_paid_write( key: &XorName, proof_of_payment: &[u8], + payment_verifier: &PaymentVerifier, paid_list: &PaidList, p2p_node: &Arc, config: &ReplicationConfig, -) { - // Rule 6: Node that validates PoP adds K to PaidForList(self). - if let Err(e) = paid_list.insert(key).await { - warn!("Failed to add key {} to PaidForList: {e}", hex::encode(key)); - } +) -> bool { + let marked = record_paid_key( + key, + payment_verifier.verified_payment_contested(key), + paid_list, + ) + .await; // Rules 7-8: PaidNotify to every member of PaidCloseGroup(K). send_paid_notify(key, proof_of_payment, p2p_node, config).await; + marked + || payment_verifier + .price_contested(key, proof_of_payment) + .await } /// Rules 2-3: send fresh offers to the close group and schedule the delayed -/// possession check (ADR-0003) for the responsible peers. +/// possession check (ADR-0003) for the responsible peers, unless `contested` +/// (see [`announce_paid_write`]). /// /// `pending_offer` is the caller's permit from the pending-offer semaphore; /// it is held with the encoded offer until the last handle to it is dropped. @@ -117,6 +169,7 @@ pub(crate) async fn send_fresh_offers( key: &XorName, data: Vec, proof_of_payment: Vec, + contested: bool, pending_offer: OwnedSemaphorePermit, ) { let self_id = *ctx.p2p_node.peer_id(); @@ -210,12 +263,21 @@ pub(crate) async fn send_fresh_offers( // Schedule the delayed possession check (ADR-0003) for the responsible // close-group peers. A closed receiver (engine shutting down) is ignored. + // Not for a contested write: a peer may have refused it (ADR-0006). if !target_peers.is_empty() { ctx.dispatched.fetch_add(1, Ordering::Relaxed); - let _ = ctx.possession_check_tx.send(PossessionCheckEvent { - key: *key, - peers: target_peers, - }); + if contested { + info!( + "Fresh replication of {}: no possession check, the payment is contested \ + under the price floor", + hex::encode(key) + ); + } else { + let _ = ctx.possession_check_tx.send(PossessionCheckEvent { + key: *key, + peers: target_peers, + }); + } } } @@ -275,6 +337,52 @@ async fn send_paid_notify( mod tests { use super::*; + /// ADR-0006: a contested payment marks a newly recorded key, an + /// uncontested one records it plainly, and the first record of a key + /// decides: a later record, either way, changes nothing. Each call + /// reports the winning record's mark, or its own when that is set, so a + /// write whose plain decision lost to a marked record is still left + /// unjudged. + #[tokio::test] + async fn the_first_record_of_a_key_decides_its_mark() { + let dir = tempfile::tempdir().expect("tempdir"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let contested = [0x91u8; 32]; + let plain = [0x92u8; 32]; + + let reported = vec![ + record_paid_key(&contested, true, &paid_list).await, + record_paid_key(&plain, false, &paid_list).await, + record_paid_key(&contested, false, &paid_list).await, + record_paid_key(&plain, true, &paid_list).await, + ]; + + let marks: Vec<(bool, bool)> = [contested, plain] + .iter() + .map(|key| { + ( + paid_list.contains(key).expect("contains"), + paid_list.is_price_contested(key).expect("read mark"), + ) + }) + .collect(); + assert_eq!(marks, vec![(true, true), (true, false)]); + assert_eq!(reported, vec![true, false, true, true]); + } + + /// ADR-0006: a plain write is left unjudged while the paid list cannot be + /// read, since the record that won may be marked. + #[tokio::test] + async fn a_write_whose_mark_cannot_be_read_is_left_unjudged() { + let dir = tempfile::tempdir().expect("tempdir"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let plain = [0x93u8; 32]; + assert!(!record_paid_key(&plain, false, &paid_list).await); + + let _reader = paid_list.hold_reader_for_tests().expect("hold reader"); + assert!(record_paid_key(&plain, false, &paid_list).await); + } + /// The permit is released with the last handle to the encoded offer, not /// the first: a slice the transport still holds keeps the offer counted. #[test] diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 62937413..2845c847 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -2487,11 +2487,14 @@ impl ReplicationEngine { /// /// Unlike the PUT path, this waits for a pending-offer permit before /// returning, and it offers the caller's bytes rather than reading them - /// back from storage. + /// back from storage. Test-only: production writes enter through the PUT + /// handler's fresh-write channel. + #[cfg(any(test, feature = "test-utils"))] pub async fn replicate_fresh(&self, key: &XorName, data: &[u8], proof_of_payment: &[u8]) { - fresh::announce_paid_write( + let contested = fresh::announce_paid_write( key, proof_of_payment, + &self.payment_verifier, &self.paid_list, &self.p2p_node, &self.config, @@ -2509,6 +2512,7 @@ impl ReplicationEngine { key, data.to_vec(), proof_of_payment.to_vec(), + contested, pending_offer, ) .await; @@ -2550,6 +2554,7 @@ impl ReplicationEngine { ) { let p2p = Arc::clone(&self.p2p_node); let paid_list = Arc::clone(&self.paid_list); + let payment_verifier = Arc::clone(&self.payment_verifier); let config = Arc::clone(&self.config); let shutdown = self.shutdown.clone(); @@ -2565,9 +2570,10 @@ impl ReplicationEngine { // Stage one never waits for chunk back-pressure: the paid-list // entry and PaidNotify are what let peers repair the key later, // so every queued write gets them at arrival rate. - fresh::announce_paid_write( + let contested = fresh::announce_paid_write( &event.key, &event.payment_proof, + &payment_verifier, &paid_list, &p2p, &config, @@ -2577,6 +2583,7 @@ impl ReplicationEngine { .send(fresh::FreshOfferEvent { key: event.key, payment_proof: event.payment_proof, + contested, read_attempts: 0, }) .is_err() @@ -2689,6 +2696,7 @@ impl ReplicationEngine { &event.key, data, event.payment_proof, + event.contested, pending_offer, ) .await; @@ -3539,6 +3547,7 @@ impl ReplicationEngine { fn start_audit_loop(&mut self) { let p2p = Arc::clone(&self.p2p_node); let storage = Arc::clone(&self.storage); + let paid_list = Arc::clone(&self.paid_list); let config = Arc::clone(&self.config); let shutdown = self.shutdown.clone(); let sync_history = Arc::clone(&self.sync_history); @@ -3573,6 +3582,7 @@ impl ReplicationEngine { audit::audit_tick_with_repair_proofs( &p2p, &storage, + &paid_list, &config, &history, &repair_proofs, @@ -3598,6 +3608,7 @@ impl ReplicationEngine { audit::audit_tick_with_repair_proofs( &p2p, &storage, + &paid_list, &config, &history, &repair_proofs, @@ -6602,10 +6613,15 @@ async fn handle_fresh_offer( } } - // Rule 6: add to PaidForList. - if let Err(e) = ctx.paid_list.insert(key).await { - warn!("Failed to add key to PaidForList: {e}"); - } + // Rule 6: add to PaidForList, marked if the price floor contested the + // payment when this node verified it, before the record is stored and + // can be hinted. + fresh::record_paid_key( + key, + ctx.payment_verifier.verified_payment_contested(key), + &ctx.paid_list, + ) + .await; // Store the record. match ctx.storage.put(key, data).await { @@ -6811,9 +6827,15 @@ async fn handle_paid_notify( } } - if let Err(e) = paid_list.insert(¬ify.key).await { - warn!("Failed to add paid notify key to PaidForList: {e}"); - } + // Marked if the price floor contested the payment when this node verified + // it, so a record later repaired onto this node is not used to judge its + // peers either. + fresh::record_paid_key( + ¬ify.key, + payment_verifier.verified_payment_contested(¬ify.key), + paid_list, + ) + .await; Ok(()) } diff --git a/src/replication/neighbor_sync.rs b/src/replication/neighbor_sync.rs index b1f0d859..d035f32f 100644 --- a/src/replication/neighbor_sync.rs +++ b/src/replication/neighbor_sync.rs @@ -13,6 +13,7 @@ use saorsa_core::identity::PeerId; use saorsa_core::P2PNode; use crate::ant_protocol::XorName; +use crate::error::Result; use crate::replication::config::{ReplicationConfig, REPLICATION_PROTOCOL_ID}; use crate::replication::paid_list::PaidList; use crate::replication::protocol::{ @@ -61,27 +62,50 @@ pub(crate) struct PeerSyncHints { /// This intentionally scans all locally stored records, not only records for /// which this node is still responsible. Out-of-range records retained by /// prune hysteresis therefore continue to repair their new close group before -/// this node is allowed to delete them. +/// this node is allowed to delete them. Price-contested records are left out +/// (ADR-0006). pub async fn build_replica_hints_for_peer( peer: &PeerId, storage: &Arc, + paid_list: &PaidList, p2p_node: &Arc, close_group_size: usize, ) -> Vec { - build_replica_hints_for_peer_with_close_groups(peer, storage, p2p_node, close_group_size) - .await - .into_iter() - .map(|hint| hint.key) - .collect() + build_replica_hints_for_peer_with_close_groups( + peer, + storage, + paid_list, + p2p_node, + close_group_size, + ) + .await + .into_iter() + .map(|hint| hint.key) + .collect() +} + +/// Stored keys this node may hint as replicas: every stored key except those +/// marked price-contested in the paid list (see `fresh::record_paid_key`), +/// which go out as paid hints only. A replica hint records a repair proof the +/// responsible audit later judges the peer on, and makes this node the sole +/// hinter a peer penalises when its quorum check fails; a paid hint does +/// neither. The marks are read from memory, so this scans nothing on disk; if +/// they could not be read when the paid list opened, no key is hinted as a +/// replica. +async fn replica_hint_keys(storage: &ChunkStore, paid_list: &PaidList) -> Result> { + let mut keys = storage.all_keys().await?; + paid_list.retain_unmarked(&mut keys); + Ok(keys) } pub(crate) async fn build_replica_hints_for_peer_with_close_groups( peer: &PeerId, storage: &Arc, + paid_list: &PaidList, p2p_node: &Arc, close_group_size: usize, ) -> Vec { - let all_keys = match storage.all_keys().await { + let all_keys = match replica_hint_keys(storage, paid_list).await { Ok(keys) => keys, Err(e) => { warn!("Failed to read stored keys for hint construction: {e}"); @@ -125,7 +149,7 @@ pub(crate) async fn build_sync_hints_for_peers( return hints_by_peer; } - let all_keys = match storage.all_keys().await { + let all_keys = match replica_hint_keys(storage, paid_list).await { Ok(keys) => keys, Err(e) => { warn!("Failed to read stored keys for batch hint construction: {e}"); @@ -521,6 +545,7 @@ pub(crate) async fn handle_sync_request_with_proofs( let sent_replica_hints = build_replica_hints_for_peer_with_close_groups( sender, storage, + paid_list, p2p_node, config.close_group_size, ) @@ -558,6 +583,7 @@ pub(crate) async fn handle_sync_request_with_proofs( mod tests { use super::*; use crate::replication::types::PeerSyncRecord; + use crate::storage::ChunkStoreConfig; use std::collections::HashMap; /// Build a `PeerId` from a single byte (zero-padded to 32 bytes). @@ -567,6 +593,44 @@ mod tests { PeerId::from_bytes(bytes) } + // -- replica_hint_keys --------------------------------------------------- + + /// ADR-0006: a stored key marked price-contested is never hinted as a + /// replica; every other stored key still is. The filter reads no paid-list + /// database, so it works while this thread cannot read one. + #[tokio::test] + async fn price_contested_keys_are_not_hinted_as_replicas() { + let dir = tempfile::tempdir().expect("tempdir"); + let storage = ChunkStore::new(ChunkStoreConfig { + root_dir: dir.path().join("chunks"), + ..ChunkStoreConfig::test_default() + }) + .await + .expect("chunk store"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let contents: [&[u8]; 3] = [b"plain record", b"contested record", b"unpaid record"]; + let [plain, contested, unpaid] = contents.map(ChunkStore::compute_address); + for (key, content) in [plain, contested, unpaid].iter().zip(contents) { + storage.put(key, content).await.expect("put"); + } + paid_list.insert(&plain).await.expect("insert"); + paid_list + .insert_price_contested(&contested) + .await + .expect("mark"); + + let _reader = paid_list.hold_reader_for_tests().expect("hold reader"); + assert!(paid_list.is_price_contested(&contested).is_err()); + let mut hinted = replica_hint_keys(&storage, &paid_list) + .await + .expect("read keys"); + hinted.sort_unstable(); + let mut expected = vec![plain, unpaid]; + expected.sort_unstable(); + + assert_eq!(hinted, expected); + } + // -- select_sync_batch --------------------------------------------------- #[test] diff --git a/src/replication/paid_list.rs b/src/replication/paid_list.rs index 26d01210..d196cd52 100644 --- a/src/replication/paid_list.rs +++ b/src/replication/paid_list.rs @@ -11,7 +11,11 @@ //! ``` //! //! One unnamed database stores set membership: key = 32-byte `XorName`, -//! value = empty byte slice. +//! value = empty byte slice, or one marker byte for a key whose +//! payment the price floor contested here (ADR-0006). Releases before that +//! marker only ever test presence, so they read a marked key as paid. The +//! marked keys are also held in memory, so hint construction never scans the +//! database for them. //! //! ## Out-of-range timestamps //! @@ -22,24 +26,55 @@ use crate::ant_protocol::XorName; use crate::error::{Error, Result}; -use crate::logging::{debug, trace, warn}; +use crate::logging::{debug, info, trace, warn}; use heed::types::Bytes; -use heed::{Database, Env, EnvOpenOptions}; -use parking_lot::RwLock; -use std::collections::HashMap; +#[cfg(test)] +use heed::WithTls; +use heed::{Database, Env, EnvOpenOptions, RoTxn}; +use parking_lot::{Mutex, RwLock}; +use std::collections::{HashMap, HashSet}; use std::path::Path; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::Arc; use std::time::Instant; use tokio::task::spawn_blocking; use tokio_util::task::TaskTracker; use crate::ant_protocol::XORNAME_LEN; +/// Value stored for a key whose payment the price floor contested here (see +/// `fresh::record_paid_key`). +const PRICE_CONTESTED_VALUE: &[u8] = &[1]; + /// Default LMDB map size for the paid list: 256 MiB. /// -/// The paid list stores only 32-byte keys with empty values, so this is -/// generous even for very large close-group memberships. +/// The paid list stores only 32-byte keys with empty or one-byte values, so +/// this is generous even for very large close-group memberships. const DEFAULT_MAP_SIZE: usize = 256 * 1_024 * 1_024; +/// What the open task returns: the environment, the database and the keys +/// marked price-contested in it, if they could be read. +type OpenedPaidList = (Env, Database, Result>); + +/// Every key whose entry in `db` carries the price-contested mark. +fn read_price_contested_keys(db: &Database, txn: &RoTxn) -> Result> { + let mut keys = HashSet::new(); + for result in db + .iter(txn) + .map_err(|e| Error::Storage(format!("Failed to iterate paid-list: {e}")))? + { + let (key_bytes, value) = + result.map_err(|e| Error::Storage(format!("Failed to read paid-list entry: {e}")))?; + if value.is_empty() { + continue; + } + if let Ok(key) = XorName::try_from(key_bytes) { + keys.insert(key); + } + } + Ok(keys) +} + /// Persistent paid-for-list backed by LMDB. /// /// Tracks which keys this node believes are paid-authorized. @@ -47,8 +82,26 @@ const DEFAULT_MAP_SIZE: usize = 256 * 1_024 * 1_024; pub struct PaidList { /// LMDB environment. env: Env, - /// The unnamed default database (key = `XorName` bytes, value = empty). + /// The unnamed default database (key = `XorName` bytes, value = empty, or + /// the price-contested marker). db: Database, + /// In-memory copy of the keys whose entry carries the price-contested + /// mark, or `None` if they could not be read on open. Changed only by the + /// blocking task that writes the database, under `write_order`, so a + /// cancelled caller cannot leave it stale. Readers clone the `Arc` and + /// filter outside the lock; a write while such a snapshot is alive + /// copies the set first (`Arc::make_mut`). + price_contested: Arc>>>>, + /// Held by every database write from before its write transaction until + /// it has updated `price_contested`, so the copy changes in commit order. + /// A new mark goes in just before its commit and a removed one just after, + /// so the copy never lacks a committed mark. A mark whose commit fails + /// stays, since that commit may still have reached the disk: when in + /// doubt, judge no one. + write_order: Arc>, + /// Mark reads that have failed since the last one that succeeded, so a + /// run of failures is logged once (see `counts_as_price_contested`). + failed_mark_reads: AtomicU64, /// In-memory: when each paid key first went out of `PaidCloseGroup` range. /// Cleared on restart (safe: hysteresis clock restarts from zero). paid_out_of_range: RwLock>, @@ -85,7 +138,7 @@ impl PaidList { // Constructor-only blocking task: it runs before `self` (and its // `blocking_tracker`) exists, so it is deliberately untracked. The // constructor awaits it right here, so it cannot outlive this call. - let (env, db) = spawn_blocking(move || -> Result<(Env, Database)> { + let (env, db, price_contested) = spawn_blocking(move || -> Result { // SAFETY: `EnvOpenOptions::open()` is unsafe because LMDB uses // memory-mapped I/O and relies on OS file-locking to prevent // corruption from concurrent access by multiple processes. We @@ -110,15 +163,36 @@ impl PaidList { .map_err(|e| Error::Storage(format!("Failed to create paid-list database: {e}")))?; wtxn.commit() .map_err(|e| Error::Storage(format!("Failed to commit db creation: {e}")))?; + // A failed read leaves its transaction unable to commit, so the + // scan gets one of its own, dropped uncommitted. A write + // transaction takes no reader slot on this thread. + let price_contested = env + .write_txn() + .map_err(|e| Error::Storage(format!("Failed to create write txn: {e}"))) + .and_then(|scan| read_price_contested_keys(&db, &scan)); - Ok((env, db)) + Ok((env, db, price_contested)) }) .await .map_err(|e| Error::Storage(format!("Paid-list init task failed: {e}")))??; + // A paid list whose marks cannot be read still opens; the node then + // sends no replica hints until it restarts (see `retain_unmarked`). + let price_contested = price_contested + .inspect_err(|e| { + warn!( + "Cannot read the paid list's price-contested marks; sending no replica \ + hints until restart: {e}" + ); + }) + .ok() + .map(Arc::new); let paid_list = Self { env, db, + price_contested: Arc::new(RwLock::new(price_contested)), + write_order: Arc::new(Mutex::new(())), + failed_mark_reads: AtomicU64::new(0), paid_out_of_range: RwLock::new(HashMap::new()), record_out_of_range: RwLock::new(HashMap::new()), paid_prune_cursor: RwLock::new(0), @@ -139,6 +213,25 @@ impl PaidList { /// /// Returns an error if the LMDB write transaction fails. pub async fn insert(&self, key: &XorName) -> Result { + self.insert_with_value(key, &[]).await + } + + /// Insert `key` as paid at a price the floor contested here (see + /// `fresh::record_paid_key`). Like [`Self::insert`], an existing entry is + /// left exactly as it is, so a key already recorded can never gain the + /// mark later. The mark survives restarts and later inserts, and goes + /// with the entry on [`Self::remove`]. + /// + /// Returns `true` if the key was newly added, `false` if it already existed. + /// + /// # Errors + /// + /// Returns an error if the LMDB write transaction fails. + pub async fn insert_price_contested(&self, key: &XorName) -> Result { + self.insert_with_value(key, PRICE_CONTESTED_VALUE).await + } + + async fn insert_with_value(&self, key: &XorName, value: &'static [u8]) -> Result { // Fast-path: avoid write transaction if key already present. if self.contains(key)? { trace!("Paid-list key {} already present", hex::encode(key)); @@ -148,10 +241,13 @@ impl PaidList { let key_owned = *key; let env = self.env.clone(); let db = self.db; + let price_contested = Arc::clone(&self.price_contested); + let write_order = Arc::clone(&self.write_order); let was_new = self .blocking_tracker .spawn_blocking(move || -> Result { + let ordered = write_order.lock(); let mut wtxn = env .write_txn() .map_err(|e| Error::Storage(format!("Failed to create write txn: {e}")))?; @@ -167,11 +263,17 @@ impl PaidList { return Ok(false); } - db.put(&mut wtxn, &key_owned, &[]) + db.put(&mut wtxn, &key_owned, value) .map_err(|e| Error::Storage(format!("Failed to insert into paid-list: {e}")))?; + if !value.is_empty() { + if let Some(keys) = price_contested.write().as_mut() { + Arc::make_mut(keys).insert(key_owned); + } + } wtxn.commit().map_err(|e| { Error::Storage(format!("Failed to commit paid-list insert: {e}")) })?; + drop(ordered); Ok(true) }) @@ -179,12 +281,87 @@ impl PaidList { .map_err(|e| Error::Storage(format!("Paid-list insert task failed: {e}")))??; if was_new { - debug!("Added key {} to paid-list", hex::encode(key)); + debug!( + "Added key {} to paid-list{}", + hex::encode(key), + if value.is_empty() { + "" + } else { + " as price-contested" + } + ); } Ok(was_new) } + /// Whether `key` is paid at a price the floor contested here (see + /// [`Self::insert_price_contested`]). + /// + /// Reads the database, which is authoritative: the in-memory copy behind + /// [`Self::retain_unmarked`] can hold a mark whose commit is under way or + /// failed, or be missing. A caller that has just seen a racing write's + /// entry sees its mark here as well (see `fresh::record_paid_key`). + /// + /// # Errors + /// + /// Returns an error if the LMDB read transaction fails. + pub fn is_price_contested(&self, key: &XorName) -> Result { + let rtxn = self + .env + .read_txn() + .map_err(|e| Error::Storage(format!("Failed to create read txn: {e}")))?; + let marked = self + .db + .get(&rtxn, key.as_ref()) + .map_err(|e| Error::Storage(format!("Failed to read paid-list entry: {e}")))? + .is_some_and(|value| !value.is_empty()); + Ok(marked) + } + + /// Whether `key` must be treated as price-contested: its entry carries the + /// mark, or the mark cannot be read (when in doubt, judge no one). While + /// reads fail no peer is judged on any key they hit, so the first failure + /// of a run is a warning, and the read that ends the run reports how many + /// failed. + pub fn counts_as_price_contested(&self, key: &XorName) -> bool { + match self.is_price_contested(key) { + Ok(marked) => { + let failed = self.failed_mark_reads.swap(0, Ordering::Relaxed); + if failed > 0 { + info!("Price-contested marks are readable again, after {failed} failed reads"); + } + marked + } + Err(e) => { + if self.failed_mark_reads.fetch_add(1, Ordering::Relaxed) == 0 { + warn!( + "Cannot read the price-contested mark of key {}; counting unreadable \ + marks as set until a read succeeds: {e}", + hex::encode(key) + ); + } else { + debug!( + "Price-contested mark of key {} still unreadable: {e}", + hex::encode(key) + ); + } + true + } + } + } + + /// Drop from `keys` every key marked price-contested, using the in-memory + /// copy rather than the database. If the marks could not be read on open, + /// no key is known to be unmarked, so every key is dropped. + pub fn retain_unmarked(&self, keys: &mut Vec) { + let snapshot = self.price_contested.read().clone(); + match snapshot { + Some(marked) => keys.retain(|key| !marked.contains(key)), + None => keys.clear(), + } + } + /// Remove a key from the paid-for set. /// /// Also clears any in-memory out-of-range timestamps for this key. @@ -198,10 +375,13 @@ impl PaidList { let key_owned = *key; let env = self.env.clone(); let db = self.db; + let price_contested = Arc::clone(&self.price_contested); + let write_order = Arc::clone(&self.write_order); let existed = self .blocking_tracker .spawn_blocking(move || -> Result { + let ordered = write_order.lock(); let mut wtxn = env .write_txn() .map_err(|e| Error::Storage(format!("Failed to create write txn: {e}")))?; @@ -211,6 +391,12 @@ impl PaidList { wtxn.commit().map_err(|e| { Error::Storage(format!("Failed to commit paid-list delete: {e}")) })?; + if let Some(keys) = price_contested.write().as_mut() { + if keys.contains(&key_owned) { + Arc::make_mut(keys).remove(&key_owned); + } + } + drop(ordered); Ok(deleted) }) .await @@ -415,10 +601,13 @@ impl PaidList { let keys_owned: Vec = keys.to_vec(); let env = self.env.clone(); let db = self.db; + let price_contested = Arc::clone(&self.price_contested); + let write_order = Arc::clone(&self.write_order); let removed_keys = self .blocking_tracker .spawn_blocking(move || -> Result> { + let ordered = write_order.lock(); let mut wtxn = env .write_txn() .map_err(|e| Error::Storage(format!("Failed to create write txn: {e}")))?; @@ -435,6 +624,15 @@ impl PaidList { wtxn.commit() .map_err(|e| Error::Storage(format!("Failed to commit batch remove: {e}")))?; + if let Some(keys) = price_contested.write().as_mut() { + if removed.iter().any(|key| keys.contains(key)) { + let keys = Arc::make_mut(keys); + for key in &removed { + keys.remove(key); + } + } + } + drop(ordered); Ok(removed) }) @@ -480,18 +678,46 @@ impl PaidList { self.blocking_tracker.wait().await; self.blocking_tracker.reopen(); } + + /// Hold a read transaction on the calling thread. LMDB gives each thread + /// one reader slot, so until it is dropped every paid-list read on this + /// thread fails, which lets a test drive the paths a mark that cannot be + /// read takes. + #[cfg(test)] + pub(crate) fn hold_reader_for_tests(&self) -> Result> { + self.env + .read_txn() + .map_err(|e| Error::Storage(format!("Failed to create read txn: {e}"))) + } } #[cfg(test)] #[allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] mod tests { use super::*; + #[cfg(target_pointer_width = "64")] + use std::fs::OpenOptions; + #[cfg(target_pointer_width = "64")] + use std::io::{Read, Seek, SeekFrom, Write}; + #[cfg(feature = "logging")] + use tracing::subscriber::with_default; + #[cfg(feature = "logging")] + use tracing_subscriber::fmt::writer::{MakeWriter, MutexGuardWriter}; /// Short probe used to prove `wait_idle` is still blocked on a parked op. const WAIT_IDLE_BLOCKED_PROBE: std::time::Duration = std::time::Duration::from_millis(200); /// Generous ceiling for `wait_idle` to complete once the op is released. const WAIT_IDLE_COMPLETE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); + /// The in-memory copy of the marks, which must have been read on open. + fn marked_in_memory(pl: &PaidList) -> HashSet { + pl.price_contested + .read() + .as_deref() + .cloned() + .expect("marks read on open") + } + async fn create_test_paid_list() -> (PaidList, tempfile::TempDir) { let temp_dir = tempfile::TempDir::new().expect("create temp dir"); let paid_list = PaidList::new(temp_dir.path()) @@ -572,6 +798,264 @@ mod tests { } } + /// ADR-0006: the price-contested mark is written only with a new entry, so + /// an already recorded key can never gain it. It survives a restart and a + /// later plain insert, and goes with the entry on removal. + #[tokio::test] + async fn price_contested_mark_persists_until_the_key_is_removed() { + let temp_dir = tempfile::TempDir::new().expect("create temp dir"); + let marked: XorName = [0x51; 32]; + let recorded_first: XorName = [0x52; 32]; + + { + let pl = PaidList::new(temp_dir.path()) + .await + .expect("create paid list"); + assert!(pl.insert(&recorded_first).await.expect("insert")); + assert!(pl.insert_price_contested(&marked).await.expect("mark")); + assert!(!pl + .insert_price_contested(&recorded_first) + .await + .expect("mark an existing key")); + assert!(!pl.insert(&marked).await.expect("re-insert")); + } + + let pl = PaidList::new(temp_dir.path()) + .await + .expect("reopen paid list"); + assert_eq!(pl.count().expect("count"), 2); + assert!(pl.contains(&marked).expect("contains")); + assert!(pl.is_price_contested(&marked).expect("read mark")); + assert!(!pl.is_price_contested(&recorded_first).expect("read mark")); + assert_eq!(marked_in_memory(&pl), HashSet::from([marked])); + + pl.remove(&marked).await.expect("remove"); + pl.insert(&marked).await.expect("insert again"); + assert!(!pl.is_price_contested(&marked).expect("read mark")); + assert!(marked_in_memory(&pl).is_empty()); + } + + /// ADR-0006: the in-memory marks follow every insert, removal and batch + /// removal. + #[tokio::test] + async fn the_in_memory_marks_follow_every_write() { + let (pl, _temp) = create_test_paid_list().await; + let first: XorName = [0x61; 32]; + let second: XorName = [0x62; 32]; + let plain: XorName = [0x63; 32]; + pl.insert_price_contested(&first).await.expect("mark"); + pl.insert_price_contested(&second).await.expect("mark"); + pl.insert(&plain).await.expect("insert"); + assert_eq!(marked_in_memory(&pl), HashSet::from([first, second])); + let mut keys = vec![first, plain, second]; + pl.retain_unmarked(&mut keys); + assert_eq!(keys, vec![plain]); + pl.remove(&first).await.expect("remove"); + assert_eq!(marked_in_memory(&pl), HashSet::from([second])); + pl.remove_batch(&[second, plain]) + .await + .expect("remove batch"); + assert!(marked_in_memory(&pl).is_empty()); + } + + /// ADR-0006: while a database write holds the write order, a marked + /// insert or a batch removal neither commits nor changes the in-memory + /// copy, so writes and the copy move in one order. + #[tokio::test] + async fn the_in_memory_marks_change_in_commit_order() { + let (pl, _temp) = create_test_paid_list().await; + let key: XorName = [0x6B; 32]; + let (release_tx, holder) = hold_write_order(&pl).await; + + let insert = pl.insert_price_contested(&key); + tokio::pin!(insert); + let blocked = tokio::time::timeout(WAIT_IDLE_BLOCKED_PROBE, &mut insert).await; + assert!(blocked.is_err(), "the insert must wait for the write order"); + assert!(!pl.contains(&key).expect("contains")); + assert!(marked_in_memory(&pl).is_empty()); + + drop(release_tx); + holder.await.expect("holder task"); + assert!(insert.await.expect("insert")); + assert_eq!(marked_in_memory(&pl), HashSet::from([key])); + + let (release_tx, holder) = hold_write_order(&pl).await; + + let batch = [key]; + let removal = pl.remove_batch(&batch); + tokio::pin!(removal); + let blocked = tokio::time::timeout(WAIT_IDLE_BLOCKED_PROBE, &mut removal).await; + assert!( + blocked.is_err(), + "the removal must wait for the write order" + ); + assert!(pl.contains(&key).expect("contains")); + assert_eq!(marked_in_memory(&pl), HashSet::from([key])); + + drop(release_tx); + holder.await.expect("holder task"); + assert_eq!(removal.await.expect("remove batch"), 1); + assert!(marked_in_memory(&pl).is_empty()); + } + + /// Byte offsets in an LMDB meta page on a 64-bit target: the magic number + /// follows the 16-byte page header, the free list's record keeps the page + /// size at 40, and the main database's root page number sits at 128. + #[cfg(target_pointer_width = "64")] + const LMDB_MAGIC_AT: u64 = 16; + #[cfg(target_pointer_width = "64")] + const LMDB_PAGE_SIZE_AT: u64 = 40; + #[cfg(target_pointer_width = "64")] + const LMDB_MAIN_ROOT_AT: u64 = 128; + + /// ADR-0006: a paid list whose marks cannot be read still opens, with no + /// marks in memory, so the node hints no key as a replica. Both meta pages + /// point the database's root past the end of the file, so the scan for + /// the marks fails as it would on a lost page. + #[cfg(target_pointer_width = "64")] + #[tokio::test] + async fn a_paid_list_whose_marks_cannot_be_read_still_opens() { + let temp_dir = tempfile::TempDir::new().expect("create temp dir"); + let marked: XorName = [0x70; 32]; + { + let pl = PaidList::new(temp_dir.path()) + .await + .expect("create paid list"); + pl.insert_price_contested(&marked).await.expect("mark"); + } + + let mut data = OpenOptions::new() + .read(true) + .write(true) + .open(temp_dir.path().join("paid_list.mdb").join("data.mdb")) + .expect("open data file"); + let mut read_u32 = |at: u64| { + let mut word = [0u8; 4]; + data.seek(SeekFrom::Start(at)).expect("seek"); + data.read_exact(&mut word).expect("read"); + u32::from_ne_bytes(word) + }; + let page_size = u64::from(read_u32(LMDB_PAGE_SIZE_AT)); + for meta in [0, page_size] { + assert_eq!( + read_u32(meta + LMDB_MAGIC_AT), + 0xBEEF_C0DE, + "LMDB meta page" + ); + } + for meta in [0, page_size] { + data.seek(SeekFrom::Start(meta + LMDB_MAIN_ROOT_AT)) + .expect("seek"); + data.write_all(&(1u64 << 40).to_ne_bytes()).expect("write"); + } + data.sync_all().expect("sync"); + drop(data); + + let pl = PaidList::new(temp_dir.path()) + .await + .expect("a paid list whose marks cannot be read must open"); + assert!(pl.price_contested.read().is_none()); + assert!(pl.is_price_contested(&marked).is_err()); + let mut keys = vec![marked]; + pl.retain_unmarked(&mut keys); + assert_eq!(keys, Vec::::new()); + } + + /// ADR-0006: a paid list whose marks could not be read on open shows no + /// key as unmarked, so the node hints no key as a replica, while writes and + /// point reads of the marks still work. + #[tokio::test] + async fn marks_unread_on_open_hint_no_replicas() { + let (pl, _temp) = create_test_paid_list().await; + let marked: XorName = [0x6C; 32]; + let plain: XorName = [0x6D; 32]; + *pl.price_contested.write() = None; + + let mut keys = vec![plain]; + pl.retain_unmarked(&mut keys); + assert_eq!(keys, Vec::::new()); + pl.insert_price_contested(&marked).await.expect("mark"); + pl.insert(&plain).await.expect("insert"); + pl.remove_batch(&[plain]).await.expect("remove batch"); + assert!(pl.price_contested.read().is_none()); + assert!(pl.counts_as_price_contested(&marked)); + assert!(!pl.counts_as_price_contested(&plain)); + } + + /// ADR-0006: a key counts as price-contested when it is marked, or when + /// its mark cannot be read; an unmarked or unpaid key that reads fine + /// does not. + #[tokio::test] + async fn an_unreadable_mark_counts_as_price_contested() { + let (pl, _temp) = create_test_paid_list().await; + let marked: XorName = [0x64; 32]; + let plain: XorName = [0x65; 32]; + let unpaid: XorName = [0x66; 32]; + pl.insert_price_contested(&marked).await.expect("mark"); + pl.insert(&plain).await.expect("insert"); + + assert!(pl.counts_as_price_contested(&marked)); + assert!(!pl.counts_as_price_contested(&plain)); + assert!(!pl.counts_as_price_contested(&unpaid)); + + let _reader = pl.hold_reader_for_tests().expect("hold reader"); + assert!(pl.counts_as_price_contested(&plain)); + assert!(pl.counts_as_price_contested(&unpaid)); + } + + /// In-memory sink a test subscriber writes its lines to. + #[cfg(feature = "logging")] + #[derive(Clone, Default)] + struct LogSink(Arc>>); + + #[cfg(feature = "logging")] + impl<'a> MakeWriter<'a> for LogSink { + type Writer = MutexGuardWriter<'a, Vec>; + + fn make_writer(&'a self) -> Self::Writer { + self.0.make_writer() + } + } + + /// ADR-0006: a run of mark reads that fail logs one warning, not one per + /// key, and the read that ends the run reports how many failed. + #[cfg(feature = "logging")] + #[tokio::test] + async fn a_run_of_unreadable_marks_warns_once() { + let (pl, _temp) = create_test_paid_list().await; + let first: XorName = [0x6E; 32]; + let second: XorName = [0x6F; 32]; + pl.insert(&first).await.expect("insert"); + let sink = LogSink::default(); + let subscriber = tracing_subscriber::fmt() + .with_writer(sink.clone()) + .with_ansi(false) + .finish(); + + with_default(subscriber, || { + { + let _reader = pl.hold_reader_for_tests().expect("hold reader"); + for key in [first, second, first] { + assert!(pl.counts_as_price_contested(&key)); + } + } + assert!(!pl.counts_as_price_contested(&first)); + assert!(!pl.counts_as_price_contested(&second)); + }); + + let logs = String::from_utf8(sink.0.lock().expect("sink").clone()).expect("utf-8"); + assert_eq!( + logs.matches("Cannot read the price-contested mark").count(), + 1, + "{logs}" + ); + assert_eq!( + logs.matches("readable again, after 3 failed reads").count(), + 1, + "{logs}" + ); + } + #[tokio::test] async fn test_all_keys() { let (pl, _temp) = create_test_paid_list().await; @@ -757,6 +1241,29 @@ mod tests { (release_tx, holder) } + /// Hold the paid list's write order on a blocking thread, so the next + /// write waits before its write transaction. + /// + /// Returns once the order is held. Dropping the returned sender releases + /// it. + async fn hold_write_order( + pl: &PaidList, + ) -> ( + tokio::sync::oneshot::Sender<()>, + tokio::task::JoinHandle<()>, + ) { + let write_order = Arc::clone(&pl.write_order); + let (release_tx, release_rx) = tokio::sync::oneshot::channel::<()>(); + let (held_tx, held_rx) = tokio::sync::oneshot::channel::<()>(); + let holder = tokio::task::spawn_blocking(move || { + let _ordered = write_order.lock(); + let _ = held_tx.send(()); + let _ = release_rx.blocking_recv(); + }); + held_rx.await.expect("write order held"); + (release_tx, holder) + } + /// Dropping an `insert` awaiter does not cancel its `spawn_blocking` LMDB /// transaction; `wait_idle` must wait for that detached write, and the /// paid list must remain usable afterwards. diff --git a/src/replication/pointer.rs b/src/replication/pointer.rs index 7afa5565..88aca32e 100644 --- a/src/replication/pointer.rs +++ b/src/replication/pointer.rs @@ -1450,6 +1450,13 @@ impl PointerReplication { warn!("A fresh pointer write did not parse; not replicating it"); return; }; + // Decided from the proof being forwarded, before it moves into the + // offer: a state the price floor contests here is not used to judge + // this node's peers (ADR-0006). + let contested = self + .payments + .price_contested(&state.address, &write.payment_proof) + .await; let self_id = *self.p2p.peer_id(); let targets: Vec = self .p2p @@ -1504,6 +1511,14 @@ impl PointerReplication { hex::encode(state.address), targets.len() ); + if contested { + info!( + "Fresh pointer {}: no possession check, the payment is contested under \ + the price floor", + hex::encode(state.address) + ); + return; + } self.schedule_possession_check(state, targets); } diff --git a/src/replication/pruning.rs b/src/replication/pruning.rs index da8d7f9f..06052436 100644 --- a/src/replication/pruning.rs +++ b/src/replication/pruning.rs @@ -342,6 +342,7 @@ struct PruneAuditReportState { #[derive(Clone, Copy)] struct PruneAuditContext<'a> { storage: &'a Arc, + paid_list: &'a PaidList, p2p_node: &'a Arc, config: &'a ReplicationConfig, sync_state: &'a Arc>, @@ -360,6 +361,7 @@ pub async fn run_prune_pass_with_context(ctx: PrunePassContext<'_>) -> PruneResu let (paid_count, paid_stats) = prune_paid_entries( ctx.self_id, ctx.paid_list, + ctx.storage, ctx.p2p_node, ctx.config, now, @@ -508,16 +510,8 @@ async fn prune_stored_records(ctx: &PrunePassContext<'_>) -> (usize, RecordPrune ) .await; - let present_by_key = collect_record_prune_proofs( - &audit_selection.candidates, - stored_keys.len(), - ctx.storage, - ctx.p2p_node, - ctx.config, - ctx.sync_state, - ctx.audit_challenge_coordinator, - ) - .await; + let present_by_key = + collect_record_prune_proofs(&audit_selection.candidates, stored_keys.len(), ctx).await; let (keys_to_delete, revalidated_cleared, audit_below_threshold) = revalidated_record_prune_keys(&audit_selection.candidates, &present_by_key, ctx).await; stats.cleared += revalidated_cleared; @@ -801,6 +795,7 @@ fn select_record_prune_audits( async fn prune_paid_entries( self_id: &PeerId, paid_list: &Arc, + storage: &ChunkStore, p2p_node: &Arc, config: &ReplicationConfig, now: Instant, @@ -849,7 +844,9 @@ async fn prune_paid_entries( let elapsed = now .checked_duration_since(first_seen) .unwrap_or(Duration::ZERO); - if elapsed >= config.prune_hysteresis_duration { + if elapsed >= config.prune_hysteresis_duration + && !keeps_price_contested_mark(key, paid_list, storage) + { match select_paid_prune_candidate( key, &closest, @@ -886,7 +883,7 @@ async fn prune_paid_entries( let confirmed_by_key = collect_paid_prune_confirmations(&expired_candidates, p2p_node, config).await; - let (paid_keys_to_delete, revalidated_cleared) = revalidated_paid_prune_keys( + let (mut paid_keys_to_delete, revalidated_cleared) = revalidated_paid_prune_keys( &expired_candidates, &confirmed_by_key, self_id, @@ -895,12 +892,24 @@ async fn prune_paid_entries( config, ) .await; + // Checked again after the network round, in case the record arrived + // meanwhile. + paid_keys_to_delete.retain(|key| !keeps_price_contested_mark(key, paid_list, storage)); stats.cleared += revalidated_cleared; stats.pruned = delete_paid_entries(&paid_keys_to_delete, paid_list).await; (paid_keys.len(), stats) } +/// Whether the paid entry for `key` must outlive the paid-prune rules: it +/// carries the price-contested mark (see `fresh::record_paid_key`) and this +/// node still holds the record the mark protects. State that cannot be read +/// counts as both (see `PaidList::counts_as_price_contested`). Such an entry +/// is not even put to the paid group, so it costs no verification budget. +fn keeps_price_contested_mark(key: &XorName, paid_list: &PaidList, storage: &ChunkStore) -> bool { + paid_list.counts_as_price_contested(key) && storage.exists(key).unwrap_or(true) +} + fn select_paid_prune_candidate( key: &XorName, closest: &[PeerId], @@ -1202,14 +1211,13 @@ async fn delete_stored_records( /// local, and the retained record continues to participate in normal /// neighbor-sync repair because replica hint construction walks all locally /// stored keys, including out-of-range keys retained by hysteresis. +/// +/// A price-contested key still needs its proofs before it is deleted, but no +/// absent peer is scored for it (see `fresh::record_paid_key`). async fn collect_record_prune_proofs( candidates: &[RecordPruneCandidate], local_stored_key_count: usize, - storage: &Arc, - p2p_node: &Arc, - config: &ReplicationConfig, - sync_state: &Arc>, - audit_challenge_coordinator: &Arc, + ctx: &PrunePassContext<'_>, ) -> HashMap> { if candidates.is_empty() { return HashMap::new(); @@ -1220,11 +1228,12 @@ async fn collect_record_prune_proofs( let challenges = build_peer_audit_challenges(candidates, max_keys_per_challenge); let report_state = PruneAuditReportState::default(); let audit_context = PruneAuditContext { - storage, - p2p_node, - config, - sync_state, - audit_challenge_coordinator, + storage: ctx.storage, + paid_list: ctx.paid_list, + p2p_node: ctx.p2p_node, + config: ctx.config, + sync_state: ctx.sync_state, + audit_challenge_coordinator: ctx.audit_challenge_coordinator, report_state: &report_state, }; let mut requests = stream::iter(challenges) @@ -1618,15 +1627,7 @@ async fn peer_proves_records( "failed" }; if !audit_failure_reported - && report_prune_audit_failure_once( - &peer, - &key, - context.p2p_node, - context.config, - context.report_state, - failure_class, - ) - .await + && report_prune_audit_failure_once(&peer, &key, context, failure_class).await { audit_failure_reported = true; } @@ -1671,16 +1672,7 @@ async fn receive_prune_audit_response( }; for key in challenge_keys { - if report_prune_audit_failure_once( - peer, - key, - context.p2p_node, - context.config, - context.report_state, - failure_class, - ) - .await - { + if report_prune_audit_failure_once(peer, key, context, failure_class).await { break; } } @@ -1964,11 +1956,25 @@ fn audit_digest_proves_key( async fn report_prune_audit_failure_once( peer: &PeerId, key: &XorName, - p2p_node: &Arc, - config: &ReplicationConfig, - report_state: &PruneAuditReportState, + context: PruneAuditContext<'_>, audit_failure_class: &'static str, ) -> bool { + let PruneAuditContext { + paid_list, + p2p_node, + config, + report_state, + .. + } = context; + // No peer is scored for a price-contested key (see + // `fresh::record_paid_key`); a mark that cannot be read counts as set. + if paid_list.counts_as_price_contested(key) { + debug!( + "Prune audit: not scoring {peer} for {}, a price-contested key", + hex::encode(key) + ); + return false; + } let should_report = peer_is_currently_responsible(peer, key, p2p_node, config).await && reserve_prune_audit_failure_report(report_state, peer).await; if !should_report { @@ -2118,6 +2124,53 @@ mod tests { RecordPruneCandidate { key, target_peers } } + /// ADR-0006: a price-contested paid entry is kept while its record is + /// held, and prunes like any other once the record is gone. Unmarked + /// entries are unaffected. + #[tokio::test] + async fn a_price_contested_paid_entry_outlives_paid_pruning_while_its_record_is_held() { + let dir = TempDir::new().expect("temp dir"); + let storage = ChunkStore::new(ChunkStoreConfig { + root_dir: dir.path().join("chunks"), + ..ChunkStoreConfig::test_default() + }) + .await + .expect("chunk store"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let held_bytes = b"held price-contested record".to_vec(); + let held = ChunkStore::compute_address(&held_bytes); + storage.put(&held, &held_bytes).await.expect("put"); + let gone: XorName = [0x81; 32]; + let plain: XorName = [0x82; 32]; + paid_list.insert_price_contested(&held).await.expect("mark"); + paid_list.insert_price_contested(&gone).await.expect("mark"); + paid_list.insert(&plain).await.expect("insert"); + + assert!(keeps_price_contested_mark(&held, &paid_list, &storage)); + assert!(!keeps_price_contested_mark(&gone, &paid_list, &storage)); + assert!(!keeps_price_contested_mark(&plain, &paid_list, &storage)); + + // A held record whose mark cannot be read is kept as if marked. + let plain_held_bytes = b"held plain record".to_vec(); + let plain_held = ChunkStore::compute_address(&plain_held_bytes); + storage + .put(&plain_held, &plain_held_bytes) + .await + .expect("put"); + paid_list.insert(&plain_held).await.expect("insert"); + assert!(!keeps_price_contested_mark( + &plain_held, + &paid_list, + &storage + )); + let _reader = paid_list.hold_reader_for_tests().expect("hold reader"); + assert!(keeps_price_contested_mark( + &plain_held, + &paid_list, + &storage + )); + } + /// A prune audit judges holders against this node's own copy. A copy that /// no longer hashes to its key would fail every honest holder and hold the /// prune back forever, so it is not used as a reference: the read takes it diff --git a/src/storage/handler.rs b/src/storage/handler.rs index 62e2db75..6212adf4 100644 --- a/src/storage/handler.rs +++ b/src/storage/handler.rs @@ -864,9 +864,9 @@ impl AntProtocol { self.quote_generator.record_store(); // 7. Notify replication engine for fresh fan-out. - // Only emit when a real proof is present — cached-as-verified - // PUTs have no proof to forward, and the chunk would have - // already replicated on the original write that carried one. + // Only emit when a real proof is present. A cached-as-verified + // PUT normally carries none; one that does forwards it + // unverified, and the fan-out prices it as it stands (ADR-0006). if let (Some(ref tx), Some(proof)) = (&self.fresh_write_tx, request.payment_proof) { // ADR-0004: strip any commitment sidecars before forwarding // to replication — the ADR specifies persisted/replicated @@ -1249,7 +1249,6 @@ mod tests { cache_capacity: 100_000, close_group_size: crate::ant_protocol::CLOSE_GROUP_SIZE, local_rewards_address: rewards_address, - price_floor: crate::payment::PriceFloorConfig::default(), }; let payment_verifier = Arc::new(PaymentVerifier::new(payment_config)); let metrics_tracker = QuotingMetricsTracker::new(100); diff --git a/tests/e2e/data_types/chunk.rs b/tests/e2e/data_types/chunk.rs index 650b8e08..96480f88 100644 --- a/tests/e2e/data_types/chunk.rs +++ b/tests/e2e/data_types/chunk.rs @@ -64,8 +64,8 @@ mod tests { use crate::{TestHarness, TestNetwork}; use ant_node::payment::{ - EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, PriceFloorConfig, - QuoteGenerator, QuotingMetricsTracker, + EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, QuoteGenerator, + QuotingMetricsTracker, }; use ant_node::storage::{AntProtocol, ChunkStore, ChunkStoreConfig}; use ant_node::ReplicationConfig; @@ -445,7 +445,6 @@ mod tests { cache_capacity: 100, close_group_size: ReplicationConfig::default().close_group_size, local_rewards_address: rewards_address, - price_floor: PriceFloorConfig::default(), }); let metrics_tracker = QuotingMetricsTracker::new(100); let quote_generator = QuoteGenerator::new(rewards_address, metrics_tracker); diff --git a/tests/e2e/harness.rs b/tests/e2e/harness.rs index 786b18c5..efa0531f 100644 --- a/tests/e2e/harness.rs +++ b/tests/e2e/harness.rs @@ -6,12 +6,45 @@ use super::anvil::TestAnvil; use super::testnet::{TestNetwork, TestNetworkConfig, TestNode, TestnetError}; use ant_node::client::XorName; -use evmlib::common::TxHash; +use ant_node::payment::{serialize_single_node_proof, PaymentProof}; +use evmlib::common::{Amount, TxHash}; +use evmlib::{EncodedPeerId, PaymentQuote, ProofOfPayment, RewardsAddress}; use saorsa_core::P2PNode; use std::collections::HashMap; use std::sync::{Arc, Mutex}; +use std::time::SystemTime; use tracing::{info, warn}; +/// A possession or audit failure is reported at weight 5, which cuts a peer's +/// trust by `1 - (1 - 0.124)^5`, about 48%; lighter events cut 12% to 33%. A +/// fall to this fraction of a peer's score or below is read as that penalty. +pub const POSSESSION_PENALTY_TRUST_RATIO: f64 = 0.6; + +/// A single-node payment proof that parses, naming `content`, whose one quote +/// is never valid: no signature, no settlement. For tests that need a node to +/// see a proof the price floor could price, without anyone accepting it. +#[allow(clippy::expect_used)] +pub fn parseable_single_node_proof(content: XorName) -> Vec { + let quote = PaymentQuote { + content: xor_name::XorName(content), + timestamp: SystemTime::now(), + price: Amount::from(1u64), + rewards_address: RewardsAddress::new([0x11; 20]), + committed_key_count: 0, + commitment_pin: None, + pub_key: Vec::new(), + signature: Vec::new(), + }; + serialize_single_node_proof(&PaymentProof { + proof_of_payment: ProofOfPayment { + peer_quotes: vec![(EncodedPeerId::new([0x22; 32]), quote)], + }, + tx_hashes: Vec::new(), + commitment_sidecars: Vec::new(), + }) + .expect("serialize a single-node proof") +} + /// How many port ranges a network is brought up on before its bind failure /// is reported. /// diff --git a/tests/e2e/mod.rs b/tests/e2e/mod.rs index ec2a74c7..cdec4e56 100644 --- a/tests/e2e/mod.rs +++ b/tests/e2e/mod.rs @@ -78,6 +78,9 @@ mod fetch_responsibility_recheck; #[cfg(test)] mod first_audit_ab; +#[cfg(test)] +mod price_floor_ab; + #[cfg(test)] mod fresh_offer_capacity; diff --git a/tests/e2e/pointer_replication.rs b/tests/e2e/pointer_replication.rs index 8548a7c0..586ccf0c 100644 --- a/tests/e2e/pointer_replication.rs +++ b/tests/e2e/pointer_replication.rs @@ -8,6 +8,7 @@ #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] +use super::harness::parseable_single_node_proof; use super::testnet::{TestNetworkConfig, TestNode}; use super::TestHarness; use ant_node::ant_protocol::chunk::{ @@ -32,8 +33,10 @@ use bytes::Bytes; use saorsa_core::identity::PeerId; use saorsa_pqc::api::sig::{ml_dsa_65, MlDsaPublicKey, MlDsaSecretKey}; use serial_test::serial; +use std::collections::HashMap; use std::sync::Arc; use std::time::Duration; +use tokio::sync::RwLock; /// How long to wait for replication to reach a node. const SETTLE: Duration = Duration::from_secs(30); @@ -475,6 +478,98 @@ async fn the_possession_check_penalises_only_a_member_that_dropped_the_record() harness.teardown().await.expect("teardown"); } +/// ADR-0006: a fresh pointer write the offering node cannot price is not +/// used to judge its peers. The checker's price floor is wired (a gossip cache +/// is attached) but it has no commitment of its own, so it cannot price the +/// single-node proof it forwards, and no possession check may follow. The +/// control write that follows forwards a proof the floor never prices, and its +/// check does penalise the members, so the first result is not vacuous. +#[tokio::test] +#[serial] +async fn a_fresh_pointer_write_the_floor_cannot_price_is_not_used_to_judge_peers() { + // Long enough for the scheduled check (at most 500 ms here) and its probes + // to have run, short enough that unrelated trust events are unlikely. + const CONTESTED_WATCH: Duration = Duration::from_secs(5); + let mut config = TestNetworkConfig::minimal(); + config.replication_config = Some(ReplicationConfig { + possession_check_delay_min: Duration::from_millis(200), + possession_check_delay_max: Duration::from_millis(500), + ..ReplicationConfig::default() + }); + let harness = TestHarness::setup_with_config(config).await.expect("setup"); + harness.warmup_dht().await.expect("warmup"); + + let checker = 3; + // The checker only judges peers that understand pointers. + let everyone: Vec = (0..harness.node_count()).collect(); + exchange_hints(&harness, &everyone, &[checker]).await; + + let checker_node = harness.test_node(checker).expect("node"); + checker_node + .ant_protocol + .as_ref() + .expect("protocol") + .payment_verifier_arc() + .attach_commitment_cache(Arc::new(RwLock::new(HashMap::new()))); + let checker_p2p = checker_node.p2p_node.as_ref().expect("p2p"); + let members: Vec = others(&harness, &[checker]) + .into_iter() + .map(|i| peer(harness.test_node(i).expect("node"))) + .collect(); + + let (pk, sk) = owner(); + let contested = signed(&pk, &sk, 0, 1); + let control = signed(&pk, &sk, 1, 2); + let mut outcomes = Vec::new(); + for (record, proof, watch) in [ + ( + &contested, + parseable_single_node_proof(contested.state_id()), + CONTESTED_WATCH, + ), + (&control, DUMMY_PROOF.to_vec(), SETTLE), + ] { + let before: Vec = members.iter().map(|p| checker_p2p.peer_trust(p)).collect(); + store(checker_node) + .put_bytes(&record.to_bytes()) + .await + .expect("put"); + // Nothing is paid anywhere, so every member refuses the offer and is + // absent when, and only if, the checker probes it. + replication(checker_node) + .replicate_fresh(PointerFreshWrite { + record: record.to_bytes(), + payment_proof: proof, + }) + .await; + + // A pointer possession failure is a weight-1 trust event, so any fall + // counts. + let deadline = tokio::time::Instant::now() + watch; + let mut penalised = false; + while tokio::time::Instant::now() < deadline { + penalised = members + .iter() + .zip(before.iter()) + .any(|(p, &b)| checker_p2p.peer_trust(p) < b - f64::EPSILON); + if penalised { + break; + } + tokio::time::sleep(POLL).await; + } + outcomes.push(penalised); + } + + assert_eq!( + outcomes, + vec![false, true], + "a write the floor cannot price must not be used to penalise members, while the \ + control write must" + ); + + harness.teardown().await.expect("teardown"); +} + /// A transfer written to one node reaches the whole group, and after that no /// node takes a second final state — not even one whose target sorts first, /// which the previous merge rule let displace the first everywhere (ADR-0018). diff --git a/tests/e2e/price_floor_ab.rs b/tests/e2e/price_floor_ab.rs new file mode 100644 index 00000000..c8522344 --- /dev/null +++ b/tests/e2e/price_floor_ab.rs @@ -0,0 +1,743 @@ +//! ADR-0006 price-floor A/B workload test, env-gated and never run in CI. +//! +//! Builds a production-wired local testnet whose nodes hold production-sized +//! commitments, then drives two workloads against it with real on-chain +//! payments (Anvil): +//! +//! - **honest** uploads: quotes from the close group, the median settled 3x, +//! the chunk PUT to every close-group storer; +//! - **near-empty quoter** uploads: a node that joined after seeding (no +//! commitment, so it quotes the baseline price) is paid 3x its quote, and the +//! chunk is PUT to it and to the mature storers. +//! +//! After a tail long enough for fresh replication, `PaidNotify` and the +//! (shortened) possession checks to play out, it reports who stored each key +//! and whose paid list holds it. Compare the `PF-METRIC` lines, plus the log +//! counts the runner greps for (possession failures naming a near-empty key, +//! contested fan-outs, floor refusals), across builds. It returns at once +//! unless `PRICE_FLOOR_AB=1` is set, so CI only compiles it. +//! +//! ```bash +//! PRICE_FLOOR_AB=1 RUST_LOG=info \ +//! cargo test --test e2e --features test-utils price_floor_ab_workload -- --nocapture +//! ``` +//! +//! Neighbour commitments are delivered straight into each node's gossip cache +//! after seeding, exactly the signed commitments neighbour sync would carry. +//! At production-sized key sets the in-process testnet's sync rounds do not +//! deliver them within any useful time, and the floor skips every admission +//! until they arrive. Everything after that (quotes, payments, PUTs, fresh +//! replication, `PaidNotify`, possession checks) runs on the live paths. +//! +//! It fails, rather than reporting, when its own preconditions do not hold: +//! fewer than 15 known neighbour commitments on the median node (the floor +//! would skip everything) or fewer uploads completed than requested. It also +//! asserts what the floor promises: every honest upload reaches a storage +//! quorum with no floor refusal, no peer that refused a near-empty write or +//! that the quoter fanned it out to loses penalty-sized trust in the quoter's +//! view, and every near-empty write stays off a quorum, every refusal being +//! the floor's (a mature storer whose own price is low enough may accept it, +//! as the cap allows). +//! +//! Knobs (env): `PF_NODES`, `PF_KEYS_PER_NODE`, `PF_HONEST`, `PF_NEAR_EMPTY`, +//! `PF_TAIL_SECS`. + +#![allow( + clippy::unwrap_used, + clippy::expect_used, + clippy::panic, + clippy::unnested_or_patterns, + clippy::cast_precision_loss, + clippy::too_many_lines +)] + +use ant_node::ant_protocol::{ + ChunkMessage, ChunkMessageBody, ChunkPutRequest, ChunkPutResponse, ChunkQuoteRequest, + ChunkQuoteResponse, CLOSE_GROUP_SIZE, +}; +use ant_node::client::{compute_address, send_and_await_chunk_response}; +use ant_node::payment::quote::CommitmentSource; +use ant_node::payment::{serialize_single_node_proof, PaymentProof, PRICE_FLOOR_TOLERANCE_PERCENT}; +use ant_node::replication::commitment_state::PeerCommitmentRecord; +use ant_node::ReplicationConfig; +use bytes::Bytes; +use evmlib::common::Amount; +use evmlib::wallet::Wallet; +use evmlib::{EncodedPeerId, PaymentQuote, ProofOfPayment}; +use rand::Rng; +use saorsa_core::identity::PeerId; +use saorsa_core::P2PNode; +use serial_test::serial; +use std::collections::{BTreeSet, HashMap}; +use std::sync::Arc; +use std::time::{Duration, Instant}; +use tracing_subscriber::EnvFilter; + +use super::anvil::TestAnvil; +use super::harness::POSSESSION_PENALTY_TRUST_RATIO; +use super::testnet::{TestNetwork, TestNetworkConfig, TestNode}; + +/// The verifier accepts a settlement of at least 3x the median quote price. +const PAYMENT_MULTIPLIER: u64 = 3; + +/// Per-request timeout for quote/PUT round trips. +const REQUEST_TIMEOUT: Duration = Duration::from_secs(20); + +/// Substring of the node's price-floor refusal. +const FLOOR_REFUSAL: &str = "below the close-group price floor"; + +/// Size of each synthetic seeding chunk. +const SEED_CHUNK_BYTES: usize = 64; + +/// Neighbour commitments a node must know before its floor evaluates at all +/// (`PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS`, private to the verifier). +const FLOOR_MIN_NEIGHBOURS: usize = 15; + +/// Stores a chunk upload needs (`CLOSE_GROUP_MAJORITY`). +const STORE_QUORUM: usize = 4; + +fn env_usize(name: &str, default: usize) -> usize { + std::env::var(name) + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(default) +} + +fn xor_distance(a: &[u8; 32], b: &[u8; 32]) -> [u8; 32] { + let mut out = [0u8; 32]; + for (o, (x, y)) in out.iter_mut().zip(a.iter().zip(b.iter())) { + *o = x ^ y; + } + out +} + +/// Indices of `peers` sorted by XOR distance to `address`, closest first. +fn closest_indices(address: &[u8; 32], peers: &[[u8; 32]]) -> Vec { + let mut order: Vec = (0..peers.len()).collect(); + order.sort_by_key(|&i| peers.get(i).map(|p| xor_distance(address, p))); + order +} + +/// Wire one node exactly like production `node.rs`: commitment source into the +/// quote generator, gossip cache into the verifier, monetized-pin sender into +/// the verifier. The e2e testnet does not wire these by default. +fn wire_like_production(node: &TestNode) { + if let (Some(protocol), Some(engine)) = + (node.ant_protocol.as_ref(), node.replication_engine.as_ref()) + { + let concrete = Arc::clone(engine.commitment_state()); + let source: Arc = concrete; + protocol.attach_commitment_source(source); + protocol + .payment_verifier_arc() + .attach_commitment_cache(Arc::clone(engine.last_commitment_by_peer())); + protocol + .payment_verifier_arc() + .attach_monetized_pin_sender(engine.monetized_pin_sender()); + } +} + +fn peer_bytes(node: &TestNode) -> [u8; 32] { + *node + .p2p_node + .as_ref() + .expect("node p2p") + .peer_id() + .as_bytes() +} + +async fn request_quote( + client: &Arc, + peer: &PeerId, + address: [u8; 32], + data_size: u64, +) -> Result { + let request_id: u64 = rand::thread_rng().gen(); + let message = ChunkMessage { + request_id, + body: ChunkMessageBody::QuoteRequest(ChunkQuoteRequest { + address, + data_size, + data_type: 0, + }), + }; + let message_bytes = message + .encode() + .map_err(|e| format!("encode quote request: {e}"))?; + let quote_bytes = send_and_await_chunk_response( + client, + peer, + message_bytes, + request_id, + REQUEST_TIMEOUT, + &[], + |body| match body { + ChunkMessageBody::QuoteResponse(ChunkQuoteResponse::Success { quote, .. }) => { + Some(Ok(quote)) + } + ChunkMessageBody::QuoteResponse(ChunkQuoteResponse::Error(e)) => { + Some(Err(format!("quote error: {e:?}"))) + } + _ => None, + }, + |e| format!("send quote request: {e}"), + || "quote request timed out".to_string(), + ) + .await?; + rmp_serde::from_slice("e_bytes).map_err(|e| format!("decode quote: {e}")) +} + +/// One storer's answer to a paid PUT. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum PutOutcome { + Stored, + FloorRefused, + OtherError, +} + +async fn paid_put( + client: &Arc, + peer: &PeerId, + address: [u8; 32], + data: &[u8], + proof_bytes: Vec, +) -> (PutOutcome, String) { + let request_id: u64 = rand::thread_rng().gen(); + let message = ChunkMessage { + request_id, + body: ChunkMessageBody::PutRequest(ChunkPutRequest::with_payment( + address, + Bytes::copy_from_slice(data), + proof_bytes, + )), + }; + let Ok(message_bytes) = message.encode() else { + return (PutOutcome::OtherError, "encode put".to_string()); + }; + let result = send_and_await_chunk_response( + client, + peer, + message_bytes, + request_id, + REQUEST_TIMEOUT, + &[], + |body| match body { + ChunkMessageBody::PutResponse(ChunkPutResponse::Success { .. }) + | ChunkMessageBody::PutResponse(ChunkPutResponse::AlreadyExists { .. }) => Some(Ok(())), + ChunkMessageBody::PutResponse(ChunkPutResponse::PaymentRequired { message }) => { + Some(Err(format!("payment required: {message}"))) + } + ChunkMessageBody::PutResponse(ChunkPutResponse::Error(e)) => { + Some(Err(format!("put error: {e:?}"))) + } + _ => None, + }, + |e| format!("send put: {e}"), + || "put timed out".to_string(), + ) + .await; + match result { + Ok(()) => (PutOutcome::Stored, String::new()), + Err(e) if e.contains(FLOOR_REFUSAL) => (PutOutcome::FloorRefused, e), + Err(e) => (PutOutcome::OtherError, e), + } +} + +/// Settle `amount` for `quote` on chain and build the single-quote proof. +async fn pay(wallet: &Wallet, issuer: &PeerId, quote: &PaymentQuote, amount: Amount) -> Vec { + let (tx_by_quote, _gas) = wallet + .pay_for_quotes(vec![(quote.hash(), quote.rewards_address, amount)]) + .await + .expect("pay_for_quotes on Anvil"); + let proof = PaymentProof { + proof_of_payment: ProofOfPayment { + peer_quotes: vec![(EncodedPeerId::new(*issuer.as_bytes()), quote.clone())], + }, + tx_hashes: tx_by_quote.values().copied().collect(), + commitment_sidecars: vec![], + }; + serialize_single_node_proof(&proof).expect("serialize proof") +} + +/// How many of `nodes` store `key`, and how many hold it in their paid list. +fn holders_and_votes(nodes: &[TestNode], key: &[u8; 32]) -> (usize, usize) { + let mut holders = 0; + let mut votes = 0; + for node in nodes { + if let Some(protocol) = node.ant_protocol.as_ref() { + if protocol.storage().exists(key).unwrap_or(false) { + holders += 1; + } + } + if let Some(engine) = node.replication_engine.as_ref() { + if engine.paid_list().contains(key).unwrap_or(false) { + votes += 1; + } + } + } + (holders, votes) +} + +fn median(mut xs: Vec) -> usize { + xs.sort_unstable(); + xs.get(xs.len() / 2).copied().unwrap_or(0) +} + +async fn sync_rounds(network: &TestNetwork, rounds: usize) { + for _ in 0..rounds { + for node in network.nodes() { + if let Some(engine) = node.replication_engine.as_ref() { + engine.trigger_neighbor_sync(); + } + } + tokio::time::sleep(Duration::from_secs(4)).await; + } +} + +async fn cache_sizes(network: &TestNetwork) -> Vec { + let mut sizes = Vec::new(); + for node in network.nodes() { + if let Some(engine) = node.replication_engine.as_ref() { + sizes.push(engine.last_commitment_by_peer().read().await.len()); + } + } + sizes +} + +/// Put every node's current signed commitment into every other node's gossip +/// cache, as neighbour sync would. +async fn deliver_commitments(network: &TestNetwork) { + let commitments: Vec<(PeerId, _)> = network + .nodes() + .iter() + .filter_map(|node| { + let engine = node.replication_engine.as_ref()?; + let current = engine.commitment_state().current()?; + Some(( + PeerId::from_bytes(peer_bytes(node)), + current.commitment().clone(), + )) + }) + .collect(); + let now = Instant::now(); + for node in network.nodes() { + let own = PeerId::from_bytes(peer_bytes(node)); + if let Some(engine) = node.replication_engine.as_ref() { + let mut cache = engine.last_commitment_by_peer().write().await; + for (peer, commitment) in &commitments { + if *peer != own { + cache.insert( + *peer, + PeerCommitmentRecord::from_verified(commitment.clone(), now), + ); + } + } + } + } +} + +#[tokio::test(flavor = "multi_thread")] +#[serial] +async fn price_floor_ab_workload() { + if std::env::var("PRICE_FLOOR_AB").ok().as_deref() != Some("1") { + eprintln!("price_floor_ab_workload skipped: set PRICE_FLOOR_AB=1 to run"); + return; + } + let _ = tracing_subscriber::fmt() + .with_env_filter( + EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")), + ) + .with_writer(std::io::stderr) + .try_init(); + + let nodes = env_usize("PF_NODES", 24); + let keys_per_node = env_usize("PF_KEYS_PER_NODE", 4_500); + let honest_uploads = env_usize("PF_HONEST", 12); + let near_empty_uploads = env_usize("PF_NEAR_EMPTY", 8); + + let tail_secs = env_usize("PF_TAIL_SECS", 90); + eprintln!( + "PF-CONFIG nodes={nodes} keys_per_node={keys_per_node} honest={honest_uploads} \ + near_empty={near_empty_uploads} tail_secs={tail_secs}" + ); + + let anvil = TestAnvil::new().await.expect("start anvil"); + let wallet = anvil.create_funded_wallet().expect("funded wallet"); + let replication_config = ReplicationConfig { + neighbor_sync_interval_min: Duration::from_secs(5), + neighbor_sync_interval_max: Duration::from_secs(10), + possession_check_delay_min: Duration::from_secs(20), + possession_check_delay_max: Duration::from_secs(30), + subtree_round1_responder_cooldown: Duration::from_millis(1), + ..ReplicationConfig::default() + }; + let mut config = TestNetworkConfig { + node_count: nodes, + payment_enforcement: true, + replication_config: Some(replication_config), + ..TestNetworkConfig::default() + }; + config.evm_network = Some(anvil.to_network()); + let mut network = TestNetwork::new(config).await.expect("build network"); + network.start().await.expect("start network"); + network.warmup_dht().await.expect("warmup dht"); + for node in network.nodes() { + wire_like_production(node); + } + + let client_index = nodes.saturating_sub(1); + let client = network + .nodes() + .get(client_index) + .and_then(|n| n.p2p_node.as_ref()) + .map(Arc::clone) + .expect("client node p2p"); + + // ---- Seed: write synthetic chunks straight into the stores of the seven + // XOR-closest nodes (each node's own close group in a full routing table), + // then rotate commitments, so every node commits to a production-sized key + // set without thousands of on-chain payments. + let peers: Vec<[u8; 32]> = network.nodes().iter().map(peer_bytes).collect(); + let seed_chunks = keys_per_node * nodes / CLOSE_GROUP_SIZE; + let mut rng = rand::thread_rng(); + let mut assignments: Vec)>> = vec![Vec::new(); nodes]; + for i in 0..seed_chunks { + let mut content = vec![0u8; SEED_CHUNK_BYTES]; + rng.fill(&mut content[..]); + if let Some(prefix) = content.get_mut(..8) { + prefix.copy_from_slice(&(i as u64).to_le_bytes()); + } + let address = compute_address(&content); + for index in closest_indices(&address, &peers) + .into_iter() + .take(CLOSE_GROUP_SIZE) + { + if let Some(slot) = assignments.get_mut(index) { + slot.push((address, content.clone())); + } + } + } + let mut writers = Vec::new(); + for (node, chunks) in network.nodes().iter().zip(assignments) { + if let Some(protocol) = node.ant_protocol.as_ref() { + let storage = protocol.storage(); + writers.push(tokio::spawn(async move { + for (address, content) in chunks { + storage.put(&address, &content).await.expect("seed put"); + } + })); + } + } + for writer in writers { + writer.await.expect("seed writer"); + } + for node in network.nodes() { + if let Some(engine) = node.replication_engine.as_ref() { + engine.rebuild_commitment_now().await.expect("rebuild"); + } + } + let counts: Vec = network + .nodes() + .iter() + .filter_map(|n| n.replication_engine.as_ref()) + .filter_map(|e| e.commitment_state().current()) + .map(|c| c.commitment().key_count as usize) + .collect(); + eprintln!( + "PF-METRIC phase=seed chunks={seed_chunks} committed_nodes={} key_count_min={} \ + key_count_median={} key_count_max={}", + counts.len(), + counts.iter().min().copied().unwrap_or(0), + median(counts.clone()), + counts.iter().max().copied().unwrap_or(0) + ); + + deliver_commitments(&network).await; + let sizes = cache_sizes(&network).await; + eprintln!( + "PF-METRIC phase=gossip mode=delivered cache_min={} cache_median={} cache_max={}", + sizes.iter().min().copied().unwrap_or(0), + median(sizes.clone()), + sizes.iter().max().copied().unwrap_or(0) + ); + assert!( + median(sizes.clone()) >= FLOOR_MIN_NEIGHBOURS, + "precondition unmet: the median node knows fewer than {FLOOR_MIN_NEIGHBOURS} neighbour \ + commitments, so the floor would skip every admission" + ); + + // ---- Honest uploads. + eprintln!("PF-METRIC phase=build tolerance_percent={PRICE_FLOOR_TOLERANCE_PERCENT}"); + let mut honest_keys = Vec::new(); + let (mut honest_stored, mut honest_floor, mut honest_other) = (0usize, 0usize, 0usize); + for i in 0..honest_uploads { + let mut data = vec![0u8; 4096]; + rng.fill(&mut data[..]); + let address = compute_address(&data); + let storers: Vec = closest_indices(&address, &peers) + .into_iter() + .filter(|&j| j != client_index) + .take(CLOSE_GROUP_SIZE) + .filter_map(|j| peers.get(j).map(|p| PeerId::from_bytes(*p))) + .collect(); + let mut quotes = Vec::new(); + for peer in &storers { + if let Ok(q) = request_quote(&client, peer, address, data.len() as u64).await { + quotes.push((*peer, q)); + } + } + quotes.sort_by_key(|(_, q)| q.price); + let Some((issuer, median_quote)) = quotes.get(quotes.len() / 2).cloned() else { + eprintln!("PF-WARN honest {i}: no quotes"); + continue; + }; + let amount = median_quote.price * Amount::from(PAYMENT_MULTIPLIER); + let proof = pay(&wallet, &issuer, &median_quote, amount).await; + let (mut stored, mut floor, mut other) = (0usize, 0usize, 0usize); + for peer in &storers { + match paid_put(&client, peer, address, &data, proof.clone()).await { + (PutOutcome::Stored, _) => stored += 1, + (PutOutcome::FloorRefused, _) => floor += 1, + (PutOutcome::OtherError, e) => { + other += 1; + eprintln!("PF-WARN honest {i} put to {peer}: {e}"); + } + } + } + eprintln!( + "PF-METRIC phase=honest upload={i} key={} quotes={} paid_price={} stored={stored} \ + floor_refused={floor} other_error={other}", + hex::encode(address), + quotes.len(), + median_quote.price + ); + assert!( + stored >= STORE_QUORUM && floor == 0, + "honest upload {i} must reach a storage quorum with no floor refusal" + ); + honest_stored += stored; + honest_floor += floor; + honest_other += other; + honest_keys.push(address); + } + + // ---- A node joins after seeding: no data, no commitment, baseline quotes. + let before = network.node_count(); + let fresh_index = match network.add_node().await { + Ok(index) => index, + // The node is started and registered before the harness waits on its + // replication bootstrap, which can outlast the harness's fixed timeout + // on a seeded network. The node is still usable: it holds no + // commitment either way, which is all this phase needs. + Err(e) if network.node_count() > before => { + eprintln!("PF-WARN fresh node bootstrap: {e}; continuing with it"); + before + } + Err(e) => panic!("add fresh node: {e}"), + }; + if let Some(node) = network.nodes().get(fresh_index) { + wire_like_production(node); + } + let fresh_peer_bytes = network + .nodes() + .get(fresh_index) + .map(peer_bytes) + .expect("fresh node"); + let fresh_peer = PeerId::from_bytes(fresh_peer_bytes); + sync_rounds(&network, 3).await; + let mut all_peers = peers.clone(); + all_peers.push(fresh_peer_bytes); + let fresh_p2p = network + .nodes() + .get(fresh_index) + .and_then(|n| n.p2p_node.as_ref()) + .map(Arc::clone) + .expect("fresh node p2p"); + let trust_before: HashMap<[u8; 32], f64> = peers + .iter() + .map(|p| (*p, fresh_p2p.peer_trust(&PeerId::from_bytes(*p)))) + .collect(); + let mut refusers: BTreeSet<[u8; 32]> = BTreeSet::new(); + let mut fan_out_targets: BTreeSet<[u8; 32]> = BTreeSet::new(); + let fan_out_group_size = ReplicationConfig::default().close_group_size; + + // ---- Near-empty quoter uploads: pay the fresh node's baseline quote 3x. + let mut near_empty_keys = Vec::new(); + let (mut ne_mature_stored, mut ne_mature_floor, mut ne_mature_other, mut ne_fresh_stored) = + (0usize, 0usize, 0usize, 0usize); + let mut attempts = 0usize; + while near_empty_keys.len() < near_empty_uploads && attempts < near_empty_uploads * 20 { + attempts += 1; + let mut data = vec![0u8; 4096]; + rng.fill(&mut data[..]); + let address = compute_address(&data); + let order = closest_indices(&address, &all_peers); + // Only addresses where the fresh node is among the eight closest, so + // every receiver counts it K-close and the floor is what decides. + let Some(rank) = order.iter().position(|&j| j == fresh_index) else { + continue; + }; + if rank >= 8 { + continue; + } + let quote = match request_quote(&client, &fresh_peer, address, data.len() as u64).await { + Ok(q) => q, + Err(e) => { + eprintln!("PF-WARN near-empty quote: {e}"); + continue; + } + }; + let amount = quote.price * Amount::from(PAYMENT_MULTIPLIER); + let proof = pay(&wallet, &fresh_peer, "e, amount).await; + let mature: Vec = order + .iter() + .copied() + .filter(|&j| j != client_index && j != fresh_index) + .take(CLOSE_GROUP_SIZE) + .filter_map(|j| all_peers.get(j).map(|p| PeerId::from_bytes(*p))) + .collect(); + // The quoter first, as a client paying it would: its store is what + // triggers the fan-out whose possession check is under test. + let (fresh_outcome, fresh_err) = + paid_put(&client, &fresh_peer, address, &data, proof.clone()).await; + // The fan-out goes to the quoter's own view of the close group, which + // can differ from the order above and can include the client node. + fan_out_targets.extend( + fresh_p2p + .dht_manager() + .find_closest_nodes_local_with_self(&address, fan_out_group_size) + .await + .iter() + .filter(|n| n.peer_id != fresh_peer) + .map(|n| *n.peer_id.as_bytes()), + ); + let (mut stored, mut floor, mut other) = (0usize, 0usize, 0usize); + for peer in &mature { + match paid_put(&client, peer, address, &data, proof.clone()).await { + (PutOutcome::Stored, _) => stored += 1, + (PutOutcome::FloorRefused, _) => { + floor += 1; + refusers.insert(*peer.as_bytes()); + } + (PutOutcome::OtherError, e) => { + other += 1; + eprintln!("PF-WARN near-empty put to {peer}: {e}"); + } + } + } + assert_eq!( + fresh_outcome, + PutOutcome::Stored, + "the near-empty quoter must store its own write: {fresh_err}" + ); + // A mature storer whose own price is at most the payment over + // 3 x the tolerance accepts by design (the cap), so a few may store it + // in a low-count region. What the floor promises is that the write + // does not reach a storage quorum, and that every refusal is the + // floor's. The quoter's own store counts toward the quorum too. + assert!( + other == 0 && floor + stored == mature.len() && stored + 1 < STORE_QUORUM, + "the floor must keep a near-empty payment off a storage quorum, refusing it \ + on the floor (stored={stored} floor={floor} other={other})" + ); + eprintln!( + "PF-METRIC phase=near_empty upload={} key={} fresh_rank={rank} paid_price={} \ + mature_stored={stored} mature_floor_refused={floor} mature_other_error={other} \ + fresh_node={fresh_outcome:?}", + near_empty_keys.len(), + hex::encode(address), + quote.price + ); + ne_mature_stored += stored; + ne_mature_floor += floor; + ne_mature_other += other; + ne_fresh_stored += usize::from(fresh_outcome == PutOutcome::Stored); + near_empty_keys.push(address); + } + + assert_eq!( + honest_keys.len(), + honest_uploads, + "every requested honest upload must complete" + ); + assert_eq!( + near_empty_keys.len(), + near_empty_uploads, + "precondition unmet: not every requested near-empty upload was attempted" + ); + + // ---- Tail: fresh offers, PaidNotify and possession checks play out. + eprintln!("PF-METRIC phase=tail sleeping_secs={tail_secs}"); + tokio::time::sleep(Duration::from_secs(tail_secs as u64)).await; + + let mut honest_holders = Vec::new(); + let mut honest_votes = Vec::new(); + for key in &honest_keys { + let (h, v) = holders_and_votes(network.nodes(), key); + eprintln!( + "PF-METRIC phase=after kind=honest key={} holders={h} paid_votes={v}", + hex::encode(key) + ); + honest_holders.push(h); + honest_votes.push(v); + } + let mut ne_holders = Vec::new(); + let mut ne_votes = Vec::new(); + for key in &near_empty_keys { + let (h, v) = holders_and_votes(network.nodes(), key); + assert!( + h < STORE_QUORUM, + "the floor must keep a near-empty write off a storage quorum" + ); + eprintln!( + "PF-METRIC phase=after kind=near_empty key={} holders={h} paid_votes={v}", + hex::encode(key) + ); + ne_holders.push(h); + ne_votes.push(v); + } + + // The node that stored the near-empty writes fanned them out; it must not + // have penalised any peer that refused them, nor any peer it fanned them + // out to. + let mut min_ratio = f64::INFINITY; + let mut penalty_sized = 0usize; + let judged: Vec<&[u8; 32]> = refusers.union(&fan_out_targets).collect(); + for &peer in &judged { + let before = trust_before.get(peer).copied().unwrap_or(0.5); + let after = fresh_p2p.peer_trust(&PeerId::from_bytes(*peer)); + let ratio = if before > 0.0 { after / before } else { 1.0 }; + min_ratio = min_ratio.min(ratio); + if ratio <= POSSESSION_PENALTY_TRUST_RATIO { + penalty_sized += 1; + } + } + eprintln!( + "PF-METRIC phase=trust refusers={} fan_out_targets={} judged={} \ + min_trust_ratio={min_ratio:.3} penalty_sized_drops={penalty_sized}", + refusers.len(), + fan_out_targets.len(), + judged.len() + ); + assert_eq!( + penalty_sized, 0, + "no refuser or fan-out target may be penalised for a write under the price floor" + ); + + eprintln!( + "PF-SUMMARY honest_uploads={} honest_stored={honest_stored} honest_floor_refused={honest_floor} \ + honest_other_error={honest_other} honest_holders_median={} honest_paid_votes_median={} \ + near_empty_uploads={} near_empty_mature_stored={ne_mature_stored} \ + near_empty_mature_floor_refused={ne_mature_floor} near_empty_mature_other_error={ne_mature_other} \ + near_empty_fresh_stored={ne_fresh_stored} near_empty_holders_median={} \ + near_empty_paid_votes_median={}", + honest_keys.len(), + median(honest_holders), + median(honest_votes), + near_empty_keys.len(), + median(ne_holders), + median(ne_votes), + ); + + network.shutdown().await.expect("shutdown network"); +} diff --git a/tests/e2e/replication.rs b/tests/e2e/replication.rs index 6ae3fb5b..9cb1b01f 100644 --- a/tests/e2e/replication.rs +++ b/tests/e2e/replication.rs @@ -5,6 +5,7 @@ #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] +use super::harness::{parseable_single_node_proof, POSSESSION_PENALTY_TRUST_RATIO}; use super::testnet::{TestNetworkConfig, DEFAULT_CHUNK_OPERATION_TIMEOUT_SECS}; use super::TestHarness; use ant_node::ant_protocol::{ChunkMessage, ChunkMessageBody, ChunkPutRequest, ChunkPutResponse}; @@ -16,10 +17,11 @@ use ant_node::replication::config::{ REPLICATION_PROTOCOL_ID, }; use ant_node::replication::fresh::FreshWriteEvent; +use ant_node::replication::neighbor_sync; use ant_node::replication::protocol::{ compute_audit_digest, AuditChallenge, AuditResponse, FetchRequest, FetchResponse, - FreshReplicationOffer, FreshReplicationResponse, NeighborSyncRequest, ReplicationMessage, - ReplicationMessageBody, VerificationRequest, ABSENT_KEY_DIGEST, + FreshReplicationOffer, FreshReplicationResponse, NeighborSyncRequest, PaidNotify, + ReplicationMessage, ReplicationMessageBody, VerificationRequest, ABSENT_KEY_DIGEST, }; use ant_node::replication::pruning; use ant_node::replication::scheduling::ReplicationQueues; @@ -30,7 +32,7 @@ use bytes::Bytes; use saorsa_core::identity::PeerId; use saorsa_core::{P2PNode, TrustEvent}; use serial_test::serial; -use std::collections::HashSet; +use std::collections::{HashMap, HashSet}; use std::fs; use std::sync::Arc; use std::time::Duration; @@ -551,6 +553,330 @@ async fn fresh_write_pipeline_never_offers_a_corrupt_chunk() { harness.teardown().await.expect("teardown"); } +/// ADR-0006: a fresh write this node cannot price is not used to judge its +/// peers. Node A's price floor is wired (a gossip cache is attached) but A has +/// no commitment of its own, so it cannot price the single-node proof it +/// forwards. +/// +/// - When A's verification of the payment skipped the floor too (stood in for +/// here, as these tests settle nothing on chain), the scheduled possession +/// check must not run, and the key is marked in A's paid list, so A hints it +/// to its peers as paid only, never as a replica they could later be +/// audited on. +/// - When A's verification was not contested but the proof A forwards is, the +/// check must not run either, and the key is recorded plainly. +/// - When a marked record of the key won before the write recorded its own +/// plain decision, as a concurrent `PaidNotify` or fresh offer can, the +/// winning mark keeps the check from running even with a proof the floor +/// never prices. +/// - The control write that follows forwards a proof the floor never prices, +/// on a key with no mark: its check does penalise the same kind of absent +/// peer and its key is hinted as a replica, so the other results are not +/// vacuous. +#[tokio::test] +#[serial] +async fn possession_check_skips_a_write_the_floor_cannot_price() { + let mut net_config = TestNetworkConfig::small(); + net_config.replication_config = Some(ReplicationConfig { + possession_check_delay_min: Duration::from_millis(200), + possession_check_delay_max: Duration::from_millis(500), + ..ReplicationConfig::default() + }); + let harness = TestHarness::setup_with_config(net_config) + .await + .expect("setup"); + harness.warmup_dht().await.expect("warmup"); + + let a_idx = 3; + let a = harness.test_node(a_idx).expect("node a"); + let p2p_a = a.p2p_node.as_ref().expect("p2p a"); + let engine_a = a.replication_engine.as_ref().expect("engine a"); + let protocol_a = a.ant_protocol.as_ref().expect("proto a"); + protocol_a + .payment_verifier_arc() + .attach_commitment_cache(Arc::new(RwLock::new(HashMap::new()))); + let self_a = *p2p_a.peer_id(); + let close_group_size = ReplicationConfig::default().close_group_size; + + let paid_list_a = engine_a.paid_list(); + let mut outcomes = Vec::new(); + for (content, parseable, admitted_contested, recorded_contested) in [ + ( + &b"adr-0006 contested fan-out payload"[..], + true, + true, + false, + ), + ( + &b"adr-0006 contested proof fan-out payload"[..], + true, + false, + false, + ), + ( + &b"adr-0006 marked record fan-out payload"[..], + false, + false, + true, + ), + ( + &b"adr-0006 control fan-out payload"[..], + false, + false, + false, + ), + ] { + let address = compute_address(content); + let proof = if parseable { + parseable_single_node_proof(address) + } else { + dummy_payment_proof() + }; + if admitted_contested { + protocol_a + .payment_verifier_arc() + .note_contested_admission_for_tests(&address); + } + if recorded_contested { + paid_list_a + .insert_price_contested(&address) + .await + .expect("record the winning mark"); + } + let close_group: Vec = p2p_a + .dht_manager() + .find_closest_nodes_local_with_self(&address, close_group_size) + .await + .iter() + .filter(|n| n.peer_id != self_a) + .map(|n| n.peer_id) + .collect(); + let close_peer = *close_group + .first() + .expect("expected a non-empty close group"); + let trust_before: Vec = close_group.iter().map(|p| p2p_a.peer_trust(p)).collect(); + + // A client PUTs the chunk to A, through A's handler and fresh-write + // pipeline. Only A holds the payment in its cache, so the close group + // refuses A's offers and is absent when, and only if, A probes it. + harness.prepopulate_payment_cache_for_peer(&self_a, &address); + put_cached_chunk(&harness, a_idx, content, proof).await; + + let recorded_by = tokio::time::Instant::now() + PROPAGATION_TIMEOUT; + while !paid_list_a.contains(&address).unwrap_or(false) + && tokio::time::Instant::now() < recorded_by + { + tokio::time::sleep(PROPAGATION_POLL_INTERVAL).await; + } + let marked = paid_list_a + .is_price_contested(&address) + .expect("read the mark"); + let replica_hinted = neighbor_sync::build_replica_hints_for_peer( + &close_peer, + &protocol_a.storage(), + paid_list_a, + p2p_a, + close_group_size, + ) + .await + .contains(&address); + let paid_hinted = neighbor_sync::build_paid_hints_for_peer( + &close_peer, + paid_list_a, + p2p_a, + ReplicationConfig::default().paid_list_close_group_size, + ) + .await + .contains(&address); + + let deadline = tokio::time::Instant::now() + PROPAGATION_TIMEOUT; + let mut penalised = false; + while tokio::time::Instant::now() < deadline { + penalised = close_group + .iter() + .zip(trust_before.iter()) + .any(|(peer, &before)| { + p2p_a.peer_trust(peer) <= before * POSSESSION_PENALTY_TRUST_RATIO + }); + if penalised { + break; + } + tokio::time::sleep(PROPAGATION_POLL_INTERVAL).await; + } + outcomes.push((penalised, marked, replica_hinted, paid_hinted)); + } + + assert_eq!( + outcomes, + vec![ + (false, true, false, true), + (false, false, true, true), + (false, true, false, true), + (true, false, true, true), + ], + "a write the floor cannot price, or whose record is marked, must not be used \ + to judge peers (no penalty; a marked key is hinted as paid only), while the \ + control write must" + ); + + harness.teardown().await.expect("teardown"); +} + +/// How a write reaches the node that records its paid key. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum RecordPath { + /// A client PUT, recorded by the storer's fresh-write pipeline. + Put, + /// A fresh offer on its own, with no `PaidNotify` before it. + FreshOffer, + /// A `PaidNotify` on its own. + PaidNotify, +} + +/// ADR-0006: every path that records a paid key marks it exactly when this +/// node's own verification of the payment was contested, and never from the +/// proof that came with the message, which a cache hit leaves unchecked. +/// +/// Each node's floor is wired but cannot price (it has no commitment of its +/// own), so every proof below would be contested if it were priced. Payments +/// are admitted from the payment cache, and the verified contested admission +/// is stood in for on the keys that have one. Each path records one key with +/// that admission, which must be marked, and one without, which must not be. +#[tokio::test] +#[serial] +async fn every_record_path_marks_a_key_from_its_own_verified_decision() { + let harness = TestHarness::setup_minimal().await.expect("setup"); + harness.warmup_dht().await.expect("warmup"); + for i in 0..harness.node_count() { + if let Some(protocol) = harness.test_node(i).and_then(|n| n.ant_protocol.as_ref()) { + protocol + .payment_verifier_arc() + .attach_commitment_cache(Arc::new(RwLock::new(HashMap::new()))); + } + } + + let (source_idx, receiver_idx) = (3, 1); + let source_p2p = harness + .test_node(source_idx) + .and_then(|n| n.p2p_node.as_ref()) + .expect("source p2p"); + let receiver_peer = *harness + .test_node(receiver_idx) + .and_then(|n| n.p2p_node.as_ref()) + .expect("receiver p2p") + .peer_id(); + + let mut outcomes = Vec::new(); + for (path, admitted_contested) in [ + (RecordPath::Put, true), + (RecordPath::Put, false), + (RecordPath::FreshOffer, true), + (RecordPath::FreshOffer, false), + (RecordPath::PaidNotify, true), + (RecordPath::PaidNotify, false), + ] { + let content = format!("adr-0006 record path {path:?} {admitted_contested}").into_bytes(); + let address = compute_address(&content); + let forged_proof = parseable_single_node_proof(address); + harness.prepopulate_payment_cache_everywhere(&address); + let recorder_idx = if path == RecordPath::Put { + source_idx + } else { + receiver_idx + }; + let recorder = harness.test_node(recorder_idx).expect("recorder"); + if admitted_contested { + recorder + .ant_protocol + .as_ref() + .expect("recorder protocol") + .payment_verifier_arc() + .note_contested_admission_for_tests(&address); + } + + match path { + RecordPath::Put => { + put_cached_chunk(&harness, source_idx, &content, forged_proof).await; + } + RecordPath::FreshOffer => { + let response = send_replication_request( + source_p2p, + &receiver_peer, + ReplicationMessage { + request_id: rand::random(), + body: ReplicationMessageBody::FreshReplicationOffer( + FreshReplicationOffer { + key: address, + data: content.clone(), + proof_of_payment: forged_proof, + }, + ), + }, + PROPAGATION_TIMEOUT, + ) + .await; + assert!( + matches!( + response.body, + ReplicationMessageBody::FreshReplicationResponse( + FreshReplicationResponse::Accepted { .. } + ) + ), + "the receiver must accept a cached fresh offer: {:?}", + response.body + ); + } + RecordPath::PaidNotify => { + let notify = ReplicationMessage { + request_id: rand::random(), + body: ReplicationMessageBody::PaidNotify(PaidNotify { + key: address, + proof_of_payment: forged_proof, + }), + }; + source_p2p + .send_message( + &receiver_peer, + REPLICATION_PROTOCOL_ID, + Bytes::from(notify.encode().expect("encode PaidNotify")), + &[], + ) + .await + .expect("send PaidNotify"); + } + } + + let paid_list = recorder + .replication_engine + .as_ref() + .expect("recorder engine") + .paid_list(); + let deadline = tokio::time::Instant::now() + PROPAGATION_TIMEOUT; + while !paid_list.contains(&address).unwrap_or(false) + && tokio::time::Instant::now() < deadline + { + tokio::time::sleep(PROPAGATION_POLL_INTERVAL).await; + } + assert!( + paid_list.contains(&address).unwrap_or(false), + "{path:?} never recorded the key" + ); + let marked = paid_list + .is_price_contested(&address) + .expect("read the mark"); + outcomes.push((path, admitted_contested, marked)); + } + + for (path, admitted_contested, marked) in outcomes { + assert_eq!( + marked, admitted_contested, + "{path:?}: the mark must follow this node's verified decision, never the attached proof" + ); + } + + harness.teardown().await.expect("teardown"); +} + /// Proof bytes for tests that only need to reach the pre-payment gates. fn dummy_payment_proof() -> Vec { vec![DUMMY_PAYMENT_PROOF_BYTE; DUMMY_PAYMENT_PROOF_LEN] @@ -578,14 +904,26 @@ async fn store_paid_chunk(harness: &TestHarness, source_idx: usize, content: &[u /// handler stores it and emits the fresh-write event itself, carrying the /// dummy proof it was paid with. async fn put_paid_chunk(harness: &TestHarness, source_idx: usize, content: &[u8]) -> XorName { + harness.prepopulate_payment_cache_everywhere(&compute_address(content)); + put_cached_chunk(harness, source_idx, content, dummy_payment_proof()).await +} + +/// PUT a chunk through the source node's handler, attaching `proof`. The +/// caller has put the payment in the source's cache, so the proof itself is +/// never checked. +async fn put_cached_chunk( + harness: &TestHarness, + source_idx: usize, + content: &[u8], + proof: Vec, +) -> XorName { let address = compute_address(content); - harness.prepopulate_payment_cache_everywhere(&address); let request = ChunkMessage { request_id: rand::random(), body: ChunkMessageBody::PutRequest(ChunkPutRequest::with_payment( address, Bytes::copy_from_slice(content), - dummy_payment_proof(), + proof, )), }; let protocol = harness @@ -1516,6 +1854,91 @@ async fn test_prune_pass_requires_remote_confirmation_before_delete() { harness.teardown().await.expect("teardown"); } +/// ADR-0006: a peer missing a key whose payment the price floor contested on +/// the pruner may have refused it under its own floor, so the prune audit +/// keeps the record (its proofs are still missing) but scores no one. The +/// control pass, the same shape with an unmarked key, does penalise the +/// absent peers, so the first result is not vacuous. +#[tokio::test] +#[serial] +async fn prune_audit_scores_no_peer_for_a_price_contested_key() { + let harness = TestHarness::setup_minimal().await.expect("setup"); + harness.warmup_dht().await.expect("warmup"); + + let pruner_idx = 3; + let close_group_size = 2; + let config = prune_test_config(close_group_size); + let sync_state = Arc::new(RwLock::new(NeighborSyncState::new_cycle(vec![]))); + let repair_proofs = Arc::new(RwLock::new(RepairProofs::new())); + let audit_challenge_coordinator = Arc::new(AuditChallengeCoordinator::new()); + + let pruner = harness.test_node(pruner_idx).expect("pruner"); + let pruner_p2p = Arc::clone(pruner.p2p_node.as_ref().expect("pruner p2p")); + let pruner_storage = pruner.ant_protocol.as_ref().expect("protocol").storage(); + let pruner_paid_list = Arc::clone( + pruner + .replication_engine + .as_ref() + .expect("pruner replication engine") + .paid_list(), + ); + let pruner_peer = *pruner_p2p.peer_id(); + + let mut outcomes = Vec::new(); + for (label, contested) in [("price-contested", true), ("unmarked", false)] { + let (content, address, targets) = + find_remote_prune_candidate(&harness, pruner_idx, close_group_size, label).await; + pruner_storage + .put(&address, &content) + .await + .expect("put record on pruner"); + if contested { + pruner_paid_list + .insert_price_contested(&address) + .await + .expect("mark"); + } + let trust_before: Vec = targets.iter().map(|p| pruner_p2p.peer_trust(p)).collect(); + + // No target holds the record: every one is absent when audited. + let result = pruning::run_prune_pass_with_context(pruning::PrunePassContext { + self_id: &pruner_peer, + storage: &pruner_storage, + paid_list: &pruner_paid_list, + p2p_node: &pruner_p2p, + config: &config, + sync_state: &sync_state, + repair_proofs: &repair_proofs, + allow_remote_prune_audits: true, + commitment_state: None, + audit_challenge_coordinator: &audit_challenge_coordinator, + }) + .await; + assert!( + pruner_storage.exists(&address).expect("exists"), + "a record no target proves must stay local" + ); + assert_eq!(result.records_pruned, 0); + + let penalised = targets + .iter() + .zip(trust_before.iter()) + .any(|(peer, &before)| { + pruner_p2p.peer_trust(peer) <= before * POSSESSION_PENALTY_TRUST_RATIO + }); + outcomes.push(penalised); + } + + assert_eq!( + outcomes, + vec![false, true], + "an absent peer must not be scored for a price-contested key, and must be for an \ + unmarked one" + ); + + harness.teardown().await.expect("teardown"); +} + /// Pruner-retention veto (ADR-0002): a chunk the pruner is no longer responsible /// for, but which is still committed under a recently-gossiped commitment, must /// NOT be deleted — the storage-commitment audit's round-2 byte challenge could diff --git a/tests/e2e/testnet.rs b/tests/e2e/testnet.rs index ca33a901..ec0d11a3 100644 --- a/tests/e2e/testnet.rs +++ b/tests/e2e/testnet.rs @@ -1174,7 +1174,6 @@ impl TestNetwork { cache_capacity: TEST_PAYMENT_CACHE_CAPACITY, close_group_size: replication_config.close_group_size, local_rewards_address: rewards_address, - price_floor: ant_node::payment::PriceFloorConfig::default(), }; let payment_verifier = PaymentVerifier::new(payment_config); diff --git a/tests/poc_price_floor_live.rs b/tests/poc_price_floor_live.rs index 3ba4d877..545a4be4 100644 --- a/tests/poc_price_floor_live.rs +++ b/tests/poc_price_floor_live.rs @@ -6,74 +6,84 @@ //! //! - a **real deployed payment vault** on a local Anvil chain, paid with a //! **real on-chain `pay_for_quotes` transaction**, read back through the -//! production `completedPayments` contract call (NOT the test override); and -//! - the **real `ResponderCommitmentState`** as the floor's commitment source, -//! so the local price is derived through the same production -//! `current_binding_snapshot` → `calculate_price(committed_key_count)` path -//! this node's own `QuoteGenerator` prices from. +//! production `completedPayments` contract call (NOT the test override); +//! - the **real `ResponderCommitmentState`** as this node's own commitment +//! source, read through the same production `current_binding_snapshot` → +//! `calculate_price(committed_key_count)` path this node's own +//! `QuoteGenerator` prices from; and +//! - a gossip cache of **real signed neighbour commitments**, the close-group +//! view the floor prices against. //! //! What each test asserts maps directly to a PR claim: -//! - shadow mode never rejects, even when the settlement is below floor; -//! - enforcement rejects the cheapest-of-K one-quote settlement; -//! - enforcement accepts an honest at-floor overpayment; +//! - the floor refuses a real 3x-baseline settlement; +//! - an honest at-floor overpayment is admitted, and one atto less is not; //! - reading the floor does not mutate commitment answerability. //! //! Run: `cargo test --test poc_price_floor_live --features test-utils`. #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] +use std::collections::HashMap; use std::sync::Arc; -use std::time::SystemTime; +use std::time::{Instant, SystemTime}; +use ant_node::payment::quote::CommitmentSource; use ant_node::payment::{ calculate_price, serialize_single_node_proof, EvmVerifierConfig, PaymentProof, PaymentStatus, - PaymentVerifier, PaymentVerifierConfig, PriceFloorConfig, VerificationContext, + PaymentVerifier, PaymentVerifierConfig, VerificationContext, PRICE_FLOOR_TOLERANCE_PERCENT, +}; +use ant_node::replication::commitment_state::{ + BuiltCommitment, PeerCommitmentRecord, ResponderCommitmentState, }; -use ant_node::replication::commitment_state::{BuiltCommitment, ResponderCommitmentState}; use ant_node::CLOSE_GROUP_SIZE; use evmlib::common::Amount; use evmlib::testnet::Testnet; use evmlib::wallet::Wallet; -use evmlib::{EncodedPeerId, PaymentQuote, ProofOfPayment, RewardsAddress}; +use evmlib::{EncodedPeerId, Network, PaymentQuote, ProofOfPayment, RewardsAddress}; +use saorsa_core::identity::PeerId; use saorsa_pqc::api::sig::{ml_dsa_65, MlDsaPublicKey, MlDsaSecretKey}; +use tokio::sync::RwLock; -/// Committed key count for the receiver in the floor tests. Chosen large -/// enough that `calculate_price(count)` sits well above twice the baseline, so -/// a baseline-priced (cheapest-of-K) settlement is genuinely below a 50% -/// floor. Asserted in-test so a pricing-curve change fails loudly. -const RECEIVER_COMMITTED_KEYS: u32 = 500_000; +/// Committed key count for the receiver. Above every neighbour, so the +/// own-price cap does not bind and the reference is the group median. +const RECEIVER_COMMITTED_KEYS: u32 = 8_000; -/// The receiver's own commitment-derived local price, via the real pricing -/// curve — the exact value the production floor computes. -fn receiver_local_price() -> Amount { - calculate_price(RECEIVER_COMMITTED_KEYS as usize) -} +/// The receiver's close group: fifteen neighbours, exactly the floor's minimum +/// sample, shaped like a production group around a ~4,000-key median. +const NEIGHBOUR_KEY_COUNTS: [u32; 15] = [ + 3_000, 3_200, 3_400, 3_600, 3_800, 4_000, 4_200, 4_400, 4_600, 4_800, 5_000, 5_200, 5_400, + 5_600, 5_800, +]; + +/// Lower median of the neighbours plus the receiver's own (larger) count: the +/// eighth smallest of sixteen. +const GROUP_MEDIAN_KEYS: u32 = 4_400; /// 3x baseline: an honest 3x settlement of the *cheapest* possible quote -/// (committed count 0). This is the cheapest-of-K amount a modified client can -/// settle while still paying an honest 3x of the quote it chose. -fn cheapest_of_k_settlement() -> Amount { +/// (committed count 0). This is the near-empty-quoter amount a modified client +/// can settle while still paying an honest 3x of the quote it chose. +fn baseline_settlement() -> Amount { calculate_price(0) * Amount::from(3u64) } -/// The settled amount an at-floor honest overpayment must reach under a 50% -/// floor: `3 x 50% x local_price`. +/// The settled amount an at-floor honest overpayment must reach: +/// `3 x tolerance x` the group median price. fn at_floor_settlement() -> Amount { - receiver_local_price() * Amount::from(50u64) / Amount::from(100u64) * Amount::from(3u64) + calculate_price(GROUP_MEDIAN_KEYS as usize) * Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT) + / Amount::from(100u64) + * Amount::from(3u64) } fn keypair() -> (MlDsaPublicKey, MlDsaSecretKey) { ml_dsa_65().generate_keypair().unwrap() } -/// Build the receiver's real commitment source, rotated to a commitment over -/// `RECEIVER_COMMITTED_KEYS` keys so `current_binding_snapshot` reports that -/// count — the same object type production wires into both the quote generator -/// and (via this PR) the verifier's floor. -fn receiver_commitment_source() -> Arc { +/// A real signed commitment over `key_count` keys, and its issuer's peer id +/// (BLAKE3 of the public key, matching production binding). +fn signed_commitment(key_count: u32) -> ([u8; 32], BuiltCommitment) { let (public_key, secret_key) = keypair(); let peer_id_bytes = *blake3::hash(&public_key.to_bytes()).as_bytes(); - let entries: Vec<([u8; 32], [u8; 32])> = (0..RECEIVER_COMMITTED_KEYS) + let entries: Vec<([u8; 32], [u8; 32])> = (0..key_count) .map(|i| { let mut k = [0u8; 32]; k[..4].copy_from_slice(&i.to_be_bytes()); @@ -83,7 +93,16 @@ fn receiver_commitment_source() -> Arc { let built = BuiltCommitment::build(entries, &peer_id_bytes, &secret_key, &public_key.to_bytes()) .unwrap(); - assert_eq!(built.commitment().key_count, RECEIVER_COMMITTED_KEYS); + assert_eq!(built.commitment().key_count, key_count); + (peer_id_bytes, built) +} + +/// Build the receiver's real commitment source, rotated to a commitment over +/// `RECEIVER_COMMITTED_KEYS` keys so `current_binding_snapshot` reports that +/// count — the same object type production wires into both the quote generator +/// and the verifier's floor. +fn receiver_commitment_source() -> Arc { + let (_, built) = signed_commitment(RECEIVER_COMMITTED_KEYS); let state = ResponderCommitmentState::new(); state.rotate(built); Arc::new(state) @@ -125,19 +144,38 @@ fn serialize_proof(peer_quotes: Vec<(EncodedPeerId, PaymentQuote)>) -> Vec { serialize_single_node_proof(&proof).unwrap() } -/// Build a verifier wired to the live Anvil vault and the real commitment -/// source, with the given floor policy. No `P2PNode` is attached, so the -/// issuer K-closest check fail-opens under `test-utils` — that isolates the -/// on-chain settlement read and the floor as the load-bearing checks. -fn live_verifier(network: evmlib::Network, floor: PriceFloorConfig) -> PaymentVerifier { +/// Build a verifier wired to the live Anvil vault, the real commitment source +/// and a close-group view of real signed neighbour commitments. No `P2PNode` +/// is attached, so the K-closest set comes from the test override: the +/// neighbours plus every quote issuer in `issuers`. +fn live_verifier( + network: Network, + source: Arc, + issuers: &[&EncodedPeerId], +) -> PaymentVerifier { let verifier = PaymentVerifier::new(PaymentVerifierConfig { evm: EvmVerifierConfig { network }, cache_capacity: 128, close_group_size: CLOSE_GROUP_SIZE, local_rewards_address: RewardsAddress::new([0x11; 20]), - price_floor: floor, }); - verifier.attach_local_commitment_source(receiver_commitment_source()); + let dyn_source: Arc = source; + verifier.attach_local_commitment_source(dyn_source); + + let received_at = Instant::now(); + let mut cache = HashMap::new(); + let mut k_closest = Vec::with_capacity(NEIGHBOUR_KEY_COUNTS.len() + issuers.len()); + for count in NEIGHBOUR_KEY_COUNTS { + let (peer, built) = signed_commitment(count); + cache.insert( + PeerId::from_bytes(peer), + PeerCommitmentRecord::from_verified(built.commitment().clone(), received_at), + ); + k_closest.push(peer); + } + k_closest.extend(issuers.iter().map(|issuer| *issuer.as_bytes())); + verifier.attach_commitment_cache(Arc::new(RwLock::new(cache))); + verifier.set_paid_quote_k_closest_for_tests(k_closest); verifier } @@ -158,103 +196,82 @@ async fn pay_and_build_proof( serialize_proof(vec![(peer_id, quote)]) } -/// Shadow mode (the shipping default) must NEVER reject on the floor, even when -/// the real on-chain settlement is genuinely below the receiver's floor. -#[tokio::test(flavor = "multi_thread")] -async fn shadow_mode_accepts_below_floor_real_settlement() { - let testnet = Testnet::new().await.unwrap(); +fn anvil_wallet(testnet: &Testnet) -> (Network, Wallet) { let network = testnet.to_network(); let wallet = Wallet::new_from_private_key( network.clone(), &testnet.default_wallet_private_key().unwrap(), ) .unwrap(); - let verifier = live_verifier(network, PriceFloorConfig::default()); - assert!(!verifier.price_floor_config().enforce, "default is shadow"); - assert!( - cheapest_of_k_settlement() < at_floor_settlement(), - "test fixture must keep the cheap settlement below the floor" - ); - - let xorname = [0xA1u8; 32]; - let (peer_id, quote) = signed_quote(xorname, calculate_price(0), RewardsAddress::new([2; 20])); - let proof = pay_and_build_proof(&wallet, peer_id, quote, cheapest_of_k_settlement()).await; - - let status = verifier - .verify_payment(&xorname, Some(&proof), VerificationContext::ClientPut) - .await - .expect("shadow mode must accept a real below-floor settlement"); - assert_eq!(status, PaymentStatus::PaymentVerified); + (network, wallet) } -/// Enforcement rejects the cheapest-of-K one-quote settlement: a real 3x-baseline -/// on-chain payment against a fuller receiver, read from the real vault. +/// The floor refuses a real 3x-baseline settlement, paid to a near-empty +/// quoter, at a receiver that knows its close group. #[tokio::test(flavor = "multi_thread")] -async fn enforced_floor_rejects_cheapest_of_k_real_settlement() { +async fn rejects_a_real_below_floor_settlement() { let testnet = Testnet::new().await.unwrap(); - let network = testnet.to_network(); - let wallet = Wallet::new_from_private_key( - network.clone(), - &testnet.default_wallet_private_key().unwrap(), - ) - .unwrap(); - let verifier = live_verifier( - network, - PriceFloorConfig { - enforce: true, - tolerance_percent: 50, - }, + let (network, wallet) = anvil_wallet(&testnet); + assert!( + baseline_settlement() < at_floor_settlement(), + "test fixture must keep the baseline settlement below the floor" ); - let xorname = [0xA2u8; 32]; - let (peer_id, quote) = signed_quote(xorname, calculate_price(0), RewardsAddress::new([3; 20])); - let proof = pay_and_build_proof(&wallet, peer_id, quote, cheapest_of_k_settlement()).await; + let xorname = [0xA1u8; 32]; + let (peer_id, quote) = signed_quote(xorname, calculate_price(0), RewardsAddress::new([2; 20])); + let verifier = live_verifier(network, receiver_commitment_source(), &[&peer_id]); + let proof = pay_and_build_proof(&wallet, peer_id, quote, baseline_settlement()).await; let err = verifier .verify_payment(&xorname, Some(&proof), VerificationContext::ClientPut) .await - .expect_err("a real 3x-baseline settlement must not clear a fuller receiver's floor"); + .expect_err("a real 3x-baseline settlement must not clear the floor"); assert!( - err.to_string().contains("below this node's price floor"), + err.to_string() + .contains("below the close-group price floor"), "unexpected rejection reason: {err}" ); } -/// Enforcement accepts an honest at-floor overpayment: the client overpays the -/// cheap quote to `3 x 50% x local_price`, settled for real on-chain. +/// Enforcement accepts an honest at-floor overpayment and refuses one atto +/// less: the client overpays the cheap quote to exactly `3 x tolerance x` the +/// group median, settled for real on-chain. #[tokio::test(flavor = "multi_thread")] async fn enforced_floor_accepts_at_floor_real_overpayment() { let testnet = Testnet::new().await.unwrap(); - let network = testnet.to_network(); - let wallet = Wallet::new_from_private_key( - network.clone(), - &testnet.default_wallet_private_key().unwrap(), - ) - .unwrap(); + let (network, wallet) = anvil_wallet(&testnet); let source = receiver_commitment_source(); + assert!( + calculate_price(RECEIVER_COMMITTED_KEYS as usize) + > calculate_price(GROUP_MEDIAN_KEYS as usize), + "the receiver must sit above its group so the reference is the group median" + ); // Snapshot the commitment hash BEFORE verification so we can prove the // floor read did not rotate/refresh/age retention (claim: non-mutating). let hash_before = source.current().map(|c| c.hash()); - let verifier = PaymentVerifier::new(PaymentVerifierConfig { - evm: EvmVerifierConfig { - network: network.clone(), - }, - cache_capacity: 128, - close_group_size: CLOSE_GROUP_SIZE, - local_rewards_address: RewardsAddress::new([0x11; 20]), - price_floor: PriceFloorConfig { - enforce: true, - tolerance_percent: 50, - }, - }); - let concrete = Arc::clone(&source); - let dyn_source: Arc = concrete; - verifier.attach_local_commitment_source(dyn_source); - let xorname = [0xA3u8; 32]; - let (peer_id, quote) = signed_quote(xorname, calculate_price(0), RewardsAddress::new([4; 20])); - let proof = pay_and_build_proof(&wallet, peer_id, quote, at_floor_settlement()).await; + let short_name = [0xA3u8; 32]; + let (short_peer, short_quote) = + signed_quote(short_name, calculate_price(0), RewardsAddress::new([4; 20])); + let xorname = [0xA4u8; 32]; + let (peer_id, quote) = signed_quote(xorname, calculate_price(0), RewardsAddress::new([5; 20])); + let verifier = live_verifier(network, Arc::clone(&source), &[&short_peer, &peer_id]); + + let one_short = at_floor_settlement() - Amount::from(1u64); + let short_proof = pay_and_build_proof(&wallet, short_peer, short_quote, one_short).await; + let err = verifier + .verify_payment( + &short_name, + Some(&short_proof), + VerificationContext::ClientPut, + ) + .await + .expect_err("one atto below the floor must be refused"); + assert!(err + .to_string() + .contains("below the close-group price floor")); + let proof = pay_and_build_proof(&wallet, peer_id, quote, at_floor_settlement()).await; let status = verifier .verify_payment(&xorname, Some(&proof), VerificationContext::ClientPut) .await From a044676a41a39cbb7fe11cac158d7db5473c5ffa Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 22 Oct 2026 14:58:33 +0900 Subject: [PATCH 2/5] fix(replication): drop in-memory price-contested marks a later write clears The paid list keeps its price-contested keys in memory, so building replica hints reads nothing on disk. A marked insert puts its key in that copy just before its commit and keeps it if the commit fails, since the commit may still have reached the disk. When it had not, the copy held a mark the database lacked, and nothing removed it: a later plain insert of the key wrote an unmarked entry, and a batch removal cleared only the keys the database reported deleted. Replica hints then left out a key that every point read, the audit included, treated as unmarked, until a restart. A plain insert that adds the key now drops any in-memory mark for it once its commit succeeds, still under the write order, and a removal, single or batched, drops the mark of every key it names, held or not. A failed commit still leaves the copy as it was, so the copy stays a superset of the committed marks, and equals them for every key a successful write touched. Tests mark a key in memory only, as such a failed commit leaves it, and check that a plain insert, a batch removal and a single removal of the key each clear it, while a key whose entry carries the mark keeps it. --- .../ADR-0006-receiver-side-revenue-floor.md | 13 ++- src/replication/paid_list.rs | 91 ++++++++++++++++--- 2 files changed, 86 insertions(+), 18 deletions(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index c7194743..ab07ac54 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -381,8 +381,9 @@ replication), arriving in 48 upload bursts. Of those, 1,801 admissions over accepted the key. The node also keeps its marked keys in memory, so leaving them out of the hints it builds for a peer reads nothing on disk. Each paid-list write updates that copy in commit order, a new - mark just before its commit, so the copy never lacks a committed mark. - If the marks cannot be read when the node starts, it still starts, as + mark just before its commit, so the copy never lacks a committed mark, + and a write that commits the key unmarked, a plain insert that adds it + or a removal, drops any mark the copy holds for it. If the marks cannot be read when the node starts, it still starts, as before, and hints no key as a replica until a restart can read them; - the responsible audit never samples a marked key, which also covers one hinted in the moment before its mark was written; @@ -589,7 +590,9 @@ out of 1,801: the paid list, so unmarked. When a marked insert's commit is what failed, the in-memory copy keeps the mark, since the commit may still have reached the disk, so this node does not hint the key as a replica - while the mark stays; a restart rebuilds the copy from the database. + while the mark stays. A later plain insert that adds the key, or a + removal of it, drops the mark once that write commits, and a restart + rebuilds the copy from the database. - Paid-list pruning checks for the record before and after its network round, but a record admitted between that check and the delete would lose its mark. Both store paths require this node inside the same width @@ -667,7 +670,9 @@ out of 1,801: held; - the in-memory copy of the marks is filled on opening and follows every write and removal, and a marked insert and a batch removal change it - only in the order writes take; building + only in the order writes take; a mark held in memory alone, as a failed + commit can leave it, goes with a later plain insert of the key and with + its removal, single or batched; building replica hints works while the paid list cannot be read at all, and a paid list whose marks cannot be read on opening still opens and hints no replica; diff --git a/src/replication/paid_list.rs b/src/replication/paid_list.rs index d196cd52..0369921f 100644 --- a/src/replication/paid_list.rs +++ b/src/replication/paid_list.rs @@ -34,6 +34,7 @@ use heed::{Database, Env, EnvOpenOptions, RoTxn}; use parking_lot::{Mutex, RwLock}; use std::collections::{HashMap, HashSet}; use std::path::Path; +use std::slice; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Arc; use std::time::Instant; @@ -75,6 +76,20 @@ fn read_price_contested_keys(db: &Database, txn: &RoTxn) -> Result Ok(keys) } +/// Drop `keys` from the in-memory marks, after a commit that left none of +/// them marked in the database. The set is copied only if one of them is in +/// it. +fn unmark_in_memory(marks: &RwLock>>>, keys: &[XorName]) { + if let Some(marked) = marks.write().as_mut() { + if keys.iter().any(|key| marked.contains(key)) { + let marked = Arc::make_mut(marked); + for key in keys { + marked.remove(key); + } + } + } +} + /// Persistent paid-for-list backed by LMDB. /// /// Tracks which keys this node believes are paid-authorized. @@ -97,7 +112,8 @@ pub struct PaidList { /// A new mark goes in just before its commit and a removed one just after, /// so the copy never lacks a committed mark. A mark whose commit fails /// stays, since that commit may still have reached the disk: when in - /// doubt, judge no one. + /// doubt, judge no one. Any later write that commits the key unmarked, a + /// plain insert that adds it or a removal, drops that mark. write_order: Arc>, /// Mark reads that have failed since the last one that succeeded, so a /// run of failures is logged once (see `counts_as_price_contested`). @@ -273,6 +289,9 @@ impl PaidList { wtxn.commit().map_err(|e| { Error::Storage(format!("Failed to commit paid-list insert: {e}")) })?; + if value.is_empty() { + unmark_in_memory(&price_contested, slice::from_ref(&key_owned)); + } drop(ordered); Ok(true) @@ -391,11 +410,7 @@ impl PaidList { wtxn.commit().map_err(|e| { Error::Storage(format!("Failed to commit paid-list delete: {e}")) })?; - if let Some(keys) = price_contested.write().as_mut() { - if keys.contains(&key_owned) { - Arc::make_mut(keys).remove(&key_owned); - } - } + unmark_in_memory(&price_contested, slice::from_ref(&key_owned)); drop(ordered); Ok(deleted) }) @@ -624,14 +639,9 @@ impl PaidList { wtxn.commit() .map_err(|e| Error::Storage(format!("Failed to commit batch remove: {e}")))?; - if let Some(keys) = price_contested.write().as_mut() { - if removed.iter().any(|key| keys.contains(key)) { - let keys = Arc::make_mut(keys); - for key in &removed { - keys.remove(key); - } - } - } + // Every key of the batch, not only those the database held: a + // mark whose commit failed can be in memory alone. + unmark_in_memory(&price_contested, &keys_owned); drop(ordered); Ok(removed) @@ -898,6 +908,59 @@ mod tests { assert!(marked_in_memory(&pl).is_empty()); } + /// Mark `key` in memory only, with no entry in the database: what a marked + /// insert whose commit failed before reaching the disk leaves. + fn mark_in_memory_only(pl: &PaidList, key: XorName) { + let mut marks = pl.price_contested.write(); + let marks = marks.as_mut().expect("marks read on open"); + Arc::make_mut(marks).insert(key); + } + + /// ADR-0006: a mark left in memory alone by a failed commit goes with a + /// later plain insert that adds the key, so the copy agrees with the + /// database again and the key is hinted as a replica. A plain insert of a + /// key whose entry carries the mark leaves both as they are. + #[tokio::test] + async fn a_plain_insert_drops_a_mark_held_in_memory_only() { + let (pl, _temp) = create_test_paid_list().await; + let stale: XorName = [0x71; 32]; + let marked: XorName = [0x72; 32]; + pl.insert_price_contested(&marked).await.expect("mark"); + mark_in_memory_only(&pl, stale); + let mut keys = vec![stale]; + pl.retain_unmarked(&mut keys); + assert_eq!(keys, Vec::::new()); + + assert!(pl.insert(&stale).await.expect("insert")); + assert!(!pl.insert(&marked).await.expect("insert")); + assert!(!pl.is_price_contested(&stale).expect("read mark")); + assert!(pl.is_price_contested(&marked).expect("read mark")); + assert_eq!(marked_in_memory(&pl), HashSet::from([marked])); + let mut keys = vec![stale, marked]; + pl.retain_unmarked(&mut keys); + assert_eq!(keys, vec![stale]); + } + + /// ADR-0006: a mark left in memory alone by a failed commit goes with a + /// later removal of the key, single or batched, although the database + /// holds no entry to delete. A mark the removal does not name stays. + #[tokio::test] + async fn a_removal_drops_a_mark_held_in_memory_only() { + let (pl, _temp) = create_test_paid_list().await; + let batched: XorName = [0x73; 32]; + let single: XorName = [0x74; 32]; + let marked: XorName = [0x75; 32]; + pl.insert_price_contested(&marked).await.expect("mark"); + mark_in_memory_only(&pl, batched); + mark_in_memory_only(&pl, single); + + assert_eq!(pl.remove_batch(&[batched]).await.expect("remove batch"), 0); + assert_eq!(marked_in_memory(&pl), HashSet::from([single, marked])); + assert!(!pl.remove(&single).await.expect("remove")); + assert_eq!(marked_in_memory(&pl), HashSet::from([marked])); + assert!(pl.is_price_contested(&marked).expect("read mark")); + } + /// Byte offsets in an LMDB meta page on a 64-bit target: the magic number /// follows the 16-byte page header, the free list's record keeps the page /// size at 40, and the main database's root page number sits at 128. From 7a735be906ae69d1e4fce4d2318dffe8c0769768 Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 22 Oct 2026 16:37:28 +0900 Subject: [PATCH 3/5] docs(adr): count the reconciliation mutations in ADR-0006 The mutation run on the price-floor code now covers the in-memory mark reconciliation too: 54 mutations, of which 53 fail at least one unit or e2e test. The one survivor is unchanged: prune-audit scoring reading a mark that cannot be read as unset, which only the e2e prune test reaches, and that test cannot make a paid-list read fail. --- docs/adr/ADR-0006-receiver-side-revenue-floor.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index ab07ac54..165e7e44 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -700,9 +700,9 @@ out of 1,801: cache with a forged proof the floor would contest, mark the key exactly when the recording node's own verified decision was contested, and never from that proof. -- 48 mutations of the floor, the contest decision, the verifier's memory, +- 54 mutations of the floor, the contest decision, the verifier's memory, the mark, its in-memory copy and every check that reads the mark were run - against these unit and e2e tests on this amendment's code, and 47 fail at + against these unit and e2e tests on this amendment's code, and 53 fail at least one of them. The one that survives makes prune-audit scoring count a mark that cannot be read as unset. Only the e2e prune test reaches that check, and it cannot make a node's paid-list read fail; the same read From 2252074964d97d1254d3476a62ab2d040c488480 Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 29 Oct 2026 10:35:18 +0900 Subject: [PATCH 4/5] docs(adr): say a fan-out, not the record, reads the current view ADR-0006's validation list said the contest decision follows the view when a write is recorded, and the unit test pinning it said the same. That holds for the proof a fan-out forwards, chunk or pointer, which is priced against the view when the write is fanned out. The paid-list mark a chunk is recorded with comes from the decision its verification made, which the verifier remembers, not from the view at record time. Both texts now say so. No code changes. --- docs/adr/ADR-0006-receiver-side-revenue-floor.md | 4 +++- src/payment/verifier.rs | 8 ++++---- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index 165e7e44..1ca5ba49 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -656,7 +656,9 @@ out of 1,801: - a write below the contest threshold, or one the node cannot price, is contested, and one at the median is not. The threshold is 60% of the uncapped median to the atto, it stays on the median when the cap binds, - and it follows the view when the write is recorded; + and a fan-out judges the proof it forwards against the view when it fans + out, while a chunk's paid-list mark keeps the decision its verification + made; - the tolerance is 30%: exactly `3 x 30%` of the group median clears it and one atto less does not; - a node with no floor wired contests nothing; diff --git a/src/payment/verifier.rs b/src/payment/verifier.rs index a8f57c84..45e8b003 100644 --- a/src/payment/verifier.rs +++ b/src/payment/verifier.rs @@ -5048,10 +5048,10 @@ mod tests { assert!(!verifier.price_contested(&xorname, &proof_bytes).await); } - /// The decision is taken from the proof against the current view when the - /// write is recorded (or, for a pointer, fanned out), so a write stored - /// after the group's prices rose is judged against the view peers will - /// price it against now. + /// A fan-out, chunk or pointer, judges the proof it forwards against the + /// view when it fans out, so a write fanned out after the group's prices + /// rose is judged against the view peers will price it against now. (A + /// chunk's paid-list mark keeps the decision its verification made.) #[tokio::test] async fn the_contest_follows_the_current_view() { let verifier = floor_test_verifier(); From 16e1fc77a561c92421ca94eb8d6d93292d895216 Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 29 Oct 2026 10:37:45 +0900 Subject: [PATCH 5/5] test: name the tokio types the new test helpers use by import The paid-list helper that holds the write order spelled out tokio::sync::oneshot and tokio::task::JoinHandle inline, and the verifier test for a node with no floor wired built its gossip cache as tokio::sync::RwLock inline. The paid-list tests now import oneshot and JoinHandle, the helper waits for the order with an assertion instead of expect, and the verifier test uses CommitmentCache::default(). No behaviour changes. --- src/payment/verifier.rs | 2 +- src/replication/paid_list.rs | 17 +++++++---------- 2 files changed, 8 insertions(+), 11 deletions(-) diff --git a/src/payment/verifier.rs b/src/payment/verifier.rs index 45e8b003..12ca85c2 100644 --- a/src/payment/verifier.rs +++ b/src/payment/verifier.rs @@ -5044,7 +5044,7 @@ mod tests { let verifier = create_test_verifier(); assert!(!verifier.price_contested(&xorname, &proof_bytes).await); - verifier.attach_commitment_cache(Arc::new(tokio::sync::RwLock::new(HashMap::new()))); + verifier.attach_commitment_cache(CommitmentCache::default()); assert!(!verifier.price_contested(&xorname, &proof_bytes).await); } diff --git a/src/replication/paid_list.rs b/src/replication/paid_list.rs index 0369921f..a6d9c27b 100644 --- a/src/replication/paid_list.rs +++ b/src/replication/paid_list.rs @@ -709,6 +709,8 @@ mod tests { use std::fs::OpenOptions; #[cfg(target_pointer_width = "64")] use std::io::{Read, Seek, SeekFrom, Write}; + use tokio::sync::oneshot; + use tokio::task::JoinHandle; #[cfg(feature = "logging")] use tracing::subscriber::with_default; #[cfg(feature = "logging")] @@ -1309,21 +1311,16 @@ mod tests { /// /// Returns once the order is held. Dropping the returned sender releases /// it. - async fn hold_write_order( - pl: &PaidList, - ) -> ( - tokio::sync::oneshot::Sender<()>, - tokio::task::JoinHandle<()>, - ) { + async fn hold_write_order(pl: &PaidList) -> (oneshot::Sender<()>, JoinHandle<()>) { let write_order = Arc::clone(&pl.write_order); - let (release_tx, release_rx) = tokio::sync::oneshot::channel::<()>(); - let (held_tx, held_rx) = tokio::sync::oneshot::channel::<()>(); - let holder = tokio::task::spawn_blocking(move || { + let (release_tx, release_rx) = oneshot::channel::<()>(); + let (held_tx, held_rx) = oneshot::channel::<()>(); + let holder = spawn_blocking(move || { let _ordered = write_order.lock(); let _ = held_tx.send(()); let _ = release_rx.blocking_recv(); }); - held_rx.await.expect("write order held"); + assert!(held_rx.await.is_ok(), "write order held"); (release_tx, holder) }