Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,5 @@ target/
__pycache__/
.marketplace/
.shop-order/

.usdt/
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Added
- Add local Orchestra and LayerZero providers with bridge lifecycle controls, real Arbitrum credits/refunds and a Core bridge smoke suite
- Add an isolated `usdt` Compose profile with pinned Anvil/Alto/gateway images, a containerized Core wallet, real payment/proof smoke checks, controlled paymaster failures and private-network Shop verification
- Run the Shop's marketplace on Synonym's staging with our own Paykit Server v0.1.0-rc11 behind a Cloudflare quick tunnel (`./shop-mixed`, profile `shop-mixed`)
- Run a paid Shop order on our own Paykit Server, Lock Server, marketplace service and Shop with Synonym's staging homeserver (`./shop-order`, profile `shop-order`)
- Rebuild the Paykit fixtures on the paykit-rs version a Bitkit pull request pins with `scripts/follow-app-paykit`
Expand All @@ -17,13 +19,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Issue payment requests payable through an LNURL-pay endpoint, with a separate proposal expiry

### Changed
- Keep USDT test history on the local fork, pace and cache archive reads, and preserve upstream throttling for retries so ordinary E2E runs need no paid Alchemy features
- Pin the local USDT gateway to `3d8c0b6`, including sanitized configuration errors and local LayerZero tracking support
- Publish every host port at 1024 or above, in the 23000-23999 range for the services Bitkit does not dial by a fixed number (README "Ports" lists them)
- Move the payment request fixture peers to paykit-rs rc65 on Pubky 0.14.0, the SDK current Bitkit builds pin (the `fixture-issuer` and `rc56-peer` service names stay)
- Move the marketplace fixture to the Pubky 0.14.0 homeserver, which grants the `LOCK` write locks current Bitkit builds take, with Paykit Server `0ffd4da` (pubky/paykit-server#46, paykit-rs rc65, locks-core rc8) and the driver on `@synonymdev/pubky` 0.14.0
- Show ready-to-copy settle and cancel commands in `holdinvoice` output
- Simplify LND funding step in README to a single command instead of clipboard-based two-step flow

### Fixed
- Pair the Shop and Lock Server denomination-aware setup API, including the seller provisioning helper
- Bound the Shop build Node heap and serialize app builds to limit peak Docker memory use
- Preserve the payment server's network namespace when starting Shop apps so the USDT verifier keeps its RPC connection
- Fund the payment request issuer's wallet before `/pay`, which failed with insufficient funds on a fresh regtest chain
- Clear stale X display locks in `scripts/trezor-emulator` before starting the emulator, fixing `RuntimeError('Emulator process died')` caused by Xvfb refusing to start over a leftover `/tmp/.X<n>-lock`
- Validate LNURL-withdraw callback invoices by millisatoshis (`num_msat`) to preserve msat precision for min/max range checks
Expand Down
39 changes: 35 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,37 @@

A complete Docker-based development environment for Bitcoin and Lightning Network development, featuring a LNURL server for Lightning payments and testing guides using the Bitkit app.

## Local USDT testing

Run the shared Arbitrum fork, Alto and gateway without spending mainnet funds:

```sh
./usdt-fixture build
export USDT_TEST_ENV_FILE=/absolute/path/to/private.env
./usdt-fixture up
./usdt-fixture check
./usdt-fixture smoke
./usdt-fixture bridge-smoke
eval "$(./usdt-fixture env)"
./usdt-fixture fund 0xYOUR_TEST_WALLET 10
```

The private file contains `ARBITRUM_RPC_URL=https://...` with historical-state access.
An Alchemy Free Arbitrum endpoint is sufficient within its quota: the fixture paces
upstream reads and excludes pre-fork mainnet logs. Payment history and recovery
cover transfers made locally after the fork; use disposable test wallets.
Requires Docker Compose, Git access to the pinned Core/service repositories, jq
and tar. Builds and tools run inside containers; no host Node or Rust is needed.
Use `./usdt-fixture logs` to inspect services and `./usdt-fixture down` to remove
this isolated stack, its disposable wallets and fork-read cache. Keep existing staging VSS,
Pubky and rate services. The stack also starts local Orchestra and LayerZero providers. After the
app registers a bridge receive address, use `./usdt-fixture orchestra deposit
ADDRESS polygon 3.5`, then `./usdt-fixture orchestra advance DEPOSIT_ID completed`
to deliver local tokens. `orchestra list` shows outgoing quotes; `layerzero
SOURCE_TX DELIVERED` updates a real USDT0 source message. See [USDT fixture](docs/usdt-fixture.md) for versions, app setup, pending
payment controls, proof verification, the optional Shop overlay and simulation limits.


## Services

- **Bitcoin Core** (regtest): Bitcoin node for development
Expand Down Expand Up @@ -642,10 +673,8 @@ The headless seller and buyer of the `marketplace` profile need the local testne
#### Shop Order on Our Own Servers

The `shop-order` profile adds the rest of the Shop to `shop-mixed`, so a Bitkit buyer can pay a Pubky Ring seller's digital listing
and the same marketplace order becomes paid: the Lock Server v0.1.0-rc10, the marketplace service (pubky-marketplace-service#98,
`005b0707`) and the Shop (pubky-marketplace#154, `fbe3babb`), the commits the Android and iOS order-paid acceptance ran on, with
Postgres, a quick tunnel per public URL, and keys made once and kept. Synonym's staging Shop stays the first route; use this when it
cannot serve the build under test. Needs Docker, `curl`, `jq`, `git` and `python3`, and about 6 GB free for the Shop's build.
and the same marketplace order becomes paid. The pinned Shop, Lock Server and marketplace service share the upstream Paykit
contract, with Postgres, a quick tunnel per public URL, and staging Pubky services. Sources and images are pinned in `shop-order`.

```bash
SHOP_ORDER_PROVISIONING=1 ./shop-order fetch # pinned sources, plus the seller's provisioning page
Expand All @@ -654,6 +683,8 @@ SHOP_ORDER_PROVISIONING=1 ./shop-order fetch # pinned sources, plus the seller
./shop-order down --keep-data # stop; keeps databases, keys and the seller's setup
```

For local USDT settlement, start `./usdt-fixture up` first and set `SHOP_ORDER_USDT=1` for every Shop command. Both the app and Paykit Server then verify payments on the same fork. Seller setup in Bitkit must include the optional USDT address. Fund the buyer with `./usdt-fixture fund ADDRESS 20`, then pay the Shop request in USDT. Keep staging Pubky and rate services.

See [docs/shop-order.md](docs/shop-order.md) for the seller and buyer steps, the pass criteria and every setting.

#### Bech32 LNURL Pay
Expand Down
130 changes: 123 additions & 7 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -712,11 +712,10 @@ services:
command: ["init-and-stay"]

# ---------------------------------------------------------------------------
# A Shop order paid end to end on our own servers with Synonym's staging homeserver (profile "shop-order", on top of
# shop-mixed's Paykit Server rc11): the Lock Server v0.1.0-rc10, the marketplace service and the Shop at the commits that ran
# the Android and iOS order-paid acceptance (10 Oct), Postgres for the three, and a quick tunnel for each URL a phone or the
# seller's browser opens. ./shop-order fetches each source at its exact commit into .shop-order/src and refuses another
# revision; docs/shop-order.md has the flow. Secrets are made by shop-order-keys in the state volume, never committed.
# Local Shop checkout with Synonym's staging homeserver (profile "shop-order", on top of shop-mixed): Paykit, Locks,
# the marketplace service, Shop and Postgres, with tunnels for the phone and seller browser. ./shop-order fetches
# the paired source revisions into .shop-order/src; docs/shop-order.md has the flow. Secrets are made once by
# shop-order-keys in the state volume, never committed.
# ---------------------------------------------------------------------------
shop-order-keys:
profiles: [shop-order]
Expand Down Expand Up @@ -754,7 +753,7 @@ services:

shop-order-locks:
profiles: [shop-order]
image: bitkit-docker/locks-server:${SHOP_ORDER_LOCKS_REV:-01cfeca14c7b5d385d8c3536c0cb4e1af81b458c}
image: bitkit-docker/locks-server:${SHOP_ORDER_LOCKS_REV:-be3f1b4d83b7318585d724fa6a429e1e5b3b0399}
build:
context: ./.shop-order/src/locks
restart: "no"
Expand Down Expand Up @@ -815,9 +814,11 @@ services:

shop-order-shop:
profiles: [shop-order]
image: bitkit-docker/pubky-marketplace:${SHOP_ORDER_SHOP_REV:-fbe3babba0c3c05990571221b5d4dc0c31788e68}
image: bitkit-docker/pubky-marketplace:${SHOP_ORDER_SHOP_REV:-af2866f58c8706e275a3028e41168c5976305808}
build:
context: ./.shop-order/src/shop
args:
NODE_MAX_OLD_SPACE_SIZE: ${SHOP_ORDER_NODE_HEAP_MB:-3072}
restart: "no"
user: "0:0" # reads the BFF secrets in the state volume (mode 0600)
depends_on:
Expand Down Expand Up @@ -864,7 +865,118 @@ services:
ports:
- "127.0.0.1:23141:23141"

# USDT test chain and provider fixtures. Opt in with ./usdt-fixture.
# Shared loopback matches Core/gateway local endpoint validation.
usdt-fixture:
profiles: [usdt]
image: bitkit-docker/usdt-fixture:${USDT_IMAGE_TAG:-unbuilt}
build:
context: ./.usdt/build
dockerfile: usdt/Dockerfile
target: fixture
# File-backed Compose secrets retain host ownership on Linux.
user: "0:0"
init: true
restart: "no"
secrets:
- usdt_upstream
environment:
USDT_GATEWAY_PORT: ${USDT_GATEWAY_PORT:-23453}
USDT_CONTROL_PORT: ${USDT_CONTROL_PORT:-23454}
ports:
- "127.0.0.1:${USDT_GATEWAY_PORT:-23453}:23453"
- "127.0.0.1:${USDT_CONTROL_PORT:-23454}:23454"
healthcheck:
test: ["CMD", "node", "cli.cjs", "status"]
interval: 2s
timeout: 5s
retries: 60
start_period: 10s

usdt-node:
profiles: [usdt]
image: bitkit-docker/usdt-fixture:${USDT_IMAGE_TAG:-unbuilt}
init: true
restart: "no"
network_mode: service:usdt-fixture
depends_on:
usdt-fixture:
condition: service_started
entrypoint: [/bin/sh, -c]
volumes:
- usdt_fork_cache:/home/node/.foundry
command:
- node wait.cjs upstream && exec anvil --host 127.0.0.1 --port 23450 --fork-url http://127.0.0.1:23455 --fork-block-number "$$(node -p "require('./pins.json').forkBlock")" --chain-id 42161 --hardfork prague --silent --block-time 1 --compute-units-per-second 150 --fork-retry-backoff 1000 --no-fork-node-info

usdt-alto:
profiles: [usdt]
image: bitkit-docker/usdt-fixture:${USDT_IMAGE_TAG:-unbuilt}
init: true
restart: "no"
network_mode: service:usdt-fixture
depends_on:
usdt-fixture:
condition: service_healthy
entrypoint: [/bin/sh, -c]
command:
- |
KEY="$$(node -p "require('viem').toHex(require('./contracts.cjs').executor.getHdKey().privateKey)")"
exec node node_modules/@pimlico/alto/esm/cli/alto.js --entrypoints 0x4337084D9E255Ff0702461CF8895CE9E3b5Ff108 --rpc-url http://127.0.0.1:23450 --executor-private-keys "$$KEY" --utility-private-key "$$KEY" --port 23451 --chain-type default --safe-mode false --enable-debug-endpoints true --min-executor-balance 0 --log-level warn --json true
healthcheck:
test: ["CMD", "node", "-e", "require('./rpc.cjs').rpc('http://127.0.0.1:23451','eth_supportedEntryPoints').catch(()=>process.exit(1))"]
interval: 2s
timeout: 5s
retries: 30

usdt-gateway:
profiles: [usdt]
image: bitkit-docker/usdt-gateway:${USDT_IMAGE_TAG:-unbuilt}
build:
context: ./.usdt/build
dockerfile: usdt/Dockerfile
target: gateway
init: true
restart: "no"
network_mode: service:usdt-fixture
depends_on:
usdt-alto:
condition: service_healthy
environment:
NODE_ENV: test
HOST: 0.0.0.0
PORT: 23453
ARBITRUM_RPC_URL: http://127.0.0.1:23450
LOCAL_PROVIDER_URL: http://127.0.0.1:23452
LOCAL_ORCHESTRA_URL: http://127.0.0.1:23456
LOCAL_LAYERZERO_URL: http://127.0.0.1:23457
ORCHESTRA_API_KEY: local-orchestra-fixture
ORCHESTRA_BRIDGE_SECRET: abababababababababababababababababababababababababababababababab
ORCHESTRA_BRIDGE_NETWORKS: ethereum,polygon,base,bsc,tron,solana,plasma
ORCHESTRA_DEPOSIT_NETWORKS: ethereum,polygon,base,bsc,tron,solana
USDT_BRIDGE_NETWORKS: ethereum,polygon,plasma,stable
REQUESTS_PER_MINUTE: 10000
GLOBAL_REQUESTS_PER_MINUTE: 100000
healthcheck:
test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:23453/healthz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"]
interval: 2s
timeout: 5s
retries: 30

usdt-tools:
profiles: [usdt]
image: bitkit-docker/usdt-fixture:${USDT_IMAGE_TAG:-unbuilt}
network_mode: service:usdt-fixture
depends_on:
usdt-gateway:
condition: service_healthy
volumes:
- usdt_state:/state
- ./.usdt/evidence:/evidence:ro
entrypoint: ["node", "cli.cjs"]

volumes:
usdt_fork_cache:
usdt_state:
trezor_emulator_state:
trezor_emulator_logs:
bitcoin_home:
Expand All @@ -879,4 +991,8 @@ volumes:
shop_mixed_state:
shop_order_postgres_data:

secrets:
usdt_upstream:
file: ${USDT_TEST_ENV_FILE:-/dev/null}

networks: {}
12 changes: 4 additions & 8 deletions docs/shop-order.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,12 @@ test, for example a Paykit version the staging Paykit Server does not run.

| Component | Revision | Built from |
| --- | --- | --- |
| Shop | pubky/pubky-marketplace#154, fbe3babba0c3c05990571221b5d4dc0c31788e68 | the repository's Dockerfile |
| Shop | pubky/pubky-marketplace, af2866f58c8706e275a3028e41168c5976305808 | the repository's Dockerfile |
| Marketplace service | pubky/pubky-marketplace-service#98, 005b0707e6047b388ce032f4b51a2e0ed9e3a752 | the repository's Dockerfile |
| Lock Server | pubky/locks v0.1.0-rc10, 01cfeca14c7b5d385d8c3536c0cb4e1af81b458c | the repository's Dockerfile |
| Lock Server | pubky/locks#75, be3f1b4d83b7318585d724fa6a429e1e5b3b0399 | the repository's Dockerfile |
| Paykit Server | v0.1.0-rc11, 662dca0619a9aa2962bcd677bd5ddd4563cd2784 (paykit-rs ad3c7224 = rc72) | `marketplace/shop-mixed` |

The Shop was tested at fbe3babb; pubky/pubky-marketplace#154 later moved to 84934516 for review changes to the seller-readiness
source and the buyer key-record gate, with the Locks payment path unchanged. The pin stays at the tested commit until #154 merges.

`./shop-order fetch` checks out each source at exactly that commit and refuses another. These are the commits the Android
(aa330959, devDebug) and iOS (bb02e835, Debug) order-paid acceptance ran on, on 10 Oct.
`./shop-order fetch` checks out each source at its pinned commit. The Shop and Lock Server both use the denomination-aware seller setup API. Keep these revisions paired when overriding them.

## Run

Expand All @@ -32,7 +28,7 @@ SHOP_ORDER_PROVISIONING=1 ./shop-order fetch # the pinned sources, plus the se
./shop-order health # readiness, revisions, Paykit's trusted keys
```

The first build compiles three Rust services and the Shop, and the Shop's build needs about 6 GB of memory. Each start gets new
The first build compiles three Rust services and the Shop. Startup builds the apps sequentially, and the Shop build uses a 3 GB Node heap per process. Next.js also starts workers, so allow additional Docker memory; an 8 GB VM can exhaust memory while the full stack is running. `SHOP_ORDER_NODE_HEAP_MB` overrides the per-process heap, but lowering it can make compilation fail. Each start gets new
quick-tunnel URLs; `up` writes them to `.shop-order/run/urls.env` and every origin setting is filled from them.

## Seller
Expand Down
Loading