diff --git a/.agents/agents.md b/.agents/agents.md index f4d0fdb..a2abb8a 100644 --- a/.agents/agents.md +++ b/.agents/agents.md @@ -6,106 +6,106 @@ This document serves as an architectural, operational, and development guide for ## 1. Architecture & Ecosystem Overview -`Lightning-Python-Tools` is a modular collection of automation daemons, background services, CLI utilities, and cron jobs designed for Lightning Network (LN) node operators running **LND**. It integrates seamlessly with key Lightning infrastructure components: +`Lightning-Python-Tools` is a modular collection of automation daemons, systemd timer units, CLI utilities, and cron jobs for Lightning Network node operators running **LND**. It integrates with the following infrastructure components: -- **LND (`lncli`) / Balance of Satoshi (`bos`)**: Channel operations, UTXO management, invoice generation, and on-chain fee estimation. -- **Amboss Space & Magma Market**: Liquidity marketplace integration using the modern Amboss Magma GraphQL API (`https://magma.amboss.tech/graphql`) and Amboss Space metadata API (`https://api.amboss.space/graphql`). -- **LNDg**: Lightning Network node management dashboard, database sync, fee policies, and rebalancer coordination. -- **PeerSwap**: Trustless off-chain to on-chain (L-BTC / BTC) peer rebalancing monitoring and dashboard integration. -- **Boltz & Lightning Loop**: Automated submarine swap-out operations for off-chain to on-chain liquidity rebalancing. -- **LNbits**: Sub-wallet allowance management and automated budget allocations. -- **Telegram Bot API**: Real-time alerts, interactive manual approval callbacks, and remote command execution. +- **LND (`lncli`) / Balance of Satoshis (`bos`)**: Channel operations, UTXO inspection/consolidation, invoice generation, route probing, and fee estimation. +- **Amboss Space & Magma Market**: Liquidity marketplace integration using the Amboss Magma GraphQL API (`https://magma.amboss.tech/graphql`) and Amboss Space GraphQL API (`https://api.amboss.space/graphql`). +- **LNDg**: Node management dashboard REST API (and optional PostgreSQL database) for channel balances, forward timestamps, fee policies, notes, and auto-rebalancer coordination. +- **charge-lnd**: Rule file generation (`fee_adjuster.txt`, `magma-channels_*.txt`, `magma-finished.txt`, `0_fee.txt`) for external policy enforcement. +- **PeerSwap (`peerswapd` / `pscli`)**: Layer-1 BTC and Liquid (L-BTC) peer swap monitoring, Telegram bot control, and LNDg note synchronization. +- **Lightning Loop (`litloop` / `loopd`) & Boltz (`boltzd` / `boltzcli`)**: Economically scored Loop Out swaps with live prepay route probes and Boltz reverse submarine swaps to Liquid L-BTC. +- **LNbits**: Scheduled fiat-denominated satoshi allowances across child sub-wallets. +- **Telegram Bot API (`pyTelegramBotAPI`)**: Real-time alerts, interactive order approval callbacks, error log streaming, and bot commands. --- ## 2. Directory & Module Breakdown -### 2.1 `Magma/` — Amboss Magma Liquidity Market Automation - -Amboss Magma is a decentralized Lightning liquidity marketplace where node operators buy and sell inbound channels. +### 2.1 `Magma/` - Amboss Magma Liquidity Market Automation | File | Type | Description | | :--- | :--- | :--- | -| `magma_sale_process.py` | Systemd Daemon / CLI | **Full-lifecycle seller automation**: Continuously monitors sales orders with `WAITING_FOR_SELLER_APPROVAL` and `WAITING_FOR_CHANNEL_OPEN`. Computes on-chain fee requirements via mempool API, verifies available UTXOs (filtering Loop locks), creates pre-image HODL invoices, opens channels via `lncli` or `bos`, auto-rejects banned peer pubkeys, confirms funding transaction outpoints to Amboss Magma via `market.order.seller.add_transaction`, and handles interactive Telegram approval callbacks. | -| `magma_market_fee.py` | Cronjob / CLI / Systemd | **Dynamic pricing & offer optimizer**: Gathers public Magma sell offers via `market.offer.offers`, filters out low seller scores (< threshold), analyzes percentile fee distributions (base fee & PPM), calculates annualized yield (APR), reserves required on-chain capital, and dynamically creates (`market.offer.create`), updates (`market.offer.update`), or toggles (`market.offer.toggle`) Magma sell offers. | -| `magma_config.ini` | Configuration | Defines pricing percentiles, minimum seller score filters, duration templates, and on-chain reserve thresholds for Magma. | -| `magma_sale.MD` | Documentation | Step-by-step setup guide for running `magma_sale_process.py` as a persistent `systemd` service. | +| `magma_sale_process.py` | Systemd Daemon / CLI | **Full-lifecycle seller automation**: Monitors sales orders in `WAITING_FOR_SELLER_APPROVAL` and `WAITING_FOR_CHANNEL_OPEN`. Checks mempool fees and unlocked UTXOs, generates Lightning invoices via `lncli addinvoice`, handles interactive Telegram approval callbacks (`/processmagmaorders`) or timeout auto-approval, opens channels via `lncli`, auto-rejects banned pubkeys, and confirms funding outpoints via `market.order.seller.add_transaction`. | +| `magma_market_fee.py` | Cronjob / CLI | **Dynamic pricing & offer optimizer**: Queries public Magma sell offers via `market.offer.offers`, filters by minimum seller score, analyzes percentile fee distributions, computes annualized yield (APR), allocates on-chain capital, and creates (`market.offer.create`), updates (`market.offer.update`), or toggles (`market.offer.toggle`) Magma offers. Supports `--dry-run` and `--force`. | +| `magma_config.ini.example` | Config Template | Copy to `Magma/magma_config.ini` (gitignored). Defines pricing percentiles, seller score filters, capital allocation shares, and `[magma_offer_*]` templates. | +| `magma_sale.md` | Documentation | Setup, systemd unit configuration, troubleshooting, and simulation guide for `magma_sale_process.py`. | -#### Amboss Magma GraphQL Endpoints +#### Amboss GraphQL Endpoints - **Magma Liquidity Market API**: `https://magma.amboss.tech/graphql` (Sales, Orders, Offers, Mutations). -- **Amboss Space API**: `https://api.amboss.space/graphql` (Node lookups, aliases, extended info). +- **Amboss Space API**: `https://api.amboss.space/graphql` (Node lookups, aliases, peer fee metrics). --- -### 2.2 `LNDg/` — LNDg Dashboard & Node Coordination - -Tools designed to augment and automate the [LNDg](https://github.com/cryptoshred/lndg) management interface. +### 2.2 `LNDg/` - LNDg Dashboard & Node Coordination | File | Type | Description | | :--- | :--- | :--- | -| `amboss_pull.py` | Cronjob / CLI | Pulls Magma sell orders from Amboss and synchronizes channel details, maturity dates, and auto-fee triggers into the local LNDg database and notes. | -| `channel_base-fee.py` | Cronjob / CLI | Evaluates channel conditions (such as fee thresholds and flow metrics) and adjusts base fees dynamically in LNDg. | -| `channel_fee-pull.py` | Cronjob / CLI | Exports channel fee policies and base fees from LNDg into standardized text/JSON formats for external automation tools. | -| `swap_out-candidates.py` | CLI / Tool | Evaluates channel capacity, local balances, and fee rates to identify prime candidates for submarine swap-outs; supports exporting to `.bos` tags format. | -| `mempool_rebalancer_trigger.py`| Systemd / Service | Tracks half-hour mempool fee rates (sat/vB). Automatically disables LNDg Auto-Rebalancer during high-fee congestion and re-enables it when fees normalize. | -| `disabled_fee-accelerator.py` | Cronjob / CLI | Adjusts fees on channels flagged as disabled or low-outbound by LNDg to incentivize liquidity rebalancing and prevent stale channels. | -| `offline_summary.py` | Cronjob / Alert | Scans peer states and compiles summaries of offline channels and inactive peers for Telegram / log reporting. | +| `amboss_pull.py` | Cronjob / CLI | Pulls Magma sell orders from Amboss, resolves channel SCIDs (via Amboss Space batch lookup with mathematical SCID fallback), writes lease maturity notes and auto-fee/rebalance flags to LNDg, and exports `charge-lnd` fee-cap rule files. | +| `mempool_rebalancer_trigger.py` | Systemd Daemon | Polls half-hour mempool fee rates (`sat/vB`) and updates LNDg `AR-Enabled` (`0` during high-fee spikes, `1` when fees normalize). | +| `channel_base-fee.py` | Cronjob / CLI | Resets `base_fee` to `0` across open, active channels in LNDg (excluding pubkeys in `base_fee_ignore`). | +| `channel_fee-pull.py` | Cronjob / CLI | Exports open channel IDs with `local_fee_rate == 0` from LNDg to `~/.chargelnd/.config/0_fee.txt` for `charge-lnd`. | +| `disabled_fee-accelerator.py` | Cronjob / CLI | Incrementally bumps `local_fee_rate` in LNDg for peer groups with low aggregate local liquidity so channels disabled by `charge-lnd` can attract rebalance inflows. | +| `swap_out-candidates.py` | CLI Tool | Filters high-local-balance, low-PPM channels from LNDg as swap-out candidates and optionally exports to `data/low-fee-high-local.log` (`-e`) or `bos/tags.json` (`-b`). | +| `offline_summary.py` | CLI Tool | Queries LNDg's PostgreSQL `gui_peerevents` table (`--month YYYY-MM`) and prints average and total peer offline hours (requires optional `psycopg2` and `pandas`). | --- -### 2.3 `Peerswap/` — PeerSwap Liquidity & Monitoring - -Integration scripts for [PeerSwap](https://github.com/ElementsProject/peerswap) (L-BTC / BTC on-chain swaps). +### 2.3 `Peerswap/` - PeerSwap Liquidity & Monitoring | File | Type | Description | | :--- | :--- | :--- | -| `peerswap-lndg_push.py` | Cronjob / CLI | Queries `pscli listpeers` and `pscli listswaps`, aggregates swap volumes and counts per peer, and injects the telemetry directly into LNDg channel cards and dashboard notes. | -| `ps_peers.py` | CLI Tool | Formats and displays a clean tabular overview of L-BTC on-chain balances, active PeerSwap peers, channel capacities, and available swap liquidity. | -| `peerswap-bot.py` | Telegram Bot | Interactive bot for monitoring PeerSwap daemon health, initiating swaps, and sending alerts upon swap completion or failure. | +| `peerswap-bot.py` | Systemd Daemon | Interactive Telegram bot for checking L-BTC balances/addresses, listing peers and swap requests, and executing `swapin` / `swapout` / `lbtcsend` commands. | +| `peerswap-lndg_push.py` | Cronjob / CLI | Queries `pscli listpeers` and `lncli listchannels`, aggregates swap counts, volumes, and paid fees per peer, and writes them into LNDg channel notes. | +| `ps_peers.py` | CLI Tool | Displays a terminal summary table of L-BTC balance, trusted PeerSwap peers, supported assets, channel balances, and fee rates. | +| `README.md` | Documentation | Bare-metal, RaspiBlitz, and MiniBolt setup guide for `elementsd` and `peerswapd`. | --- -### 2.4 `Other/` — Advanced Routing, Consolidations & Swap Automation +### 2.4 `Other/` - Fee Curves, Rebalance Guard, Swaps & UTXO Consolidation | File | Type | Description | | :--- | :--- | :--- | -| `fee_adjuster.py` | Systemd / Cronjob | **Advanced Liquidity Fee Controller**: Dynamically adjusts channel routing fees based on liquidity balance curves, applies progressive discounts for stuck outbound capacity, adds demand premiums, and protects newly opened channels with cooldown guards. | -| `lnd_utxo_consolidator.py` | CLI / Daemon | Monitors the mempool for low-fee windows (e.g. < 10 sat/vB) and consolidates small, fragmented LND wallet UTXOs into single high-value outputs to save future fees. | -| `boltz_swap-out.py` | CLI / Daemon | Performs trustless submarine swap-outs via the Boltz Exchange API, converting excess Lightning outbound liquidity into Liquid BTC (L-BTC). | -| `swap_out-loop.py` | CLI / Daemon | Submarine swap-out automation leveraging Lightning Labs Loop daemon (`loopd` / `loop out`). | -| `swap_wallet.py` | CLI Tool | Automated Lightning payment scheduler and non-custodial batch payout tool for wallet-to-wallet transfers with rate limits and fee caps. | +| `fee_adjuster.py` | Systemd Timer / Cronjob | **Dynamic Liquidity Fee Controller**: Adjusts outbound and inbound channel fees using Amboss Space statistics, 5-band liquidity curves, and stuck-channel decay, with `charge-lnd` export and inbound discount protection. Documented in `Other/fee_adjuster.md`; systemd installer in `Other/install_fee_adjuster_service.sh`. | +| `rebalance_guard.py` | Cronjob / CLI | **Standalone Rebalance Guard**: Audits all open LNDg channels across native Auto-Fees and `fee_adjuster.py`. Locks `ar_out_target = 100%` when an inbound discount is active and restores baseline targets (`data/rebalance_targets_baseline.json`) via dynamic hysteresis once liquidity recovers. | +| `swap_out-loop.py` | CLI Tool | **Economically Optimized Loop Out**: Ranks channels by net effective PPM, runs live fake-hash prepay route probes (with 2-hop direct Loop fallback), supports multi-channel MPP selection (`litloop` / `loopd`), interactive TTY multi-select, and native `loop_sqlite.db` history/CSV accounting. Documented in `Other/swap_out-loop.md`. | +| `boltz_swap-out.py` | CLI Tool | Executes reverse submarine swaps (Lightning to Liquid L-BTC) via `boltzd` / `boltzcli` using LNDg candidate channels, multi-batch payment retries, and `--debug` dry-run mode. | +| `swap_wallet.py` | CLI Tool | Batches Lightning payouts to a Lightning Address via `bos send` through high-local-balance peers under configurable fee caps. | +| `lnd_utxo_consolidator.py` | CLI Tool | Interactive helper that inspects `lncli listunspent` and builds a verified `lncli sendcoins` UTXO consolidation command for manual review (never broadcasts directly). | --- -### 2.5 `LNBits/` — Micro-Wallet & Allowance Management +### 2.5 `LNBits/` - Allowance Management | File | Type | Description | | :--- | :--- | :--- | -| `pocketmoney.py` | Cronjob / Systemd | Automated allowance daemon that distributes scheduled satoshi payments from a master LNbits wallet to child sub-wallets based on configurable fiat/crypto budgets. | +| `pocketmoney.py` | Systemd Timer / Daemon | Distributes scheduled fiat-denominated satoshi allowances from a parent LNbits wallet to child wallets defined in `wallets.json` (`--run-once` supported). Systemd timer installer in `LNBits/install_pocketmoney_service.sh`. | --- ## 3. Configuration Conventions -Configuration files follow standard INI (`configparser`) and JSON formats: +All user-specific configuration files are ignored by `.gitignore` and initialized from `.example` templates: -- `config.ini` / `config.ini.example`: Core repository configuration including: - - `[credentials]`: Amboss tokens, LNbits API keys. - - `[telegram]`: Telegram bot token and user/group chat IDs. - - `[paths]`: Path to `lncli`, `bos`, and LNDg database. - - `[magma]`: Invoice expiry, fee limits, and auto-approval settings. -- `Magma/magma_config.ini`: Granular offer templates, duration brackets, and capital reserve limits. -- `Other/feeConfig.json`: Liquidity curve points, discount steps, and channel tags for `fee_adjuster.py`. +- `config.ini` (from `config.ini.example`): Shared configuration across scripts (`[credentials]`, `[lndg]`, `[pubkey]`, `[no-swapout]`, `[paths]`, `[telegram]`, `[info]`, `[system]`, `[urls]`, `[parameters]`, `[magma]`, `[LNBits]`, `[Schedule]`, `[lnd]`, `[loop]`, `[database]`). +- `Magma/magma_config.ini` (from `Magma/magma_config.ini.example`): Offer templates, duration brackets, seller score filter, and capital allocation shares for `Magma/magma_market_fee.py`. +- `feeConfig.json` (from `feeConfig.json.example`): Peer groups, liquidity bands, stuck-channel settings, and `inbound_protection` thresholds for `Other/fee_adjuster.py` and `Other/rebalance_guard.py`. +- `wallets.json` (from `wallets.json.example`): Child wallet IDs, invoice keys, and fiat allowance amounts for `LNBits/pocketmoney.py`. --- ## 4. Testing & Quality Standards -- **Unit Testing Framework**: `pytest` with `pytest-mock` and `requests-mock`. -- **Test Locations**: `tests/` directory: - - `tests/LNDg/test_amboss_pull.py`: Magma orders sync to LNDg, mathematical SCID conversion fallback, Space API batch ID lookup, fee-cap clustering, and LNDg REST updates. -- `tests/Magma/test_magma_sale_process.py`: Sales order lifecycle, GraphQL queries/mutations, invoice generation, UTXO vbyte calculations, and Telegram alerts. - - `tests/Magma/test_magma_market_fee.py`: Public offers analysis, APR calculations, offer creation/updates/toggles, and dry-run simulations. - - `tests/test_fee_adjuster.py`: Liquidity curve calculations, stuck-channel discounts, and fee protections. -- **CI/CD Matrix**: `.github/workflows/tests.yml` executes on every push and pull request across Python `3.11`, `3.12`, and `3.13`. -- **Development Policy**: All new features and bugfixes must adhere to strict Test-Driven Development (TDD) and ensure 100% passing tests before merging. +- **Testing Stack**: `pytest`, `pytest-cov`, `pytest-mock`, and `requests-mock` (installed via `pip install -r requirements-dev.txt`, configured in `pyproject.toml`). +- **Test Suite (`tests/`)**: + - `tests/LNDg/test_amboss_pull.py`: Magma order sync to LNDg, SCID conversion fallback, Amboss Space batch lookup, fee-cap clustering, and REST updates. + - `tests/Magma/test_magma_sale_process.py`: Order lifecycle, GraphQL queries/mutations, `lncli addinvoice`, UTXO vbyte fee validation, and Telegram callbacks. + - `tests/Magma/test_magma_market_fee.py`: Public offer analysis, APR math, offer create/update/toggle logic, and `--dry-run` mode. + - `tests/Magma/test_magma_simulation.py` & `tests/simulate_magma_sale.py`: End-to-end offline and live-Telegram Magma order simulation harness. + - `tests/Other/test_swap_out_loop.py`: Economic PPM scoring, route prepay probing, 2-hop fallback, multi-channel batching, TTY UI, and `loop_sqlite.db` history/CSV export. + - `tests/test_amboss_graphql_contracts.py`: AST-level Amboss Magma and Space GraphQL field/input schema contract checks. + - `tests/test_ca_trust_hardening.py`: Enforces `certifi>=2026.10.19`, `requests>=2.34.2`, `urllib3>=2.8.0`, and OS CA trust bundle bindings in systemd installers. + - `tests/test_fee_adjuster.py`: Liquidity band adjustments, stuck-channel step-downs, inbound fee discount caps, `ar_out_target` locking, and Amboss error handling. + - `tests/test_rebalance_guard.py`: Cross-channel rebalance guard auditing and dynamic hysteresis baseline restoration. + - `tests/test_repo_hygiene.py`: LNbits URL normalization/fail-closed config checks, import-time side-effect guards, dependency hygiene, and `README.md` link integrity. +- **CI/CD Matrix**: `.github/workflows/tests.yml` runs on every push and pull request across Python `3.11`, `3.12`, and `3.13` with pip caching and `pytest-cov` reporting. +- **Development Policy**: Follow Test-Driven Development (TDD), maintain zero broken CI checks, and ensure all tests pass before merging. diff --git a/.github/SECURITY.md b/.github/SECURITY.md deleted file mode 120000 index 9d57138..0000000 --- a/.github/SECURITY.md +++ /dev/null @@ -1 +0,0 @@ -../SECURITY.md \ No newline at end of file diff --git a/.gitignore b/.gitignore index 9c32971..0d81b79 100644 --- a/.gitignore +++ b/.gitignore @@ -28,6 +28,7 @@ ENV/ # Configuration and Secrets config.ini +Magma/magma_config.ini feeConfig.json wallets.json .env diff --git a/LNBits/pocketmoney.py b/LNBits/pocketmoney.py index 2b5d596..7f4b72c 100644 --- a/LNBits/pocketmoney.py +++ b/LNBits/pocketmoney.py @@ -10,6 +10,14 @@ from telebot import TeleBot import argparse +def normalize_lnbits_base_url(raw_url): + """Normalize LNbits base URL so /api/v1/... endpoints are never doubled.""" + url = (raw_url or "").strip().rstrip("/") + if url.endswith("/api/v1"): + url = url[: -len("/api/v1")].rstrip("/") + return url + + # Get the path to the parent directory parent_dir = os.path.dirname(os.path.abspath(__file__)) @@ -19,15 +27,15 @@ config.read(config_file_path) # Parse scheduling preferences -recurrence_raw = config["Schedule"]["recurrence"] +recurrence_raw = config.get("Schedule", "recurrence", fallback="weekly") RECURRENCE = recurrence_raw.split("#")[0].strip().lower() # Verify RECURRENCE if RECURRENCE not in ["daily", "weekly"]: raise ValueError("Invalid recurrence setting. Choose from 'daily' or 'weekly'.") -TIME = config["Schedule"]["time"].strip() -DAY_OF_WEEK = config["Schedule"].get("day_of_week", "friday").strip() +TIME = config.get("Schedule", "time", fallback="09:00").strip() +DAY_OF_WEEK = config.get("Schedule", "day_of_week", fallback="friday").strip().lower() # Check if TIME is valid try: @@ -36,17 +44,18 @@ raise ValueError("Invalid time format. Use HH:MM (e.g., 09:00).") # Other configuration variables -ADMIN_KEY = config["LNBits"]["admin_key"] -LNBITS_URL = config["LNBits"]["base_url"] -TOKEN = config["telegram"]["lnbits_bot_token"] -CHAT_ID = config["telegram"]["telegram_user_id"] +ADMIN_KEY = config.get("LNBits", "admin_key", fallback="").strip() +LNBITS_URL = normalize_lnbits_base_url(config.get("LNBits", "base_url", fallback="")) +TOKEN = config.get("telegram", "lnbits_bot_token", fallback="").strip() +CHAT_ID = config.get("telegram", "telegram_user_id", fallback="").strip() THRESHOLD_BALANCE = 1000000 -# Initialize Telegram bot -bot = TeleBot(TOKEN) +# Initialize Telegram bot (safe placeholder when imported without config.ini in tests) +bot = TeleBot(TOKEN if (TOKEN and ":" in TOKEN) else "123456:fake_token_for_testing") # Set up logging log_file_path = os.path.join(os.path.dirname(__file__), "..", "logs", "pocketmoney.log") +os.makedirs(os.path.dirname(log_file_path), exist_ok=True) handler = RotatingFileHandler(log_file_path, maxBytes=10 * 1024 * 1024, backupCount=5) logging.basicConfig( level=logging.DEBUG, @@ -233,6 +242,20 @@ def schedule_transfers(): raise ValueError("Invalid recurrence setting. Choose from 'daily' or 'weekly'.") +def validate_runtime_config(): + missing = [] + if not LNBITS_URL: + missing.append("[LNBits] base_url") + if not ADMIN_KEY: + missing.append("[LNBits] admin_key") + if not TOKEN or ":" not in TOKEN: + missing.append("[telegram] lnbits_bot_token") + if not CHAT_ID: + missing.append("[telegram] telegram_user_id") + if missing: + raise ValueError(f"Missing or invalid config.ini settings: {', '.join(missing)}") + + def main(): parser = argparse.ArgumentParser(description="Pocket Money Scheduler") parser.add_argument( @@ -241,6 +264,7 @@ def main(): help="Run the transfer once and exit, best for cronjobs", ) args = parser.parse_args() + validate_runtime_config() if args.run_once: scheduled_transfer() diff --git a/LNDg/amboss_pull.py b/LNDg/amboss_pull.py index bdd2881..2e0b24f 100644 --- a/LNDg/amboss_pull.py +++ b/LNDg/amboss_pull.py @@ -36,7 +36,9 @@ LNDG_BASE_URL = config.get("lndg", "lndg_api_url", fallback="http://localhost:8889") # Output Paths -CHARGE_LND_PATH = config.get("paths", "charge_lnd_path", fallback="/tmp/charge-lnd") +CHARGE_LND_PATH = os.path.expanduser( + config.get("paths", "charge_lnd_path", fallback="/tmp/charge-lnd") +) FINISHED_FILE_NAME = "magma-finished.txt" FEE_CAP_FILE_PATTERN = re.compile(r"^magma-channels_(\d+)\.txt$") LOG_FILE_PATH = os.path.join(parent_dir, "..", "logs", "amboss-LNDg_changes.log") diff --git a/LNDg/channel_base-fee.py b/LNDg/channel_base-fee.py index 0188388..858177b 100644 --- a/LNDg/channel_base-fee.py +++ b/LNDg/channel_base-fee.py @@ -175,6 +175,7 @@ def modify_channels(channels): if response.status_code == 200: # Log the changes + os.makedirs(os.path.dirname(log_file_path), exist_ok=True) with open(log_file_path, "a") as log_file: log_file.write( f"{timestamp}: Changed 'local_base_fee' to 0 for channel {chan_id}\n" diff --git a/LNDg/channel_fee-pull.py b/LNDg/channel_fee-pull.py index 99be28c..d34e912 100644 --- a/LNDg/channel_fee-pull.py +++ b/LNDg/channel_fee-pull.py @@ -56,6 +56,7 @@ def get_chan_ids_to_write(): chan_ids = get_chan_ids_to_write() print(f"Channel-IDs: {chan_ids}") +os.makedirs(os.path.dirname(file_path), exist_ok=True) if chan_ids: with open(file_path, "w") as file: for chan_id in chan_ids: diff --git a/LNDg/disabled_fee-accelerator.py b/LNDg/disabled_fee-accelerator.py index 52cff0d..46047b9 100644 --- a/LNDg/disabled_fee-accelerator.py +++ b/LNDg/disabled_fee-accelerator.py @@ -48,6 +48,7 @@ # File path for the log file log_file_path = os.path.join(parent_dir, "..", "logs", "lndg-fee-accelerator.log") +os.makedirs(os.path.dirname(log_file_path), exist_ok=True) # Logfile definition logging.basicConfig(filename=log_file_path, level=logging.DEBUG) diff --git a/LNDg/mempool_rebalancer_trigger.py b/LNDg/mempool_rebalancer_trigger.py index d604389..eafefd6 100644 --- a/LNDg/mempool_rebalancer_trigger.py +++ b/LNDg/mempool_rebalancer_trigger.py @@ -63,6 +63,7 @@ def get_current_timestamp(): # Logfile definition log_file_path = os.path.join(parent_dir, "..", "logs", "mempool_rebalancer_trigger.log") +os.makedirs(os.path.dirname(log_file_path), exist_ok=True) logging.basicConfig(filename=log_file_path, level=logging.DEBUG) diff --git a/LNDg/swap_out-candidates.py b/LNDg/swap_out-candidates.py index 05adf1e..a9f109e 100644 --- a/LNDg/swap_out-candidates.py +++ b/LNDg/swap_out-candidates.py @@ -243,6 +243,7 @@ def write_bos_tags(): ] } + os.makedirs(os.path.dirname(file_path_to_bos), exist_ok=True) with open(file_path_to_bos, "w") as file: json.dump(tags_data, file, indent=2) @@ -262,6 +263,7 @@ def main(): if args.file_export: chan_ids = get_chan_ids_to_write() if chan_ids: + os.makedirs(os.path.dirname(file_path), exist_ok=True) with open(file_path, "w") as file: file.write(",".join(chan_ids) + "\n") print(f"Channel-ID data written to {file_path}") diff --git a/Magma/magma_config.ini b/Magma/magma_config.ini.example similarity index 94% rename from Magma/magma_config.ini rename to Magma/magma_config.ini.example index fd2b2a2..d2c4275 100644 --- a/Magma/magma_config.ini +++ b/Magma/magma_config.ini.example @@ -1,5 +1,5 @@ -# Magma/magma_config.ini -# This file contains Magma-specific settings for the magma_market_fee.py script. +# Magma/magma_config.ini.example +# Copy to Magma/magma_config.ini and customize settings for Magma/magma_market_fee.py. [magma_autoprice] # Enable/disable the magma auto-pricing script @@ -48,4 +48,4 @@ target_apr_min = 7.0 target_apr_max = 18.0 capital_allocation_share = 0.6 -# ... add more [magma_offer_TEMPLATENAME] sections as needed ... \ No newline at end of file +# ... add more [magma_offer_TEMPLATENAME] sections as needed ... diff --git a/Magma/magma_market_fee.py b/Magma/magma_market_fee.py index b1123c8..bc815ec 100644 --- a/Magma/magma_market_fee.py +++ b/Magma/magma_market_fee.py @@ -22,10 +22,10 @@ # - Includes a --dry-run mode to simulate actions without making live API changes. # # How to Run: -# 1. Ensure Python 3 is installed. -# 2. Install required Python packages: `pip install requests telebot configparser` +# 1. Ensure Python 3.11+ is installed. +# 2. Install required Python packages: `pip install -r requirements.txt` # 3. Ensure `../config.ini` exists and is configured (see config.ini.example). -# 4. Create `Magma/magma_config.ini` for Magma-specific settings. +# 4. Copy `Magma/magma_config.ini.example` to `Magma/magma_config.ini` and customize offer settings. # 5. Make the script executable: `chmod +x magma_market_fee.py` # 6. Run the script: `python /path/to/Magma/magma_market_fee.py` # For dry run: `python /path/to/Magma/magma_market_fee.py --dry-run` @@ -1338,7 +1338,8 @@ def main(): if not os.path.exists(MAGMA_CONFIG_FILE_PATH): logging.critical( - f"CRITICAL: Magma-specific configuration file {MAGMA_CONFIG_FILE_PATH} not found. Exiting." + f"CRITICAL: Magma-specific configuration file {MAGMA_CONFIG_FILE_PATH} not found " + "(copy Magma/magma_config.ini.example to Magma/magma_config.ini). Exiting." ) return magma_specific_config.read(MAGMA_CONFIG_FILE_PATH) diff --git a/Magma/magma_sale.MD b/Magma/magma_sale.MD deleted file mode 100644 index a708650..0000000 --- a/Magma/magma_sale.MD +++ /dev/null @@ -1,227 +0,0 @@ -# Running Magma Auto-Sale Script with systemd - -This document outlines how to run the `magma_sale_process.py` script as a persistent background service using `systemd` on a Linux system. This ensures the script automatically starts on boot and restarts if it encounters issues, utilizing a Python virtual environment for dependency management. - -## Prerequisites - -1. **Python 3 & virtualenv:** Ensure Python 3 and the `virtualenv` package are installed on your system. - ```bash - sudo apt update - sudo apt install python3 python3-pip virtualenv - ``` -2. **Project Setup:** - * Clone or download the `Lightning-Python-Tools` repository if you haven't already. - ```bash - # Example: - # git clone https://github.com/TrezorHannes/Lightning-Python-Tools.git - # cd Lightning-Python-Tools - ``` - * Navigate to the root of the `Lightning-Python-Tools` directory. - -3. **Create and Activate Virtual Environment:** - It's highly recommended to use a Python virtual environment. - ```bash - # In the Lightning-Python-Tools directory - virtualenv -p python3 .venv - source .venv/bin/activate - ``` - Your shell prompt should change to indicate the virtual environment is active (e.g., `(.venv) your-user@host:...$`). - -4. **Install Required Python Packages:** - With the virtual environment active, install the necessary packages: - ```bash - # Ensure your virtual environment is active - pip install pyTelegramBotAPI requests schedule - ``` - (Note: If there's a `requirements.txt` specific to Magma or the broader project that includes these, you might use `pip install -r requirements.txt` instead). - -5. **Script Location:** The `magma_sale_process.py` script is expected to be in the `Magma/` subdirectory (e.g., `/opt/lightning-tools/Magma/magma_sale_process.py`). - -6. **Configuration File:** - * Copy `config.ini.example` to `config.ini` in the root of the `Lightning-Python-Tools` directory (e.g. `/opt/lightning-tools/config.ini`). - * Fill in all required details in `config.ini` (API tokens, Telegram bot info, LND/BOS paths, etc.). - -7. **LND & BOS (Optional):** Ensure `lncli` (and `bos` if used) are accessible, either in the system's PATH or by providing the full path in `config.ini`. -8. **Log Directory:** The script will attempt to create a `logs` directory in its parent directory (e.g., `/opt/lightning-tools/logs/`). Ensure the user running the script has write permissions there. - -## systemd Service File - -Create a systemd service file to manage the script. - -1. **Create the service file:** - Open a new file named `magma-sale.service` in `/etc/systemd/system/`: - ```bash - sudo nano /etc/systemd/system/magma-sale.service - ``` - -2. **Paste the following content:** - Adjust `User`, `Group`, `WorkingDirectory`, and `ExecStart` paths according to your setup. The `ExecStart` path **must** point to the Python interpreter inside your virtual environment. - - ```ini - [Unit] - Description=Magma Channel Auto-Sale Service - # Add other dependencies if LND runs as a service - After=network.target LND.service - - [Service] - Type=simple - # Replace with the user that should run the script - User=your_username - # Replace with the group for the user - Group=your_groupname - - # Set the working directory to the root of your Lightning-Python-Tools project - # This ensures relative paths like config.ini and logs/ work correctly, - # and that Magma/magma_sale_process.py can be found. - # Example: if your project is at /opt/lightning-tools/ - WorkingDirectory=/opt/lightning-tools/ - Environment="SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt" - Environment="REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt" - - # Path to the Python interpreter IN YOUR VIRTUAL ENVIRONMENT and the script - # Example: /opt/lightning-tools/.venv/bin/python3 /opt/lightning-tools/Magma/magma_sale_process.py - ExecStart=/opt/lightning-tools/.venv/bin/python3 /opt/lightning-tools/Magma/magma_sale_process.py - - Restart=always - RestartSec=10 - StandardOutput=journal - StandardError=journal - SyslogIdentifier=magma-sale - - [Install] - WantedBy=multi-user.target - ``` - - **Notes on the service file:** - * `User` and `Group`: It's recommended to run services under a dedicated non-root user. - * `WorkingDirectory`: Crucial for the script to find `config.ini` (in this directory) and the `logs` directory (also in this directory), and the script itself in the `Magma/` subdirectory. If your project is at `/home/admin/tools/Lightning-Python-Tools/`, then `WorkingDirectory` should be `/home/admin/tools/Lightning-Python-Tools/`. - * `ExecStart`: **Critically important.** This must be the absolute path to the `python3` executable *within your `.venv` directory*, followed by the absolute path to `Magma/magma_sale_process.py`. - * `After=network.target LND.service`: If your LND node runs as a systemd service (e.g., `LND.service`), adding it here ensures Magma starts after LND. Adjust service name if needed. - * `Restart=always`: Ensures the script restarts automatically. - * `StandardOutput=journal` and `StandardError=journal`: Directs script output to the systemd journal. - -3. **Save and close the file.** - -## Enabling and Managing the Service - -1. **Reload systemd daemon:** - This makes systemd aware of the new service file. - ```bash - sudo systemctl daemon-reload - ``` - -2. **Enable the service:** - This makes the service start automatically on boot. - ```bash - sudo systemctl enable magma-sale.service - ``` - -3. **Start the service:** - ```bash - sudo systemctl start magma-sale.service - ``` - -4. **Check the status of the service:** - ```bash - sudo systemctl status magma-sale.service - ``` - You should see output indicating the service is active (running). - -5. **View logs:** - To view the logs generated by the script (which are now directed to the journal): - ```bash - sudo journalctl -u magma-sale.service -f - ``` - (Use `-f` to follow the logs in real-time. Remove `-f` to see all past logs. Add `-n LINES` (e.g., `-n 100`) to see the last N lines.) - -## Stopping or Restarting the Service - -* **To stop the service:** - ```bash - sudo systemctl stop magma-sale.service - ``` -* **To restart the service (e.g., after updating the script or config.ini):** - ```bash - sudo systemctl restart magma-sale.service - ``` - -## Troubleshooting - -### Critical Error Flag Halts Bot - -The script uses a critical error flag file (`logs/magma_sale_process-critical-error.flag` relative to `WorkingDirectory`) to halt operations if a systemic or unrecoverable error occurs. This requires manual intervention. - -**Symptoms:** -* The bot sends a Telegram message like "☠️ Magma Bot STARTUP FAILED: Critical error flag found..." -* The `magma-sale.service` status or logs (via `journalctl -u magma-sale.service`) will indicate that the flag file exists and the bot has suspended operations. - -**Resolution:** -1. **Investigate the Logs:** Check the script's log file (`logs/magma-sale-process.log`) and the systemd journal (`journalctl -u magma-sale.service`) for details about the error that caused the flag to be created. -2. **Resolve the Underlying Issue:** Address the root cause of the error. This might involve code changes, configuration updates, or fixing external dependencies. -3. **Delete the Flag File:** Once the issue is believed to be resolved, manually delete the flag file. - ```bash - # Example, adjust path if your WorkingDirectory is different - rm /opt/lightning-tools/logs/magma_sale_process-critical-error.flag - # Or from within the WorkingDirectory: - # rm logs/magma_sale_process-critical-error.flag - ``` -4. **Restart the Service:** - ```bash - sudo systemctl restart magma-sale.service - ``` - -This setup provides a robust way to run your Magma auto-sale script using best practices for Python environments. Remember to adjust paths and usernames in the example service file to match your specific environment. - -## Telegram Bot Setup - -To allow manual triggering of the Magma sale process via Telegram, you need to configure your bot with BotFather. - -1. **Open Telegram and find BotFather:** If you haven't already, start a chat with `@BotFather`. -2. **Edit your bot's commands:** - * Send `/mybots` to BotFather and select your Magma bot from the list. - * Click on "Edit Bot". - * Click on "Edit Commands". - -3. **Set the command:** - BotFather will ask you to send the list of commands. For this script, you only need to register one command. Send the following text to BotFather: - - ``` - processmagmaorders - Manually checks for new Magma offers and processes paid orders. - ``` - - * `processmagmaorders` is the command itself (without the leading `/`). - * The text after the hyphen is the description that users will see in Telegram when they type `/`. - -Once set, users (specifically the `telegram_user_id` configured in your `config.ini`) will be able to use the `/runnow` command to trigger the script's main processing logic on demand. The script also uses inline keyboard buttons for approving or rejecting new offers; these do not need to be registered as commands with BotFather as they are contextually generated by the bot. - -## Simulating a Magma Buy Order End-to-End - -Because Magma sales are scarce per quarter and Amboss does not offer a sandbox dry-run environment on mainnet, an end-to-end simulation harness is provided in `tests/simulate_magma_sale.py`. - -The simulation tests the full order lifecycle safely without opening channels or charging real sats: -1. **Timeout Auto-Approval**: Ingests an order in `WAITING_FOR_SELLER_APPROVAL` with Amboss unsettled zero-fee schema (`fees.seller.sats == 0`), verifies fixed/variable fee fallback calculation, verifies queueing in `pending_user_confirmations`, triggers timeout auto-approval without `KeyError`, generates the seller invoice, and executes the Amboss acceptance flow. -2. **Manual Inline Callback Approval**: Ingests the offer, simulates the user clicking the `[✅ Approve Offer]` inline keyboard button, acknowledges callback, edits message text, and executes the approval pipeline. -3. **Telegram Error Logging**: Dispatches a test exception to ensure `TelegramLoggingHandler` formats and streams traceback alerts to your Telegram chat. - -### Running Offline Automated Simulation (Mock Mode) -Safe for CI and local automated testing without touching Telegram or external networks: - -```bash -# Run full simulation suite -.venv/bin/python tests/simulate_magma_sale.py - -# Or via pytest -.venv/bin/pytest tests/Magma/test_magma_simulation.py -``` - -### Running Live Interactive Telegram Simulation -To verify how prompts, timeout updates, and error alerts actually look and behave in your Telegram app, use the `--live-telegram` flag. This uses your `config.ini` bot credentials while safely mocking Amboss mutations and LND invoice creation: - -```bash -# Live interactive test with a 8-second timeout countdown -.venv/bin/python tests/simulate_magma_sale.py --live-telegram --timeout-seconds 8 - -# Live test of only the Telegram error logger stream -.venv/bin/python tests/simulate_magma_sale.py --live-telegram --scenario error -``` - diff --git a/Magma/magma_sale.md b/Magma/magma_sale.md new file mode 100644 index 0000000..86fc074 --- /dev/null +++ b/Magma/magma_sale.md @@ -0,0 +1,175 @@ +# Running Magma Auto-Sale Script with systemd + +This document outlines how to run `Magma/magma_sale_process.py` as a persistent background service using `systemd` on Linux. Running under `systemd` starts the daemon automatically on boot and restarts it on transient failures while isolating dependencies inside the project virtual environment. + +## Prerequisites + +1. **Python 3.11+ & `venv`**: + ```bash + sudo apt update + sudo apt install python3 python3-venv python3-pip + ``` + +2. **Project Setup**: + Clone `Lightning-Python-Tools` and enter the repository root: + ```bash + git clone https://github.com/TrezorHannes/Lightning-Python-Tools.git + cd Lightning-Python-Tools + ``` + +3. **Create and Activate Virtual Environment**: + ```bash + python3 -m venv .venv + source .venv/bin/activate + ``` + +4. **Install Required Python Packages**: + Install the pinned dependencies from `requirements.txt` (including hardened `certifi` and `urllib3` versions): + ```bash + pip install --upgrade pip + pip install -r requirements.txt + ``` + +5. **Configuration File**: + - Copy `config.ini.example` to `config.ini` in the root of `Lightning-Python-Tools`: + ```bash + cp config.ini.example config.ini + chmod 600 config.ini + ``` + - Populate `[credentials]` (`amboss_authorization`), `[telegram]` (`magma_bot_token`, `telegram_user_id`), `[paths]` (`lncli_path`), `[system]` (`full_path_bos`, `log_level`), `[pubkey]` (`banned_magma_pubkeys`), and `[magma]`. + +6. **LND & BOS (Optional)**: + Ensure `lncli` (and optionally `bos`) are accessible via the paths configured in `config.ini`. + +7. **Log Directory**: + The script automatically creates `logs/` in the repository root and writes rotating logs to `logs/magma-sale-process.log`. Ensure the service user has write permissions to the repository directory. + +## systemd Service File + +1. **Create the service file**: + ```bash + sudo nano /etc/systemd/system/magma-sale.service + ``` + +2. **Paste the following content** (adjust `User`, `Group`, `WorkingDirectory`, `After`, and `ExecStart` to match your host): + ```ini + [Unit] + Description=Magma Channel Auto-Sale Service + After=network.target lnd.service + + [Service] + Type=simple + User=your_username + Group=your_groupname + + # Repository root directory (required so config.ini and logs/ resolve cleanly) + WorkingDirectory=/opt/lightning-tools + Environment="SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt" + Environment="REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt" + + # Absolute path to .venv python3 and Magma/magma_sale_process.py + ExecStart=/opt/lightning-tools/.venv/bin/python3 /opt/lightning-tools/Magma/magma_sale_process.py + + Restart=always + RestartSec=10 + StandardOutput=journal + StandardError=journal + SyslogIdentifier=magma-sale + + [Install] + WantedBy=multi-user.target + ``` + + **Service file notes**: + - `User` and `Group`: Run under a dedicated non-root user with access to `lncli` macaroons/TLS cert. + - `WorkingDirectory`: Set to the root of `Lightning-Python-Tools` so `config.ini` and `logs/` resolve cleanly. + - `Environment`: Binds `SSL_CERT_FILE` and `REQUESTS_CA_BUNDLE` to the OS CA trust bundle (`/etc/ssl/certs/ca-certificates.crt`). + - `ExecStart`: Must point to `.venv/bin/python3` inside your virtual environment. + - `After=network.target lnd.service`: Adjust `lnd.service` if your LND unit uses a different name (for example `LND.service`). + +## Enabling and Managing the Service + +```bash +# Reload systemd unit definitions +sudo systemctl daemon-reload + +# Enable on boot and start immediately +sudo systemctl enable --now magma-sale.service + +# Check service status +sudo systemctl status magma-sale.service + +# Follow live logs +sudo journalctl -u magma-sale.service -f +``` + +### Stopping or Restarting + +```bash +sudo systemctl stop magma-sale.service +sudo systemctl restart magma-sale.service +``` + +## Troubleshooting + +### Critical Error Flag Halts Bot + +`magma_sale_process.py` writes a critical error flag file (`logs/magma_sale_process-critical-error.flag` relative to the repository root) to halt automated execution if an unrecoverable error occurs (such as a failed post-open Amboss channel point confirmation). + +**Symptoms**: +- Telegram alert indicating startup or order processing halted due to a critical error flag. +- `journalctl -u magma-sale.service` reports that the critical error flag file exists and operations are suspended. + +**Resolution**: +1. Inspect `logs/magma-sale-process.log` and `sudo journalctl -u magma-sale.service -n 100` to identify the root cause. +2. Resolve the underlying issue (for example, verify channel point state on Amboss or LND connectivity). +3. Remove the flag file: + ```bash + rm /opt/lightning-tools/logs/magma_sale_process-critical-error.flag + ``` +4. Restart the service: + ```bash + sudo systemctl restart magma-sale.service + ``` + +## Telegram Bot Setup + +To trigger order checks manually from Telegram, register the command with `@BotFather`: + +1. Open Telegram, chat with `@BotFather`, send `/mybots`, and select your Magma bot. +2. Choose **Edit Bot** -> **Edit Commands** and send: + ```text + processmagmaorders - Manually checks for new Magma offers and processes paid orders. + ``` +3. Once set, the authorized user (`telegram_user_id` in `config.ini`) can send `/processmagmaorders` in Telegram to trigger an immediate order check cycle. Inline approval and rejection buttons for incoming orders are generated dynamically by the bot and require no extra BotFather configuration. + +## Simulating a Magma Buy Order End-to-End + +Because Magma sales are infrequent and Amboss does not provide a mainnet sandbox, an end-to-end simulation harness is included in `tests/simulate_magma_sale.py`. + +The simulation exercises the order lifecycle without opening channels or spending real funds: +1. **Timeout Auto-Approval**: Ingests an order in `WAITING_FOR_SELLER_APPROVAL` with Amboss's unsettled zero-fee schema (`fees.seller.sats == 0`), verifies fee fallback calculation, queues confirmation, triggers timeout auto-approval, generates the seller invoice, and runs the Amboss acceptance flow. +2. **Manual Inline Callback Approval**: Simulates clicking the inline approval button, acknowledges the callback, updates message text, and runs the approval pipeline. +3. **Telegram Error Logging**: Dispatches a test exception to verify `TelegramLoggingHandler` traceback delivery. + +### Offline Automated Simulation (Mock Mode) +Safe for CI and local testing without touching Telegram or external networks: + +```bash +# Run standalone simulation suite +.venv/bin/python3 tests/simulate_magma_sale.py + +# Or via pytest +.venv/bin/pytest tests/Magma/test_magma_simulation.py +``` + +### Live Interactive Telegram Simulation +To test live prompts, timeout countdowns, and error alerts in your Telegram client while keeping Amboss mutations and `lncli` calls mocked: + +```bash +# Live interactive test with an 8-second timeout countdown +.venv/bin/python3 tests/simulate_magma_sale.py --live-telegram --timeout-seconds 8 + +# Live test of only the Telegram error logger stream +.venv/bin/python3 tests/simulate_magma_sale.py --live-telegram --scenario error +``` diff --git a/Magma/magma_sale_process.py b/Magma/magma_sale_process.py index de9d934..d4460b1 100644 --- a/Magma/magma_sale_process.py +++ b/Magma/magma_sale_process.py @@ -26,8 +26,8 @@ # - (Optional) Attempts to record income via BOS `send` command after successful channel open. # # How to Run: -# 1. Ensure Python 3 is installed. -# 2. Install required Python packages: `pip install pyTelegramBotAPI requests schedule` +# 1. Ensure Python 3.11+ is installed. +# 2. Install required Python packages: `pip install -r requirements.txt` # 3. Copy `config.ini.example` to `config.ini` in the script's parent directory and fill in # all necessary details (API tokens, Telegram bot info, LND/BOS paths, etc.). # 4. Make the script executable: `chmod +x magma_sale_process.py` @@ -46,7 +46,7 @@ # - `pyTelegramBotAPI` library # - `schedule` library -# Import Lybraries +# Import Libraries import requests import telebot import json diff --git a/Other/fee_adjuster.md b/Other/fee_adjuster.md index 03c08cb..b438dc4 100644 --- a/Other/fee_adjuster.md +++ b/Other/fee_adjuster.md @@ -1,91 +1,85 @@ -## Guidelines for the Fee Adjuster Script - -This script automates the adjustment of channel fees based on network conditions, peer behavior, -and local liquidity using data from the Amboss API and LNDg API. - -### Configuration Settings: -- base_adjustment_percentage: Percentage adjustment applied to the selected fee base (median, mean, etc.). -- group_adjustment_percentage: Additional adjustment based on node group membership. -- max_cap: Maximum allowed fee rate (in ppm). -- trend_sensitivity: Multiplier for the influence of Amboss fee trends on the adjustment. -- fee_base: Statistical measure from Amboss used as the fee calculation base ("median", "mean", etc.). -- groups: Node categories for applying differentiated strategies via group_adjustment_percentage. -- fee_bands: (Optional) Dynamic fee adjustments based on local liquidity. - - enabled: true/false. - - discount: Negative percentage adjustment for high local balance (80-100%). - - premium: Positive percentage adjustment for low local balance (0-40%). -- stuck_channel_adjustment: (Optional) Gradually reduces fees for channels without recent forwards. - - enabled: true/false. - - stuck_time_period: Number of days defining one 'stuck period' interval (e.g., 7). -- inbound_protection: (Optional) Configures protection against unprofitable rebalance feedback loops. - - enabled: true/false. - - max_inbound_discount_ppm: Hard cap on negative inbound fee discounts (default: 250 ppm). - - lock_ar_out_target_on_discount: Automatically sets ar_out_target = 100% when an inbound discount is active to prevent rebalance drainage. - - default_restored_ar_out_target: Restores ar_out_target (e.g. 75%) once channel balance is restored and discount is removed. - - restore_liquidity_threshold: Local liquidity ratio required before unlocking (default: 75.0%). - -### Inbound Discount Protection & Rebalance Guard: -When inbound discounts are offered on depleted channels, `fee_adjuster.py` and `rebalance_guard.py` ensure: -1. Inbound discounts are capped at `max_inbound_discount_ppm` to prevent routing payments at net losses against exit channels. -2. The channel is locked (`ar_out_target = 100%`) while offering discounts so LNDg will not cannibalize refilling liquidity as a rebalance donor. -3. When local liquidity recovers and discounts are lifted, standard `ar_out_target` settings are restored. - -### Standalone Rebalance Guard Tool: -`rebalance_guard.py` audits all open channels in LNDg across both native `af.py` and `fee_adjuster.py` channels: -```bash -# Dry run check -python3 Other/rebalance_guard.py --dry-run +# Guidelines for the Fee Adjuster Script (`Other/fee_adjuster.py`) -# Live execution -python3 Other/rebalance_guard.py -``` -Allows tailored fee strategies for nodes in specific categories (e.g., "sink", "expensive"). +`Other/fee_adjuster.py` automates outbound and inbound channel fee adjustments based on Amboss Space network statistics, forward activity, and local liquidity balances via the Amboss GraphQL API and LNDg REST API. -### Fee Bands: -Adjusts fees based on local balance ratio, dividing liquidity into 5 bands (0-20%, 20-40%, 40-60%, 60-80%, 80-100%). A graduated adjustment is applied between the configured discount (high local balance) and premium (low local balance). The premium is capped at the 20-40% liquidity band to avoid excessively high fees on nearly drained channels. +## Configuration (`feeConfig.json`) -### Stuck Channel Adjustment: -This feature incrementally reduces fees for channels that haven't forwarded payments recently. -For each multiple of the `stuck_time_period` (in days) that a peer's channels have gone without an *outbound* forwarding, the fee band is moved down by one level (towards the maximum discount). -The adjustment is capped at moving down 4 bands (reaching the maximum discount band). -If an outbound forward is detected for any channel of the peer, the stuck adjustment is reset to 0 bands down. -This adjustment is automatically skipped if the aggregate local liquidity for the peer is below 20%, preventing discounts on heavily imbalanced channels needing rebalancing. The script queries the LNDg API to find the timestamp of the last outbound forward for the peer. +Copy `feeConfig.json.example` to `feeConfig.json` in the repository root: -### Usage: -- Configure nodes and their settings in `feeConfig.json`. -- Run the script to automatically adjust fees based on configured settings. -- Requires a running LNDg instance for local channel details and fee updates. +- `Terminal_output`: Print adjustment summary tables to stdout (`true`/`false`). +- `LNDg_fee_update`: Push calculated outbound/inbound fees directly to the LNDg API (`true`/`false`). +- `write_charge_lnd_file`: Export rule stanzas to `/fee_adjuster.txt` (`true`/`false`). +- `update_channel_notes`: Write fee calculation summaries into LNDg channel notes (`true`/`false`). +- `inbound_protection`: Protects channels from unprofitable circular rebalance loops: + - `lock_ar_out_target_on_discount`: Locks `ar_out_target` to `lock_target` whenever a negative inbound fee discount is active (`default: true`). + - `lock_target`: Target outbound auto-rebalance percentage applied while locked (`default: 100`). + - `default_restored_ar_out_target`: Fallback `ar_out_target` restored if no prior baseline was recorded (`default: 75`). + - `restore_liquidity_threshold`: Minimum local liquidity percentage required before unlocking (`default: 75.0`). + - `max_inbound_discount_ppm`: Optional cap on negative inbound fee discounts in ppm (`null` by default for uncapped, or set e.g. `250` to cap at `-250 ppm`). +- `groups`: Named peer strategy profiles (for example `sink`, `expensive`) defining: + - `group_adjustment_percentage`: Multiplier offset applied to the selected `fee_base`. + - `max_cap`: Maximum allowed outbound fee rate in ppm. + - `trend_sensitivity`: Multiplier weighting Amboss fee trend deltas. + - `fee_base`: Statistical measure from Amboss (`"median"`, `"mean"`, etc.). + - `fee_delta_threshold`: Minimum change in ppm required before pushing an update to LNDg. + - `inbound_auto_fee_enabled`: Enables dynamic negative inbound fee discounts scaled by LNDg `ar_max_cost` in low-liquidity bands (`true`/`false`). + - `fee_bands`: Liquidity-based fee adjustments across 5 balance bands (`0-20%`, `20-40%`, `40-60%`, `60-80%`, `80-100%`): + - `enabled`: `true`/`false`. + - `discount`: Negative percentage adjustment for high local balance (`80-100%`). + - `premium`: Positive percentage adjustment for low local balance (capped at the `20-40%` band). + - `stuck_channel_adjustment`: Progressively steps down the liquidity band for channels without recent outbound forwards: + - `enabled`: `true`/`false`. + - `stuck_time_period`: Days without an outbound forward per 1-band step-down (capped at 4 bands). + - `min_local_balance_for_stuck_discount`: Minimum aggregate peer local liquidity ratio (for example `0.2` for `20%`) required before applying stuck discounts. + - `min_updates_for_discount`: Minimum channel `num_updates` threshold to avoid discounting newly opened channels. +- `nodes`: Per-peer list mapping `pubkey` to a `group` (with optional per-node overrides or custom `fee_conditions`). -### Test Suite: -New features and refactors are guarded by a suite of unit tests. To run them locally: +## Inbound Discount Protection & Standalone Rebalance Guard -```bash -# Activate your venv first if not active -source .venv/bin/activate +When inbound discounts are active on depleted channels, `Other/fee_adjuster.py` and `Other/rebalance_guard.py` coordinate to prevent rebalance cannibalization: +1. Optional `max_inbound_discount_ppm` caps negative inbound discounts so transit forwards do not route at a net loss. +2. Discounted channels are locked (`ar_out_target = 100%`), recording their prior baseline target in `data/rebalance_targets_baseline.json`. +3. Once local liquidity recovers to `restore_liquidity_threshold` and the inbound discount is lifted, the channel's baseline `ar_out_target` is automatically restored. -# Install test dependencies -pip install -r requirements-dev.txt +Run the standalone guard across all open LNDg channels (covering both native LNDg Auto-Fees and `fee_adjuster.py`): +```bash +# Dry-run audit +.venv/bin/python3 Other/rebalance_guard.py --dry-run -# Run the tests (pytest auto-discovers tests in the current directory) -pytest +# Live execution +.venv/bin/python3 Other/rebalance_guard.py ``` -### Command Line Arguments: -- --debug: Enable detailed debug output, including stuck channel check results. +## Command-Line Arguments + +- `--debug`: Prints detailed fee and stuck-channel diagnostics while skipping live LNDg API mutations and `charge-lnd` file writes. + +## `charge-lnd` Integration -### Charge-lnd Details: -Configure charge-lnd to use the output file: +Ensure `charge_lnd_path` is set under `[paths]` in `config.ini` (defaults to `~/.chargelnd/.config`), and reference the generated file in `charge-lnd.config`: ```ini -# charge-lnd.config -[🤖 FeeAdjuster Import] +[FeeAdjuster Import] strategy = use_config -config_file = file:///path/to/your/charge-lnd/.config/fee_adjuster.txt +config_file = file:///home/admin/.chargelnd/.config/fee_adjuster.txt ``` -### Installation: -Add to crontab or use the systemd installer script: +## Installation (systemd Timer or Crontab) + +Use the included installer script to set up `fee_adjuster.service` and an hourly `fee_adjuster.timer`: +```bash +chmod +x Other/install_fee_adjuster_service.sh +sudo ./Other/install_fee_adjuster_service.sh +``` + +Or schedule via `crontab -e`: +```cron +0 * * * * /path/to/Lightning-Python-Tools/.venv/bin/python3 /path/to/Lightning-Python-Tools/Other/fee_adjuster.py >> /path/to/Lightning-Python-Tools/logs/cron_fee.log 2>&1 +``` + +## Running Tests + ```bash -crontab -e -# Add the following line (adjust path as needed) -0 * * * * /path/Lightning-Python-Tools/.venv/bin/python3 /path/Lightning-Python-Tools/Other/fee_adjuster.py >/dev/null 2>&1 -``` \ No newline at end of file +source .venv/bin/activate +pip install -r requirements-dev.txt +pytest tests/test_fee_adjuster.py tests/test_rebalance_guard.py -v +``` diff --git a/Other/fee_adjuster.py b/Other/fee_adjuster.py index 8aeb2cd..3c4bff3 100644 --- a/Other/fee_adjuster.py +++ b/Other/fee_adjuster.py @@ -121,6 +121,7 @@ def __init__(self, message, status_code=None, response_data=None): # File path for the log file log_file_path = os.path.join(parent_dir, "..", "logs", "fee-adjuster.log") +os.makedirs(os.path.dirname(log_file_path), exist_ok=True) # Logfile definition logging.basicConfig(filename=log_file_path, level=logging.DEBUG) @@ -1824,8 +1825,14 @@ def main(): and updated_any_channel and not skip_charge_lnd_file_write ): + charge_lnd_dir = os.path.expanduser( + config.get( + "paths", "charge_lnd_path", fallback="~/.chargelnd/.config" + ) + ) + os.makedirs(charge_lnd_dir, exist_ok=True) charge_lnd_file_path = os.path.join( - config["paths"]["charge_lnd_path"], "fee_adjuster.txt" + charge_lnd_dir, "fee_adjuster.txt" ) # Pass the inbound fee details to write_charge_lnd_file # calculated_inbound_ppm_for_peer is the peer-level summary calculated earlier diff --git a/Other/rebalance_guard.py b/Other/rebalance_guard.py index 3c89e31..56003dc 100644 --- a/Other/rebalance_guard.py +++ b/Other/rebalance_guard.py @@ -26,6 +26,7 @@ config_file_path = os.path.join(parent_dir, "..", "config.ini") fee_config_file_path = os.path.join(parent_dir, "..", "feeConfig.json") log_file_path = os.path.join(parent_dir, "..", "logs", "rebalance-guard.log") +os.makedirs(os.path.dirname(log_file_path), exist_ok=True) # Setup logging logging.basicConfig( diff --git a/Peerswap/README.md b/Peerswap/README.md index 1ac6b18..c005174 100644 --- a/Peerswap/README.md +++ b/Peerswap/README.md @@ -346,7 +346,7 @@ $ rm elements-backup.dat elements-wallet.tar.gz There is [a frontend](https://github.com/Impa10r/peerswap-web), which I haven't installed since I prefer the commandline. But it looks great and continously gets improved **Tools** -I'm working on a couple of python tools to make things easier to manage. This guide, as well as those scripts, [can be found here on my GH repo **[Peerswap-Python-Tools]**](https://github.com/TrezorHannes/Peerswap-Python-Tools). +I'm working on a couple of python tools to make things easier to manage. This guide, as well as those scripts, [can be found here in **Lightning-Python-Tools/Peerswap**](https://github.com/TrezorHannes/Lightning-Python-Tools/tree/main/Peerswap). Lastly, I'm not in any way affiliated. I just think it's another tool in our set of armory to ensure a healthy and profitable ecosystem in lightning is possible for every runner out there. If this guide was of any help, I'd appreciate if you share the article with others, give me a follow on X [![Twitter URL](https://img.shields.io/twitter/url/https/twitter.com/HandsdownI.svg?style=social&label=Follow%20%40HodlmeTight1337)](https://twitter.com/HodlmeTight1337) or [nostr](https://njump.me/npub1ch25m5lkk8kfepr63f0jnpd9te8l9f585pfpr2g2ma4pre9rmlrqlu0yjy), perhaps even donating some sats to [hakuna@getalby.com](https://getalby.com/p/hakuna) diff --git a/Peerswap/peerswap-bot.py b/Peerswap/peerswap-bot.py index 21ab684..ec36750 100644 --- a/Peerswap/peerswap-bot.py +++ b/Peerswap/peerswap-bot.py @@ -40,6 +40,7 @@ # Logging log_file_path = os.path.join(parent_dir, '..', 'logs', 'peerswap-bot.log') +os.makedirs(os.path.dirname(log_file_path), exist_ok=True) logging.basicConfig(filename=log_file_path, level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s') bot = telebot.TeleBot(BOT_TOKEN) diff --git a/Peerswap/peerswap-lndg_push.py b/Peerswap/peerswap-lndg_push.py index 0cb627c..fd1b0f8 100644 --- a/Peerswap/peerswap-lndg_push.py +++ b/Peerswap/peerswap-lndg_push.py @@ -35,6 +35,7 @@ # File path for the log file log_file_path = os.path.join(parent_dir, "..", "logs", "peerswap-LNDg_changes.log") +os.makedirs(os.path.dirname(log_file_path), exist_ok=True) # Logfile definition logging.basicConfig(filename=log_file_path, level=logging.DEBUG) diff --git a/Peerswap/ps_peers.py b/Peerswap/ps_peers.py index 2ae119c..29703d5 100644 --- a/Peerswap/ps_peers.py +++ b/Peerswap/ps_peers.py @@ -93,6 +93,6 @@ def main(): print(table) -get_lbtc_balance() + if __name__ == "__main__": - main() \ No newline at end of file + main() diff --git a/README.md b/README.md index 0918aca..9117184 100644 --- a/README.md +++ b/README.md @@ -1,90 +1,93 @@ -# ⚡ Lightning Python Tools +# Lightning Python Tools [![Tests](https://github.com/TrezorHannes/Lightning-Python-Tools/actions/workflows/tests.yml/badge.svg)](https://github.com/TrezorHannes/Lightning-Python-Tools/actions/workflows/tests.yml) +[![Python 3.11 | 3.12 | 3.13](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue.svg)](.github/workflows/tests.yml) +[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) [![Security Policy](https://img.shields.io/badge/Security-Policy-blue.svg)](SECURITY.md) -A comprehensive suite of production-grade automation scripts, background daemons, liquidity optimizers, and monitoring utilities for **LND** Lightning Network node operators. +A modular suite of automation scripts, background daemons, liquidity optimizers, and monitoring utilities for **LND** Lightning Network node operators. --- -## 📑 Table of Contents -- [Overview](#-overview) -- [Architecture & Script Directory](#-architecture--script-directory) +## Table of Contents +- [Overview](#overview) +- [Architecture & Script Directory](#architecture--script-directory) - [1. Magma (Amboss Liquidity Market)](#1-magma-amboss-liquidity-market) - [2. LNDg Integration & Fee Management](#2-lndg-integration--fee-management) - [3. PeerSwap Automation](#3-peerswap-automation) - - [4. Swaps & Capital Management](#4-swaps--capital-management) + - [4. Swaps, Fees & UTXO Management](#4-swaps-fees--utxo-management) - [5. LNbits & Micro-services](#5-lnbits--micro-services) -- [Prerequisites & Installation](#-prerequisites--installation) -- [Configuration Guide](#-configuration-guide) -- [Running as Background Services](#-running-as-background-services) +- [Prerequisites & Installation](#prerequisites--installation) +- [Configuration Guide](#configuration-guide) +- [Running as Background Services](#running-as-background-services) + - [Automated Systemd Service Installers](#automated-systemd-service-installers) - [Systemd Daemon Example](#systemd-daemon-example) - [Crontab Example](#crontab-example) -- [Testing & Quality Assurance](#-testing--quality-assurance) -- [Security & Vulnerability Reporting](#-security--vulnerability-reporting) +- [Testing & Quality Assurance](#testing--quality-assurance) +- [Security & Vulnerability Reporting](#security--vulnerability-reporting) --- -## 🌟 Overview +## Overview -Operating an institutional or routing Lightning node requires robust, automated tooling across multiple dimensions: -- **Inbound Liquidity Auctioning**: Autonomous pricing and clearance on the Amboss Magma liquidity market. -- **Dynamic Fee Optimization**: Real-time channel fee adjustment based on remote market statistics, flow metrics, and stuck liquidity. -- **Inbound Discount Protection & Rebalance Guard**: Prevents circular rebalance loops from draining channels with active fee discounts. -- **Submarine & Liquid Swaps**: Programmatic balance management across on-chain, Liquid (L-BTC), and Lightning. -- **Automated PeerSwap Operations**: Triggering rebalances and swaps based on live peer flow. +Operating a routing Lightning node benefits from consistent automation across liquidity pricing, channel policy tuning, and capital rebalancing: +- **Inbound Liquidity Sales**: Autonomous offer pricing, order approval, invoice creation, and channel opening on the Amboss Magma liquidity market. +- **Dynamic Fee Optimization**: Channel fee adjustments driven by Amboss network statistics, forward flow activity, liquidity bands, and stuck-channel decay. +- **Inbound Discount Protection & Rebalance Guard**: Locks `ar_out_target = 100%` on discounted channels to prevent circular rebalance loops from draining refilling liquidity, restoring targets via dynamic hysteresis once balances recover. +- **Submarine & Liquid Swaps**: Economically scored Loop Out swaps (`litloop` / `loopd`) with live prepay route probing, plus Boltz reverse swaps (Lightning to Liquid L-BTC). +- **PeerSwap Operations**: Peer eligibility tracking, LNDg telemetry sync, and interactive swap management over Telegram. --- -## 🛠️ Architecture & Script Directory +## Architecture & Script Directory ### 1. Magma (Amboss Liquidity Market) -Automate your inbound liquidity sales on the Amboss Magma marketplace using the modern Magma GraphQL API (`https://magma.amboss.tech/graphql`). +Automate inbound liquidity sales on the Amboss Magma marketplace via the Magma GraphQL API (`https://magma.amboss.tech/graphql`). | Script | Execution Mode | Purpose | | :--- | :--- | :--- | -| [`Magma/magma_sale_process.py`](Magma/magma_sale_process.py) | `systemd service` | **Seller Auto-Pilot Daemon**: Listens for orders awaiting seller approval or channel open, checks UTXO and mempool fees, creates pre-image HODL invoices, opens channels, confirms funding transactions to Amboss, and auto-rejects banned peers. | -| [`Magma/magma_market_fee.py`](Magma/magma_market_fee.py) | `cronjob` / `service` | **Dynamic APR & Fee Synchronizer**: Scrapes active Magma public market offers, analyzes percentile rates, computes APR across timeframes (30d/60d/90d), and updates or toggles your Magma sell offers. Supports dry-run simulations. | +| [`Magma/magma_sale_process.py`](Magma/magma_sale_process.py) | `systemd service` | **Seller Auto-Pilot Daemon**: Monitors orders awaiting seller approval or channel open, validates on-chain UTXOs and mempool fee bounds, generates Lightning invoices via `lncli addinvoice`, supports interactive Telegram approval/timeout auto-approval, opens channels, confirms funding outpoints to Amboss, and auto-rejects banned peers. See [`Magma/magma_sale.md`](Magma/magma_sale.md). | +| [`Magma/magma_market_fee.py`](Magma/magma_market_fee.py) | `cronjob` / `CLI` | **Dynamic APR & Offer Optimizer**: Queries public Magma market offers, filters by seller score, computes percentile rates and annualized yield (APR) across offer templates in `Magma/magma_config.ini` ([example](Magma/magma_config.ini.example)), and creates, updates, or toggles sell offers (with `--dry-run` support). | --- ### 2. LNDg Integration & Fee Management -Synchronize metrics with your local LNDg dashboard and automate channel policies. +Synchronize marketplace metadata and automate channel policies against a local [LNDg](https://github.com/cryptosharks131/lndg) instance. | Script | Execution Mode | Purpose | | :--- | :--- | :--- | -| [`LNDg/amboss_pull.py`](LNDg/amboss_pull.py) | `cronjob` / `one-off` | Pulls Magma orders from Amboss, writes channel details into LNDg GUI, and configures auto-fee triggers upon channel maturity. | -| [`LNDg/mempool_rebalancer_trigger.py`](LNDg/mempool_rebalancer_trigger.py) | `systemd service` | Monitors live mempool block fee rates; triggers LNDg rebalancer when fees dip below user-defined thresholds. | -| [`LNDg/channel_base-fee.py`](LNDg/channel_base-fee.py) | `cronjob` / `one-off` | Normalizes or updates base fees across all open channels in LNDg. | -| [`LNDg/channel_fee-pull.py`](LNDg/channel_fee-pull.py) | `cronjob` | Pulls channel fee configurations from remote references and synchronizes into LNDg. | -| [`LNDg/disabled_fee-accelerator.py`](LNDg/disabled_fee-accelerator.py) | `cronjob` | Identifies disabled/inactive channels and escalates fees or triggers alerts. | -| [`LNDg/swap_out-candidates.py`](LNDg/swap_out-candidates.py) | `CLI tool` | Identifies channels with heavy local balances as candidates for off-loading liquidity via swap-out. | -| [`LNDg/offline_summary.py`](LNDg/offline_summary.py) | `cronjob` / `alert` | Sends Telegram reports summarizing offline channel downtime and liquidity impacts. | +| [`LNDg/amboss_pull.py`](LNDg/amboss_pull.py) | `cronjob` / `CLI` | Pulls active Magma sell orders from Amboss, writes maturity details into LNDg channel notes, toggles auto-fees/auto-rebalance upon lease expiration, and exports `charge-lnd` fee rules. | +| [`LNDg/mempool_rebalancer_trigger.py`](LNDg/mempool_rebalancer_trigger.py) | `systemd service` | Monitors half-hour mempool fee rates (`sat/vB`) and toggles LNDg `AR-Enabled`, pausing auto-rebalancing during high-fee spikes and re-enabling it when fees normalize. | +| [`LNDg/channel_base-fee.py`](LNDg/channel_base-fee.py) | `cronjob` / `one-off` | Resets `base_fee` to `0` across open, active channels in LNDg (excluding pubkeys in `base_fee_ignore`). | +| [`LNDg/channel_fee-pull.py`](LNDg/channel_fee-pull.py) | `cronjob` / `CLI` | Exports open channel IDs currently set to `0 ppm` in LNDg to `~/.chargelnd/.config/0_fee.txt` for `charge-lnd` rule ingestion. | +| [`LNDg/disabled_fee-accelerator.py`](LNDg/disabled_fee-accelerator.py) | `cronjob` / `CLI` | Incrementally bumps `local_fee_rate` in LNDg for depleted peer groups so locally initiated channels disabled by `charge-lnd` at low liquidity can attract rebalance flow. | +| [`LNDg/swap_out-candidates.py`](LNDg/swap_out-candidates.py) | `CLI tool` | Lists high-local-balance, low-PPM channels from LNDg as swap-out candidates, with optional export to `data/low-fee-high-local.log` (`-e`) or `bos/tags.json` (`-b`). | +| [`LNDg/offline_summary.py`](LNDg/offline_summary.py) | `CLI tool` | Queries LNDg's PostgreSQL `gui_peerevents` table (`--month YYYY-MM`) and prints a summary table of average and total peer offline hours (requires optional `psycopg2` and `pandas`). | --- ### 3. PeerSwap Automation -Automate Layer-1 / Liquid swaps with connected peers. +Monitor and execute Layer-1 BTC and Liquid (L-BTC) swaps with connected peers. For bare-metal, RaspiBlitz, and MiniBolt installation instructions for `elementsd` and `peerswapd`, see [`Peerswap/README.md`](Peerswap/README.md). | Script | Execution Mode | Purpose | | :--- | :--- | :--- | -| [`Peerswap/peerswap-bot.py`](Peerswap/peerswap-bot.py) | `systemd service` | **PeerSwap Telegram Bot**: Interactive management bot to monitor peer swap eligibility, submit swaps, and view historical swap status. | -| [`Peerswap/peerswap-lndg_push.py`](Peerswap/peerswap-lndg_push.py) | `cronjob` | Ingests PeerSwap metrics and logs them directly into LNDg notes and database records. | -| [`Peerswap/ps_peers.py`](Peerswap/ps_peers.py) | `CLI tool` | Summarizes liquidity distribution and active status of all PeerSwap-compatible peers. | +| [`Peerswap/peerswap-bot.py`](Peerswap/peerswap-bot.py) | `systemd service` | **PeerSwap Telegram Bot**: Interactive bot for checking L-BTC balances, listing eligible peers, initiating `swapin`/`swapout` operations, and inspecting swap status. | +| [`Peerswap/peerswap-lndg_push.py`](Peerswap/peerswap-lndg_push.py) | `cronjob` / `CLI` | Aggregates `pscli` swap counts, satoshi volumes, and paid fees per peer and writes them into LNDg channel notes. | +| [`Peerswap/ps_peers.py`](Peerswap/ps_peers.py) | `CLI tool` | Displays a terminal summary table of L-BTC balance, trusted PeerSwap peers, supported assets, channel balances, and local/remote fee rates. | --- -### 4. Swaps & Capital Management -Manage on-chain UTXOs, submarine swaps, and dynamic channel fee pricing. +### 4. Swaps, Fees & UTXO Management +Manage dynamic fee curves, rebalance protection, submarine swaps, and on-chain UTXO consolidation. | Script | Execution Mode | Purpose | | :--- | :--- | :--- | -| [`Other/fee_adjuster.py`](Other/fee_adjuster.py) | `cronjob` / `service` | **Dynamic Fee Optimizer**: Adjusts channel fees based on Amboss Space network fee trends, stuck channel detection, and liquidity percentages. Includes inbound discount protection and global rebalance guard auditing. | -| [`Other/rebalance_guard.py`](Other/rebalance_guard.py) | `cronjob` / `CLI tool` | **Standalone Rebalance Guard**: Audits all open LNDg channels across both native Auto-Fees and `fee_adjuster.py`. Protects channels with active inbound discounts by setting `ar_out_target = 100%`, and automatically restores baseline targets using **Dynamic Hysteresis** once liquidity recovers. | -| [`Other/boltz_swap-out.py`](Other/boltz_swap-out.py) | `CLI tool` | Automates submarine swap-outs through the Boltz exchange (Lightning to Liquid L-BTC). | -| [`Other/swap_wallet.py`](Other/swap_wallet.py) | `CLI tool` | Batches automated payouts or drain payments over Lightning to a designated Lightning Address. | -| [`Other/swap_out-loop.py`](Other/swap_out-loop.py) | `CLI tool` | Economically optimized Loop Out liquidity rebalancing via Lightning Labs Loop (`litloop` / `loopd`) with live route probing, multi-channel outbound route selection (MPP diversity), economic scoring, interactive UI, and SQLite accounting. See [documentation](Other/swap_out-loop.md). | -| [`Other/lnd_utxo_consolidator.py`](Other/lnd_utxo_consolidator.py) | `CLI tool` | Safely consolidates fragmented on-chain LND UTXOs during low-mempool fee environments. | +| [`Other/fee_adjuster.py`](Other/fee_adjuster.py) | `systemd timer` / `cronjob` | **Dynamic Fee Optimizer**: Adjusts outbound and inbound channel fees using Amboss Space peer fee metrics, 5-band liquidity curves, and stuck-channel step-downs, with optional `charge-lnd` rule export and inbound discount protection. See [`Other/fee_adjuster.md`](Other/fee_adjuster.md). | +| [`Other/rebalance_guard.py`](Other/rebalance_guard.py) | `cronjob` / `CLI tool` | **Standalone Rebalance Guard**: Audits all open LNDg channels across native Auto-Fees and `fee_adjuster.py`. Locks `ar_out_target = 100%` while an inbound discount is active and restores baseline targets via dynamic hysteresis once liquidity recovers (`--dry-run` supported). | +| [`Other/swap_out-loop.py`](Other/swap_out-loop.py) | `CLI tool` | **Economically Optimized Loop Out**: Scores candidate channels by net effective PPM (server fee + sweep fee + probed routing fee + opportunity cost), performs live fake-hash prepay probes via `lncli`, supports multi-channel MPP selection (`litloop` / `loopd`), interactive terminal UI, and native `loop_sqlite.db` history/CSV accounting. See [`Other/swap_out-loop.md`](Other/swap_out-loop.md). | +| [`Other/boltz_swap-out.py`](Other/boltz_swap-out.py) | `CLI tool` | Automates reverse submarine swaps (Lightning to Liquid L-BTC) via `boltzd` / `boltzcli`, selecting candidate outbound channels from LNDg with multi-batch payment retries and `--debug` dry-run mode. | +| [`Other/swap_wallet.py`](Other/swap_wallet.py) | `CLI tool` | Batches scheduled Lightning payouts to a Lightning Address via `bos send` through high-local-balance peers under configurable fee rate caps. | +| [`Other/lnd_utxo_consolidator.py`](Other/lnd_utxo_consolidator.py) | `CLI tool` | Interactive helper that inspects `lncli listunspent`, filters UTXOs by size and strategy, and prints a verified `lncli sendcoins` consolidation command for manual operator execution (never broadcasts directly). | --- @@ -92,19 +95,22 @@ Manage on-chain UTXOs, submarine swaps, and dynamic channel fee pricing. | Script | Execution Mode | Purpose | | :--- | :--- | :--- | -| [`LNBits/pocketmoney.py`](LNBits/pocketmoney.py) | `cronjob` | Automated pocket money allowances: tops up linked LNbits wallets on a regular schedule. | +| [`LNBits/pocketmoney.py`](LNBits/pocketmoney.py) | `systemd timer` / `daemon` | Automated recurring satoshi allowances: converts fiat budgets (`USD`, `EUR`, `GBP`, etc.) via LNbits exchange rates, pays child wallets configured in `wallets.json`, and sends Telegram balance alerts (`--run-once` supported for systemd timers/cron). | --- -## 🚀 Prerequisites & Installation +## Prerequisites & Installation ### System Requirements -- Python 3.10, 3.11, or 3.12 -- Linux environment (Debian/Ubuntu/Raspberry Pi OS) -- Running **LND** node (`lncli` accessible) -- (Optional) Installed `bos` (Balance of Satoshis) and `peerswapd` - -### 1. Clone & Set Up Virtual Environment +- **Python**: `3.11`, `3.12`, or `3.13` +- **OS**: Linux (Debian, Ubuntu, RaspiBlitz / Raspberry Pi OS) +- **Core Node**: Running **LND** node (`lncli` accessible) +- **Optional Integrations** (depending on which scripts you use): + - [LNDg](https://github.com/cryptosharks131/lndg) REST API (used by `LNDg/` scripts, `fee_adjuster.py`, `rebalance_guard.py`, `swap_out-loop.py`, and `boltz_swap-out.py`) + - `bos` (Balance of Satoshis), `charge-lnd`, `litd` / `loopd` (`litloop`), `peerswapd` (`pscli`), `boltzd` (`boltzcli`) + - Optional Python packages `psycopg2-binary` and `pandas` (only required for `LNDg/offline_summary.py` against a PostgreSQL LNDg database) + +### Clone & Set Up Virtual Environment ```bash git clone https://github.com/TrezorHannes/Lightning-Python-Tools.git cd Lightning-Python-Tools @@ -113,32 +119,60 @@ cd Lightning-Python-Tools python3 -m venv .venv source .venv/bin/activate -# Upgrade pip and install all required dependencies +# Upgrade pip and install runtime dependencies pip install --upgrade pip pip install -r requirements.txt ``` --- -## ⚙️ Configuration Guide +## Configuration Guide + +All local configuration files containing credentials or node-specific rules are ignored by Git (`.gitignore`). Copy the templates you need: + +```bash +cp config.ini.example config.ini +cp Magma/magma_config.ini.example Magma/magma_config.ini # Required for Magma/magma_market_fee.py +cp feeConfig.json.example feeConfig.json # Required for Other/fee_adjuster.py & Other/rebalance_guard.py +cp wallets.json.example wallets.json # Required for LNBits/pocketmoney.py +chmod 600 config.ini Magma/magma_config.ini feeConfig.json wallets.json +``` -1. Create your local `config.ini`: - ```bash - cp config.ini.example config.ini - ``` -2. Populate the required sections: - - `[credentials]`: Amboss API token (`amboss_authorization`), LNbits keys, LNDg credentials. - - `[telegram]`: Telegram bot token and chat ID for notifications. - - `[lndg]`: LNDg local API URL (e.g. `http://localhost:8889`). - - `[paths]`: Absolute path to `lncli`, `bos`, and output rule directories. - - `[magma]`: Min/max fee caps, invoice expiry, auto-approval thresholds. +### Configuration Files Overview +1. **[`config.ini`](config.ini.example)** (shared core configuration): + - `[credentials]`: Amboss API token (`amboss_authorization`) and LNDg login (`lndg_username`, `lndg_password`). + - `[lndg]`: Local LNDg API URL (e.g., `http://localhost:8889`). + - `[paths]` & `[system]`: Paths to `lncli`, `pscli`, `boltzcli`, `bos` (`full_path_bos`), `boltzd` TLS cert/macaroon, `charge_lnd_path`, and `log_level`. + - `[lnd]` & `[loop]`: LND gRPC connection details and Loop Out (`litloop`, `loop_db_path`, confirmation target, liquidity ratios, probe worker count, and routing fee leeway). + - `[magma]`: Invoice expiry, route hints, on-chain fee cap ratio, initial channel PPM, peer connection retries, and polling interval. + - `[pubkey]` & `[no-swapout]`: Comma-separated pubkey exclusion lists (`base_fee_ignore`, `banned_magma_pubkeys`, `swapout_blacklist`). + - `[telegram]`: Bot tokens (`magma_bot_token`, `peerswap_bot_token`, `lnbits_bot_token`) and authorized `telegram_user_id` / `chat_id`. + - `[LNBits]` & `[Schedule]`: LNbits server URL (`base_url`, without `/api/v1`), parent wallet `admin_key`, and allowance recurrence (`daily` or `weekly`). + - `[database]`: Optional PostgreSQL settings (`name`, `host`, `port`) for `LNDg/offline_summary.py`. +2. **[`feeConfig.json`](feeConfig.json.example)**: Peer groups (`sink`, `expensive`, etc.), Amboss fee bases (`median`/`mean`), 5-band liquidity discounts/premiums, stuck-channel intervals, and `inbound_protection` thresholds. +3. **[`Magma/magma_config.ini`](Magma/magma_config.ini.example)**: Auto-pricing toggle, LND on-chain balance allocation fraction, seller-score filter, target competitiveness percentile, and `[magma_offer_*]` channel size/duration/APR templates. +4. **[`wallets.json`](wallets.json.example)**: Child wallet IDs, invoice keys, and fiat allowance amounts/currencies for `LNBits/pocketmoney.py`. --- -## 🔄 Running as Background Services +## Running as Background Services + +### Automated Systemd Service Installers +Helper scripts are included to generate and enable `systemd` oneshot services and timers (pre-configured with OS CA trust bundle environment variables): + +```bash +# Install hourly fee_adjuster.service + fee_adjuster.timer +chmod +x Other/install_fee_adjuster_service.sh +sudo ./Other/install_fee_adjuster_service.sh + +# Install scheduled pocketmoney.service + pocketmoney.timer (reads [Schedule] from config.ini) +chmod +x LNBits/install_pocketmoney_service.sh +sudo ./LNBits/install_pocketmoney_service.sh +``` ### Systemd Daemon Example -Create `/etc/systemd/system/magma-sale.service`: +For long-running daemons such as `Magma/magma_sale_process.py` (full guide in [`Magma/magma_sale.md`](Magma/magma_sale.md)), create `/etc/systemd/system/magma-sale.service`: + ```ini [Unit] Description=Amboss Magma Channel Auto-Sale Service @@ -157,44 +191,54 @@ RestartSec=10 [Install] WantedBy=multi-user.target ``` + Enable and start the service: ```bash sudo systemctl daemon-reload -sudo systemctl enable magma-sale.service -sudo systemctl start magma-sale.service +sudo systemctl enable --now magma-sale.service ``` ### Crontab Example ```cron -# Run fee adjuster every 30 minutes -*/30 * * * * /home/admin/Lightning-Python-Tools/.venv/bin/python3 /home/admin/Lightning-Python-Tools/Other/fee_adjuster.py >> /home/admin/cron_fee.log 2>&1 +# Run dynamic fee adjuster every 30 minutes (if not using the systemd timer) +*/30 * * * * /home/admin/Lightning-Python-Tools/.venv/bin/python3 /home/admin/Lightning-Python-Tools/Other/fee_adjuster.py >> /home/admin/Lightning-Python-Tools/logs/cron_fee.log 2>&1 # Run standalone rebalance guard audit hourly -0 * * * * /home/admin/Lightning-Python-Tools/.venv/bin/python3 /home/admin/Lightning-Python-Tools/Other/rebalance_guard.py >> /home/admin/cron_guard.log 2>&1 +0 * * * * /home/admin/Lightning-Python-Tools/.venv/bin/python3 /home/admin/Lightning-Python-Tools/Other/rebalance_guard.py >> /home/admin/Lightning-Python-Tools/logs/cron_guard.log 2>&1 -# Sync Amboss Magma channel data to LNDg hourly -0 * * * * /home/admin/Lightning-Python-Tools/.venv/bin/python3 /home/admin/Lightning-Python-Tools/LNDg/amboss_pull.py >> /home/admin/cron_amboss.log 2>&1 +# Sync Amboss Magma channel orders to LNDg hourly +15 * * * * /home/admin/Lightning-Python-Tools/.venv/bin/python3 /home/admin/Lightning-Python-Tools/LNDg/amboss_pull.py >> /home/admin/Lightning-Python-Tools/logs/cron_amboss.log 2>&1 ``` --- -## 🧪 Testing & Quality Assurance +## Testing & Quality Assurance -The codebase includes full test coverage with unit tests and live GraphQL schema verification: +Unit tests, Amboss GraphQL contract checks, CA trust hardening checks, and Magma sale lifecycle simulations are executed across Python `3.11`, `3.12`, and `3.13` in GitHub Actions. ```bash -# Run the entire test suite source .venv/bin/activate -pytest tests/ -v + +# Install test and coverage dependencies +pip install -r requirements-dev.txt + +# Run the full test suite (configured via pyproject.toml) +pytest + +# Run with module coverage report +pytest --cov=LNDg --cov=Magma --cov=Other --cov=Peerswap --cov=LNBits + +# Run the standalone offline Magma buy-order lifecycle simulation +python3 tests/simulate_magma_sale.py ``` --- -## 🔒 Security & Vulnerability Reporting +## Security & Vulnerability Reporting -For our full responsible disclosure policy, supported versions, and contact channels, please review [`SECURITY.md`](SECURITY.md). +For our responsible disclosure policy, supported versions, and contact channels, review [`SECURITY.md`](SECURITY.md). - **Vulnerability Disclosure**: Report security vulnerabilities privately via [GitHub Security Advisories](https://github.com/TrezorHannes/Lightning-Python-Tools/security/advisories) or email `security@tunnelsats.com`. -- **Secrets Management**: Never commit `config.ini`, `.env`, or credential files to Git. -- **Permissions**: Restrict file permissions on `config.ini`: `chmod 600 config.ini`. -- **Least Privilege**: Use read-only or scoped API keys and macaroons whenever possible. +- **Secrets Management**: Never commit `config.ini`, `feeConfig.json`, `wallets.json`, `.env`, or macaroon files to Git. +- **Permissions**: Restrict file permissions on sensitive config files (`chmod 600 config.ini`). +- **Least Privilege**: Use read-only or scoped API keys and LND macaroons wherever possible. diff --git a/config.ini.example b/config.ini.example index cc9b137..20803c7 100644 --- a/config.ini.example +++ b/config.ini.example @@ -29,6 +29,8 @@ boltzcli_path = /usr/local/bin/boltzcli # This is boltzd's own TLS certificate, not LND's. boltzd_tlscert_path = /home/boltz/.boltz/tls.cert boltzd_admin_macaroon_path = /home/boltz/.boltz/macaroons/admin.macaroon +# Output directory for charge-lnd fee rule files generated by fee_adjuster.py and amboss_pull.py +charge_lnd_path = ~/.chargelnd/.config [telegram] chat_id = @@ -38,6 +40,7 @@ telegram_user_id = lnbits_bot_token = [info] +# Local LND node identity pubkey (used to exclude own offers in Magma market analysis and BOS accounting) node = [system] @@ -77,8 +80,8 @@ polling_interval_minutes = 10 ## LNBits Pocket Money Settings [LNBits] -# http:// for localhost or https:// for your external domain -base_url = https://yourdomain.com/api/v1 +# Base URL of your LNbits instance (http:// for localhost or https:// for external domain, without /api/v1) +base_url = https://yourdomain.com # admin key for the main LNBits wallet (parent wallet) admin_key = admin_key_1 @@ -125,3 +128,10 @@ max_channels = 3 # Loop daemon connection parameters (used if loop_command is 'loop' instead of 'litloop') # rpcserver = localhost:8443 # tlscertpath = ~/.lit/tls.cert + +[database] +# Optional: PostgreSQL connection settings used by LNDg/offline_summary.py +name = lndg +host = localhost +port = 5432 + diff --git a/requirements.txt b/requirements.txt index cc45d92..0453ea6 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,8 +1,6 @@ certifi>=2026.10.23 -configparser==7.3.0 prettytable pyTelegramBotAPI>=4.37.1 requests>=2.34.2 schedule>=1.2.2 -telebot>=0.0.5 urllib3>=2.8.0 diff --git a/tests/test_repo_hygiene.py b/tests/test_repo_hygiene.py new file mode 100644 index 0000000..78e4256 --- /dev/null +++ b/tests/test_repo_hygiene.py @@ -0,0 +1,120 @@ +"""Tests for repository configuration, URL normalization, import safety, and README links.""" + +import importlib +import json +from pathlib import Path +import re +from unittest.mock import MagicMock + +import pytest + +import pocketmoney + + +REPO_ROOT = Path(__file__).resolve().parent.parent + + +@pytest.mark.parametrize( + ("raw_url", "expected"), + [ + ("https://lnbits.example.com", "https://lnbits.example.com"), + ("https://lnbits.example.com/", "https://lnbits.example.com"), + ("https://lnbits.example.com/api/v1", "https://lnbits.example.com"), + ("https://lnbits.example.com/api/v1/", "https://lnbits.example.com"), + (" http://localhost:5000/api/v1 ", "http://localhost:5000"), + ], +) +def test_normalize_lnbits_base_url_prevents_double_api_v1(raw_url, expected): + assert pocketmoney.normalize_lnbits_base_url(raw_url) == expected + + +def test_pocketmoney_get_exchange_rate_uses_single_api_v1_prefix(monkeypatch, requests_mock): + monkeypatch.setattr( + pocketmoney, + "LNBITS_URL", + pocketmoney.normalize_lnbits_base_url("https://lnbits.example.com/api/v1"), + ) + requests_mock.get("https://lnbits.example.com/api/v1/rate/EUR", json={"rate": 1500}) + + assert pocketmoney.get_exchange_rate("EUR") == 1500 + + +def test_pocketmoney_validate_runtime_config_fails_closed(monkeypatch): + monkeypatch.setattr(pocketmoney, "LNBITS_URL", "") + monkeypatch.setattr(pocketmoney, "ADMIN_KEY", "") + monkeypatch.setattr(pocketmoney, "TOKEN", "invalid-token") + monkeypatch.setattr(pocketmoney, "CHAT_ID", "") + + with pytest.raises(ValueError, match="Missing or invalid config.ini settings"): + pocketmoney.validate_runtime_config() + + +def test_ps_peers_has_no_import_time_subprocess_side_effects(monkeypatch): + mock_run = MagicMock(side_effect=AssertionError("subprocess.run called at import time")) + monkeypatch.setattr("subprocess.run", mock_run) + + module = importlib.import_module("ps_peers") + importlib.reload(module) + mock_run.assert_not_called() + + responses = { + ("lncli", "getinfo"): {"identity_pubkey": "02local"}, + ("pscli", "listpeers"): { + "peers": [ + { + "node_id": "03peer", + "swaps_allowed": True, + "supported_assets": ["lbtc"], + "channels": [ + { + "channel_id": "123", + "local_balance": "500000", + "remote_balance": "500000", + "active": True, + } + ], + } + ] + }, + ("pscli", "lbtc-getbalance"): {"sat_amount": "250000"}, + ("lncli", "listchannels"): { + "channels": [{"remote_pubkey": "03peer", "peer_alias": "PeerNode"}] + }, + ("lncli", "getchaninfo", "--chan_id", "123"): { + "node1_pub": "02local", + "node2_pub": "03peer", + "node1_policy": {"fee_rate_milli_msat": "150"}, + "node2_policy": {"fee_rate_milli_msat": "200"}, + }, + } + + def fake_run(cmd, **_kwargs): + res = MagicMock() + res.stdout = json.dumps(responses[tuple(cmd)]).encode("utf-8") + return res + + monkeypatch.setattr("subprocess.run", fake_run) + module.main() + + +def test_requirements_excludes_stdlib_and_shadow_packages(): + req_lines = [ + line.strip().lower() + for line in (REPO_ROOT / "requirements.txt").read_text(encoding="utf-8").splitlines() + if line.strip() and not line.strip().startswith("#") + ] + pkg_names = {re.split(r"[><=!~\s]", line, maxsplit=1)[0] for line in req_lines} + + assert "configparser" not in pkg_names, "configparser is part of Python 3 stdlib" + assert "telebot" not in pkg_names, "pyTelegramBotAPI already provides the telebot module" + assert "pytelegrambotapi" in pkg_names + + +def test_readme_relative_file_links_exist(): + readme_text = (REPO_ROOT / "README.md").read_text(encoding="utf-8") + targets = re.findall(r"\]\(([^)]+)\)", readme_text) + for target in targets: + if target.startswith(("http://", "https://", "#", "mailto:")): + continue + rel_path = target.split("#", maxsplit=1)[0] + assert (REPO_ROOT / rel_path).exists(), f"Broken relative link in README.md: {target}"