diff --git a/README.md b/README.md index a013a15a8..3052c9c6b 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ try (QuestDB db = QuestDB.connect("ws::addr=localhost:9000;")) { .doubleColumn("price", 2615.54) .doubleColumn("amount", 0.00044) .atNow(); - // close() flushes pending rows and returns the Sender to the pool. + // close() returns the Sender to the pool; rows ship once an auto-flush threshold trips (age, rows or bytes), or when the idle slot is settled. } } ``` @@ -98,8 +98,15 @@ snippets below borrow from its pools. ### Ingest Data -Borrow a `Sender`, write rows, and close it. `close()` flushes pending rows and returns the sender to the pool — the -underlying connection is only torn down when the `QuestDB` handle itself is closed. +Borrow a `Sender`, write rows, and close it. Close releases, flush ships, drain waits: `close()` returns the sender to +the pool and ships the slot's pending rows once the oldest is older than `auto_flush_interval` (100 ms by default) or +another auto-flush threshold trips, otherwise it only releases; rows left in the slot go out when an auto-flush +threshold trips on a later append or `close()`, or when the pool's housekeeper settles the idle slot, typically within +one `auto_flush_interval`. A lease that holds the slot without appending or closing, for example one that waits for input +inside the `try` block, holds those rows back until it does: borrow once the data is in hand. Borrowing per +message is the intended pattern and batches like a sender held for the whole run. Call `flush()` to ship now and +`drain()` to wait for the server. The underlying connection is only torn down when the `QuestDB` handle itself is +closed. ```java import io.questdb.client.Sender; @@ -110,15 +117,29 @@ try (Sender sender = db.borrowSender()) { .doubleColumn("price", 42_500.50) .longColumn("size", 100) .atNow(); - // No explicit flush() needed: close() flushes for you. + // close() returns the sender; the pool ships the row by threshold or on its next settle. } ``` ### Sending Batches of Rows The sender buffers rows and sends them in batches. For high throughput, write many rows on one borrowed sender and -flush per batch rather than per row: each flush has a fixed cost, so batching raises throughput. `close()` flushes the -final partial batch. +flush per batch rather than per row: each flush has a fixed cost, so batching raises throughput. The final partial batch ships on the +pool's next settle; call `flush()` before `close()` if you need it on the wire now. + +If you borrow per message and want each message in the store-and-forward ring the moment `close()` returns, set +`flush_on_close=on` together with a commit window (`commit_rows`, `commit_interval_ms`): every close ships one frame +flagged defer-commit, the server appends it, and the pool commits once per window instead of once per close. A rejected +frame then rolls back the whole open group, so every lease since the last commit loses its rows. + +A server rejection covers a whole frame. With the default `flush_on_close=off` one frame carries the rows of every +lease that closed since the last ship, so one rejected row discards the other leases' rows in that frame as well, +after their `close()` calls returned. The default error handler logs the rejection with the frame's sequence span, and +the error is thrown by the next call on that slot, possibly to a lease that wrote none of the rejected rows. A +retriable rejection (write error, internal error) replays the frame; a terminal one (schema mismatch, parse error, +security error) retires the slot, and with `sf_dir` its unacked frames stay in the slot directory for recovery. A +lease whose rows must not share a frame calls `flush()` before `close()`, or the handle sets `flush_on_close=on`. +`reset()` on a borrowed sender discards only the rows appended through that lease. ```java try (Sender sender = db.borrowSender()) { @@ -134,7 +155,7 @@ try (Sender sender = db.borrowSender()) { sender.flush(); } } - // Remaining rows are flushed by close(). + // Remaining rows ship on the pool's next settle; flush() ships them now. } ``` @@ -542,6 +563,9 @@ schema::key1=value1;key2=value2; | `acquire_timeout_ms` | `5000` | How long `borrowSender()`/`borrowQuery()` waits for a free slot | | `idle_timeout_ms` | `60000` | How long a pooled connection may stay idle before it is reaped | | `max_lifetime_ms` | `1800000` | Maximum lifetime of a pooled connection before it is recycled | +| `flush_on_close` | `off` | `on` makes a pooled `close()` flush before releasing (one frame and one commit per close, the pre-1.4 behaviour). Note: "flush" means published to the store-and-forward ring, not acknowledged. Close-time errors are wrapped; the delegate's error is the cause; with `on` and no commit window each lease ships its own frame, so a server rejection cannot discard other leases' rows; with `off` leases share frames and a rejection covers all of them; with a commit window a rejection rolls back the whole open group, every lease since the last commit | +| `commit_rows` | _(unset)_ | Commit window: with `flush_on_close=on`, closes ship frames flagged defer-commit and the server commits once per window of this many rows. Acks arrive at the commit; `drain()` after `flush()` | +| `commit_interval_ms` | _(unset)_ | Commit window age in milliseconds; setting either window key enables the window, the other defaults to 1000 rows / 100 ms. An idle slot's open group is committed at the settle cadence, so the interval only bounds groups on busy slots | ### Query routing keys diff --git a/core/src/main/java/io/questdb/client/QuestDB.java b/core/src/main/java/io/questdb/client/QuestDB.java index cf669a166..e6dee7a82 100644 --- a/core/src/main/java/io/questdb/client/QuestDB.java +++ b/core/src/main/java/io/questdb/client/QuestDB.java @@ -137,8 +137,25 @@ static QuestDB connect(CharSequence configurationString, HttpTokenProvider token /** * Borrows a {@link Sender} from the pool. The caller MUST call * {@link Sender#close()} on the returned instance to release it back to - * the pool. {@code close()} on a pooled Sender flushes pending rows - * before returning to the pool; a real disconnect only happens at + * the pool. Close releases, flush ships, drain waits: {@code close()} on a + * pooled Sender returns the slot and by default ships the slot's pending + * rows once the oldest is older than {@code auto_flush_interval} or another + * auto-flush threshold trips, otherwise it only releases; rows left in the + * slot go out when an auto-flush threshold trips on a later append or + * release, or when the housekeeper settles the idle slot, within about one + * {@code auto_flush_interval} of it going idle. A lease that holds the slot + * without appending or closing holds those rows back until it does. Call + * {@link Sender#flush()} to ship now and + * {@link Sender#drain(long)} to wait for the server. Set + * {@code flush_on_close=on} to have {@code close()} flush, as before 1.4. + * With the default, one frame carries the rows of every lease that closed + * since the last ship, so a server rejection of that frame discards those + * leases' rows together; the error is logged by the error handler and thrown + * by a later call on that slot. Call {@link Sender#flush()} before close, or + * set {@code flush_on_close=on}, when a lease's rows must not share a frame. + * {@link Sender#reset()} on a pooled Sender discards only that lease's rows. + * Note that in this client "flush" means published to the store-and-forward + * ring, not acknowledged by the server. A real disconnect only happens at * {@link #close()} on this {@code QuestDB} handle. *

* Allocation: zero at steady state -- the returned instance is a diff --git a/core/src/main/java/io/questdb/client/QuestDBBuilder.java b/core/src/main/java/io/questdb/client/QuestDBBuilder.java index 11364f9ba..a2ed24b7e 100644 --- a/core/src/main/java/io/questdb/client/QuestDBBuilder.java +++ b/core/src/main/java/io/questdb/client/QuestDBBuilder.java @@ -54,12 +54,18 @@ public final class QuestDBBuilder { static final long DEFAULT_ACQUIRE_TIMEOUT_MILLIS = 5_000; + static final long DEFAULT_COMMIT_INTERVAL_MILLIS = 100; + static final int DEFAULT_COMMIT_ROWS = 1000; static final long DEFAULT_HOUSEKEEPER_INTERVAL_MILLIS = 5_000; static final long DEFAULT_IDLE_TIMEOUT_MILLIS = 60_000; static final long DEFAULT_MAX_LIFETIME_MILLIS = 30 * 60_000L; static final int DEFAULT_POOL_MAX = 4; static final int DEFAULT_POOL_MIN = 1; static final long DEFAULT_QUERY_CLOSE_TIMEOUT_MILLIS = 5_000; + // Settle cadence when auto_flush_interval is not set: the WebSocket + // sender's own default auto-flush interval, so rows left in an idle slot + // wait no longer than rows in a busy one. + static final long DEFAULT_SETTLE_INTERVAL_MILLIS = 100; // Every valid pool value is >= 0, so -1 unambiguously marks "not set // explicitly". The public pool setters are the only writers of these @@ -67,11 +73,15 @@ public final class QuestDBBuilder { private static final int UNSET = -1; private long acquireTimeoutMillis = UNSET; + private long commitIntervalMillis = UNSET; + private int commitRows = UNSET; // Optional ingest-side async callbacks. Null -> each pooled Sender uses its // loud-not-silent default. Applied to every Sender the pool builds. private SenderConnectionListener connectionListener; private BackgroundDrainerListener drainerListener; private SenderErrorHandler errorHandler; + // null = not set explicitly; the string key or the default applies. + private Boolean flushOnClose; private long housekeeperIntervalMillis = UNSET; private HttpTokenProvider httpTokenProvider; private String config; @@ -80,6 +90,9 @@ public final class QuestDBBuilder { private long queryCloseTimeoutMillis = UNSET; private int queryPoolMax = UNSET; private int queryPoolMin = UNSET; + private int resolvedCommitRows; + private long resolvedCommitIntervalMillis; + private String resolvedIngestConfig; private int senderPoolMax = UNSET; private int senderPoolMin = UNSET; @@ -114,6 +127,35 @@ public QuestDBBuilder queryCloseTimeoutMillis(long millis) { return this; } + /** + * Age of a commit window in milliseconds. Setting a commit window makes + * pooled closes ship frames flagged defer-commit: the server appends them + * and commits once per window, when either {@link #commitRows(int)} rows + * or this many milliseconds have accumulated since the last commit. Acks + * then arrive at the commit, so a {@link Sender#drain(long)} should follow + * a {@link Sender#flush()}. Requires {@code flush_on_close=on}. When only + * one of the two window settings is given, the other takes the auto-flush + * default (1000 rows, 100 ms). + */ + public QuestDBBuilder commitIntervalMillis(long millis) { + if (millis < 1) { + throw new IllegalArgumentException("commitIntervalMillis must be >= 1"); + } + this.commitIntervalMillis = millis; + return this; + } + + /** + * Rows per commit window. See {@link #commitIntervalMillis(long)}. + */ + public QuestDBBuilder commitRows(int rows) { + if (rows < 1) { + throw new IllegalArgumentException("commitRows must be >= 1"); + } + this.commitRows = rows; + return this; + } + /** * Sets the async connection-event listener applied to every pooled ingest * {@link Sender}. The listener observes connect / disconnect / failover @@ -166,6 +208,27 @@ public QuestDBBuilder errorHandler(SenderErrorHandler handler) { return this; } + /** + * Whether {@code close()} on a borrowed sender flushes its rows before + * returning the slot. Off by default: close releases, flush ships, drain + * waits, and rows left in a slot ship within one auto-flush interval of the slot going + * idle, or at the next append or close on it after that interval. On + * restores the pre-1.4 behaviour, one frame and one WAL commit per close, + * which is the slow path for applications that borrow per message. In this + * client "flush" means published to the store-and-forward ring, not + * acknowledged by the server; use {@link Sender#drain(long)} for that. + * With {@code on} and no commit window every lease ships its own frame, so + * a server rejection cannot discard other leases' rows. With {@code off} + * leases share frames and a rejection covers all of them. With a commit + * window a rejection rolls back the whole open group, every lease since the + * last commit. + * Overrides {@code flush_on_close} in the config string. + */ + public QuestDBBuilder flushOnClose(boolean flushOnClose) { + this.flushOnClose = flushOnClose; + return this; + } + /** * Builds the {@link QuestDB} handle. Validates both connect strings up * front -- so a malformed config fails here even when both pools have @@ -222,6 +285,53 @@ public QuestDB build() { resolvePoolLong(idleTimeoutMillis, "idle_timeout_ms", view, DEFAULT_IDLE_TIMEOUT_MILLIS, this::idleTimeoutMillis); resolvePoolLong(maxLifetimeMillis, "max_lifetime_ms", view, DEFAULT_MAX_LIFETIME_MILLIS, this::maxLifetimeMillis); resolvePoolLong(housekeeperIntervalMillis, "housekeeper_interval_ms", view, DEFAULT_HOUSEKEEPER_INTERVAL_MILLIS, this::housekeeperIntervalMillis); + if (flushOnClose == null) { + flushOnClose = view.getBool("flush_on_close", false); + } + int windowRows = commitRows; + if (windowRows == UNSET && view.has("commit_rows")) { + windowRows = view.getInt("commit_rows", UNSET); + if (windowRows < 1) { + throw new IllegalArgumentException("commit_rows must be >= 1, got " + windowRows); + } + } + long windowIntervalMillis = commitIntervalMillis; + if (windowIntervalMillis == UNSET && view.has("commit_interval_ms")) { + windowIntervalMillis = view.getLong("commit_interval_ms", UNSET); + if (windowIntervalMillis < 1) { + throw new IllegalArgumentException("commit_interval_ms must be >= 1, got " + windowIntervalMillis); + } + } + if (windowRows != UNSET || windowIntervalMillis != UNSET) { + if (windowRows == UNSET) { + windowRows = DEFAULT_COMMIT_ROWS; + } + if (windowIntervalMillis == UNSET) { + windowIntervalMillis = DEFAULT_COMMIT_INTERVAL_MILLIS; + } + if (!flushOnClose) { + throw new IllegalArgumentException( + "conflicting configuration: commit_rows/commit_interval_ms need flush_on_close=on in this version. " + + "The buffered variant (flush_on_close=off with a commit window) is not available yet; " + + "set flush_on_close=on or remove the commit window keys."); + } + String transaction = view.getStr("transaction"); + if (transaction != null && !"on".equalsIgnoreCase(transaction)) { + throw new IllegalArgumentException( + "conflicting configuration: a commit window needs transaction=on on the pooled senders, but " + + "transaction=" + transaction + " was set. Remove transaction= (the pool sets it) or set it to on."); + } + if (transaction == null) { + ingestConfig = injectAfterSchema(ingestConfig, "transaction=on;"); + } + } else { + windowRows = 0; + windowIntervalMillis = 0; + } + resolvedCommitRows = windowRows; + resolvedCommitIntervalMillis = windowIntervalMillis; + resolvedIngestConfig = ingestConfig; + long settleIntervalMillis = resolveSettleIntervalMillis(view, windowIntervalMillis); return new QuestDBImpl( ingestConfig, @@ -238,7 +348,11 @@ public QuestDB build() { httpTokenProvider, errorHandler, connectionListener, - drainerListener + drainerListener, + flushOnClose, + settleIntervalMillis, + windowRows, + windowIntervalMillis ); } @@ -440,6 +554,11 @@ public QuestDBBuilder senderPoolSize(int size) { return this; } + @TestOnly + public String ingestConfigForTest() { + return resolvedIngestConfig; + } + /** * Snapshot of the resolved pool config, keyed by connect-string key name. * Valid after {@link #build()} has run pool-key resolution. Drives the @@ -457,18 +576,44 @@ public java.util.Map poolConfigSnapshotForTest() { m.put("idle_timeout_ms", idleTimeoutMillis); m.put("max_lifetime_ms", maxLifetimeMillis); m.put("housekeeper_interval_ms", housekeeperIntervalMillis); + m.put("flush_on_close", flushOnClose); + m.put("commit_rows", resolvedCommitRows); + m.put("commit_interval_ms", resolvedCommitIntervalMillis); return m; } + // Inserts a key right after the "ws::" / "wss::" schema separator so the + // sender parse sees it as part of the ingest config. + private static String injectAfterSchema(String config, String keyValue) { + int sep = config.indexOf("::") + 2; + return config.substring(0, sep) + keyValue + config.substring(sep); + } + // Inject a non-blocking async initial connect right after the schema // separator so lazy_connect's build never blocks or fail-fast on a down // server. Only used when the user set no initial_connect_retry of their own // (resolveLazyConnect rejects an explicit blocking mode rather than silently // overriding it), so placement is immaterial -- there is no competing value. private static String withDefaultAsyncConnect(String config) { - int sep = config.indexOf("::"); - // sep >= 0: fromConfig() validated a ws/wss schema, so "::" is present. - return config.substring(0, sep + 2) + "initial_connect_retry=async;" + config.substring(sep + 2); + // fromConfig() validated a ws/wss schema, so "::" is present. + return injectAfterSchema(config, "initial_connect_retry=async;"); + } + + /** + * The housekeeper's settle cadence: {@code auto_flush_interval} when set, + * {@link #DEFAULT_SETTLE_INTERVAL_MILLIS} when unset. The WebSocket + * transport rejects both {@code auto_flush=off} and + * {@code auto_flush_interval=off}, so the interval is a number or unset. + * With a commit window, the shorter of that and {@code commitIntervalMillis} + * (0 = no window). + * Public only for the unit test, which lives in another package. + */ + @TestOnly + public static long resolveSettleIntervalMillis(ConfigView view, long commitIntervalMillis) { + // getLong uses the connect-string number syntax (underscore separators + // included), the same one the Sender parse already accepted. + long fromAutoFlush = Math.max(1L, view.getLong("auto_flush_interval", DEFAULT_SETTLE_INTERVAL_MILLIS)); + return commitIntervalMillis > 0 ? Math.min(fromAutoFlush, commitIntervalMillis) : fromAutoFlush; } private static void requireWebSocketSchema(CharSequence config, String role) { diff --git a/core/src/main/java/io/questdb/client/Sender.java b/core/src/main/java/io/questdb/client/Sender.java index 7617aa643..48c757094 100644 --- a/core/src/main/java/io/questdb/client/Sender.java +++ b/core/src/main/java/io/questdb/client/Sender.java @@ -276,6 +276,16 @@ static Sender fromEnv() { */ void atNow(); + /** + * Runs the same row, byte and age thresholds every append runs and ships the pending rows + * when one trips. At a pooled release the age threshold is the one that usually trips, which + * lets a pool ship rows a lease left behind once they are older than + * {@code auto_flush_interval}. It does nothing while a row is in progress. + * Only the WebSocket QWP transport implements it. + */ + default void autoFlushIfDue() { + } + /** * Block until the server has acknowledged every frame up to {@code targetFsn}, * or until {@code timeoutMillis} elapses. Pair with {@link #flushAndGetSequence()} @@ -415,6 +425,21 @@ default Sender charColumn(CharSequence name, char value) { throw new LineSenderException("current protocol version does not support char"); } + /** + * Throws the error this sender has latched, if any: the terminal error of + * the WebSocket I/O loop (server rejection, exhausted reconnect budget, + * authentication failure) or a connection error. Publishes nothing and + * never blocks. Used by the connection pool to decide whether a returned + * sender may be handed to the next borrower. + *
+ * Only the WebSocket QWP transport latches errors; on other transports this + * is a no-op. + * + * @throws LineSenderException if a terminal or connection error is latched + */ + default void checkError() { + } + /** * Close this Sender. *
@@ -570,6 +595,22 @@ default long flushAndGetSequence() { return -1L; } + /** + * Ships the pending rows as frames carrying the defer-commit flag: the + * server appends them to its WAL writers and withholds both the commit and + * the acknowledgement until the group is closed. A later {@link #flush()}, + * {@link #close()}, or (outside transactional mode) an auto-flush that + * ships an unflagged frame closes the group. Used by the connection pool's + * commit window. Only the WebSocket QWP transport supports deferred commits. + * + * @return the highest FSN published by this call, or {@code -1} if nothing was pending + * @throws UnsupportedOperationException on transports without deferred commits + * @throws LineSenderException if the transport has latched a terminal error + */ + default long flushDeferred() { + throw new UnsupportedOperationException("deferred flush requires the WebSocket transport"); + } + /** * Add a GEOHASH column value from pre-packed bits and an explicit bit precision. *

@@ -639,6 +680,30 @@ default long getAckedFsn() { return -1L; } + /** + * Whether frames shipped with {@link #flushDeferred()}, or by auto-flush in + * {@code transaction=on} mode, are still waiting for the commit that a + * {@link #flush()} sends. Only the WebSocket QWP transport tracks this. + * + * @return {@code true} while a deferred group is open + */ + default boolean hasDeferredFrames() { + return false; + } + + /** + * Whether a row is in progress: at least one column value has been set for + * the current row (just {@link #table(CharSequence)} does not count) and the + * row is not yet finished with {@link #at}, {@link #atNow()} or {@link #cancelRow()}. + * Only the WebSocket QWP transport tracks this; other transports return + * {@code false}. + * + * @return {@code true} while a row is in progress + */ + default boolean hasRowInProgress() { + return false; + } + /** * Add a column with a 32-bit signed integer value. * @@ -704,6 +769,32 @@ default Sender long256Column(CharSequence name, long l0, long l1, long l2, long */ Sender longColumn(CharSequence name, long value); + /** + * Marks the current buffer state as the point {@link #resetToSavepoint()} + * rolls back to: rows appended after this call are discarded by that + * method, rows appended before it are kept. Take it between rows, not + * while a row is in progress. Any flush that ships the pending rows + * empties the savepoint, so a later {@code resetToSavepoint()} discards + * every pending row like {@link #reset()}. Pure bookkeeping: it runs no + * error check. Default no-op. + *

+ * A pooled sender ignores this call: the pool marks a savepoint when it + * hands a slot to a borrower, and {@code resetToSavepoint()} on the pooled + * sender rolls back to that mark. + */ + default void markSavepoint() { + } + + /** + * Number of rows appended with {@link #at}/{@link #atNow()} that have not + * been shipped to the server yet. Only the WebSocket QWP transport counts + * pending rows; other transports return {@code 0}. + * + * @return rows buffered and not yet flushed + */ + default int pendingRowCount() { + return 0; + } /** * Advisory request to start a fresh symbol-dictionary epoch. The reset @@ -753,13 +844,42 @@ default void resetSymbolDictionary() { * data format) and want to continue sending new data without retrying the problematic data. * After calling this method, you can start building a new row by calling {@link #table(CharSequence)}. *
- * Note: This method is only available for HTTP transport. TCP transport doesn't support - * this operation. + * Supported by the HTTP, WebSocket and UDP transports; TCP does not support it. On a + * pooled sender this discards only the rows appended through the current lease, see + * {@link #resetToSavepoint()}. * * @see #flush() */ void reset(); + /** + * Discards the rows appended since the last {@link #markSavepoint()} and + * keeps the rows buffered before it. Without a savepoint, or after a flush + * emptied it, this discards every pending row like {@link #reset()}, and + * also removes the column definitions the server has not seen yet. Default + * calls {@code reset()}. + *

+ * On a pooled sender this is the lease-scoped {@code reset()}: it rolls + * back to the savepoint the pool marked at borrow, so a borrower can drop + * its own bad batch without discarding rows earlier borrowers left in the + * shared slot buffer. + */ + default void resetToSavepoint() { + reset(); + } + + /** + * Monotonic count of rows finished with {@link #at}/{@link #atNow()} over + * this sender's life, whether or not they have shipped. The connection + * pool measures its commit window on deltas of this value. Only the + * WebSocket QWP transport counts; other transports return {@code 0}. + * + * @return rows appended so far + */ + default long rowsAppended() { + return 0L; + } + /** * Add a column with a 16-bit signed integer value. * @@ -1053,7 +1173,7 @@ final class LineSenderBuilder { private static final int DEFAULT_TCP_PORT = 9009; private static final int DEFAULT_UDP_PORT = 9007; private static final int DEFAULT_WEBSOCKET_PORT = 9000; - private static final int DEFAULT_WS_AUTO_FLUSH_BYTES = 0; + private static final int DEFAULT_WS_AUTO_FLUSH_BYTES = QwpWebSocketSender.AUTO_FLUSH_BYTES_SERVER_CAP; private static final long DEFAULT_WS_AUTO_FLUSH_INTERVAL_NANOS = 100_000_000L; // 100ms private static final int DEFAULT_WS_AUTO_FLUSH_ROWS = 1_000; // Cadence at which the SF cursor I/O loop sends a keepalive PING @@ -1354,7 +1474,8 @@ public LineSenderBuilder authTimeoutMillis(long millis) { *
* This is only used when communicating over WebSocket transport. *
- * Default value is 0, which disables byte-based auto-flush. + * Unset by default, which flushes at 90 percent of the batch cap the server advertises, + * and not by bytes when it advertises none. 0 disables byte-based auto-flush. * * @param bytes maximum bytes per batch * @return this instance for method chaining diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/BatchTooLargeForCapException.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/BatchTooLargeForCapException.java index b9ef97e8e..cd363c3ce 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/BatchTooLargeForCapException.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/BatchTooLargeForCapException.java @@ -38,7 +38,36 @@ */ public class BatchTooLargeForCapException extends LineSenderException { + // Length of getMessage() when endDetail() ran: the detail ends there and the + // standalone remedy follows. -1 until then, when the whole message is the detail. + private int detailLength = -1; + public BatchTooLargeForCapException(CharSequence message) { super(message); } + + /** + * Marks the end of the detail: the text appended so far says what was rejected + * and the sizes involved, and the text appended after this call is the remedy for + * a standalone sender, which {@link #getDetail()} leaves out. + * + * @return this exception, to append the remedy + */ + public BatchTooLargeForCapException endDetail() { + detailLength = getMessage().length(); + return this; + } + + /** + * The message without the standalone remedy ("call reset() and keep this sender"), + * for a caller that handles the rejection itself, such as the connection pool, + * which retires the slot. Equals {@link #getMessage()} when the message carries no + * such remedy. + * + * @return what was rejected and the sizes involved + */ + public String getDetail() { + String message = getMessage(); + return detailLength < 0 || detailLength > message.length() ? message : message.substring(0, detailLength); + } } diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpColumnWriter.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpColumnWriter.java index f8cc669fb..ae9215baf 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpColumnWriter.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpColumnWriter.java @@ -316,7 +316,20 @@ private void writeTimestampColumn(long addr, int count, boolean useGorilla) { } void encodeTable(QwpTableBuffer tableBuffer, boolean useGlobalSymbols, boolean useGorilla) { - encodeTable(tableBuffer, tableBuffer.getRowCount(), null, null, null, useGlobalSymbols, useGorilla); + if (tableBuffer.hasHeldRow()) { + // the table buffer keeps the held row in its columns; this writer encodes only the rows before it + encodeTable( + tableBuffer, + tableBuffer.getRowCount(), + tableBuffer.getHeldValueCounts(), + tableBuffer.getHeldStringDataSizes(), + tableBuffer.getHeldSymbolDictionarySizes(), + useGlobalSymbols, + useGorilla + ); + } else { + encodeTable(tableBuffer, tableBuffer.getRowCount(), null, null, null, useGlobalSymbols, useGorilla); + } } void encodeTable( diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java index f591b565f..28d1b325a 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/QwpWebSocketSender.java @@ -98,7 +98,7 @@ * Configuration options: *

*

@@ -130,6 +130,16 @@ */ public class QwpWebSocketSender implements Sender { + /** + * {@code autoFlushBytes} value meaning "no byte budget of its own": flush at + * 90 percent of whatever batch cap the server advertises, and at no byte + * threshold while it advertises none. The config-string default for the + * WebSocket transport. An explicit {@code auto_flush_bytes=off} is {@code 0} + * and keeps byte flushing off whatever the server advertises. Rows appended + * before the first handshake carry no byte trigger; the first append after it + * flushes the backlog. + */ + public static final int AUTO_FLUSH_BYTES_SERVER_CAP = -1; public static final long DEFAULT_AUTH_TIMEOUT_MS = 15_000L; // Soft per-batch byte budget. Trips a flush when raw column-buffer bytes // cross the threshold, well before the server's 16 MB wire cap. Wide-row @@ -454,6 +464,13 @@ public class QwpWebSocketSender implements Sender { // Optional owning-pool notification, relayed after the engine confirms the // flock release and slotLockReleased is visible to pool re-probes. private volatile Runnable slotLockReleaseListener; + // Batch-cap rejection from a flush attempt's pre-flight. Cleared by reset(), + // resetToSavepoint() and by the next flush attempt whose batch passes the + // pre-flight, even when that attempt then fails for another reason, so an old + // rejection never outlives a batch the cap accepts. Not a terminal: the + // standalone contract lets the caller reset() and carry on. checkError() + // reports it so a pool can retire a slot whose batch no flush will ship. + private BatchTooLargeForCapException retainedBatchRejection; // Engine whose close() could not complete during sender close() — its // cleanup is pending on a worker/I/O-thread exit path. isSlotLockReleased() // re-probes it so a late flock release becomes visible to the owning pool. @@ -461,6 +478,16 @@ public class QwpWebSocketSender implements Sender { // while no engine is retained. private volatile CursorSendEngine retainedEngine; private int pendingRowCount; + // Monotonic for the sender's life; reset() does not clear it. Counts rows whether or not they shipped. + private long rowsAppended; + // Pending counters and batch symbol watermark at the last markSavepoint(); + // the counters are zero and the watermark is -1 once a flush ships the pending + // rows, at which point resetToSavepoint() discards every pending row like + // reset() and also removes the column definitions the server has not seen yet. + private int savepointBatchMaxSymbolId = -1; + private long savepointFirstPendingRowTimeNanos; + private long savepointPendingBytes; + private int savepointPendingRowCount; private SenderProgressDispatcher progressDispatcher; // Async-delivery sink for ack-watermark advances. Default no-op; a // setProgressHandler call before connect() swaps in a real handler. @@ -679,12 +706,13 @@ private QwpWebSocketSender( this.autoFlushBytes = autoFlushBytes; // Until the handshake completes, honor the configured budget verbatim. // applyServerBatchSizeLimit() clamps this on connect once the server's - // X-QWP-Max-Batch-Size is known. - this.effectiveAutoFlushBytes = autoFlushBytes; + // X-QWP-Max-Batch-Size is known. A cap-bound sender (-1) has no budget of + // its own before that. + this.effectiveAutoFlushBytes = Math.max(0, autoFlushBytes); this.autoFlushIntervalNanos = autoFlushIntervalNanos; this.globalSymbolDictionary = new GlobalSymbolDictionary(); - int microbatchBufferSize = Math.max(DEFAULT_MICROBATCH_BUFFER_SIZE, autoFlushBytes * 2); + int microbatchBufferSize = Math.max(DEFAULT_MICROBATCH_BUFFER_SIZE, Math.max(0, autoFlushBytes) * 2); try { this.buffer0 = new MicrobatchBuffer(microbatchBufferSize); this.buffer1 = new MicrobatchBuffer(microbatchBufferSize); @@ -1366,7 +1394,7 @@ public static QwpWebSocketSender createForTesting(String host, int port, String * @param host server host (not connected) * @param port server port (not connected) * @param autoFlushRows rows per batch (0 = no limit) - * @param autoFlushBytes bytes per batch (0 = no limit) + * @param autoFlushBytes bytes per batch (0 = no limit, -1 = 90 percent of the server's advertised cap) * @param autoFlushIntervalNanos age before flush in nanos (0 = no limit) * @return unconnected sender */ @@ -1447,6 +1475,17 @@ public void atNow() { } } + @Override + public void autoFlushIfDue() { + checkNotClosed(); + if (currentTableBuffer != null && currentTableBuffer.hasInProgressRow()) { + return; + } + if (shouldAutoFlush()) { + flushPendingRows(transactional); + } + } + /** * Blocks until {@link #getAckedFsn()} {@code >= targetFsn}, or until * {@code timeoutMillis} elapses. Polls on a 50us park; surfaces I/O loop @@ -1656,6 +1695,15 @@ public QwpWebSocketSender charColumn(CharSequence columnName, char value) { return this; } + @Override + public void checkError() { + checkNotClosed(); + BatchTooLargeForCapException rejected = retainedBatchRejection; + if (rejected != null) { + throw rejected; + } + } + /** * Closes the sender: flushes user-thread state into the engine, drains * acked data within {@code close_flush_timeout}, stops the I/O loop, @@ -2238,6 +2286,33 @@ public long flushAndGetSequence() { return afterFsn > beforeFsn ? fsnEpochBase + afterFsn : -1L; } + @Override + public long flushDeferred() { + checkNotClosed(); + checkRecycleFailure(); + if (cursorEngine != null) { + cursorEngine.checkDurability(); + } + ensureNoInProgressRow(); + ensureConnected(); + + long beforeFsn = cursorEngine != null ? cursorEngine.publishedFsn() : -1L; + + // Same publish path as flushAndGetSequence() with the flag forced on: + // the frame is appended server-side, not committed, and its ack is + // withheld until a later unflagged frame or commit message. The commit + // boundary is deliberately NOT advanced here. + flushPendingRows(true); + if (activeBuffer != null && activeBuffer.hasData()) { + sealAndSwapBuffer(); + } + cursorSendLoop.checkError(); + checkConnectionError(); + + long afterFsn = cursorEngine != null ? cursorEngine.publishedFsn() : -1L; + return afterFsn > beforeFsn ? fsnEpochBase + afterFsn : -1L; + } + /** * Flushes pending rows and blocks until the server has acknowledged * every frame published so far (the current published-FSN watermark), @@ -2389,7 +2464,9 @@ public int getActiveBackgroundDrainers() { } /** - * Returns the auto-flush byte threshold. + * Returns the configured auto-flush byte threshold: a positive budget, {@code 0} for off, + * or {@link #AUTO_FLUSH_BYTES_SERVER_CAP}. See {@link #getEffectiveAutoFlushBytes()} for + * the value in force. */ public int getAutoFlushBytes() { return autoFlushBytes; @@ -2899,6 +2976,16 @@ public long getTotalServerErrors() { return counters.serverErrors.get(); } + @Override + public boolean hasDeferredFrames() { + return hasDeferredMessages; + } + + @Override + public boolean hasRowInProgress() { + return currentTableBuffer != null && currentTableBuffer.hasInProgressRow(); + } + /** * Adds an INT column value to the current row. * @@ -3108,6 +3195,21 @@ public QwpWebSocketSender longColumn(CharSequence columnName, long value) { return this; } + @Override + public void markSavepoint() { + ObjList keys = tableBuffers.keys(); + for (int i = 0, n = keys.size(); i < n; i++) { + QwpTableBuffer buf = tableBuffers.get(keys.getQuick(i)); + if (buf != null) { + buf.markSavepoint(); + } + } + savepointBatchMaxSymbolId = currentBatchMaxSymbolId; + savepointPendingRowCount = pendingRowCount; + savepointPendingBytes = pendingBytes; + savepointFirstPendingRowTimeNanos = firstPendingRowTimeNanos; + } + /** * Returns a {@link CursorWebSocketSendLoop.ReconnectFactory} that, on each * call, performs the multi-endpoint walk and returns a freshly connected @@ -3133,6 +3235,11 @@ public CursorWebSocketSendLoop.ReconnectFactory newBackgroundReconnectFactory( return new ReconnectSupplier(stopFlag, "drainer stop requested during connect"); } + @Override + public int pendingRowCount() { + return pendingRowCount; + } + /** * Test seam: installs the per-attempt WebSocket client factory override * consulted by {@code newWebSocketClient()} inside the connect walk. @@ -3184,25 +3291,7 @@ public void reset() { buf.reset(); } } - // Drop the batch's symbol watermark along with the rows that raised it. A - // later flush encodes the delta section as - // [sentMaxSymbolId+1 .. currentBatchMaxSymbolId], so a watermark left behind - // by the discarded batch keeps re-encoding that batch's symbols: a single-row - // batch after reset() would still carry the whole abandoned range and hit the - // very cap rejection reset() is documented to clear, leaving the sender unable - // to flush anything at all. -1 is the same value resetTableBuffersAfterFlush - // leaves behind after a successful flush, and sendCommitMessage already reads - // it as an empty delta. - currentBatchMaxSymbolId = -1; - reclaimUnsentSymbolIds(); - pendingBytes = 0; - pendingRowCount = 0; - firstPendingRowTimeNanos = 0; - currentTableBuffer = null; - currentTableBufferSnapshotBytes = 0; - currentTableName = null; - cachedTimestampColumn = null; - cachedTimestampNanosColumn = null; + discardPendingBatchState(); } /** @@ -3236,6 +3325,53 @@ public void resetSymbolDictionary() { } } + @Override + public void resetToSavepoint() { + checkNotClosed(); + // Always roll back, also with an empty savepoint: after a flush each + // buffer's mark sits at "no rows, server-known columns", so the rollback + // drops the column definitions the lease created that the server has not + // seen, which reset() would keep and ship null-padded. + ObjList keys = tableBuffers.keys(); + for (int i = 0, n = keys.size(); i < n; i++) { + QwpTableBuffer buf = tableBuffers.get(keys.getQuick(i)); + if (buf != null) { + buf.rollbackToSavepoint(); + } + } + if (savepointPendingRowCount == 0) { + // No rows before the mark, or a flush since emptied it: every pending + // row is the caller's own and the sender-level clears of reset() apply. + discardPendingBatchState(); + return; + } + // Rows before the mark reference only symbol ids at or below the + // watermark recorded at the mark, and the watermark only grows within a + // batch, so the batch range is restored to it. In delta mode the ids the + // discarded rows registered above it are reclaimed too (the floor keeps + // the ring's and the durable dictionary's ids), so the next new symbol + // does not re-ship the orphaned range. Full-dict mode cannot reclaim: + // its frames already on the ring define those ids. Its orphans stay in + // the dictionary, which reset() cannot shrink there either; they are + // only re-shipped once a later row's new symbol id lies above them. + currentBatchMaxSymbolId = savepointBatchMaxSymbolId; + reclaimUnsentSymbolIds(savepointBatchMaxSymbolId + 1); + pendingRowCount = savepointPendingRowCount; + pendingBytes = savepointPendingBytes; + firstPendingRowTimeNanos = savepointFirstPendingRowTimeNanos; + currentTableBuffer = null; + currentTableBufferSnapshotBytes = 0; + currentTableName = null; + cachedTimestampColumn = null; + cachedTimestampNanosColumn = null; + retainedBatchRejection = null; + } + + @Override + public long rowsAppended() { + return rowsAppended; + } + /** * Register an async listener for connection-state transitions: initial * connect, primary failover, endpoint attempt failures, the full address @@ -3768,6 +3904,10 @@ public boolean wasEverConnected() { return hasLoopEverConnected || (l != null && l.hasEverConnected()); } + private static long byteBudgetForCap(int cap) { + return (long) cap * 9 / 10; + } + private static Throwable captureCloseError(Throwable terminalError, Throwable t) { if (terminalError == null) { return t; @@ -3816,29 +3956,32 @@ private static List singleEndpoint(String host, int port) { * in {@link #sendRow} fires against the freshly advertised value. The * byte trigger, however, is only adjusted when the user left it enabled: * an explicit {@code auto_flush_bytes=off} (autoFlushBytes == 0) is - * preserved even when the server advertises a cap, so applications that - * opted out keep the contract they asked for. + * preserved even when the server advertises a cap, and a sender built with + * {@link #AUTO_FLUSH_BYTES_SERVER_CAP} takes the 90 percent budget as its only byte trigger, + * so applications that opted out keep the contract they asked for. */ @TestOnly public void applyServerBatchSizeLimit(int advertisedMaxBatchSize) { serverMaxBatchSize = advertisedMaxBatchSize; - if (autoFlushBytes <= 0) { - // User opted out of byte-based auto-flush; respect that even when - // the server advertises a cap. The single-row guard still protects - // against oversize individual rows via serverMaxBatchSize. + if (autoFlushBytes == 0) { + // Explicit auto_flush_bytes=off: respect it even when the server + // advertises a cap. The single-row guard still protects against + // oversize individual rows via serverMaxBatchSize. effectiveAutoFlushBytes = 0; return; } if (advertisedMaxBatchSize <= 0) { - effectiveAutoFlushBytes = autoFlushBytes; + // Older build, no header: the configured budget verbatim, and no + // byte trigger at all for a sender bound to a cap it cannot see. + effectiveAutoFlushBytes = Math.max(0, autoFlushBytes); return; } // Cap at 90% of the server's hard limit. Raw column-buffer bytes are // a conservative proxy for wire size (compression usually shrinks the // payload), but schema and dict-delta overhead can push the wire size // above the raw total in pathological cases. - long safeBudget = (long) advertisedMaxBatchSize * 9 / 10; - if (autoFlushBytes < safeBudget) { + long safeBudget = byteBudgetForCap(advertisedMaxBatchSize); + if (autoFlushBytes > 0 && autoFlushBytes < safeBudget) { effectiveAutoFlushBytes = autoFlushBytes; } else { effectiveAutoFlushBytes = (int) safeBudget; @@ -3865,6 +4008,58 @@ private void atNanos(long timestampNanos) { sendRow(); } + /** + * Runs the auto-flush {@link #sendRow()} starts once a row trips a + * threshold. The byte trigger fires after the crossing row is committed, so + * that row alone can push the batch over the server batch cap, for example + * when earlier pooled leases left rows behind. On that rejection this method + * holds the row back and ships the rows before it, and + * {@link #resetTableBuffersAfterFlush()} keeps the row buffered alone. A + * held row that still trips a threshold on its own is flushed at once, so a + * rejection of its own frame reaches this caller. When the rows before it do + * not fit either, the row goes back into the batch, and the method retains + * and throws the first rejection, which describes the whole batch. + */ + private void autoFlushAfterRow() { + try { + flushPendingRows(transactional); + } catch (BatchTooLargeForCapException rejection) { + // The hold-back runs only where the sender used to throw. Every cap + // rejection precedes the first publish, so the whole batch is still + // buffered and the retry below sees exactly the rows before this one. + // Once the rows before it ship, the held row gets the normal auto-flush + // check once, with no further hold-back: at or above the byte budget the + // next append would flush it anyway, and flushing it now puts a rejection + // of its own frame on this row's at() call, which a pooled close() rolls + // back while the slot stays in service. + if (pendingRowCount < 2 || !currentTableBuffer.holdBackLastRow()) { + throw rejection; + } + try { + // Ships every row but the held one; resetTableBuffersAfterFlush + // resolves the hold by retaining the held row. + flushPendingRows(transactional); + } catch (Throwable t) { + // restoreHeldRow() does nothing when resetTableBuffersAfterFlush + // already retained the row. + currentTableBuffer.restoreHeldRow(); + if (t instanceof BatchTooLargeForCapException) { + retainedBatchRejection = rejection; + throw rejection; + } + throw t; + } + // With auto_flush_bytes=off shouldAutoFlush() has no byte test, so a held row + // whose own frame exceeds the advertised cap would stay in the slot after its + // lease. Flush it at once when it reaches the same 90 percent budget the + // default byte trigger uses. + int cap = serverMaxBatchSize; + if (shouldAutoFlush() || (cap > 0 && getPendingBytes() >= byteBudgetForCap(cap))) { + flushPendingRows(transactional); + } + } + } + /** * Resolves the connect timeout for one {@code buildAndConnect} walk. * Foreground connects honour the configured value verbatim: 0 (the @@ -4452,6 +4647,14 @@ private void checkTableSelected() { } } + private void clearSavepointAndRejection() { + savepointBatchMaxSymbolId = -1; + savepointPendingRowCount = 0; + savepointPendingBytes = 0; + savepointFirstPendingRowTimeNanos = 0; + retainedBatchRejection = null; + } + private synchronized Throwable closeRemainingResources(Throwable terminalError) { if (closeCleanupStarted) { return terminalError; @@ -4622,6 +4825,30 @@ private Endpoint currentEndpoint() { return idx < 0 ? null : endpoints.get(idx); } + private void discardPendingBatchState() { + // Drop the batch's symbol watermark along with the rows that raised it. A + // later flush encodes the delta section as + // [sentMaxSymbolId+1 .. currentBatchMaxSymbolId], so a watermark left behind + // by the discarded batch keeps re-encoding that batch's symbols: a single-row + // batch after reset() would still carry the whole abandoned range and hit the + // very cap rejection reset() is documented to clear, leaving the sender unable + // to flush anything at all. -1 is what resetTableBuffersAfterFlush leaves + // behind after a flush that ships every pending row (after a hold-back flush + // the watermark stays at the batch max while the held row is pending), and + // sendCommitMessage already reads -1 as an empty delta. + currentBatchMaxSymbolId = -1; + reclaimUnsentSymbolIds(0); + pendingBytes = 0; + pendingRowCount = 0; + firstPendingRowTimeNanos = 0; + clearSavepointAndRejection(); + currentTableBuffer = null; + currentTableBufferSnapshotBytes = 0; + currentTableName = null; + cachedTimestampColumn = null; + cachedTimestampNanosColumn = null; + } + /** * Build and offer a connection event to the dispatcher. No-op when the * dispatcher has not been allocated yet (e.g. very early in connect() @@ -5032,6 +5259,7 @@ private void flushPendingRows(boolean deferCommit) { ? 0 : currentTableBuffer.getBufferedBytes(); pendingRowCount = 0; firstPendingRowTimeNanos = 0; + clearSavepointAndRejection(); return; } @@ -5095,7 +5323,13 @@ private void flushPendingRows(boolean deferCommit) { && !deltaDictEnabled && currentBatchMaxSymbolId > deltaBaseline && !splitFramesFit(cap, deltaBaseline) && splitFramesFit(cap, currentBatchMaxSymbolId)) { - publishDictionaryChunks(cap, deltaBaseline + 1, currentBatchMaxSymbolId); + try { + publishDictionaryChunks(cap, deltaBaseline + 1, currentBatchMaxSymbolId); + } catch (BatchTooLargeForCapException e) { + // Thrown before the try block below, which would latch it. + retainedBatchRejection = e; + throw e; + } dictionaryChunksAwaitCommit = true; deltaBaseline = currentBatchMaxSymbolId; combinedBodyStart = encodeCombinedFrame(tableCount, deferCommit, deltaBaseline); @@ -5110,6 +5344,8 @@ && splitFramesFit(cap, currentBatchMaxSymbolId)) { flushPendingRowsSplit(deferCommit, combinedBodyStart, cap, deltaBaseline); return; } + // The batch fits the cap: an earlier rejection no longer describes it. + retainedBatchRejection = null; // Write-ahead: durably persist this frame's new symbols BEFORE it is // published, so a recovered/orphan-drained slot can always rebuild the @@ -5131,6 +5367,9 @@ && splitFramesFit(cap, currentBatchMaxSymbolId)) { resetTableBuffersAfterFlush(); } catch (Throwable t) { + if (t instanceof BatchTooLargeForCapException) { + retainedBatchRejection = (BatchTooLargeForCapException) t; + } if (dictionaryChunksAwaitCommit) { commitOrphanedDictionaryChunks(t); } @@ -5279,14 +5518,18 @@ private void flushPendingRowsSplit( + "symbol for this data" : "call reset() to discard the retained batch and keep this sender, close the " + "sender to discard everything, or produce smaller batches"; - throw new BatchTooLargeForCapException("single table batch too large for server batch cap") - .put(" [table=").put(tableName) + BatchTooLargeForCapException rejection = new BatchTooLargeForCapException( + "single table batch too large for server batch cap"); + rejection.put(" [table=").put(tableName) .put(", messageSize=").put(messageSize) .put(", dictionaryFrameBytes=").put(frameOverhead) - .put(", serverMaxBatchSize=").put(cap).put(']') + .put(", serverMaxBatchSize=").put(cap).put(']'); + // The pool recovers on its own and reports only the detail. + rejection.endDetail() .put("; the batch is retained for retry and every flush() will " + "reject it again until a larger-cap node is reached -- ") .put(remedy); + throw rejection; } // Mirror advanceSentMaxSymbolId: once the first frame ships the batch's // new ids, the remaining frames carry an empty delta above the baseline. @@ -5294,6 +5537,9 @@ private void flushPendingRowsSplit( simBaseline = currentBatchMaxSymbolId; } } + // Every split frame fits the cap: an earlier rejection no longer describes + // the batch. + retainedBatchRejection = null; int tableBodyOffset = combinedBodyStart; // Mirrors simBaseline above and advanceSentMaxSymbolId below, so the frame @@ -5370,16 +5616,35 @@ private void onSlotLockReleased() { private void resetTableBuffersAfterFlush() { for (int i = 0, n = flushTableBuffers.size(); i < n; i++) { - flushTableBuffers.getQuick(i).reset(); + QwpTableBuffer tableBuffer = flushTableBuffers.getQuick(i); + if (!tableBuffer.hasHeldRow()) { + tableBuffer.reset(); + } } // Drop the references; the next flush re-collects. flushTableNames.clear(); flushTableBuffers.clear(); - currentBatchMaxSymbolId = -1; - pendingBytes = 0; - currentTableBufferSnapshotBytes = 0; - pendingRowCount = 0; - firstPendingRowTimeNanos = 0; + if (currentTableBuffer != null && currentTableBuffer.hasHeldRow()) { + // autoFlushAfterRow shipped every row but the one it held back, which + // stays buffered as the only pending row. currentBatchMaxSymbolId keeps + // its value: the held row's symbol ids are at or below it, and a + // full-dictionary frame must still carry them. In delta mode the frame + // that just shipped carried them, and advanceSentMaxSymbolId moved the + // baseline past them. + currentTableBuffer.retainHeldRow(); + long heldRowBytes = currentTableBuffer.getBufferedBytes(); + pendingRowCount = 1; + pendingBytes = heldRowBytes; + currentTableBufferSnapshotBytes = heldRowBytes; + firstPendingRowTimeNanos = System.nanoTime(); + } else { + currentBatchMaxSymbolId = -1; + pendingBytes = 0; + currentTableBufferSnapshotBytes = 0; + pendingRowCount = 0; + firstPendingRowTimeNanos = 0; + } + clearSavepointAndRejection(); armIfEligible(); } @@ -6700,17 +6965,19 @@ private void publishDictionaryChunk(int startId, int endId) { * * Handing either to a different string is the silent misattribution the dense * id space exists to prevent, so the floor is the max of the two. + * {@code minFloor} raises it further: resetToSavepoint() passes the ids the + * retained rows still reference. Pass 0 to reclaim everything unsent. *

* Full-dict mode is excluded. Its sections always start at id 0, so there is * no lifetime anchor to shrink and nothing to gain -- while reclaiming would * let a later frame define a different string at an id a frame already on the * ring defines, which is precisely the hazard above. */ - private void reclaimUnsentSymbolIds() { + private void reclaimUnsentSymbolIds(int minFloor) { if (!deltaDictEnabled) { return; } - int floor = sentMaxSymbolId + 1; + int floor = Math.max(sentMaxSymbolId + 1, minFloor); PersistedSymbolDict pd = cursorEngine == null ? null : cursorEngine.getPersistedSymbolDict(); if (pd != null) { int durable = pd.size(); @@ -6983,6 +7250,7 @@ private void sendRow() { firstPendingRowTimeNanos = System.nanoTime(); } pendingRowCount++; + rowsAppended++; // Advance pendingBytes by the bytes the just-committed row added to // the current table. nextRow() accumulated the total during its @@ -6994,7 +7262,7 @@ private void sendRow() { currentTableBufferSnapshotBytes = bufferedNow; if (shouldAutoFlush()) { - flushPendingRows(transactional); + autoFlushAfterRow(); } } diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorSendEngine.java b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorSendEngine.java index daf38b77f..5ae526cbe 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorSendEngine.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/client/sf/cursor/CursorSendEngine.java @@ -942,25 +942,37 @@ public long appendBlocking(long payloadAddr, int payloadLen) { // deadline clock. counters.backpressureStalls.incrementAndGet(); long deadlineNs = System.nanoTime() + appendDeadlineNanos; - while (true) { - long now = System.nanoTime(); - if (now >= deadlineNs) { - throw new io.questdb.client.cutlass.line.LineSenderException( - "cursor ring backpressured for ").put(appendDeadlineNanos / 1_000_000L) - .put(" ms - wire path is not draining (server slow / disconnected, " - + "periodic disk sync is slow, or sf_max_total_bytes is too small)"); - } - if (now - lastBackpressureLogNs >= BACKPRESSURE_LOG_THROTTLE_NANOS) { - lastBackpressureLogNs = now; - LOG.warn("cursor producer backpressured ({} stalls so far); waiting for I/O or periodic disk sync; " - + "will throw after {} ms", - counters.backpressureStalls.get(), appendDeadlineNanos / 1_000_000L); + boolean hasClearedInterrupt = false; + try { + // parkNanos returns at once while the interrupt flag is set, so the wait + // clears the flag before each park and restores it on exit. + while (true) { + long now = System.nanoTime(); + if (now >= deadlineNs) { + throw new io.questdb.client.cutlass.line.LineSenderException( + "cursor ring backpressured for ").put(appendDeadlineNanos / 1_000_000L) + .put(" ms - wire path is not draining (server slow / disconnected, " + + "periodic disk sync is slow, or sf_max_total_bytes is too small)"); + } + if (now - lastBackpressureLogNs >= BACKPRESSURE_LOG_THROTTLE_NANOS) { + lastBackpressureLogNs = now; + LOG.warn("cursor producer backpressured ({} stalls so far); waiting for I/O or periodic disk sync; " + + "will throw after {} ms", + counters.backpressureStalls.get(), appendDeadlineNanos / 1_000_000L); + } + if (Thread.interrupted()) { + hasClearedInterrupt = true; + } + LockSupport.parkNanos(50_000L); // 50 µs + fsn = ring.appendOrFsn(payloadAddr, payloadLen); + if (fsn >= 0) return fsn; + if (fsn == SegmentRing.PAYLOAD_TOO_LARGE) { + throw new MmapSegmentException("payload too large for segment"); + } } - LockSupport.parkNanos(50_000L); // 50 µs - fsn = ring.appendOrFsn(payloadAddr, payloadLen); - if (fsn >= 0) return fsn; - if (fsn == SegmentRing.PAYLOAD_TOO_LARGE) { - throw new MmapSegmentException("payload too large for segment"); + } finally { + if (hasClearedInterrupt) { + Thread.currentThread().interrupt(); } } } diff --git a/core/src/main/java/io/questdb/client/cutlass/qwp/protocol/QwpTableBuffer.java b/core/src/main/java/io/questdb/client/cutlass/qwp/protocol/QwpTableBuffer.java index 9f01f0ab8..3965bfec4 100644 --- a/core/src/main/java/io/questdb/client/cutlass/qwp/protocol/QwpTableBuffer.java +++ b/core/src/main/java/io/questdb/client/cutlass/qwp/protocol/QwpTableBuffer.java @@ -70,8 +70,24 @@ public class QwpTableBuffer implements QuietCloseable { private boolean columnDefsCacheValid; private int committedColumnCount; // columns that existed at last nextRow() private ColumnBuffer[] fastColumns; // plain array for O(1) sequential access + // held* arrays: holdBackLastRow() captures each column's state before the + // last committed row here; while isRowHeld is set, the column writer encodes + // only the rows before the held one. + private int[] heldArrayDataOffsets; + private int[] heldArrayShapeOffsets; + private long[] heldStringDataSizes; + private int[] heldSymbolDictionarySizes; + private int[] heldValueCounts; private int inProgressColumnCount; + private boolean isRowHeld; private int rowCount; + // Savepoint for rollbackToSavepoint(): the row count and column count at the + // last markSavepoint(). reset(), retainInProgressRow() and retainHeldRow() + // move it to "no rows, all current columns", since a flushed buffer's columns + // are known to the server; retainHeldRow() keeps the column count when no + // row of this table shipped. clear() zeroes it. + private int savepointColumnCount; + private int savepointRowCount; public QwpTableBuffer(String tableName) { this(tableName, null); @@ -129,7 +145,10 @@ public void clear() { columnAccessCursor = 0; committedColumnCount = 0; inProgressColumnCount = 0; + isRowHeld = false; rowCount = 0; + savepointRowCount = 0; + savepointColumnCount = 0; columnDefsCacheValid = false; cachedColumnDefs = null; } @@ -190,6 +209,30 @@ public ColumnBuffer getExistingColumn(CharSequence name, byte type) { return lookupColumn(name, type); } + /** + * Per-column string data size before the held row, indexed by column + * index. Valid only while {@link #hasHeldRow()}. + */ + public long[] getHeldStringDataSizes() { + return heldStringDataSizes; + } + + /** + * Per-column symbol dictionary size before the held row, indexed by column + * index. Valid only while {@link #hasHeldRow()}. + */ + public int[] getHeldSymbolDictionarySizes() { + return heldSymbolDictionarySizes; + } + + /** + * Per-column value count before the held row, indexed by column index. + * Valid only while {@link #hasHeldRow()}. + */ + public int[] getHeldValueCounts() { + return heldValueCounts; + } + /** * Gets or creates a column with the given name and type. *

@@ -249,6 +292,15 @@ public int getRowCount() { return rowCount; } + /** + * Row count recorded by the last {@link #markSavepoint()}; {@code 0} when + * none was taken since the buffer was last reset, cleared or + * retained a row in progress or a held row. + */ + public int getSavepointRowCount() { + return savepointRowCount; + } + /** * Returns the table name. */ @@ -256,6 +308,14 @@ public String getTableName() { return tableName; } + /** + * Returns true while {@link #holdBackLastRow()} holds the last committed + * row back, until {@link #restoreHeldRow()} or {@link #retainHeldRow()}. + */ + public boolean hasHeldRow() { + return isRowHeld; + } + /** * Returns true if a row is partially built (columns written but not yet committed * via {@link #nextRow()}). @@ -264,6 +324,57 @@ public boolean hasInProgressRow() { return inProgressColumnCount > 0; } + /** + * Holds the last committed row back: records each column's state before + * that row and decrements the row count, so the column writer encodes only + * the rows before it. The row's values stay in the column buffers. + * {@link #restoreHeldRow()} puts the row back; {@link #retainHeldRow()} + * drops the rows before it and keeps it as the only row. The caller + * resolves the hold with one of the two before it writes, cancels or rolls + * back rows. + *

+ * Columns first used by the held row are part of the prefix frame, all-null + * for the prefix rows, so the server learns them, including their type and + * any decimal scale or geohash precision, even if the held row is later + * rolled back. The local symbol dictionary of the prefix may carry entries + * only the held row uses. + * + * @return false, changing nothing, when the buffer has no committed row, + * a row is in progress, or a row is already held + */ + public boolean holdBackLastRow() { + if (rowCount == 0 || isRowHeld || hasInProgressRow()) { + return false; + } + int columnCount = columns.size(); + ensureHeldColumnCapacity(columnCount); + int lastRow = rowCount - 1; + for (int i = 0; i < columnCount; i++) { + fastColumns[i].captureStateBeforeLastRow( + lastRow, + i, + heldValueCounts, + heldStringDataSizes, + heldSymbolDictionarySizes, + heldArrayShapeOffsets, + heldArrayDataOffsets + ); + } + rowCount = lastRow; + isRowHeld = true; + return true; + } + + /** + * Remembers the current row count and column count. A later + * {@link #rollbackToSavepoint()} truncates the buffer back to this point. + * Take it between rows, not while a row is in progress. + */ + public void markSavepoint() { + savepointRowCount = rowCount; + savepointColumnCount = columns.size(); + } + /** * Advances to the next row. *

@@ -344,9 +455,63 @@ public void reset() { columnAccessCursor = 0; committedColumnCount = columns.size(); inProgressColumnCount = 0; + isRowHeld = false; + savepointRowCount = 0; + savepointColumnCount = columns.size(); rowCount = 0; } + /** + * Undoes {@link #holdBackLastRow()}: the held row counts as committed + * again. A no-op when no row is held. + */ + public void restoreHeldRow() { + if (isRowHeld) { + rowCount++; + isRowHeld = false; + } + } + + /** + * Drops the rows before the held row and keeps the held row as the only + * committed row, as {@link #retainInProgressRow} does for a row in + * progress. The savepoint moves to no rows. When rows preceded the held + * row, the caller has shipped them with every current column, so the + * savepoint also takes every current column. When the held row is the + * table's only row, nothing of the table shipped, so the savepoint keeps + * its column count, which still names only columns the server knows: none + * for a new buffer, otherwise those at the last mark or flush. A later + * {@link #rollbackToSavepoint()} then drops the columns the held row + * added. A no-op when no row is held. + */ + public void retainHeldRow() { + if (!isRowHeld) { + return; + } + boolean hasShippedRows = rowCount > 0; + for (int i = 0, n = columns.size(); i < n; i++) { + ColumnBuffer col = fastColumns[i]; + // retainTailRow clears the geohash precision of a column whose held + // value is null; the server has locked it with the shipped rows. + int geohashPrecisionBefore = col.geohashPrecision; + col.retainTailRow( + rowCount, + heldValueCounts[i], + heldArrayShapeOffsets[i], + heldArrayDataOffsets[i] + ); + col.geohashPrecision = geohashPrecisionBefore; + } + rowCount = 1; + savepointRowCount = 0; + if (hasShippedRows) { + savepointColumnCount = columns.size(); + } + committedColumnCount = columns.size(); + columnAccessCursor = 0; + isRowHeld = false; + } + public void retainInProgressRow( int[] sizeBefore, int[] valueCountBefore, @@ -368,6 +533,46 @@ public void retainInProgressRow( } } rowCount = 0; + savepointRowCount = 0; + savepointColumnCount = columns.size(); + committedColumnCount = columns.size(); + } + + /** + * Discards the rows and the columns added since {@link #markSavepoint()}, + * including a row in progress, and keeps everything before the mark. + * Columns the mark did not see are closed and removed: the server has not + * seen them either, so they must not pad the retained rows. Without a mark + * since the last {@link #reset()} this drops every row and keeps the column + * definitions, which is what {@code reset()} does, including keeping a retained + * column's geohash precision; on a buffer that was never + * marked or reset it also removes every column, since the server has seen + * none of them. + */ + public void rollbackToSavepoint() { + columnAccessCursor = 0; + inProgressColumnCount = 0; + boolean columnsRemoved = false; + for (int i = columns.size() - 1; i >= savepointColumnCount; i--) { + ColumnBuffer col = columns.getQuick(i); + if (col != null) { + col.close(); + } + columns.remove(i); + columnsRemoved = true; + } + if (columnsRemoved) { + rebuildColumnAccessStructures(); + } + for (int i = 0, n = columns.size(); i < n; i++) { + ColumnBuffer col = fastColumns[i]; + // truncateTo clears the geohash precision of a column it empties; a + // retained column is known to the server, which has locked it. + int geohashPrecisionBefore = col.geohashPrecision; + col.truncateTo(savepointRowCount); + col.geohashPrecision = geohashPrecisionBefore; + } + rowCount = savepointRowCount; committedColumnCount = columns.size(); } @@ -431,6 +636,23 @@ private ColumnBuffer createColumn(CharSequence name, byte type, boolean useNullB return col; } + private void ensureHeldColumnCapacity(int required) { + if (heldValueCounts != null && required <= heldValueCounts.length) { + return; + } + + int newCapacity = heldValueCounts != null ? heldValueCounts.length : 8; + while (newCapacity < required) { + newCapacity *= 2; + } + + heldValueCounts = new int[newCapacity]; + heldStringDataSizes = new long[newCapacity]; + heldSymbolDictionarySizes = new int[newCapacity]; + heldArrayShapeOffsets = new int[newCapacity]; + heldArrayDataOffsets = new int[newCapacity]; + } + private ColumnBuffer lookupColumn(CharSequence name, byte type) { // Fast path: predict next column in sequence int n = columns.size(); @@ -1940,5 +2162,43 @@ private void retainSymbolValue(int valueIndex) { symbolDict.put(symbol, 0); Unsafe.getUnsafe().putInt(dataBuffer.pageAddress(), 0); } + + /** + * Writes this column's state before {@code lastRow}, its last committed + * row, into slot {@code columnIndex} of each array. The symbol dictionary + * size stays the current local one: WebSocket frames encode global ids, + * so a local entry that only the last row uses rides along unused. + */ + void captureStateBeforeLastRow( + int lastRow, + int columnIndex, + int[] valueCounts, + long[] stringDataSizes, + int[] symbolDictionarySizes, + int[] arrayShapeOffsets, + int[] arrayDataOffsets + ) { + boolean isLastNull = useNullBitmap && isNull(lastRow); + int valueCountBefore = isLastNull ? valueCount : valueCount - 1; + valueCounts[columnIndex] = valueCountBefore; + stringDataSizes[columnIndex] = stringOffsets != null + ? Unsafe.getUnsafe().getInt(stringOffsets.pageAddress() + (long) valueCountBefore * Integer.BYTES) + : 0; + symbolDictionarySizes[columnIndex] = getSymbolDictionarySize(); + + int shapeOffsetBefore = arrayShapeOffset; + int dataOffsetBefore = arrayDataOffset; + if (!isLastNull && arrayDims != null) { + int nDims = arrayDims[valueCountBefore] & 0xFF; + shapeOffsetBefore -= nDims; + int elemCount = 1; + for (int d = 0; d < nDims; d++) { + elemCount *= arrayShapes[shapeOffsetBefore + d]; + } + dataOffsetBefore -= elemCount; + } + arrayShapeOffsets[columnIndex] = shapeOffsetBefore; + arrayDataOffsets[columnIndex] = dataOffsetBefore; + } } } diff --git a/core/src/main/java/io/questdb/client/impl/ConfigSchema.java b/core/src/main/java/io/questdb/client/impl/ConfigSchema.java index b97256084..f1c404c83 100644 --- a/core/src/main/java/io/questdb/client/impl/ConfigSchema.java +++ b/core/src/main/java/io/questdb/client/impl/ConfigSchema.java @@ -124,6 +124,9 @@ public final class ConfigSchema { longRange("max_lifetime_ms", Side.POOL, OPEN, OPEN_MAX, false, false); longRange("housekeeper_interval_ms", Side.POOL, OPEN, OPEN_MAX, false, false); boolOnOff("lazy_connect", Side.POOL); // facade flag: tolerant non-blocking startup (async ingest + lazy reads) + boolOnOff("flush_on_close", Side.POOL); // pooled close() ships rows (on) or only releases the slot (off, default) + intRange("commit_rows", Side.POOL, OPEN, OPEN_MAX, false, false); // commit window: rows per server commit + longRange("commit_interval_ms", Side.POOL, OPEN, OPEN_MAX, false, false); // commit window: age per server commit // RESERVED -- accepted no-op (error-policy keys reserved by the spec). str("on_internal_error", Side.RESERVED); diff --git a/core/src/main/java/io/questdb/client/impl/PoolHousekeeper.java b/core/src/main/java/io/questdb/client/impl/PoolHousekeeper.java index 0778104b9..78ce4ed2f 100644 --- a/core/src/main/java/io/questdb/client/impl/PoolHousekeeper.java +++ b/core/src/main/java/io/questdb/client/impl/PoolHousekeeper.java @@ -33,6 +33,11 @@ */ final class PoolHousekeeper { + // Wait between startup-recovery steps that did not find more work to do at + // once: a finished scan, an empty one, or a transient failure such as an + // unreachable server. The same cadence SenderPool's private driver uses, and + // deliberately not the settle cadence, which can be 50 ms. + static final long STARTUP_RECOVERY_RETRY_MILLIS = SenderPool.RECOVERY_RETRY_INTERVAL_MILLIS; // How long stop() waits for the daemon to exit. Kept ABOVE // SenderPool.RECOVERY_DRAIN_BUDGET_MILLIS so a startup-recovery drain still // in flight when close() arrives finishes well within this join (C1 fix). @@ -48,16 +53,26 @@ final class PoolHousekeeper { static final long STOP_TIMEOUT_MILLIS = 2_000; private final long intervalMillis; + private final long originNanos = System.nanoTime(); private final QueryClientPool queryPool; private final SenderPool senderPool; + private final long settleIntervalMillis; private final Object signalLock = new Object(); private final Thread thread; private volatile boolean stop; - PoolHousekeeper(SenderPool senderPool, QueryClientPool queryPool, long intervalMillis) { + /** + * @param intervalMillis reap cadence: idle and over-age slots + * @param settleIntervalMillis settle cadence: ships rows left in idle slots; + * {@code auto_flush_interval} (100 ms when + * unset), or the commit interval when that is + * shorter + */ + PoolHousekeeper(SenderPool senderPool, QueryClientPool queryPool, long intervalMillis, long settleIntervalMillis) { this.senderPool = senderPool; this.queryPool = queryPool; this.intervalMillis = intervalMillis; + this.settleIntervalMillis = Math.max(1L, settleIntervalMillis); this.thread = new Thread(this::runLoop, "questdb-pool-housekeeper"); this.thread.setDaemon(true); } @@ -122,6 +137,16 @@ void stop() { } } + /** + * {@code nowMillis + intervalMillis}, saturating at {@link Long#MAX_VALUE} + * so an interval of {@code Long.MAX_VALUE} ("never") does not wrap negative + * and either stall every other deadline or fire at once. + */ + static long deadline(long nowMillis, long intervalMillis) { + long d = nowMillis + intervalMillis; + return d < nowMillis ? Long.MAX_VALUE : d; + } + /** * Waits up to the supplied budget without letting cancellation skip the caller's remaining shutdown * work. Every {@link InterruptedException} clears the caller's flag, so remember it and spend only the @@ -151,7 +176,15 @@ static boolean joinIgnoringCallerInterrupts(Thread target, long timeoutMillis) { return callerWasInterrupted; } + private long monotonicMillis() { + return TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - originNanos); + } + private void runLoop() { + long startMillis = monotonicMillis(); + long nextSettleAt = deadline(startMillis, settleIntervalMillis); + long nextReapAt = deadline(startMillis, intervalMillis); + long nextRecoveryAt = startMillis; while (!stop) { // Per-slot startup SF recovery, driven on THIS (the reap-loop) // thread, one stranded slot per iteration. Doing it here rather than @@ -167,49 +200,76 @@ private void runLoop() { // sole residual overrun is an in-flight connect to a black-holed // host, which no interrupt breaks. See SenderPool.recoverOneSlotStep. // While recovery still has work we skip the idle wait so the backlog - // drains promptly; once done we fall back to the normal interval. - // No-op once recovery completes or the pool is closing. Best-effort: + // drains promptly; otherwise the step is retried every + // STARTUP_RECOVERY_RETRY_MILLIS until the pool reports it done, and + // stays cheap after that. A no-op once recovery completes or the + // pool is closing. Best-effort: // a recovery failure (including an Error) must never kill this // daemon, so swallow Throwable -- exactly as the reap guards below. - boolean recovering; - try { - recovering = senderPool.runStartupRecoveryStep(); - } catch (Throwable ignored) { - recovering = false; + boolean recovering = false; + if (monotonicMillis() >= nextRecoveryAt) { + try { + recovering = senderPool.runStartupRecoveryStep(); + } catch (Throwable ignored) { + recovering = false; + } + // A step with more work runs again at once. A step that finished, + // found nothing or hit a transient failure is retried on its own + // cadence, never on the settle cadence: a stranded slot against an + // unreachable server must not become a hot connect loop. + long afterStep = monotonicMillis(); + nextRecoveryAt = recovering ? afterStep : deadline(afterStep, STARTUP_RECOVERY_RETRY_MILLIS); } synchronized (signalLock) { if (stop) { return; } if (!recovering) { - try { - signalLock.wait(intervalMillis); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - return; + long nextAt = Math.min(Math.min(nextSettleAt, nextReapAt), nextRecoveryAt); + long waitMillis = nextAt - monotonicMillis(); + if (waitMillis > 0) { + try { + signalLock.wait(waitMillis); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + return; + } } } } if (stop) { return; } - try { - senderPool.reapIdle(); - } catch (Throwable ignored) { - // Defensive, intentionally unreachable in normal operation: - // SenderPool.reapIdle() already swallows per-delegate close() - // failures internally. The outer catch is a belt-and-braces - // guard. Reaping must not propagate -- it's best-effort - // housekeeping. Catch Throwable (not just RuntimeException) so - // an Error from a delegate teardown can never kill this daemon - // thread and stop all future reaping for the life of the handle. + long now = monotonicMillis(); + if (now >= nextSettleAt) { + try { + senderPool.settleIdle(); + } catch (Throwable ignored) { + // Best-effort: settleIdle() already guards per-slot + // failures; a Throwable here must never kill the daemon. + } + nextSettleAt = deadline(now, settleIntervalMillis); } - try { - queryPool.reapIdle(); - } catch (Throwable ignored) { - // Same rationale as the senderPool guard above: best-effort, - // must never propagate, and Throwable (not RuntimeException) so - // an Error from query-client teardown cannot kill the daemon. + if (now >= nextReapAt) { + try { + senderPool.reapIdle(); + } catch (Throwable ignored) { + // Defensive, intentionally unreachable in normal operation: + // SenderPool.reapIdle() already swallows per-delegate close() + // failures internally. The outer catch is a belt-and-braces + // guard. Reaping must not propagate -- it's best-effort + // housekeeping. Catch Throwable (not just RuntimeException) so + // an Error from a delegate teardown can never kill this daemon + // thread and stop all future reaping for the life of the handle. + } + try { + queryPool.reapIdle(); + } catch (Throwable ignored) { + // Same rationale as the senderPool guard above: best-effort, + // must never propagate, and Throwable (not RuntimeException) so + // an Error from query-client teardown cannot kill the daemon. + } + nextReapAt = deadline(now, intervalMillis); } } } diff --git a/core/src/main/java/io/questdb/client/impl/PooledSender.java b/core/src/main/java/io/questdb/client/impl/PooledSender.java index 095f64abd..a3f1183d5 100644 --- a/core/src/main/java/io/questdb/client/impl/PooledSender.java +++ b/core/src/main/java/io/questdb/client/impl/PooledSender.java @@ -24,9 +24,12 @@ package io.questdb.client.impl; +import io.questdb.client.LineSenderServerException; import io.questdb.client.Sender; +import io.questdb.client.cutlass.line.LineSenderException; import io.questdb.client.cutlass.line.array.DoubleArray; import io.questdb.client.cutlass.line.array.LongArray; +import io.questdb.client.cutlass.qwp.client.BatchTooLargeForCapException; import io.questdb.client.std.Decimal128; import io.questdb.client.std.Decimal256; import io.questdb.client.std.Decimal64; @@ -44,9 +47,11 @@ * {@link Sender} call to the reused {@link SenderSlot}'s delegate, validating * that generation first via {@link SenderSlot#live(long)}. *

- * Behaviour difference from a raw Sender: {@link #close()} flushes the buffer - * and returns the slot to the pool. The underlying Sender is only truly closed - * when {@link io.questdb.client.QuestDB#close()} shuts the pool down. + * Behaviour difference from a raw Sender: {@link #close()} returns the slot to + * the pool and, by default, ships nothing; rows ship by threshold or by the + * pool's housekeeper. With {@code flush_on_close=on} close flushes first. The + * underlying Sender is only truly closed when + * {@link io.questdb.client.QuestDB#close()} shuts the pool down. *

* Because the slot is reused across borrows, this wrapper -- not the slot -- * carries the lease identity. A stale handle (held after {@link #close()}, with @@ -80,6 +85,11 @@ public void atNow() { slot.live(generation).atNow(); } + @Override + public void autoFlushIfDue() { + slot.live(generation).autoFlushIfDue(); + } + @Override public boolean awaitAckedFsn(long targetFsn, long timeoutMillis) { return slot.live(generation).awaitAckedFsn(targetFsn, timeoutMillis); @@ -131,56 +141,166 @@ public Sender charColumn(CharSequence name, char value) { return this; } + @Override + public void checkError() { + slot.live(generation).checkError(); + } + /** - * Flushes pending rows and returns this lease's slot to the pool. Does not - * actually close the underlying {@link Sender}; that only happens when the - * owning {@code QuestDB} is closed. If the pool is already shutting down - * when this lease comes home, the underlying Sender is torn down here, on - * the calling thread (see {@link SenderPool#close()}). + * Returns this lease's slot to the pool. By default nothing is shipped + * (see the commit-window note below): + * the rows stay in the slot's buffer and go out when an auto-flush + * threshold trips on a later append or when the pool's housekeeper settles + * the idle slot. With {@code flush_on_close=on} the rows are flushed first, + * as before 1.4. {@link #reset()} on a lease discards only that lease's rows. + * Does not close the underlying {@link Sender}; that only + * happens when the owning {@code QuestDB} is closed. If the pool is already + * shutting down when this lease comes home, the underlying Sender is torn + * down here, on the calling thread, which ships whatever it still buffers. + *

+ * A row left in progress is cancelled, the slot is returned, and then a + * {@link LineSenderException} is thrown so the mistake is visible. A + * delegate that has latched a terminal error is retired instead of being + * returned, and a new {@link LineSenderException} is thrown here whose + * cause is the delegate's error and whose message carries the number of unshipped rows at + * the retirement. The count lives on the wrapper of a non-server failure; the typed server + * branch below carries none. A lease body that already threw the typed + * error keeps it as the primary exception, with close's wrapper + * suppressed. The typed error and its classification live on the cause. + *

+ * A server batch cap rejection ({@link BatchTooLargeForCapException}, retained + * by an auto-flush or thrown by close's own flush) does not retire the slot at + * once. QwpWebSocketSender handles a byte-trigger crossing row itself, so a + * rejection that reaches close comes, for example, from a lease whose own + * rows push the batch over the cap through an explicit flush() with the byte + * trigger off, or from a held row whose own frame exceeds the cap. The + * rejected batch can hold mostly rows earlier leases left. The lease is rolled + * back to its borrow mark, which drops only this lease's rows, and the flush is + * retried: when the rows left by earlier leases ship, the slot is returned and a + * {@link LineSenderException} whose cause is the rejection names the number of + * this lease's rows dropped. Only an inherited batch the cap still rejects + * retires the slot, with the wrapper described above. *

* Idempotent for sequential reuse: a stale generation (the lease was - * already returned and the slot possibly re-borrowed) is a no-op, and the - * pool re-checks the generation under its lock, so a duplicate close can - * never re-enqueue a slot a different borrower now owns. Concurrent + * already returned and the slot possibly re-borrowed) is a no-op. Concurrent * double-close of ONE lease from two threads is outside the Sender - * threading contract; the flush below revalidates the lease via - * {@link SenderSlot#live(long)} so a racing duplicate typically fails with - * {@code IllegalStateException} rather than flushing into a re-borrowed - * slot, but two threads racing inside a single lease are not fully - * protected. + * threading contract. + *

+ * With a commit window ({@link SenderPool#setCommitWindow}) and + * {@code flush_on_close=on}, each close ships a deferred frame; the close + * that fills the window (rows appended or age of the open group) flushes + * instead, which commits the group. Without {@code flush_on_close}, only + * the release that closes the window ships: it flushes, committing the + * group. The window's age is measured from the first release that found + * work. A failed ship retires the slot and throws the wrapper described above. + *

+ * Close-time errors are wrapped: the delegate's error is the cause. A latched + * {@link LineSenderServerException} is rethrown as a new one carrying the same + * {@code SenderError}. While the pool is shutting down, the rows are flushed here + * so a ship failure propagates to the caller. */ @Override public void close() { if (generation != slot.generation()) { return; } - // Track normal completion rather than catching a specific throwable - // type. flush() can exit abnormally with an Error (AssertionError - // under -ea, OutOfMemoryError, ...) as well as a RuntimeException; - // keying the recycle decision off normal completion treats every - // abnormal exit as unrecyclable, which is the fail-safe default. - boolean flushed = false; + SenderPool pool = slot.pool(); + boolean cancelledRow = false; + boolean recycle = false; + RuntimeException failure = null; + BatchTooLargeForCapException capRejection = null; + int droppedLeaseRows = 0; + int pendingRowsAtRetire = 0; try { - // live(), not delegate(): re-validating the generation right - // before the flush narrows the duplicate-close race documented - // above. A stale duplicate throws here, and the finally routes it - // to discardBroken, which drops it on the same stale-generation - // check under the pool lock. - slot.live(generation).flush(); - flushed = true; + // live(), not delegate(): re-validate the generation right before + // touching the delegate, narrowing the duplicate-close race. + Sender delegate = slot.live(generation); + try { + delegate.checkError(); + if (delegate.hasRowInProgress()) { + delegate.cancelRow(); + cancelledRow = true; + } + if (pool.isClosed()) { + // Pool shutting down: ship here so a failure reaches the caller. + delegate.flush(); + } else { + ship(pool, delegate); + } + recycle = true; + } catch (RuntimeException e) { + // The delegate rethrows its latched instance, which a lease + // body may already have thrown: wrap it below, after the + // slot is retired. + failure = e; + } + if (failure instanceof BatchTooLargeForCapException) { + // QwpWebSocketSender handles a byte-trigger crossing row itself, so + // this rejection comes, for example, from the lease's own rows + // pushing the batch over the cap through an explicit flush() with + // the byte trigger off, or from a held row whose own frame exceeds + // the cap. The + // rejected batch can hold mostly rows earlier leases left. + // Drop only this lease's rows and retry: the slot stays in service + // when the rest ships, and an inherited batch the cap still + // rejects falls through to the retirement below. + try { + int pendingBeforeRollback = delegate.pendingRowCount(); + delegate.resetToSavepoint(); + droppedLeaseRows = pendingBeforeRollback - delegate.pendingRowCount(); + delegate.flush(); + slot.markCommitted(delegate.rowsAppended()); + capRejection = (BatchTooLargeForCapException) failure; + failure = null; + recycle = true; + } catch (RuntimeException e) { + failure = e; + } + } } finally { - if (flushed) { - slot.pool().giveBack(this); + if (recycle) { + pool.giveBack(this); } else { - // flush() did not complete normally. Sender does not clear its - // buffer on flush failure (see Sender Javadoc), and WebSocket - // transport latches the failure for good. Either way the slot - // is unsafe to recycle: the next borrower would inherit the - // failed rows or a dead connection. The original throwable - // propagates naturally once this finally returns -- no explicit - // rethrow needed. - slot.pool().discardBroken(this); + // The probe, the cancel or the ship threw: the delegate has + // latched an error, the flush failed, or the generation went + // stale. Retire it; a wrapper around the original propagates + // once this finally returns. Rows a rollback above dropped are + // lost with the slot as well. + try { + pendingRowsAtRetire = droppedLeaseRows + slot.delegate().pendingRowCount(); + } catch (Throwable ignored) { + // best effort: the count is only for the message + } + pool.discardBroken(this); + } + } + if (failure != null) { + if (failure instanceof LineSenderServerException) { + LineSenderServerException typed = new LineSenderServerException(((LineSenderServerException) failure).getServerError()); + typed.initCause(failure); + throw typed; } + // A cap rejection's message ends with the standalone remedy, which does + // not apply to a retired slot: report only its detail. + String msg = failure instanceof BatchTooLargeForCapException + ? ((BatchTooLargeForCapException) failure).getDetail() + : failure.getMessage(); + boolean retryable = failure instanceof LineSenderException && ((LineSenderException) failure).isRetryable(); + LineSenderException wrapper = new LineSenderException("pooled sender close failed, slot retired with " + + pendingRowsAtRetire + " unshipped row(s): " + msg, retryable); + wrapper.initCause(failure); + throw wrapper; + } + if (capRejection != null) { + LineSenderException wrapper = new LineSenderException("pooled sender close dropped " + droppedLeaseRows + + " row(s) of this lease after the server batch cap rejected the batch; rows left by earlier leases shipped", + capRejection.isRetryable()); + wrapper.initCause(capRejection); + throw wrapper; + } + if (cancelledRow) { + throw new LineSenderException("row in progress at close(): the row was cancelled and the sender returned to the pool; " + + "finish rows with at(), atNow() or cancelRow() before close()"); } } @@ -240,7 +360,10 @@ public Sender doubleColumn(CharSequence name, double value) { @Override public boolean drain(long timeoutMillis) { - return slot.live(generation).drain(timeoutMillis); + Sender delegate = slot.live(generation); + boolean drained = delegate.drain(timeoutMillis); + slot.markCommitted(delegate.rowsAppended()); + return drained; } @Override @@ -251,12 +374,22 @@ public Sender floatColumn(CharSequence name, float value) { @Override public void flush() { - slot.live(generation).flush(); + Sender delegate = slot.live(generation); + delegate.flush(); + slot.markCommitted(delegate.rowsAppended()); } @Override public long flushAndGetSequence() { - return slot.live(generation).flushAndGetSequence(); + Sender delegate = slot.live(generation); + long sequence = delegate.flushAndGetSequence(); + slot.markCommitted(delegate.rowsAppended()); + return sequence; + } + + @Override + public long flushDeferred() { + return slot.live(generation).flushDeferred(); } @Override @@ -276,6 +409,16 @@ public long getAckedFsn() { return slot.live(generation).getAckedFsn(); } + @Override + public boolean hasDeferredFrames() { + return slot.live(generation).hasDeferredFrames(); + } + + @Override + public boolean hasRowInProgress() { + return slot.live(generation).hasRowInProgress(); + } + @Override public Sender intColumn(CharSequence name, int value) { slot.live(generation).intColumn(name, value); @@ -330,9 +473,22 @@ public Sender longColumn(CharSequence name, long value) { return this; } + @Override + public int pendingRowCount() { + return slot.live(generation).pendingRowCount(); + } + + /** + * Discards the rows appended through this lease and keeps the rows earlier + * leases left in the slot's buffer: the pool marks a savepoint at borrow and + * this rolls back to it ({@link Sender#resetToSavepoint()}). After a flush + * inside the lease, explicit or by auto-flush, every pending row is this + * lease's own and all of them go. Use it to drop a batch the server cannot + * accept, for example after a {@code BatchTooLargeForCapException}. + */ @Override public void reset() { - slot.live(generation).reset(); + slot.live(generation).resetToSavepoint(); } @Override @@ -340,6 +496,11 @@ public void resetSymbolDictionary() { slot.live(generation).resetSymbolDictionary(); } + @Override + public long rowsAppended() { + return slot.live(generation).rowsAppended(); + } + @Override public Sender shortColumn(CharSequence name, short value) { slot.live(generation).shortColumn(name, value); @@ -404,4 +565,37 @@ public boolean hasSameSlotForTesting(PooledSender that) { SenderSlot slot() { return slot; } + + /** + * What a release ships, per the pool's settings: + *

+ */ + private void ship(SenderPool pool, Sender delegate) { + if (pool.hasCommitWindow()) { + boolean closes = slot.windowCloses(delegate.rowsAppended(), delegate.hasDeferredFrames(), + System.nanoTime(), pool.commitRows(), pool.commitIntervalNanos()); + if (closes) { + delegate.flush(); + slot.markCommitted(delegate.rowsAppended()); + return; + } + if (pool.isFlushOnClose()) { + delegate.flushDeferred(); + } + return; + } + if (pool.isFlushOnClose()) { + delegate.flush(); + } else { + delegate.autoFlushIfDue(); + } + } } diff --git a/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java b/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java index 574d6b59e..f20e25522 100644 --- a/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java +++ b/core/src/main/java/io/questdb/client/impl/QuestDBImpl.java @@ -72,7 +72,8 @@ public QuestDBImpl( this(ingestConfig, queryConfig, senderMin, senderMax, queryMin, queryMax, acquireTimeoutMillis, idleTimeoutMillis, maxLifetimeMillis, housekeeperIntervalMillis, queryCloseTimeoutMillis, null, - errorHandler, connectionListener, drainerListener); + errorHandler, connectionListener, drainerListener, + false, housekeeperIntervalMillis, 0, 0L); } public QuestDBImpl( @@ -90,12 +91,17 @@ public QuestDBImpl( HttpTokenProvider tokenProvider, SenderErrorHandler errorHandler, SenderConnectionListener connectionListener, - BackgroundDrainerListener drainerListener + BackgroundDrainerListener drainerListener, + boolean flushOnClose, + long settleIntervalMillis, + int commitRows, + long commitIntervalMillis ) { this(ingestConfig, queryConfig, senderMin, senderMax, queryMin, queryMax, acquireTimeoutMillis, idleTimeoutMillis, maxLifetimeMillis, housekeeperIntervalMillis, queryCloseTimeoutMillis, null, null, - tokenProvider, errorHandler, connectionListener, drainerListener); + tokenProvider, errorHandler, connectionListener, drainerListener, + flushOnClose, settleIntervalMillis, commitRows, commitIntervalMillis); } // Test-only constructor exposing the senderFactory and connectHook seams: @@ -122,7 +128,8 @@ public QuestDBImpl( this(ingestConfig, queryConfig, senderMin, senderMax, queryMin, queryMax, acquireTimeoutMillis, idleTimeoutMillis, maxLifetimeMillis, housekeeperIntervalMillis, QueryClientPool.DEFAULT_CLOSE_QUERY_TIMEOUT_MILLIS, - senderFactory, connectHook, null, null, null, null); + senderFactory, connectHook, null, null, null, null, + false, housekeeperIntervalMillis, 0, 0L); } // Full constructor adding the ingest-side errorHandler/connectionListener/ @@ -147,7 +154,11 @@ public QuestDBImpl( HttpTokenProvider tokenProvider, SenderErrorHandler errorHandler, SenderConnectionListener connectionListener, - BackgroundDrainerListener drainerListener + BackgroundDrainerListener drainerListener, + boolean flushOnClose, + long settleIntervalMillis, + int commitRows, + long commitIntervalMillis ) { SenderPool builtSenderPool = null; QueryClientPool builtQueryPool = null; @@ -161,11 +172,15 @@ public QuestDBImpl( // server; the housekeeper drives it via runStartupRecoveryStep(). true, errorHandler, connectionListener, drainerListener, tokenProvider); + builtSenderPool.setFlushOnClose(flushOnClose); + if (commitRows > 0) { + builtSenderPool.setCommitWindow(commitRows, commitIntervalMillis); + } builtQueryPool = new QueryClientPool( queryConfig, queryMin, queryMax, acquireTimeoutMillis, idleTimeoutMillis, maxLifetimeMillis, connectHook, null, tokenProvider); builtQueryPool.closeQueryTimeoutMillis(queryCloseTimeoutMillis); - builtHousekeeper = new PoolHousekeeper(builtSenderPool, builtQueryPool, housekeeperIntervalMillis); + builtHousekeeper = new PoolHousekeeper(builtSenderPool, builtQueryPool, housekeeperIntervalMillis, settleIntervalMillis); builtHousekeeper.start(); } catch (Throwable e) { // Catch Throwable, not just RuntimeException: this orchestrator is the diff --git a/core/src/main/java/io/questdb/client/impl/SenderPool.java b/core/src/main/java/io/questdb/client/impl/SenderPool.java index bc93309d3..ab4ada32d 100644 --- a/core/src/main/java/io/questdb/client/impl/SenderPool.java +++ b/core/src/main/java/io/questdb/client/impl/SenderPool.java @@ -30,6 +30,7 @@ import io.questdb.client.SenderError; import io.questdb.client.SenderErrorHandler; import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.cutlass.qwp.client.BatchTooLargeForCapException; import io.questdb.client.cutlass.qwp.client.QwpWebSocketSender; import io.questdb.client.cutlass.qwp.client.sf.cursor.BackgroundDrainerListener; import io.questdb.client.cutlass.qwp.client.sf.cursor.OrphanScanner; @@ -129,10 +130,6 @@ public final class SenderPool implements AutoCloseable { // the dominant server-wide transient failure keeps its retry-the-same- // candidate behavior for the first attempts. private static final int RECOVERY_MAX_SLOT_FAILURE_STREAK = 3; - // A direct SenderPool has no PoolHousekeeper to retry a transient startup - // recovery failure. Its private driver waits this long between failed - // attempts so an unavailable server does not cause a hot retry loop. - private static final long RECOVERY_RETRY_INTERVAL_MILLIS = 1_000; // Hard cap on close()'s wait for a creation that is blocked in DNS, TCP, // TLS, or WebSocket setup. The creator retains ownership after this budget: // once construction returns, borrow() observes closed and tears down the @@ -145,6 +142,11 @@ public final class SenderPool implements AutoCloseable { // shutdown join is capped at QueryWorker.SHUTDOWN_JOIN_MILLIS (same value) // regardless of user config. static final long MAX_CLOSE_LEASE_WAIT_MILLIS = 5_000; + // A direct SenderPool has no PoolHousekeeper to retry a transient startup + // recovery failure. Its private driver waits this long between failed + // attempts so an unavailable server does not cause a hot retry loop. The + // housekeeper waits the same between its own startup-recovery steps. + static final long RECOVERY_RETRY_INTERVAL_MILLIS = 1_000; private final long acquireTimeoutMillis; private final ArrayList all; private final ArrayDeque available; @@ -246,11 +248,16 @@ public final class SenderPool implements AutoCloseable { // production; regression tests release a retired slot here to prove that // the terminal pass re-probes returned capacity before throwing. private volatile Runnable borrowWaitExpiredHook; + private volatile long commitIntervalNanos; + private volatile int commitRows; // Test seam invoked after close() handles an interrupt and confirms the // original creation-wait deadline still permits another wait. Null in // production; lifecycle tests use it to acknowledge distinct retries // without inspecting transient Condition queue membership. private volatile Runnable creationWaitRetryHook; + // On for a directly built pool, which has no housekeeper to settle rows a + // release leaves in an idle slot. QuestDBImpl sets it from flush_on_close. + private volatile boolean flushOnClose = true; // Slots removed from `all` whose delegate is still releasing its flock. // They keep reserving capacity (and their slotInUse mark) until the // flock drops, so the cap check and the slot allocator stay consistent @@ -359,6 +366,7 @@ public final class SenderPool implements AutoCloseable { private int recoveryFailStreak; private int recoveryFailStreakSlot = -1; private final IntList recoveryWarnedSlots = new IntList(); + private volatile int startupRecoveryBuildAttempts; public SenderPool( String configurationString, @@ -1238,12 +1246,22 @@ public PooledSender borrow() { throw new LineSenderException("QuestDB handle is closed"); } if (!available.isEmpty()) { - SenderSlot s = available.pollFirst(); + // LIFO: the most recently returned slot is the warmest and + // its buffer is the fullest; cold slots sink to the head + // where reapIdle() visits them first. + SenderSlot s = available.pollLast(); // Stamp a fresh lease id under the lock so the PooledSender // wrapper handed out can be told apart from any prior, // now-stale borrow of the same slot. s.bumpGeneration(); - return new PooledSender(s, s.generation()); + PooledSender lease = new PooledSender(s, s.generation()); + lock.unlock(); + // Lease-scoped reset: the borrower's reset() rolls back to here, so rows + // earlier leases left in this buffer survive it. The slot is off the free + // list under a fresh generation, so no other thread touches the delegate, + // and the mark's walk over every buffered table no longer serializes borrowers. + s.delegate().markSavepoint(); + return lease; } if (all.size() + inFlightCreations + closingSlots + leakedSlots + recoveringSlots < maxSize) { inFlightCreations++; @@ -1313,7 +1331,12 @@ public PooledSender borrow() { created.bumpGeneration(); inFlightCreations--; creationFinished.signalAll(); - return new PooledSender(created, created.generation()); + PooledSender lease = new PooledSender(created, created.generation()); + lock.unlock(); + // Every lease handed to a caller starts at a mark. The new slot is off + // the free list and owned by this borrower, so the mark runs outside the pool lock. + created.delegate().markSavepoint(); + return lease; } // Capacity-starved: re-probe retired slots BEFORE the terminal // timeout check — a deferred engine cleanup may have released a @@ -1488,6 +1511,11 @@ public void setCreationWaitRetryHookForTesting(Runnable hook) { this.creationWaitRetryHook = hook; } + @TestOnly + public int startupRecoveryBuildAttemptsForTesting() { + return startupRecoveryBuildAttempts; + } + /** * Raises the shutdown signal early -- without tearing down live delegates -- * so an in-flight startup-recovery step stops promptly between slots. Direct @@ -1586,7 +1614,8 @@ public void close() { inFlightCreations, creationWaitMillis); } // Bounded graceful wait for outstanding leases. A slot is borrowed - // iff it is in `all` but not in `available`; retireLease's + // iff it is in `all` but not in `available`, which includes the + // internal leases settleIdle() holds while it flushes; retireLease's // delegate-close section (running outside the lock on a returning // borrower's thread) is tracked by pendingLeaseTeardowns so this // method does not return while a teardown is still in flight. @@ -1625,9 +1654,18 @@ public void close() { // Close each idle delegate outside the lock so a slow real-close() // doesn't keep the pool latched. for (int i = 0; i < idleSnapshot.length; i++) { + SenderSlot s = idleSnapshot[i]; + // Rows earlier leases released into the slot; a failed close drops + // them, and nothing else reports it. + int pendingRows = pendingRowsOf(s); try { - idleSnapshot[i].delegate().close(); - } catch (Throwable ignored) { + s.delegate().close(); + } catch (Throwable t) { + if (pendingRows > 0) { + String error = t.toString(); + LOG.error("SenderPool.close(): idle slot {} closed with {} unshipped row(s); delegate close failed: {}", + s.slotIndex(), pendingRows, error); + } // Best-effort: an Error from one delegate's teardown must not // abort the loop and strand the remaining delegates unclosed. } @@ -1705,6 +1743,132 @@ public void setStartupRecoveryJoinHooksForTesting(Runnable beforeJoinHook, Runna this.afterStartupRecoveryJoinHook = afterJoinHook; } + /** + * Ships rows left behind in idle slots. With flush on close off, the + * {@code QuestDB} facade's default, a pooled {@code close()} does not flush, + * and auto-flush thresholds are only evaluated on append, + * so a slot nobody borrows again would keep its last rows until the pool + * closes. The housekeeper calls this on its settle cadence. + *

+ * The settle lists the dirty slots under the pool lock, then takes one slot at + * a time off the free list under that lock and stamps it with a fresh + * generation, which makes it an internal lease: a + * borrower arriving meanwhile takes another slot or waits, and + * {@link #close()} waits for it like any outstanding lease. The flush runs + * outside the lock. After the flush, whether it succeeded or not, the + * delegate is probed with {@link Sender#checkError()}: a latched error + * retires the slot, anything else returns it to the free list, and rows a + * failed flush left behind wait for the next settle. An {@link Error} + * thrown by the flush is not transient and retires the slot, and so does a + * {@code BatchTooLargeForCapException}: that batch belongs to no lease, a settle retry + * cannot ship it, so the slot is retired and the dropped row count logged. The slot's idle clock + * is not touched, so reaping is unaffected. + * + * @return the number of slots whose flush completed + */ + public int settleIdle() { + if (closed) { + return 0; + } + ArrayList candidates = null; + lock.lock(); + try { + if (closed) { + return 0; + } + for (SenderSlot s : available) { + if (hasUnshippedWork(s.delegate())) { + if (candidates == null) { + candidates = new ArrayList<>(); + } + candidates.add(s); + } + } + } finally { + lock.unlock(); + } + if (candidates == null) { + return 0; + } + int settled = 0; + for (int i = 0, n = candidates.size(); i < n; i++) { + SenderSlot s = candidates.get(i); + PooledSender lease; + lock.lock(); + try { + if (closed) { + break; + } + // A borrower may have taken the slot since the listing: check + // membership before reading the delegate of a leased slot. + if (!available.contains(s) || !hasUnshippedWork(s.delegate())) { + continue; + } + available.remove(s); + s.bumpGeneration(); + lease = new PooledSender(s, s.generation()); + } finally { + lock.unlock(); + } + if (settleOne(lease)) { + settled++; + } + } + return settled; + } + + private boolean settleOne(PooledSender lease) { + boolean flushed = false; + Throwable fatal = null; + BatchTooLargeForCapException overCap = null; + try { + // PooledSender.flush() already marks the commit on the slot. + lease.flush(); + lease.slot().settleFailureLogged(false); + flushed = true; + } catch (BatchTooLargeForCapException e) { + // Deterministic: the probe would report it too, but no settle retry + // can ship this batch and no lease owns it any more. Retire the + // slot below. + overCap = e; + } catch (Exception e) { + // Transient (ring full, recycle in flight) or terminal: the + // requeue below tells them apart with the probe. + logSettleFailure(lease.slot(), e); + } catch (Throwable t) { + // An Error (AssertionError, OutOfMemoryError) is not transient. + fatal = t; + } + try { + if (fatal != null) { + LOG.warn("settleIdle: flush threw an Error, retiring slot {}", lease.slot().slotIndex(), fatal); + retireLease(lease, " during settle"); + } else if (overCap != null) { + LOG.error("settleIdle: slot {} holds a batch the server cap rejects, retiring it and dropping {} pending row(s): {}", + lease.slot().slotIndex(), pendingRowsOf(lease.slot()), overCap.getDetail()); + retireLease(lease, " during settle"); + } else { + requeueSettled(lease); + } + } catch (Throwable t) { + // Keep going: the remaining leases must not stay out of `available`. + LOG.warn("settleIdle: could not requeue or retire slot {}", lease.slot().slotIndex(), t); + } + return flushed; + } + + private void logSettleFailure(SenderSlot slot, Exception e) { + int idx = slot.slotIndex(); + int pending = pendingRowsOf(slot); + String error = e.toString(); + if (!slot.settleFailureLogged()) { + slot.settleFailureLogged(true); + LOG.warn("SenderPool: settle flush failed for slot {} with {} pending row(s), will retry: {}", idx, pending, error); + } else { + LOG.debug("SenderPool: settle flush failed for slot {} with {} pending row(s), will retry: {}", idx, pending, error); + } + } + private void stopFailedStartupRecoveryDriver(Thread recoveryThread) { if (recoveryThread == null || recoveryThread == Thread.currentThread()) { return; @@ -1774,11 +1938,71 @@ private void waitForStartupRecoveryRetry() { } } + /** + * What {@link #settleIdle()} looks for in an idle slot. Phase A: buffered + * rows and deferred frames awaiting their commit. + */ + private static boolean hasUnshippedWork(Sender delegate) { + return delegate.pendingRowCount() > 0 || delegate.hasDeferredFrames(); + } + + /** Best-effort pending row count of a slot's delegate, for log lines only. */ + private static int pendingRowsOf(SenderSlot slot) { + try { + return slot.delegate().pendingRowCount(); + } catch (Throwable ignored) { + return 0; + } + } + + /** + * Returns a slot held by {@link #settleIdle()} to the free list without + * resetting its idle clock, or retires it when the delegate has latched an + * error or the pool is shutting down. Mirrors {@link #giveBack} for the + * internal lease. + */ + private void requeueSettled(PooledSender lease) { + SenderSlot s = lease.slot(); + boolean healthy = false; + try { + s.delegate().checkError(); + healthy = true; + } catch (Throwable ignored) { + // latched: retire below + } + if (!healthy) { + int pendingRows = pendingRowsOf(s); + LOG.warn("settleIdle: delegate of slot {} latched an error, retiring it with {} pending row(s)", + s.slotIndex(), pendingRows); + retireLease(lease, " during settle"); + return; + } + lock.lock(); + try { + if (!closed) { + if (s.generation() != lease.generation()) { + return; + } + s.bumpGeneration(); + assert !available.contains(s) : "slot already present in available deque on requeueSettled"; + available.addLast(s); + slotReleased.signal(); + return; + } + } finally { + lock.unlock(); + } + // Pool is shutting down: tear the delegate down here, on the settle + // thread, exactly as giveBack does for a borrower's lease. + retireLease(lease, " during settle after pool close"); + } + /** * Retires one lease on the calling (borrower's) thread: validates the * lease generation under the lock, removes the slot from {@code all}, * closes the delegate OUTSIDE the lock, and reclaims the SF slot index. - * Shared by {@link #discardBroken} (broken delegate) and by + * Shared by {@link #discardBroken} (broken delegate), by + * {@link #requeueSettled} (settle lease with a dead delegate) and by * {@link #giveBack} when the pool is shutting down (delegated teardown -- * see {@link #close()}). *

@@ -1826,9 +2050,20 @@ private void retireLease(PooledSender ps, String context) { // Close the delegate outside the lock (releases the SF flock). Always // attempt it so native resources are freed even on the defensive path // where the wrapper had already left `all`. + int pendingAtRetire = 0; try { - s.delegate().close(); + pendingAtRetire = s.delegate().pendingRowCount(); } catch (Throwable ignored) { + // best effort: the count is only for the log line + } + try { + s.delegate().close(); + } catch (Throwable t) { + if (pendingAtRetire > 0) { + String error = t.toString(); + LOG.error("SenderPool: slot {} retired with {} unshipped row(s); delegate close failed: {}", + s.slotIndex(), pendingAtRetire, error); + } // Best-effort teardown: a delegate close() can throw an Error // (e.g. an -ea AssertionError) as well as a RuntimeException. // Either way the accounting in the finally below MUST run, @@ -1857,7 +2092,10 @@ private void retireLease(PooledSender ps, String context) { /** * Closes idle slots that have exceeded {@code idleTimeoutMillis} or that * have aged past {@code maxLifetimeMillis}. Never shrinks below - * {@code minSize}. Called by the {@link PoolHousekeeper} on its tick. + * {@code minSize}. Skips a slot that still holds unshipped work: closing it + * would drop rows a release left there, so it waits until + * {@link #settleIdle()} ships them. Called by the {@link PoolHousekeeper} + * on its tick. */ public void reapIdle() { if (closed) { @@ -1882,7 +2120,7 @@ public void reapIdle() { && (now - s.idleSinceMillis()) >= idleTimeoutMillis; boolean overAge = maxLifetimeMillis < Long.MAX_VALUE && (now - s.createdAtMillis()) >= maxLifetimeMillis; - if (idleExpired || overAge) { + if ((idleExpired || overAge) && !hasUnshippedWork(s.delegate())) { it.remove(); all.remove(s); // Keep the SF slot reserved until its flock is released @@ -1952,6 +2190,38 @@ public int totalSize() { } } + /** Age per commit window in nanoseconds, or 0 when no window is configured. */ + public long commitIntervalNanos() { + return commitIntervalNanos; + } + + /** Rows per commit window, or 0 when no window is configured. */ + public int commitRows() { + return commitRows; + } + + /** Whether a commit window is configured: pooled closes ship deferred frames and commit per window. */ + public boolean hasCommitWindow() { + return commitRows > 0; + } + + /** Whether {@code close()} has started; a lease returning now is torn down instead of recycled. */ + boolean isClosed() { + return closed; + } + + /** + * Whether a pooled {@code close()} ships the lease's rows before releasing + * the slot (the pre-1.4 behaviour) or only releases it. On by default for a + * directly built pool, since nothing settles its idle slots. The + * {@code QuestDB} facade sets it from {@code flush_on_close}, off by + * default: rows then ship when an auto-flush threshold trips on a later + * append, or when the housekeeper settles the idle slot. + */ + public boolean isFlushOnClose() { + return flushOnClose; + } + /** * Snapshot of the number of SF slots currently retired because a * delegate {@code close()} returned with the slot flock still held after @@ -1972,6 +2242,25 @@ public int leakedSlotCount() { } } + /** + * Configures the commit window. Both values positive enable it; {@code 0, 0} + * disables it; negative values are rejected. Must be set before the first borrow; the pool's delegates + * must have been built with {@code transaction=on} so that auto-flush + * inside a lease ships deferred frames too (QuestDBBuilder injects it). + */ + public void setCommitWindow(int commitRows, long commitIntervalMillis) { + if (commitRows < 0 || commitIntervalMillis < 0 || (commitRows > 0) != (commitIntervalMillis > 0)) { + throw new IllegalArgumentException("commit window needs both rows and interval, or neither: rows=" + + commitRows + ", intervalMillis=" + commitIntervalMillis); + } + this.commitRows = commitRows; + this.commitIntervalNanos = TimeUnit.MILLISECONDS.toNanos(commitIntervalMillis); + } + + public void setFlushOnClose(boolean flushOnClose) { + this.flushOnClose = flushOnClose; + } + private SenderSlot createSlot(IntFunction factory, int slotIndex) { Sender delegate = factory.apply(slotIndex); try { @@ -2009,6 +2298,7 @@ private SenderSlot createUnlocked(int slotIndex) { * {@link #drainCandidateSlotForRecovery}. */ private SenderSlot createRecoverer(int slotIndex) { + startupRecoveryBuildAttempts++; return createSlot(recoverySenderFactory, slotIndex); } diff --git a/core/src/main/java/io/questdb/client/impl/SenderSlot.java b/core/src/main/java/io/questdb/client/impl/SenderSlot.java index 5d9ed6ff7..0271a1830 100644 --- a/core/src/main/java/io/questdb/client/impl/SenderSlot.java +++ b/core/src/main/java/io/questdb/client/impl/SenderSlot.java @@ -48,11 +48,18 @@ final class SenderSlot { private final Sender delegate; private final SenderPool pool; private final int slotIndex; + // Commit-window state, guarded by the lease: only the thread holding the + // slot's current lease (a borrower or the settle step) reads or writes it. + // Long.MIN_VALUE = no group open since the last commit. + private long groupOpenedAtNanos = Long.MIN_VALUE; // Index in SenderPool.retiredSlots, or -1 while this slot is not retired. // Guarded by the pool lock. The pool maintains this index across swap // removals so the delegate's release callback can recover this exact slot // without scanning every other retired slot. private int retiredIndex = -1; + // Commit-window state, guarded by the lease (see groupOpenedAtNanos). + private long rowsAppendedAtLastCommit; + private boolean settleFailureLogged; // Monotonic lease id. Mutated only under the SenderPool lock (bumped in // borrow() when the slot is handed out and in giveBack()/discardBroken() // when it is returned). A PooledSender wrapper captures it live for its @@ -109,6 +116,15 @@ Sender live(long gen) { return delegate; } + /** + * Records that everything appended so far is committed. Called after a + * committing {@code flush()} on the delegate, by a release or by the settle. + */ + void markCommitted(long rowsAppended) { + rowsAppendedAtLastCommit = rowsAppended; + groupOpenedAtNanos = Long.MIN_VALUE; + } + void markIdleAt(long nowMillis) { idleSinceMillis = nowMillis; } @@ -125,7 +141,32 @@ void retiredIndex(int retiredIndex) { this.retiredIndex = retiredIndex; } + void settleFailureLogged(boolean logged) { + this.settleFailureLogged = logged; + } + + boolean settleFailureLogged() { + return settleFailureLogged; + } + int slotIndex() { return slotIndex; } + + /** + * Whether the commit window should close at this release. Rows in the + * window are counted from appends, so the answer does not depend on + * whether rows shipped at close or by auto-flush. The first release that + * finds work after a commit opens the group's clock. + */ + boolean windowCloses(long rowsAppended, boolean deferredOpen, long nowNanos, int commitRows, long commitIntervalNanos) { + long inWindow = rowsAppended - rowsAppendedAtLastCommit; + if (inWindow <= 0 && !deferredOpen) { + return false; + } + if (groupOpenedAtNanos == Long.MIN_VALUE) { + groupOpenedAtNanos = nowNanos; + } + return inWindow >= commitRows || nowNanos - groupOpenedAtNanos >= commitIntervalNanos; + } } diff --git a/core/src/test/java/io/questdb/client/test/PoolBorrowPerRowBenchmark.java b/core/src/test/java/io/questdb/client/test/PoolBorrowPerRowBenchmark.java new file mode 100644 index 000000000..0d75d4964 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/PoolBorrowPerRowBenchmark.java @@ -0,0 +1,80 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test; + +import io.questdb.client.QuestDB; +import io.questdb.client.Sender; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.Ignore; +import org.junit.Test; + +import java.util.concurrent.TimeUnit; + +/** + * The customer's shape: one million rows, one borrow per row, against the + * ack-all test server. Run by hand; prints wall time per configuration. + * Target: the default is within noise of a single held lease; the window + * numbers answer whether frames or commits dominate. + */ +@Ignore("manual benchmark") +public class PoolBorrowPerRowBenchmark { + private static final int ROWS = 1_000_000; + + @Test + public void run() throws Exception { + time("held lease, default", "", false); + time("borrow per row, default (buffer on close)", "", true); + time("borrow per row, flush_on_close=on (today's shape)", "flush_on_close=on;", true); + time("borrow per row, flush_on_close=on + window 1000/100", "flush_on_close=on;commit_rows=1000;commit_interval_ms=100;", true); + } + + private static void time(String label, String extra, boolean borrowPerRow) throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + server.awaitStart(5, TimeUnit.SECONDS); + String cfg = "ws::addr=localhost:" + server.getPort() + + ";sender_pool_min=1;sender_pool_max=1;query_pool_min=0;close_flush_timeout_millis=60000;" + extra; + long start = System.nanoTime(); + try (QuestDB db = QuestDB.connect(cfg)) { + if (borrowPerRow) { + for (int i = 0; i < ROWS; i++) { + try (Sender s = db.borrowSender()) { + s.table("bench").symbol("sym", "ABC").doubleColumn("px", i).longColumn("qty", i).atNow(); + } + } + } else { + try (Sender s = db.borrowSender()) { + for (int i = 0; i < ROWS; i++) { + s.table("bench").symbol("sym", "ABC").doubleColumn("px", i).longColumn("qty", i).atNow(); + } + } + } + } + long millis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start); + System.out.printf("%-60s %6d ms, %d data frames%n", label, millis, log.dataFrames()); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/QuestDBBuilderTest.java b/core/src/test/java/io/questdb/client/test/QuestDBBuilderTest.java index e338aa72c..c830cb770 100644 --- a/core/src/test/java/io/questdb/client/test/QuestDBBuilderTest.java +++ b/core/src/test/java/io/questdb/client/test/QuestDBBuilderTest.java @@ -29,6 +29,8 @@ import io.questdb.client.QuestDBBuilder; import io.questdb.client.Query; import io.questdb.client.Sender; +import io.questdb.client.impl.ConfigString; +import io.questdb.client.impl.ConfigView; import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; import org.junit.Assert; import org.junit.Test; @@ -241,6 +243,166 @@ public void testTlsRootsWithUnsafeOffRejectedAtBuildWithMinZero() { ); } + @Test + public void testFlushOnCloseBuilderCallOverridesString() { + QuestDBBuilder b = QuestDB.builder() + .fromConfig("ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;flush_on_close=off;") + .flushOnClose(true); + try (QuestDB ignored = b.build()) { + Assert.assertEquals(Boolean.TRUE, b.poolConfigSnapshotForTest().get("flush_on_close")); + } + } + + @Test + public void testFlushOnCloseKeyParsedAndSnapshotted() { + QuestDBBuilder on = QuestDB.builder() + .fromConfig("ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;flush_on_close=on;"); + try (QuestDB ignored = on.build()) { + Assert.assertEquals(Boolean.TRUE, on.poolConfigSnapshotForTest().get("flush_on_close")); + } + QuestDBBuilder dflt = QuestDB.builder() + .fromConfig("ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;"); + try (QuestDB ignored = dflt.build()) { + Assert.assertEquals("default is off", Boolean.FALSE, dflt.poolConfigSnapshotForTest().get("flush_on_close")); + } + QuestDBBuilder t = QuestDB.builder() + .fromConfig("ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;flush_on_close=true;"); + try (QuestDB ignored = t.build()) { + Assert.assertEquals(Boolean.TRUE, t.poolConfigSnapshotForTest().get("flush_on_close")); + } + QuestDBBuilder f = QuestDB.builder() + .fromConfig("ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;flush_on_close=false;"); + try (QuestDB ignored = f.build()) { + Assert.assertEquals(Boolean.FALSE, f.poolConfigSnapshotForTest().get("flush_on_close")); + } + } + + @Test + public void testFlushOnCloseRejectsInvalidValue() { + assertBuildRejected( + "ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;flush_on_close=maybe;", + "flush_on_close"); + } + + @Test + public void testCommitWindowKeysParsedWithDefaults() { + QuestDBBuilder rowsOnly = QuestDB.builder() + .fromConfig("ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;flush_on_close=on;commit_rows=500;"); + try (QuestDB ignored = rowsOnly.build()) { + Assert.assertEquals(500, rowsOnly.poolConfigSnapshotForTest().get("commit_rows")); + Assert.assertEquals("interval defaults to the auto-flush default", 100L, rowsOnly.poolConfigSnapshotForTest().get("commit_interval_ms")); + } + QuestDBBuilder intervalOnly = QuestDB.builder() + .fromConfig("ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;flush_on_close=on;commit_interval_ms=250;"); + try (QuestDB ignored = intervalOnly.build()) { + Assert.assertEquals("rows default to the auto-flush default", 1000, intervalOnly.poolConfigSnapshotForTest().get("commit_rows")); + Assert.assertEquals(250L, intervalOnly.poolConfigSnapshotForTest().get("commit_interval_ms")); + } + QuestDBBuilder none = QuestDB.builder() + .fromConfig("ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;"); + try (QuestDB ignored = none.build()) { + Assert.assertEquals("no window", 0, none.poolConfigSnapshotForTest().get("commit_rows")); + Assert.assertEquals(0L, none.poolConfigSnapshotForTest().get("commit_interval_ms")); + } + } + + @Test + public void testCommitWindowBuilderCallsOverrideString() { + QuestDBBuilder b = QuestDB.builder() + .fromConfig("ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;flush_on_close=on;commit_rows=500;") + .commitRows(7) + .commitIntervalMillis(9L); + try (QuestDB ignored = b.build()) { + Assert.assertEquals(7, b.poolConfigSnapshotForTest().get("commit_rows")); + Assert.assertEquals(9L, b.poolConfigSnapshotForTest().get("commit_interval_ms")); + } + } + + @Test + public void testSettleIntervalTakesShorterOfAutoFlushAndCommitIntervals() { + ConfigView v = new ConfigView(ConfigString.parse("ws::addr=127.0.0.1:1;auto_flush_interval=250;")); + Assert.assertEquals(40L, QuestDBBuilder.resolveSettleIntervalMillis(v, 40L)); + Assert.assertEquals(250L, QuestDBBuilder.resolveSettleIntervalMillis(v, 900L)); + Assert.assertEquals(250L, QuestDBBuilder.resolveSettleIntervalMillis(v, 0L)); + ConfigView unset = new ConfigView(ConfigString.parse("ws::addr=127.0.0.1:1;")); + Assert.assertEquals(40L, QuestDBBuilder.resolveSettleIntervalMillis(unset, 40L)); + Assert.assertEquals(100L, QuestDBBuilder.resolveSettleIntervalMillis(unset, 900L)); + } + + @Test + public void testWindowInjectsTransactionOnIntoIngestConfig() { + QuestDBBuilder b = QuestDB.builder() + .fromConfig("ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;flush_on_close=on;commit_rows=10;"); + try (QuestDB ignored = b.build()) { + Assert.assertTrue(b.ingestConfigForTest().startsWith("ws::transaction=on;")); + } + QuestDBBuilder explicitOn = QuestDB.builder() + .fromConfig("ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;flush_on_close=on;commit_rows=10;transaction=on;"); + try (QuestDB ignored = explicitOn.build()) { + String resolved = explicitOn.ingestConfigForTest(); + Assert.assertEquals(resolved, resolved.indexOf("transaction=on;"), resolved.lastIndexOf("transaction=on;")); + Assert.assertTrue(resolved.contains("transaction=on;")); + } + QuestDBBuilder none = QuestDB.builder() + .fromConfig("ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;"); + try (QuestDB ignored = none.build()) { + Assert.assertFalse(none.ingestConfigForTest().contains("transaction=")); + } + } + + @Test + public void testWindowWithTransactionOffIsRejected() { + assertBuildRejected( + "ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;flush_on_close=on;commit_rows=10;transaction=off;", + "transaction"); + } + + @Test + public void testWindowWithoutFlushOnCloseIsRejected() { + assertBuildRejected( + "ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;commit_rows=10;", + "flush_on_close"); + assertBuildRejected( + "ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;flush_on_close=off;commit_interval_ms=10;", + "flush_on_close"); + } + + @Test + public void testBuildTwiceWithoutWindowStaysWindowless() { + QuestDBBuilder b = QuestDB.builder() + .fromConfig("ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;"); + for (int i = 0; i < 2; i++) { + try (QuestDB ignored = b.build()) { + Assert.assertEquals(0, b.poolConfigSnapshotForTest().get("commit_rows")); + Assert.assertEquals(0L, b.poolConfigSnapshotForTest().get("commit_interval_ms")); + Assert.assertFalse(b.ingestConfigForTest().contains("transaction=")); + } + } + } + + @Test + public void testCommitWindowStringKeysBelowOneAreRejected() { + assertBuildRejected( + "ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;flush_on_close=on;commit_rows=0;", + "commit_rows"); + assertBuildRejected( + "ws::addr=127.0.0.1:1;sender_pool_min=0;query_pool_min=0;flush_on_close=on;commit_interval_ms=-5;", + "commit_interval_ms"); + } + + @Test + public void testSettleIntervalFollowsAutoFlushInterval() { + // The WebSocket transport rejects both auto_flush=off and + // auto_flush_interval=off, so the interval is unset (the sender's + // 100 ms default) or a number (underscore separators allowed). + ConfigView dflt = new ConfigView(ConfigString.parse("ws::addr=127.0.0.1:1;")); + Assert.assertEquals(100L, QuestDBBuilder.resolveSettleIntervalMillis(dflt, 0L)); + ConfigView explicit = new ConfigView(ConfigString.parse("ws::addr=127.0.0.1:1;auto_flush_interval=250;")); + Assert.assertEquals(250L, QuestDBBuilder.resolveSettleIntervalMillis(explicit, 0L)); + ConfigView underscored = new ConfigView(ConfigString.parse("ws::addr=127.0.0.1:1;auto_flush_interval=1_000;")); + Assert.assertEquals(1_000L, QuestDBBuilder.resolveSettleIntervalMillis(underscored, 0L)); + } + @Test public void testMalformedPoolValueRejectedAtBuild() { // A non-numeric pool value is rejected at build()'s pool-key resolution, diff --git a/core/src/test/java/io/questdb/client/test/QuestDBCommitWindowTest.java b/core/src/test/java/io/questdb/client/test/QuestDBCommitWindowTest.java new file mode 100644 index 000000000..ffcab234a --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/QuestDBCommitWindowTest.java @@ -0,0 +1,251 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test; + +import io.questdb.client.LineSenderServerException; +import io.questdb.client.QuestDB; +import io.questdb.client.Sender; +import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.cutlass.qwp.client.WebSocketResponse; +import io.questdb.client.impl.PooledSender; +import io.questdb.client.test.QuestDBFlushOnCloseTest.FrameLog; +import io.questdb.client.test.cutlass.qwp.client.QwpWireTestUtils; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.Assert; +import org.junit.Test; + +import java.io.IOException; +import java.util.concurrent.TimeUnit; + +/** + * The commit window as the server sees it: flagged frames per close, one + * unflagged frame per window, a rowless commit message for an idle tail, and + * the client's reaction to a rejected frame inside a group. + */ +public class QuestDBCommitWindowTest { + + @Test + public void testAgeWindowCommitsAtNextRelease() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = QuestDB.connect(config(server.getPort(), "commit_rows=1000;commit_interval_ms=100;housekeeper_interval_ms=60000;auto_flush_interval=60000;"))) { + try (Sender s = db.borrowSender()) { + s.table("t").longColumn("v", 1L).atNow(); + } + // Hold the only slot across the window: the settle skips a leased slot (it + // would otherwise commit the idle group on its 100 ms cadence), so the + // release below is the first thing to find the group older than the window. + try (Sender s = db.borrowSender()) { + Thread.sleep(150); + s.table("t").longColumn("v", 2L).atNow(); + } + log.awaitDataFrames(2, 5_000); + Assert.assertTrue(log.deferFlag(0)); + Assert.assertFalse(log.deferFlag(1)); + } + } + } + + @Test + public void testLongLeaseAutoFlushShipsFlaggedAndFlushCommits() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = QuestDB.connect(config(server.getPort(), "auto_flush_rows=2;commit_rows=1000;commit_interval_ms=60000;housekeeper_interval_ms=60000;"))) { + try (Sender s = db.borrowSender()) { + for (int i = 0; i < 3; i++) { + s.table("t").longColumn("v", i).atNow(); // auto-flush at 2 rows: flagged + } + s.flush(); // row 3 rides in the committing frame + } + log.awaitDataFrames(2, 5_000); + Assert.assertTrue(log.deferFlag(0)); + Assert.assertFalse(log.deferFlag(1)); + Assert.assertEquals(0, log.commitFrames()); + } + } + } + + @Test + public void testRetriableCommitFailureReplaysGroupWithoutApplicationError() throws Exception { + RejectSecondFrameHandler handler = new RejectSecondFrameHandler(WebSocketResponse.STATUS_INTERNAL_ERROR); + try (TestWebSocketServer server = new TestWebSocketServer(handler)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = QuestDB.connect(config(server.getPort(), "commit_rows=2;commit_interval_ms=60000;housekeeper_interval_ms=60000;auto_flush_interval=60000;") + + "reconnect_initial_backoff_millis=10;reconnect_max_backoff_millis=50;")) { + try (Sender s = db.borrowSender()) { + s.table("t").longColumn("v", 1L).atNow(); + } + try (Sender s = db.borrowSender()) { + s.table("t").longColumn("v", 2L).atNow(); // closes the window: the committing frame is rejected + } + handler.awaitSecondConnectionFrames(2, 5_000); + Assert.assertEquals("both frames replay on the new connection", 2, handler.secondConnectionDataFrames()); + try (Sender s = db.borrowSender()) { + s.table("t").longColumn("v", 3L).atNow(); // no exception reaches the application + } + } + } + } + + @Test + public void testRowsWindowShipsFlaggedFramesThenOneCommittingFrame() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = QuestDB.connect(config(server.getPort(), "commit_rows=3;commit_interval_ms=60000;housekeeper_interval_ms=60000;"))) { + QuestDBFlushOnCloseTest.appendThreeRowsInThreeLeases(db); + log.awaitDataFrames(3, 5_000); + Assert.assertTrue(log.deferFlag(0)); + Assert.assertTrue(log.deferFlag(1)); + Assert.assertFalse("third close fills the window: unflagged, commits the group", log.deferFlag(2)); + Assert.assertEquals("no separate commit message when the closing frame carries rows", 0, log.commitFrames()); + } + } + } + + @Test + public void testSettleCommitsIdleGroupWithCommitMessage() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // settle cadence = min(auto_flush_interval, commit_interval_ms) = 100 ms + try (QuestDB db = QuestDB.connect(config(server.getPort(), "commit_rows=1000;commit_interval_ms=100;housekeeper_interval_ms=60000;"))) { + try (Sender s = db.borrowSender()) { + s.table("t").longColumn("v", 1L).atNow(); + } + log.awaitDataFrames(1, 5_000); + Assert.assertTrue(log.deferFlag(0)); + log.awaitCommitFrames(1, 3_000); + } + } + } + + @Test + public void testTerminalOnSecondFrameSurfacesOnNextCallWithItsSequence() throws Exception { + RejectSecondFrameHandler handler = new RejectSecondFrameHandler(WebSocketResponse.STATUS_SCHEMA_MISMATCH); + try (TestWebSocketServer server = new TestWebSocketServer(handler)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = QuestDB.connect(config(server.getPort(), "commit_rows=1000;commit_interval_ms=60000;housekeeper_interval_ms=60000;auto_flush_interval=60000;") + + "reconnect_max_duration_millis=500;reconnect_initial_backoff_millis=10;reconnect_max_backoff_millis=50;")) { + Sender retiredDelegate = null; + LineSenderServerException seen = null; + long deadline = System.currentTimeMillis() + 5_000; + for (long v = 1; seen == null && (v <= 2 || System.currentTimeMillis() < deadline); v++) { + // try-with-resources: the typed error stays primary, close()'s wrapper is suppressed + try (Sender s = db.borrowSender()) { + retiredDelegate = ((PooledSender) s).getDelegateForTesting(); + s.table("t").longColumn("v", v).atNow(); + } catch (LineSenderServerException e) { + seen = e; + } catch (LineSenderException e) { + // the terminal latched after the last row call: close() wraps it + if (!(e.getCause() instanceof LineSenderServerException)) { + throw e; + } + seen = (LineSenderServerException) e.getCause(); + } + Thread.sleep(20); + } + Assert.assertNotNull("the latched terminal must surface on a later call on that slot", seen); + Assert.assertEquals("it names the rejected frame", 1L, seen.getServerError().getMessageSequence()); + try (Sender s = db.borrowSender()) { + Assert.assertNotSame("the pool replaces the retired slot", retiredDelegate, ((PooledSender) s).getDelegateForTesting()); + s.table("t").longColumn("v", 4L).atNow(); + } + } + } + } + + private static String config(int port, String extra) { + return QuestDBFlushOnCloseTest.config(port, "flush_on_close=on;" + extra); + } + + /** + * First connection: withholds the ack of frame 0 (as the real server does + * for a flagged frame inside an open group) and NACKs frame 1 with the + * given status. Later connections: acks everything. Counts the frames the + * second connection receives, so a replay from the acked watermark, which + * is nothing, shows both frames again. + */ + private static final class RejectSecondFrameHandler implements TestWebSocketServer.WebSocketServerHandler { + private final byte rejectStatus; + private int connections; + private TestWebSocketServer.ClientHandler currentClient; + private TestWebSocketServer.ClientHandler firstClient; + private long nextSeq; + private int secondConnectionDataFrames; + + RejectSecondFrameHandler(byte rejectStatus) { + this.rejectStatus = rejectStatus; + } + + @Override + public synchronized void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + if (currentClient != client) { + currentClient = client; + connections++; + nextSeq = 0; + if (firstClient == null) { + firstClient = client; + } + } + long seq = nextSeq++; + if (connections >= 2 && data.length >= 8 && ((data[6] & 0xFF) | ((data[7] & 0xFF) << 8)) > 0) { + secondConnectionDataFrames++; + } + try { + if (client == firstClient) { + if (seq == 1) { + client.sendBinary(QwpWireTestUtils.buildNack(seq, rejectStatus)); + } + // seq 0: ack withheld, the frame stays replayable client-side + } else { + client.sendBinary(QwpWireTestUtils.buildAck(seq)); + } + } catch (IOException e) { + throw new RuntimeException(e); + } + } + + void awaitSecondConnectionFrames(int atLeast, long timeoutMillis) throws InterruptedException { + long deadline = System.currentTimeMillis() + timeoutMillis; + while (secondConnectionDataFrames() < atLeast && System.currentTimeMillis() < deadline) { + Thread.sleep(10); + } + } + + synchronized int secondConnectionDataFrames() { + return secondConnectionDataFrames; + } + } +} \ No newline at end of file diff --git a/core/src/test/java/io/questdb/client/test/QuestDBFlushOnCloseTest.java b/core/src/test/java/io/questdb/client/test/QuestDBFlushOnCloseTest.java new file mode 100644 index 000000000..3a9573c22 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/QuestDBFlushOnCloseTest.java @@ -0,0 +1,805 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test; + +import io.questdb.client.QuestDB; +import io.questdb.client.LineSenderServerException; +import io.questdb.client.Sender; +import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.cutlass.qwp.client.BatchTooLargeForCapException; +import io.questdb.client.cutlass.qwp.client.QwpWebSocketSender; +import io.questdb.client.cutlass.qwp.client.WebSocketResponse; +import io.questdb.client.impl.PooledSender; +import io.questdb.client.test.cutlass.qwp.client.QwpWireTestUtils; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.Assert; +import org.junit.Test; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.TimeUnit; + +/** + * Close releases, flush ships, drain waits: observed on the wire through the + * in-repo test server. Each test counts the data frames (table count above + * zero) the server receives. + */ +public class QuestDBFlushOnCloseTest { + + @Test + public void testCloseShipsNothingUntilSettleOrFlush() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // settle cadence 60 s: nothing ships inside the test window + try (QuestDB db = QuestDB.connect(config(server.getPort(), "auto_flush_interval=60000;housekeeper_interval_ms=60000;"))) { + appendThreeRowsInThreeLeases(db); + Thread.sleep(300); + Assert.assertEquals("three closes must not ship", 0, log.dataFrames()); + try (Sender s = db.borrowSender()) { + s.flush(); + } + log.awaitDataFrames(1, 5_000); + Assert.assertEquals("explicit flush ships one frame holding all three rows", 1, log.dataFrames()); + } + } + } + + @Test + public void testCrossingRowShipsTheEarlierRowsAndStaysBuffered() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // auto_flush_bytes unset: the byte trigger sits at 90 percent of the 4 KiB cap (3686 bytes) + try (QuestDB db = QuestDB.connect(config(server.getPort(), "auto_flush_interval=60000;housekeeper_interval_ms=60000;"))) { + appendThreeWideRowsAfterAWarmUp(db, log); + Sender delegate; + try (Sender s = db.borrowSender()) { + delegate = ((PooledSender) s).getDelegateForTesting(); + // the auto-flushed batch would be 4.5 KB, over the 4 KiB cap: the sender + // ships the three rows before the crossing row and keeps that row + s.table("t").stringColumn("s", wideValue(1500)).atNow(); + Assert.assertEquals("only the crossing row stays buffered", 1, s.pendingRowCount()); + } + log.awaitDataFrames(2, 5_000); + Assert.assertEquals("the three earlier rows ship as one frame with table t", 1, log.tableCount(1)); + try (Sender s = db.borrowSender()) { + Assert.assertSame("the slot stayed in service", delegate, ((PooledSender) s).getDelegateForTesting()); + Assert.assertEquals(1, s.pendingRowCount()); + s.flush(); + } + log.awaitDataFrames(3, 5_000); + } + } + } + + @Test + public void testCrossingRowWhoseOwnFrameExceedsTheCapDropsOnlyThatRow() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // auto_flush_bytes unset: the byte trigger sits at 90 percent of the 4 KiB cap (3686 bytes) + try (QuestDB db = QuestDB.connect(config(server.getPort(), "auto_flush_interval=60000;housekeeper_interval_ms=60000;"))) { + appendThreeWideRowsAfterAWarmUp(db, log); + Sender delegate = null; + boolean hasCloseThrown = false; + try (Sender s = db.borrowSender()) { + delegate = ((PooledSender) s).getDelegateForTesting(); + try { + // 4070 bytes pass the per-row guard, but the table name and schema + // take the row's own frame over the cap + s.table("u").stringColumn("z", wideValue(4070)).atNow(); + Assert.fail("the row's own frame exceeds the 4 KiB cap"); + } catch (BatchTooLargeForCapException expected) { + // the rows before it shipped; the row stays and the rejection is retained + } + } catch (LineSenderException e) { + hasCloseThrown = true; + Assert.assertTrue("close() wraps the rejection", e.getCause() instanceof BatchTooLargeForCapException); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("dropped 1 row(s) of this lease")); + } + Assert.assertTrue("close() must report the dropped row", hasCloseThrown); + log.awaitDataFrames(2, 5_000); + Assert.assertEquals("the three earlier rows ship as one frame with table t", 1, log.tableCount(1)); + try (Sender s = db.borrowSender()) { + Assert.assertSame("the slot stayed in service", delegate, ((PooledSender) s).getDelegateForTesting()); + Assert.assertEquals(0, s.pendingRowCount()); + s.table("w").longColumn("v", 2L).atNow(); + s.flush(); + } + log.awaitDataFrames(3, 5_000); + } + } + } + + @Test + public void testFlushOnCloseShipsOneFramePerClose() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = QuestDB.connect(config(server.getPort(), "flush_on_close=on;housekeeper_interval_ms=60000;"))) { + appendThreeRowsInThreeLeases(db); + log.awaitDataFrames(3, 5_000); + Assert.assertEquals(3, log.dataFrames()); + for (int i = 0; i < 3; i++) { + Assert.assertFalse("no commit window: frames are not flagged", log.deferFlag(i)); + } + } + } + } + + @Test + public void testHandleCloseShipsRowsLeftInSlot() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + QuestDB db = QuestDB.connect(config(server.getPort(), + "housekeeper_interval_ms=60000;auto_flush_interval=60000;close_flush_timeout_millis=5000;")); + appendThreeRowsInThreeLeases(db); + db.close(); + log.awaitDataFrames(1, 5_000); + } + } + + @Test + public void testLaterLeaseResetKeepsEarlierLeasesRows() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = QuestDB.connect(config(server.getPort(), "auto_flush_interval=60000;housekeeper_interval_ms=60000;"))) { + try (Sender a = db.borrowSender()) { + a.table("t1").longColumn("v", 1L).atNow(); + } + try (Sender b = db.borrowSender()) { + b.table("t2").longColumn("v", 2L).atNow(); + b.reset(); // B drops its own bad batch + } + try (Sender c = db.borrowSender()) { + Assert.assertEquals("A's row is still buffered", 1, c.pendingRowCount()); + c.flush(); + } + log.awaitDataFrames(1, 5_000); + Assert.assertEquals("one frame, one table: t1 only", 1, log.tableCount(0)); + Assert.assertEquals(1, log.dataFrames()); + } + } + } + + @Test + public void testOverCapBatchRetiresTheSlotInsteadOfPoisoningIt() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // auto_flush_bytes=off on purpose: defeat the cap-bound byte trigger so the batch outgrows the cap + try (QuestDB db = QuestDB.connect(config(server.getPort(), + "auto_flush_bytes=off;auto_flush_interval=60000;housekeeper_interval_ms=60000;"))) { + // warm-up lease: proves the connection and the advertised cap are in place + // (initial_connect_retry=async learns the cap on the I/O thread) + try (Sender s = db.borrowSender()) { + s.table("w").longColumn("v", 1L).atNow(); + s.flush(); + } + log.awaitDataFrames(1, 5_000); + String wide = wideValue(1000); + for (int i = 0; i < 6; i++) { + try (Sender s = db.borrowSender()) { + s.table("t").stringColumn("s", wide).atNow(); + } + } + Sender poisoned = null; + boolean hasCloseThrown = false; + try (Sender s = db.borrowSender()) { + poisoned = ((PooledSender) s).getDelegateForTesting(); + try { + s.flush(); + Assert.fail("six 1 KB rows do not fit a 4 KiB cap"); + } catch (BatchTooLargeForCapException expected) { + // the retained batch belongs to nobody: close() must retire the slot + } + } catch (LineSenderException e) { + hasCloseThrown = true; + Assert.assertTrue("close() wraps the retained rejection", e.getCause() instanceof BatchTooLargeForCapException); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("6 unshipped row(s)")); + } + Assert.assertTrue("close() must report the retirement", hasCloseThrown); + try (Sender s = db.borrowSender()) { + Assert.assertNotSame("the retired slot is replaced", poisoned, ((PooledSender) s).getDelegateForTesting()); + s.table("t").stringColumn("s", wide).atNow(); + s.flush(); + } + log.awaitDataFrames(2, 5_000); + Assert.assertEquals("the over-cap batch never ships", 2, log.dataFrames()); + } + } + } + + @Test + public void testOverCapFlushDropsOnlyThatLeasesRowsAndShipsTheRest() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // auto_flush_bytes=off: no byte trigger, so the sender never holds a row back + // and the lease's explicit flush() meets the whole over-cap batch + try (QuestDB db = QuestDB.connect(config(server.getPort(), + "auto_flush_bytes=off;auto_flush_interval=60000;housekeeper_interval_ms=60000;"))) { + appendThreeWideRowsAfterAWarmUp(db, log); + Sender delegate = null; + boolean hasCloseThrown = false; + try (Sender s = db.borrowSender()) { + delegate = ((PooledSender) s).getDelegateForTesting(); + String wide = wideValue(1000); + s.table("t").stringColumn("s", wide).atNow(); + s.table("t").stringColumn("s", wide).atNow(); + try { + s.flush(); + Assert.fail("the lease's two rows make the batch 5 KB, over the 4 KiB cap"); + } catch (BatchTooLargeForCapException expected) { + // the batch is retained: close() rolls the lease back and retries + } + } catch (LineSenderException e) { + hasCloseThrown = true; + Assert.assertTrue("close() wraps the rejection", e.getCause() instanceof BatchTooLargeForCapException); + Assert.assertTrue(e.getMessage(), e.getMessage().contains("dropped 2 row(s) of this lease")); + } + Assert.assertTrue("close() must report the dropped rows", hasCloseThrown); + log.awaitDataFrames(2, 5_000); + Assert.assertEquals("the three earlier rows ship as one frame with table t", 1, log.tableCount(1)); + try (Sender s = db.borrowSender()) { + Assert.assertSame("the slot stayed in service", delegate, ((PooledSender) s).getDelegateForTesting()); + Assert.assertEquals("only the lease's rows were dropped, the rest shipped", 0, s.pendingRowCount()); + } + } + } + } + + @Test + public void testPartialRowReleaseAfterTheIntervalStillShipsEarlierRows() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = QuestDB.connect(config(server.getPort(), "auto_flush_interval=200;housekeeper_interval_ms=60000;"))) { + try (Sender s = db.borrowSender()) { + s.table("t").longColumn("v", 1L).atNow(); + } + Sender delegate; + Sender s = db.borrowSender(); + delegate = ((PooledSender) s).getDelegateForTesting(); + s.table("t").longColumn("v", 2L); + Thread.sleep(400); + try { + s.close(); + Assert.fail("expected LineSenderException"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("row in progress at close()")); + } + Assert.assertEquals("the earlier lease's aged row shipped", 0, delegate.pendingRowCount()); + log.awaitDataFrames(1, 5_000); + } + } + } + + @Test + public void testReleaseShipsAnEarlierLeasesRowOnceOlderThanTheInterval() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = QuestDB.connect(config(server.getPort(), "auto_flush_interval=200;housekeeper_interval_ms=60000;"))) { + try (Sender s = db.borrowSender()) { + s.table("t").longColumn("v", 1L).atNow(); + } + Sender delegate; + try (Sender s = db.borrowSender()) { // the same slot; settle cannot reach a leased slot + delegate = ((PooledSender) s).getDelegateForTesting(); + Thread.sleep(400); + } + Assert.assertEquals("the release shipped the earlier lease's aged row", 0, delegate.pendingRowCount()); + log.awaitDataFrames(1, 5_000); + } + } + } + + @Test + public void testResetAfterACrossingRowAloneInItsTableDropsTheColumnItAdded() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // auto_flush_bytes unset: the byte trigger sits at 90 percent of the 4 KiB cap (3686 bytes) + try (QuestDB db = QuestDB.connect(config(server.getPort(), "auto_flush_interval=60000;housekeeper_interval_ms=60000;"))) { + // warm-up lease: the server learns late_table with kept_column, and late_table + // precedes t in the sender's tables, so its frame carries the batch's new + // symbols when the batch splits + try (Sender s = db.borrowSender()) { + s.table("late_table").longColumn("kept_column", 1L).atNow(); + s.flush(); + } + log.awaitDataFrames(1, 5_000); + for (int i = 0; i < 3; i++) { + try (Sender s = db.borrowSender()) { + // 1.8 KB of new symbols and 1.5 KB of strings in all, below the trigger + s.table("t").symbol("y", i + wideValue(600)).stringColumn("s", wideValue(500)).atNow(); + } + } + try (Sender s = db.borrowSender()) { + Assert.assertTrue("memory mode selects delta mode", + ((QwpWebSocketSender) ((PooledSender) s).getDelegateForTesting()).isDeltaDictEnabledForTest()); + // late_table's frame would carry this 2.6 KB row and the 1.8 KB of new symbols: + // the sender ships t's rows with the symbols and holds the row, the only row + // of late_table in the batch and below the byte trigger on its own + s.table("late_table").longColumn("kept_column", 2L).stringColumn("discarded_column", wideValue(2600)).atNow(); + Assert.assertEquals("only the crossing row stays buffered", 1, s.pendingRowCount()); + s.reset(); + } + log.awaitDataFrames(2, 5_000); + try (Sender s = db.borrowSender()) { + s.table("late_table").longColumn("kept_column", 3L).atNow(); + s.flush(); + } + log.awaitDataFrames(3, 5_000); + List frames = log.frames(); + Assert.assertTrue("late_table ships kept_column", containsAscii(frames.get(frames.size() - 1), "kept_column")); + for (byte[] frame : frames) { + Assert.assertFalse("reset() dropped the column the server never saw", containsAscii(frame, "discarded_column")); + } + } + } + } + + @Test + public void testResetAfterACrossingRowDropsOnlyThatRow() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // auto_flush_bytes unset: the byte trigger sits at 90 percent of the 4 KiB cap (3686 bytes) + try (QuestDB db = QuestDB.connect(config(server.getPort(), "auto_flush_interval=60000;housekeeper_interval_ms=60000;"))) { + appendThreeWideRowsAfterAWarmUp(db, log); + try (Sender s = db.borrowSender()) { + s.table("t").stringColumn("s", wideValue(1500)).atNow(); + // the three earlier rows shipped with the auto-flush, so the + // crossing row is the only row left and it is this lease's own + s.reset(); + Assert.assertEquals("reset() drops the crossing row", 0, s.pendingRowCount()); + } + log.awaitDataFrames(2, 5_000); + try (Sender s = db.borrowSender()) { + Assert.assertEquals(0, s.pendingRowCount()); + } + } + } + } + + @Test + public void testRowOverTheCapFailsOnlyThatRowAndKeepsTheSlot() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = QuestDB.connect(config(server.getPort(), "auto_flush_interval=60000;housekeeper_interval_ms=60000;"))) { + // warm-up lease: proves the connection and the advertised cap are in place + // (initial_connect_retry=async learns the cap on the I/O thread) + try (Sender s = db.borrowSender()) { + s.table("w").longColumn("v", 1L).atNow(); + s.flush(); + } + log.awaitDataFrames(1, 5_000); + Sender delegate; + try (Sender s = db.borrowSender()) { + delegate = ((PooledSender) s).getDelegateForTesting(); + try { + s.table("t").stringColumn("s", wideValue(5000)).atNow(); + Assert.fail("a 5 KB row cannot fit a 4 KiB cap"); + } catch (LineSenderException expected) { + Assert.assertTrue(expected.getMessage(), expected.getMessage().contains("row too large")); + } + s.table("t").longColumn("v", 1L).atNow(); // the lease carries on + } + try (Sender s = db.borrowSender()) { + Assert.assertSame("the slot stayed in service", delegate, ((PooledSender) s).getDelegateForTesting()); + Assert.assertEquals(1, s.pendingRowCount()); + s.flush(); + } + log.awaitDataFrames(2, 5_000); + } + } + } + + @Test + public void testSettleShipsOnDefaultCadenceWhenIntervalUnset() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // no auto_flush_interval: the settle cadence defaults to 100 ms, independent of the 60 s reap cadence + try (QuestDB db = QuestDB.connect(config(server.getPort(), "housekeeper_interval_ms=60000;"))) { + appendThreeRowsInThreeLeases(db); + log.awaitDataFrames(1, 2_000); + Thread.sleep(300); + Assert.assertTrue("a settle tick landing mid-loop may split the rows, got " + log.dataFrames(), + log.dataFrames() >= 1 && log.dataFrames() <= 3); + } + } + } + + @Test + public void testSettleShipsWithinAutoFlushInterval() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = QuestDB.connect(config(server.getPort(), "auto_flush_interval=100;housekeeper_interval_ms=60000;"))) { + appendThreeRowsInThreeLeases(db); + log.awaitDataFrames(1, 2_000); + Thread.sleep(300); + Assert.assertTrue("a settle tick landing mid-loop may split the rows, got " + log.dataFrames(), + log.dataFrames() >= 1 && log.dataFrames() <= 3); + } + } + } + + @Test + public void testTerminalRejectionDropsEveryLeaseRowInTheFrame() throws Exception { + RejectFirstFrameHandler handler = new RejectFirstFrameHandler(); + try (TestWebSocketServer server = new TestWebSocketServer(handler)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = QuestDB.connect(config(server.getPort(), "auto_flush_interval=60000;housekeeper_interval_ms=60000;") + + "reconnect_max_duration_millis=500;reconnect_initial_backoff_millis=10;reconnect_max_backoff_millis=50;")) { + try (Sender a = db.borrowSender()) { + a.table("t1").longColumn("v", 1L).atNow(); // a clean row + } + try (Sender b = db.borrowSender()) { + b.table("t2").longColumn("v", 2L).atNow(); // the row the server will reject + } + LineSenderServerException seen = null; + long deadline = System.currentTimeMillis() + 5_000; + for (int i = 0; seen == null && System.currentTimeMillis() < deadline; i++) { + try (Sender c = db.borrowSender()) { + if (i == 0) { + c.flush(); // ships A's and B's rows in ONE frame + } else { + c.table("t1").longColumn("v", 3L).atNow(); + } + } catch (LineSenderServerException e) { + seen = e; + } catch (LineSenderException e) { + if (!(e.getCause() instanceof LineSenderServerException)) { + throw e; + } + seen = (LineSenderServerException) e.getCause(); + } + Thread.sleep(20); + } + Assert.assertNotNull("the terminal surfaces on a later lease, not on A or B", seen); + Assert.assertEquals("both leases' rows shared the rejected frame", 1, handler.firstConnectionDataFrames()); + Assert.assertEquals(2, handler.firstConnectionTableCount()); + try (Sender d = db.borrowSender()) { + d.table("t1").longColumn("v", 4L).atNow(); + d.flush(); + } + handler.awaitLaterConnectionFrames(1, 5_000); + Assert.assertEquals("nothing is replayed after a terminal: only D's frame arrives on the fresh slot", + 1, handler.laterConnectionDataFrames()); + } + } + } + + @Test + public void testThresholdOnLaterLeaseShipsEarlierRows() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = QuestDB.connect(config(server.getPort(), + "auto_flush_rows=3;auto_flush_interval=60000;housekeeper_interval_ms=60000;"))) { + appendThreeRowsInThreeLeases(db); + log.awaitDataFrames(1, 5_000); + } + } + } + + @Test + public void testWideRowsAcrossLeasesShipAtTheServerCapNotAtTheRowThreshold() throws Exception { + FrameLog log = new FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(16 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // settle 60 s, row threshold 1000: only the cap-bound byte trigger can ship inside the test + try (QuestDB db = QuestDB.connect(config(server.getPort(), "auto_flush_interval=60000;housekeeper_interval_ms=60000;"))) { + String wide = wideValue(1000); + // initial_connect_retry=async: the cap is learned on the I/O thread's + // handshake, and rows appended before it carry no byte trigger. Keep + // appending (far below the 1000-row threshold) until a frame ships. + for (int i = 0; i < 200 && log.dataFrames() < 1; i++) { + try (Sender s = db.borrowSender()) { + s.table("t").stringColumn("s", wide).atNow(); + } + Thread.sleep(10); + } + log.awaitDataFrames(1, 5_000); + } + } + } + + static void appendThreeRowsInThreeLeases(QuestDB db) { + for (int i = 0; i < 3; i++) { + try (Sender s = db.borrowSender()) { + s.table("t").longColumn("v", i).atNow(); + } + } + } + + static void appendThreeWideRowsAfterAWarmUp(QuestDB db, FrameLog log) throws InterruptedException { + // warm-up lease: proves the connection and the advertised cap are in place + // (initial_connect_retry=async learns the cap on the I/O thread) + try (Sender s = db.borrowSender()) { + s.table("w").longColumn("v", 1L).atNow(); + s.flush(); + } + log.awaitDataFrames(1, 5_000); + String wide = wideValue(1000); + for (int i = 0; i < 3; i++) { + try (Sender s = db.borrowSender()) { + s.table("t").stringColumn("s", wide).atNow(); // 3 KB in all, below the trigger + } + } + } + + static String config(int port, String extra) { + return "ws::addr=localhost:" + port + + ";sender_pool_min=1;sender_pool_max=1" + + ";query_pool_min=0;query_pool_max=1" + + ";initial_connect_retry=async" + + ";close_flush_timeout_millis=0;" + + extra; + } + + private static boolean containsAscii(byte[] frame, String text) { + byte[] needle = text.getBytes(StandardCharsets.US_ASCII); + for (int i = 0, n = frame.length - needle.length; i <= n; i++) { + int j = 0; + while (j < needle.length && frame[i + j] == needle[j]) { + j++; + } + if (j == needle.length) { + return true; + } + } + return false; + } + + public static String wideValue(int length) { + char[] chars = new char[length]; + Arrays.fill(chars, 'x'); + return new String(chars); + } + + /** + * Parses every QWP frame in each WebSocket message (a read can carry more + * than one on replay), records its flags byte, table count and bytes, and acks + * cumulatively so the sender never backs up. Public: the sender-level + * probe test in a sub-package and the benchmark reuse it. + */ + public static final class FrameLog implements TestWebSocketServer.WebSocketServerHandler { + private static final int HEADER_SIZE = 12; + private static final int OFFSET_FLAGS = 5; + private static final int OFFSET_PAYLOAD_LENGTH = 8; + private static final int OFFSET_TABLE_COUNT = 6; + private final List flags = new ArrayList<>(); + private final List frames = new ArrayList<>(); + private final List tableCounts = new ArrayList<>(); + private TestWebSocketServer.ClientHandler currentClient; + private long nextSeq; + + public void awaitCommitFrames(int atLeast, long timeoutMillis) throws InterruptedException { + long deadline = System.currentTimeMillis() + timeoutMillis; + while (commitFrames() < atLeast && System.currentTimeMillis() < deadline) { + Thread.sleep(10); + } + Assert.assertTrue("expected at least " + atLeast + " commit frame(s) within " + timeoutMillis + " ms, got " + commitFrames(), + commitFrames() >= atLeast); + } + + public void awaitDataFrames(int atLeast, long timeoutMillis) throws InterruptedException { + long deadline = System.currentTimeMillis() + timeoutMillis; + while (dataFrames() < atLeast && System.currentTimeMillis() < deadline) { + Thread.sleep(10); + } + Assert.assertTrue("expected at least " + atLeast + " data frame(s) within " + timeoutMillis + " ms, got " + dataFrames(), + dataFrames() >= atLeast); + } + + /** + * Count of rowless frames with the defer flag clear: the commit messages. + */ + public synchronized int commitFrames() { + int n = 0; + for (int i = 0, size = tableCounts.size(); i < size; i++) { + if (tableCounts.get(i) == 0 && (flags.get(i) & 0x01) == 0) { + n++; + } + } + return n; + } + + public synchronized int dataFrames() { + int n = 0; + for (int i = 0, size = tableCounts.size(); i < size; i++) { + if (tableCounts.get(i) > 0) { + n++; + } + } + return n; + } + + /** + * Defer flag of the i-th data frame (frames with a table count above zero). + */ + public synchronized boolean deferFlag(int dataFrameIndex) { + int seen = -1; + for (int i = 0, size = tableCounts.size(); i < size; i++) { + if (tableCounts.get(i) > 0 && ++seen == dataFrameIndex) { + return (flags.get(i) & 0x01) != 0; + } + } + throw new AssertionError("no data frame at index " + dataFrameIndex); + } + + /** + * A copy of every frame {@link #onBinaryMessage} parsed, in arrival order, + * each frame's header and payload as one array. + */ + public synchronized List frames() { + return new ArrayList<>(frames); + } + + @Override + public synchronized void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + if (currentClient != client) { + currentClient = client; + nextSeq = 0; + } + int pos = 0; + int framesInMessage = 0; + while (pos + HEADER_SIZE <= data.length) { + int tableCount = (data[pos + OFFSET_TABLE_COUNT] & 0xFF) | ((data[pos + OFFSET_TABLE_COUNT + 1] & 0xFF) << 8); + long payloadLength = (data[pos + OFFSET_PAYLOAD_LENGTH] & 0xFFL) + | ((data[pos + OFFSET_PAYLOAD_LENGTH + 1] & 0xFFL) << 8) + | ((data[pos + OFFSET_PAYLOAD_LENGTH + 2] & 0xFFL) << 16) + | ((data[pos + OFFSET_PAYLOAD_LENGTH + 3] & 0xFFL) << 24); + if (payloadLength < 0 || pos + HEADER_SIZE + payloadLength > data.length) { + break; + } + flags.add(data[pos + OFFSET_FLAGS] & 0xFF); + frames.add(Arrays.copyOfRange(data, pos, pos + HEADER_SIZE + (int) payloadLength)); + tableCounts.add(tableCount); + framesInMessage++; + pos += HEADER_SIZE + (int) payloadLength; + } + nextSeq += Math.max(1, framesInMessage); + try { + client.sendBinary(QwpWireTestUtils.buildAck(nextSeq - 1)); + } catch (IOException e) { + throw new RuntimeException(e); + } + } + + /** + * Table count of the i-th data frame (frames with a table count above zero). + */ + public synchronized int tableCount(int dataFrameIndex) { + int seen = -1; + for (int i = 0, size = tableCounts.size(); i < size; i++) { + if (tableCounts.get(i) > 0 && ++seen == dataFrameIndex) { + return tableCounts.get(i); + } + } + throw new AssertionError("no data frame at index " + dataFrameIndex); + } + } + + /** + * First connection: NACKs its first data frame with SCHEMA_MISMATCH (a + * terminal) and acks nothing else. Later connections: ack everything and + * count their data frames, so a replay would show up there. + */ + private static final class RejectFirstFrameHandler implements TestWebSocketServer.WebSocketServerHandler { + private static final int OFFSET_TABLE_COUNT = 6; + private int connections; + private TestWebSocketServer.ClientHandler currentClient; + private int firstConnectionDataFrames; + private int firstConnectionTableCount; + private int laterConnectionDataFrames; + private long nextSeq; + + void awaitLaterConnectionFrames(int atLeast, long timeoutMillis) throws InterruptedException { + long deadline = System.currentTimeMillis() + timeoutMillis; + while (laterConnectionDataFrames() < atLeast && System.currentTimeMillis() < deadline) { + Thread.sleep(10); + } + } + + synchronized int firstConnectionDataFrames() { + return firstConnectionDataFrames; + } + + synchronized int firstConnectionTableCount() { + return firstConnectionTableCount; + } + + synchronized int laterConnectionDataFrames() { + return laterConnectionDataFrames; + } + + @Override + public synchronized void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + if (currentClient != client) { + currentClient = client; + connections++; + nextSeq = 0; + } + long seq = nextSeq++; + int tableCount = data.length >= 8 ? ((data[OFFSET_TABLE_COUNT] & 0xFF) | ((data[OFFSET_TABLE_COUNT + 1] & 0xFF) << 8)) : 0; + try { + if (connections == 1) { + if (tableCount > 0) { + firstConnectionDataFrames++; + firstConnectionTableCount = tableCount; + client.sendBinary(QwpWireTestUtils.buildNack(seq, WebSocketResponse.STATUS_SCHEMA_MISMATCH)); + } + // rowless frames on the first connection: no ack, no nack + } else { + if (tableCount > 0) { + laterConnectionDataFrames++; + } + client.sendBinary(QwpWireTestUtils.buildAck(seq)); + } + } catch (IOException e) { + throw new RuntimeException(e); + } + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/QuestDBHousekeeperCadenceTest.java b/core/src/test/java/io/questdb/client/test/QuestDBHousekeeperCadenceTest.java new file mode 100644 index 000000000..f117c4364 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/QuestDBHousekeeperCadenceTest.java @@ -0,0 +1,101 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ +package io.questdb.client.test; + +import io.questdb.client.QuestDB; +import io.questdb.client.Sender; +import io.questdb.client.impl.QuestDBImpl; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.Assert; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.File; +import java.util.Arrays; +import java.util.concurrent.TimeUnit; + +/** + * The housekeeper's two cadences under extreme settings: a startup + * store-and-forward recovery that keeps failing, and a reap interval of + * {@code Long.MAX_VALUE}. + */ +public class QuestDBHousekeeperCadenceTest { + + @Rule + public final TemporaryFolder temp = new TemporaryFolder(); + + @Test + public void testFailedStartupRecoveryRetriesOnItsOwnCadenceNotTheSettleCadence() throws Exception { + File sfDir = temp.newFolder("sf"); + int port; + try (TestWebSocketServer silent = new TestWebSocketServer(new SilentHandler())) { + silent.start(); + Assert.assertTrue(silent.awaitStart(5, TimeUnit.SECONDS)); + port = silent.getPort(); + // Strand one slot: the server never acks, so the frame stays on disk after close. + try (QuestDB db = QuestDB.connect("ws::addr=localhost:" + port + ";sf_dir=" + sfDir.getAbsolutePath() + + ";sender_pool_min=1;sender_pool_max=1;query_pool_min=0;query_pool_max=1" + + ";close_flush_timeout_millis=200;housekeeper_interval_ms=60000;auto_flush_interval=60000;")) { + try (Sender s = db.borrowSender()) { + s.table("recover").longColumn("v", 1L).atNow(); + s.flush(); + } + } + } + // The port is closed now: every recovery step fails fast on a refused connect. + // Settle every 50 ms; recovery must not follow that cadence. + try (QuestDB db = QuestDB.connect("ws::addr=localhost:" + port + ";sf_dir=" + sfDir.getAbsolutePath() + + ";sender_pool_min=0;sender_pool_max=1;query_pool_min=0;query_pool_max=1" + + ";housekeeper_interval_ms=1000;auto_flush_interval=50;")) { + Thread.sleep(2_600); + int attempts = ((QuestDBImpl) db).getSenderPoolForTesting().startupRecoveryBuildAttemptsForTesting(); + Assert.assertTrue("about one attempt per second expected, got " + attempts + + "; sf_dir=" + Arrays.toString(sfDir.list()), attempts >= 1 && attempts <= 5); + } + } + + @Test + public void testHousekeeperIntervalAtLongMaxStillSettles() throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (QuestDB db = QuestDB.connect(QuestDBFlushOnCloseTest.config(server.getPort(), + "housekeeper_interval_ms=" + Long.MAX_VALUE + ";"))) { + try (Sender s = db.borrowSender()) { + s.table("t").longColumn("v", 1L).atNow(); + } + log.awaitDataFrames(1, 3_000); // the 100 ms settle must still run + } + } + } + + private static final class SilentHandler implements TestWebSocketServer.WebSocketServerHandler { + @Override + public void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + // no ack: the frame stays unacked on disk + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpWebSocketSenderTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpWebSocketSenderTest.java index 3911f5c05..2252cd95f 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpWebSocketSenderTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/QwpWebSocketSenderTest.java @@ -122,6 +122,24 @@ public void testApplyServerBatchSizeLimit_optOutPreservedDespiteAdvertisement() }); } + @Test + public void testApplyServerBatchSizeLimit_serverCapBoundFollowsAdvertisedCap() throws Exception { + assertMemoryLeak(() -> { + try (QwpWebSocketSender sender = QwpWebSocketSender.createForTesting( + "localhost", 9000, + /*autoFlushRows*/ 1000, + /*autoFlushBytes*/ QwpWebSocketSender.AUTO_FLUSH_BYTES_SERVER_CAP, + /*autoFlushIntervalNanos*/ 0L)) { + Assert.assertEquals("no byte trigger before the handshake", 0, sender.getEffectiveAutoFlushBytes()); + sender.applyServerBatchSizeLimit(16 * 1024 * 1024); + Assert.assertEquals("90% of the advertised cap", (long) (16 * 1024 * 1024) * 9 / 10, sender.getEffectiveAutoFlushBytes()); + Assert.assertEquals(16 * 1024 * 1024, sender.getServerMaxBatchSize()); + sender.applyServerBatchSizeLimit(0); + Assert.assertEquals("a node that advertises no cap: no byte trigger", 0, sender.getEffectiveAutoFlushBytes()); + } + }); + } + @Test public void testAtAfterCloseThrows() throws Exception { assertMemoryLeak(() -> { diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SelfSufficientFramesTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SelfSufficientFramesTest.java index e305210b1..5d66217e5 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SelfSufficientFramesTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SelfSufficientFramesTest.java @@ -26,6 +26,7 @@ import io.questdb.client.Sender; import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.cutlass.qwp.client.BatchTooLargeForCapException; import io.questdb.client.cutlass.qwp.client.QwpWebSocketSender; import io.questdb.client.cutlass.qwp.client.sf.cursor.CursorSendEngine; import io.questdb.client.cutlass.qwp.protocol.QwpConstants; @@ -44,6 +45,7 @@ import java.nio.file.Files; import java.nio.file.Path; import java.util.ArrayList; +import java.util.Arrays; import java.util.List; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicLong; @@ -487,7 +489,7 @@ public void testFileModeSplitPersistsDictBeforeAFAILEDPublish() throws Exception Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); String config = "ws::addr=localhost:" + port + ";sf_dir=" + sfDir - + ";sf_max_segment_bytes=64;"; + + ";sf_max_segment_bytes=64;auto_flush_bytes=off;"; String pad = TestUtils.repeat("x", 60); Sender sender = Sender.fromConfig(config); try { @@ -1008,6 +1010,62 @@ public void testFullDictNearCapOversizedBodyStrandsNoChunks() throws Exception { }); } + @Test + public void testHeldCrossingRowFrameCarriesTheDictionaryInFullDictMode() throws Exception { + // A row that pushes an auto-flushed batch over the cap stays buffered while + // the rows before it ship. The sender registered the row's symbol before it + // encoded that frame, so the frame defines the id. In full-dict mode every + // frame must define every id its rows reference, so the frame that later + // ships the held row must carry the dictionary again: the sender keeps the + // batch's symbol watermark when it retains the row. + Path sfDir = temporaryFolder.newFolder("qwp-sf-held-row").toPath(); + String slot = sfDir.resolve("default").toString(); + assertMemoryLeak(() -> { + CapturingHandler handler = new CapturingHandler(); + try (TestWebSocketServer server = new TestWebSocketServer(handler)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + int port = server.getPort(); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + + // openCleanRW fails against nothing on disk, so the slot runs full-dict. + UnopenableDictFacade dictFf = new UnopenableDictFacade(); + CursorSendEngine engine = new CursorSendEngine( + slot, 4L * 1024 * 1024, CursorSendEngine.DEFAULT_APPEND_DEADLINE_NANOS, + CursorSendEngine.DEFAULT_APPEND_DEADLINE_NANOS, dictFf); + // the default auto_flush_bytes binds the byte trigger to 90% of the 4 KiB cap + try (Sender sender = QwpWebSocketSender.connect( + "localhost", port, null, 1_000_000, QwpWebSocketSender.DEFAULT_AUTO_FLUSH_BYTES, + 0L, null, false, engine)) { + Assert.assertFalse("the slot runs full-dict", + ((QwpWebSocketSender) sender).isDeltaDictEnabledForTest()); + String wide = TestUtils.repeat("x", 1000); + for (int i = 0; i < 3; i++) { + sender.table("t").symbol("y", "a").stringColumn("s", wide).atNow(); + } + // the auto-flushed batch would be 4.5 KB: the three rows before this + // one ship, and this row, which registered "b", stays buffered + sender.table("t").symbol("y", "b").stringColumn("s", TestUtils.repeat("x", 1500)).atNow(); + Assert.assertEquals(1, sender.pendingRowCount()); + sender.flush(); + sender.table("t").symbol("y", "b").atNow(); + sender.flush(); + waitFor(() -> handler.batches.size() >= 3, 10_000); + } + + Assert.assertEquals(3, handler.batches.size()); + for (int i = 0; i < 3; i++) { + byte[] frame = handler.batches.get(i); + Assert.assertEquals("frame " + i + " carries table t", 1, QwpWireTestUtils.tableCount(frame)); + List dictionary = new ArrayList<>(); + QwpWireTestUtils.accumulateDeltaDictionary(frame, dictionary); + Assert.assertEquals("frame " + i + " defines every id its rows reference", + Arrays.asList("a", "b"), dictionary); + } + } + }); + } + @Test public void testSingleSymbolLargerThanTheCapThrowsWithNothingPublished() throws Exception { // A symbol value cannot be split across frames, so one larger than the cap is a @@ -1040,7 +1098,14 @@ public void testSingleSymbolLargerThanTheCapThrowsWithNothingPublished() throws Assert.assertTrue(e.getMessage(), e.getMessage().contains("a single symbol value is too large for the server batch cap")); } + try { + sender.checkError(); + Assert.fail("the probe must report the retained batch"); + } catch (BatchTooLargeForCapException expected) { + // latched by the dictionary pre-flight + } sender.reset(); + sender.checkError(); } // Nothing may have been published -- not even a dictionary chunk. diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SenderProbeMethodsTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SenderProbeMethodsTest.java new file mode 100644 index 000000000..6cd7e2e8e --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/SenderProbeMethodsTest.java @@ -0,0 +1,631 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.cutlass.qwp.client; + +import io.questdb.client.Sender; +import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.cutlass.qwp.client.BatchTooLargeForCapException; +import io.questdb.client.cutlass.qwp.client.QwpWebSocketSender; +import io.questdb.client.test.QuestDBFlushOnCloseTest; +import io.questdb.client.test.cutlass.qwp.websocket.TestWebSocketServer; +import org.junit.Assert; +import org.junit.Test; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.TimeUnit; + +/** + * The sender methods the pool relies on, exercised on a real WebSocket sender + * against the in-repo test server, which acks every frame: the probes it runs + * at release ({@code checkError()}, {@code hasRowInProgress()} and + * {@code pendingRowCount()}), the savepoint pair ({@code markSavepoint()} and + * {@code resetToSavepoint()}) behind a lease's {@code reset()}, the deferred + * flush of the commit window, the cap-bound byte trigger and the crossing row + * the sender holds back when the cap rejects the auto-flushed batch. + */ +public class SenderProbeMethodsTest { + + @Test + public void testAutoFlushIfDueShipsOnlyRowsOlderThanTheInterval() throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (Sender sender = Sender.fromConfig("ws::addr=localhost:" + server.getPort() + + ";auto_flush_bytes=off;auto_flush_rows=1000000;auto_flush_interval=2000;close_flush_timeout_millis=0;")) { + sender.table("t").longColumn("v", 1L).atNow(); + sender.autoFlushIfDue(); + Assert.assertEquals("younger than the interval: kept", 1, sender.pendingRowCount()); + Thread.sleep(2200); + sender.autoFlushIfDue(); + Assert.assertEquals("older than the interval: shipped", 0, sender.pendingRowCount()); + log.awaitDataFrames(1, 5_000); + } + } + } + + @Test + public void testCheckErrorReportsARetainedOverCapBatchUntilReset() throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (Sender sender = Sender.fromConfig(config(server.getPort()))) { + Assert.assertEquals("the cap is known once fromConfig returns", 4 * 1024, ((QwpWebSocketSender) sender).getServerMaxBatchSize()); + String wide = QuestDBFlushOnCloseTest.wideValue(1000); + for (int i = 0; i < 6; i++) { + sender.table("t").stringColumn("s", wide).atNow(); + } + try { + sender.flush(); + Assert.fail("six 1 KB rows do not fit a 4 KiB cap"); + } catch (BatchTooLargeForCapException expected) { + // retained for retry, as the message says + Assert.assertTrue(expected.getMessage(), expected.getMessage().contains("call reset()")); + Assert.assertFalse("the detail leaves out the standalone remedy", expected.getDetail().contains("reset()")); + Assert.assertTrue(expected.getDetail(), expected.getDetail().endsWith("serverMaxBatchSize=4096]")); + } + try { + sender.checkError(); + Assert.fail("the probe must report the retained batch"); + } catch (BatchTooLargeForCapException expected) { + // the pool retires a slot on this + } + sender.reset(); + sender.checkError(); // cleared with the batch + sender.table("t").longColumn("v", 1L).atNow(); + sender.flush(); + log.awaitDataFrames(1, 5_000); + } + } + } + + @Test + public void testCheckErrorThrowsLatchedTerminal() throws Exception { + try (TestWebSocketServer server = new TestWebSocketServer(new AckAllHandler())) { + server.setRejectWithStatus(401, "Unauthorized"); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // async connect: build() returns, the I/O thread hits the 401 wall and latches a terminal + try (Sender sender = Sender.fromConfig(config(server.getPort()) + + "initial_connect_retry=async;reconnect_max_duration_millis=300;" + + "reconnect_initial_backoff_millis=10;reconnect_max_backoff_millis=50;")) { + long deadline = System.currentTimeMillis() + 5_000; + LineSenderException seen = null; + while (System.currentTimeMillis() < deadline) { + try { + sender.checkError(); + } catch (LineSenderException e) { + seen = e; + break; + } + Thread.sleep(20); + } + Assert.assertNotNull("checkError() must throw the latched auth terminal", seen); + } + } + } + + @Test + public void testCrossingRowInATransactionShipsTheRowsBeforeItDeferred() throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // transaction=on, which a commit window injects: the rows before the crossing + // row ship inside the open transaction, and only flush() commits it + try (Sender sender = Sender.fromConfig(configWithoutAutoFlushBytes(server.getPort()) + "transaction=on;")) { + String wide = QuestDBFlushOnCloseTest.wideValue(1000); + for (int i = 0; i < 3; i++) { + sender.table("t").stringColumn("s", wide).atNow(); + } + sender.table("t").stringColumn("s", QuestDBFlushOnCloseTest.wideValue(1500)).atNow(); + Assert.assertEquals("only the crossing row stays buffered", 1, sender.pendingRowCount()); + log.awaitDataFrames(1, 5_000); + Assert.assertTrue("the rows before the crossing row ship without committing", log.deferFlag(0)); + Assert.assertTrue(sender.hasDeferredFrames()); + sender.flush(); + log.awaitDataFrames(2, 5_000); + Assert.assertFalse("flush() commits the transaction with the held row", log.deferFlag(1)); + Assert.assertEquals(0, log.commitFrames()); + sender.checkError(); + } + } + } + + @Test + public void testCrossingRowShipsTheRowsBeforeIt() throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // no auto_flush_bytes key: the byte trigger sits at 90% of the 4 KiB cap (3686 bytes) + try (Sender sender = Sender.fromConfig(configWithoutAutoFlushBytes(server.getPort()))) { + String wide = QuestDBFlushOnCloseTest.wideValue(1000); + for (int i = 0; i < 3; i++) { + sender.table("t").stringColumn("s", wide).atNow(); + } + // the auto-flushed batch would be 4.5 KB, over the cap: the sender ships + // the three rows before the crossing row and keeps that row + sender.table("t").stringColumn("s", QuestDBFlushOnCloseTest.wideValue(1500)).atNow(); + Assert.assertEquals("only the crossing row stays buffered", 1, sender.pendingRowCount()); + log.awaitDataFrames(1, 5_000); + sender.flush(); + log.awaitDataFrames(2, 5_000); + } + } + } + + @Test + public void testCrossingRowWhoseEarlierRowsStillExceedTheCapThrowsAndKeepsEveryRow() throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // auto_flush_bytes=off: only the row trigger fires, at the sixth row, when the + // five rows before it already exceed the cap + try (Sender sender = Sender.fromConfig("ws::addr=localhost:" + server.getPort() + + ";auto_flush_bytes=off;auto_flush_rows=6;auto_flush_interval=60000;close_flush_timeout_millis=0;")) { + String wide = QuestDBFlushOnCloseTest.wideValue(1000); + for (int i = 0; i < 5; i++) { + sender.table("t").stringColumn("s", wide).atNow(); + } + BatchTooLargeForCapException rejection = null; + try { + sender.table("t").stringColumn("s", wide).atNow(); + Assert.fail("the five rows before the crossing row still exceed the 4 KiB cap"); + } catch (BatchTooLargeForCapException expected) { + rejection = expected; + } + Assert.assertEquals("the crossing row goes back into the batch", 6, sender.pendingRowCount()); + Thread.sleep(300); + Assert.assertEquals("nothing ships", 0, log.dataFrames()); + try { + sender.checkError(); + Assert.fail("the probe must report the retained batch"); + } catch (BatchTooLargeForCapException expected) { + Assert.assertSame("the probe reports the rejection of the whole batch", rejection, expected); + } + try { + sender.flush(); + Assert.fail("the retained batch still exceeds the cap"); + } catch (BatchTooLargeForCapException expected) { + Assert.assertEquals("flush() encodes the same six rows", rejection.getMessage(), expected.getMessage()); + } + sender.reset(); + } + } + } + + @Test + public void testCrossingRowWhoseOwnFrameExceedsTheCapThrowsAfterTheRowsBeforeItShip() throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // no auto_flush_bytes key: the byte trigger sits at 90% of the 4 KiB cap (3686 bytes) + try (Sender sender = Sender.fromConfig(configWithoutAutoFlushBytes(server.getPort()))) { + String wide = QuestDBFlushOnCloseTest.wideValue(1000); + for (int i = 0; i < 3; i++) { + sender.table("t").stringColumn("s", wide).atNow(); + } + // 4070 bytes pass the per-row guard, but the table name and schema take the + // row's own frame over the cap: the rows before it ship, and the row, alone + // over the byte trigger, fails its own flush + BatchTooLargeForCapException rejection = null; + try { + sender.table("u").stringColumn("z", QuestDBFlushOnCloseTest.wideValue(4070)).atNow(); + Assert.fail("the row's own frame exceeds the 4 KiB cap"); + } catch (BatchTooLargeForCapException expected) { + rejection = expected; + } + log.awaitDataFrames(1, 5_000); + Assert.assertEquals("the three t rows ship as one frame with table t", 1, log.tableCount(0)); + try { + sender.checkError(); + Assert.fail("the probe must report the retained row"); + } catch (BatchTooLargeForCapException expected) { + Assert.assertSame(rejection, expected); + } + Assert.assertEquals("only the crossing row stays buffered", 1, sender.pendingRowCount()); + sender.reset(); + Assert.assertEquals(0, sender.pendingRowCount()); + sender.checkError(); + } + } + } + + @Test + public void testCrossingRowWhoseOwnFrameExceedsTheCapFlushesAtOnceWithTheByteTriggerOff() throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (Sender sender = Sender.fromConfig("ws::addr=localhost:" + server.getPort() + + ";auto_flush_bytes=off;auto_flush_rows=4;auto_flush_interval=60000;close_flush_timeout_millis=0;")) { + String wide = QuestDBFlushOnCloseTest.wideValue(1000); + for (int i = 0; i < 3; i++) { + sender.table("t").stringColumn("s", wide).atNow(); + } + try { + sender.table("u").stringColumn("z", QuestDBFlushOnCloseTest.wideValue(4070)).atNow(); + Assert.fail("the row's own frame exceeds the 4 KiB cap"); + } catch (BatchTooLargeForCapException expected) { + // the held row flushed at once and its own rejection reached this call + } + log.awaitDataFrames(1, 5_000); + Assert.assertEquals("only the crossing row stays buffered", 1, sender.pendingRowCount()); + try { + sender.checkError(); + Assert.fail("the probe must report the retained row"); + } catch (BatchTooLargeForCapException expected) { + // the pool retires a slot on this + } + sender.reset(); + Assert.assertEquals(0, sender.pendingRowCount()); + sender.checkError(); + } + } + } + + @Test + public void testDefaultsOnNonWebSocketTransport() { + try (Sender sender = Sender.fromConfig("http::addr=127.0.0.1:1;protocol_version=2;auto_flush=off;")) { + sender.autoFlushIfDue(); // default: no-op + sender.checkError(); // default: no-op + Assert.assertFalse(sender.hasRowInProgress()); // default: false + Assert.assertEquals(0, sender.pendingRowCount()); // default: 0 + Assert.assertEquals(0L, sender.rowsAppended()); + Assert.assertFalse(sender.hasDeferredFrames()); + try { + sender.flushDeferred(); + Assert.fail("expected UnsupportedOperationException"); + } catch (UnsupportedOperationException expected) { + } + } + } + + @Test + public void testDeferredFlushShipsFlaggedFrameAndFlushCommits() throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (Sender sender = Sender.fromConfig(config(server.getPort()))) { + Assert.assertEquals(0L, sender.rowsAppended()); + Assert.assertFalse(sender.hasDeferredFrames()); + Assert.assertEquals(-1L, sender.flushDeferred()); // nothing pending + + sender.table("t").longColumn("v", 1L).atNow(); + Assert.assertEquals(1L, sender.rowsAppended()); + Assert.assertTrue(sender.flushDeferred() >= 0L); + Assert.assertTrue("a deferred frame is open", sender.hasDeferredFrames()); + Assert.assertEquals(0, sender.pendingRowCount()); + + sender.table("t").longColumn("v", 2L).atNow(); + sender.flush(); // closes the group + Assert.assertFalse(sender.hasDeferredFrames()); + Assert.assertEquals(2L, sender.rowsAppended()); + + log.awaitDataFrames(2, 5_000); + Assert.assertEquals("no separate commit message: the committing frame carries the rows", 0, log.commitFrames()); + Assert.assertTrue("first frame carries FLAG_DEFER_COMMIT", log.deferFlag(0)); + Assert.assertFalse("committing frame is unflagged", log.deferFlag(1)); + } + } + } + + @Test + public void testFlushAfterAnEmptySavepointResetShipsTheNextRow() throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (Sender sender = Sender.fromConfig(config(server.getPort()))) { + sender.markSavepoint(); + sender.table("t").longColumn("v", 1L).atNow(); + sender.flush(); // empties the savepoint + sender.table("t").longColumn("v", 2L).atNow(); + sender.resetToSavepoint(); // drops row 2 + sender.table("t").longColumn("v", 3L).atNow(); + sender.flush(); + log.awaitDataFrames(2, 5_000); + sender.checkError(); + } + } + } + + @Test + public void testFlushEmptiesTheSavepoint() throws Exception { + try (TestWebSocketServer server = new TestWebSocketServer(new AckAllHandler())) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (Sender sender = Sender.fromConfig(config(server.getPort()))) { + sender.table("t").longColumn("v", 1L).atNow(); + sender.markSavepoint(); + sender.table("t").longColumn("v", 2L).atNow(); + sender.flush(); // ships both: the savepoint is now empty + sender.table("t").longColumn("v", 3L).atNow(); + sender.resetToSavepoint(); + Assert.assertEquals("a flush inside the lease makes every pending row the lease's own", 0, sender.pendingRowCount()); + sender.checkError(); + } + } + } + + @Test + public void testHealthySenderReportsBufferState() throws Exception { + try (TestWebSocketServer server = new TestWebSocketServer(new AckAllHandler())) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (Sender sender = Sender.fromConfig(config(server.getPort()))) { + sender.checkError(); // healthy: no throw + Assert.assertFalse(sender.hasRowInProgress()); + Assert.assertEquals(0, sender.pendingRowCount()); + + sender.table("t").longColumn("v", 1L); // row started, not committed + Assert.assertTrue(sender.hasRowInProgress()); + Assert.assertEquals(0, sender.pendingRowCount()); + + sender.cancelRow(); + Assert.assertFalse(sender.hasRowInProgress()); + + sender.table("t").longColumn("v", 2L).atNow(); + Assert.assertFalse(sender.hasRowInProgress()); + Assert.assertEquals(1, sender.pendingRowCount()); + + sender.flush(); + Assert.assertEquals(0, sender.pendingRowCount()); + } + } + } + + @Test + public void testHeldCrossingRowWithANewSymbolKeepsTheDictionaryGapFree() throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (Sender sender = Sender.fromConfig(configWithoutAutoFlushBytes(server.getPort()))) { + Assert.assertTrue("memory mode selects delta mode", ((QwpWebSocketSender) sender).isDeltaDictEnabledForTest()); + String wide = QuestDBFlushOnCloseTest.wideValue(1000); + for (int i = 0; i < 3; i++) { + sender.table("t").symbol("y", "a").stringColumn("s", wide).atNow(); + } + // the held crossing row registers "b" before the frame of the three + // rows ahead of it ships + sender.table("t").symbol("y", "b").stringColumn("s", QuestDBFlushOnCloseTest.wideValue(1500)).atNow(); + sender.flush(); + sender.table("t").symbol("y", "b").atNow(); + sender.flush(); + log.awaitDataFrames(3, 5_000); + List dict = new ArrayList<>(); + for (byte[] frame : log.frames()) { + QwpWireTestUtils.accumulateDeltaDictionary(frame, dict, false); + } + Assert.assertEquals(Arrays.asList("a", "b"), dict); + sender.checkError(); + } + } + } + + @Test + public void testResetToSavepointAfterAnOverCapFlushClearsTheProbeAndShipsTheEarlierRows() throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (Sender sender = Sender.fromConfig(config(server.getPort()))) { + sender.table("t").longColumn("v", 1L).atNow(); + sender.markSavepoint(); + String wide = QuestDBFlushOnCloseTest.wideValue(1000); + for (int i = 0; i < 6; i++) { + sender.table("t").stringColumn("s", wide).atNow(); + } + try { + sender.flush(); + Assert.fail("the lease's rows do not fit a 4 KiB cap"); + } catch (BatchTooLargeForCapException expected) { + // retained for retry + } + try { + sender.checkError(); + Assert.fail("the probe must report the retained batch"); + } catch (BatchTooLargeForCapException expected) { + // the pool retires a slot on this + } + sender.resetToSavepoint(); + sender.checkError(); + Assert.assertEquals(1, sender.pendingRowCount()); + sender.flush(); + log.awaitDataFrames(1, 5_000); + } + } + } + + @Test + public void testResetToSavepointKeepsRowsBufferedBeforeTheMark() throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (Sender sender = Sender.fromConfig(config(server.getPort()))) { + sender.table("t1").longColumn("v", 1L).atNow(); + sender.table("t1").longColumn("v", 2L).atNow(); + sender.markSavepoint(); + sender.table("t1").longColumn("v", 3L).stringColumn("extra", "x").atNow(); + sender.table("t2").longColumn("v", 1L).atNow(); + Assert.assertEquals(4, sender.pendingRowCount()); + + sender.resetToSavepoint(); + + Assert.assertEquals("the two rows before the mark stay", 2, sender.pendingRowCount()); + sender.flush(); + log.awaitDataFrames(1, 5_000); + Assert.assertEquals("t2 was created after the mark and is gone", 1, log.tableCount(0)); + // the buffer is still usable, including the column the rollback removed + sender.table("t1").longColumn("v", 4L).stringColumn("extra", "y").atNow(); + sender.flush(); + log.awaitDataFrames(2, 5_000); + } + } + } + + @Test + public void testResetToSavepointReclaimsTheSymbolIdsOfTheDroppedRows() throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (Sender sender = Sender.fromConfig(config(server.getPort()))) { + sender.table("t").symbol("s", "a").atNow(); + sender.markSavepoint(); + String wide = QuestDBFlushOnCloseTest.wideValue(1000); + for (int i = 0; i < 6; i++) { + sender.table("t").symbol("s", wide + i).atNow(); + } + try { + sender.flush(); + Assert.fail("six wide symbols do not fit a 4 KiB cap"); + } catch (BatchTooLargeForCapException expected) { + // retained for retry + } + sender.resetToSavepoint(); + sender.flush(); + log.awaitDataFrames(1, 5_000); + sender.table("t").symbol("s", "b").atNow(); + sender.flush(); + log.awaitDataFrames(2, 5_000); + sender.checkError(); + } + } + } + + @Test + public void testResetToSavepointWithoutMarkIsReset() throws Exception { + try (TestWebSocketServer server = new TestWebSocketServer(new AckAllHandler())) { + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (Sender sender = Sender.fromConfig(config(server.getPort()))) { + sender.table("t").longColumn("v", 1L).atNow(); + sender.table("t").longColumn("v", 2L).atNow(); + sender.resetToSavepoint(); + Assert.assertEquals(0, sender.pendingRowCount()); + } + } + } + + @Test + public void testSingleRowWhoseOwnFrameExceedsTheCapThrowsAndStaysBuffered() throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(4 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + try (Sender sender = Sender.fromConfig(configWithoutAutoFlushBytes(server.getPort()))) { + // the crossing row is the only pending row: no rows before it can ship, + // so the sender holds nothing back and the rejection reaches this call + BatchTooLargeForCapException rejection = null; + try { + sender.table("u").stringColumn("z", QuestDBFlushOnCloseTest.wideValue(4070)).atNow(); + Assert.fail("the only pending row's own frame exceeds the 4 KiB cap"); + } catch (BatchTooLargeForCapException expected) { + rejection = expected; + } + Assert.assertEquals("the row stays buffered", 1, sender.pendingRowCount()); + try { + sender.checkError(); + Assert.fail("the probe must report the retained row"); + } catch (BatchTooLargeForCapException expected) { + Assert.assertSame(rejection, expected); + } + sender.reset(); + sender.table("u").longColumn("v", 1L).atNow(); + sender.flush(); + log.awaitDataFrames(1, 5_000); + sender.checkError(); + } + } + } + + @Test + public void testUnsetAutoFlushBytesShipsAtTheAdvertisedCap() throws Exception { + QuestDBFlushOnCloseTest.FrameLog log = new QuestDBFlushOnCloseTest.FrameLog(); + try (TestWebSocketServer server = new TestWebSocketServer(log)) { + server.setAdvertisedMaxBatchSize(16 * 1024); + server.start(); + Assert.assertTrue(server.awaitStart(5, TimeUnit.SECONDS)); + // no auto_flush_bytes key: the sender binds its byte trigger to 90% of the 16 KiB cap + try (Sender sender = Sender.fromConfig("ws::addr=localhost:" + server.getPort() + + ";auto_flush_rows=1000000;auto_flush_interval=60000;close_flush_timeout_millis=0;")) { + String value = QuestDBFlushOnCloseTest.wideValue(1000); + for (int i = 0; i < 20; i++) { + sender.table("t").stringColumn("s", value).atNow(); + } + // 20 KB of rows against a 14.7 KB budget: the byte trigger shipped at least once + log.awaitDataFrames(1, 5_000); + Assert.assertTrue("rows left after the byte flush", sender.pendingRowCount() > 0 && sender.pendingRowCount() < 20); + } + } + } + + private static String config(int port) { + return "ws::addr=localhost:" + port + ";auto_flush_bytes=off;auto_flush_rows=1000000;auto_flush_interval=60000;close_flush_timeout_millis=0;"; + } + + private static String configWithoutAutoFlushBytes(int port) { + return "ws::addr=localhost:" + port + ";auto_flush_rows=1000000;auto_flush_interval=60000;close_flush_timeout_millis=0;"; + } + + private static final class AckAllHandler implements TestWebSocketServer.WebSocketServerHandler { + private TestWebSocketServer.ClientHandler currentClient; + private long nextSeq; + + @Override + public synchronized void onBinaryMessage(TestWebSocketServer.ClientHandler client, byte[] data) { + if (currentClient != client) { + currentClient = client; + nextSeq = 0; + } + try { + client.sendBinary(QwpWireTestUtils.buildAck(nextSeq++)); + } catch (IOException e) { + throw new RuntimeException(e); + } + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorSendEngineTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorSendEngineTest.java index 7f802535a..3536c518d 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorSendEngineTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/client/sf/cursor/CursorSendEngineTest.java @@ -44,9 +44,12 @@ import io.questdb.client.std.Unsafe; import io.questdb.client.test.tools.TestUtils; import org.junit.After; +import org.junit.Assume; import org.junit.Before; import org.junit.Test; +import java.lang.management.ManagementFactory; +import java.lang.management.ThreadMXBean; import java.lang.reflect.Constructor; import java.lang.reflect.Field; import java.lang.reflect.InvocationTargetException; @@ -344,6 +347,41 @@ public void testCheckDurabilitySurfacesLatchedFailureToSenderEntryPoints() throw }); } + @Test + public void testAppendBlockingParksInsteadOfSpinningWhenInterrupted() throws Exception { + ThreadMXBean mx = ManagementFactory.getThreadMXBean(); + Assume.assumeTrue(mx.isCurrentThreadCpuTimeSupported()); + TestUtils.assertMemoryLeak(() -> { + long buf = Unsafe.malloc(16, MemoryTag.NATIVE_DEFAULT); + try (CursorSendEngine engine = new CursorSendEngine(tmpDir, 4096, 2 * 4096, TimeUnit.MILLISECONDS.toNanos(300))) { + while (true) { // fill the ring; nothing drains it + try { + engine.appendBlocking(buf, 16); + } catch (LineSenderException full) { + break; + } + } + Thread.currentThread().interrupt(); + long cpuStart = mx.getCurrentThreadCpuTime(); + long wallStart = System.nanoTime(); + try { + engine.appendBlocking(buf, 16); + fail("a full ring rejects after the deadline"); + } catch (LineSenderException expected) { + // backpressure deadline + } + long wall = System.nanoTime() - wallStart; + long cpu = mx.getCurrentThreadCpuTime() - cpuStart; + assertTrue("the interrupt flag survives the wait", Thread.interrupted()); + assertTrue("waited out the deadline", wall >= TimeUnit.MILLISECONDS.toNanos(300)); + assertTrue("parked, not spinning: cpu=" + cpu + " wall=" + wall, cpu < wall / 2); + } finally { + Thread.interrupted(); + Unsafe.free(buf, 16, MemoryTag.NATIVE_DEFAULT); + } + }); + } + @Test public void testAppendChecksLatchedDurabilityFailureBeforePublishing() throws Exception { TestUtils.assertMemoryLeak(() -> { diff --git a/core/src/test/java/io/questdb/client/test/cutlass/qwp/protocol/QwpTableBufferTest.java b/core/src/test/java/io/questdb/client/test/cutlass/qwp/protocol/QwpTableBufferTest.java index ab14c18d8..ba68eb7aa 100644 --- a/core/src/test/java/io/questdb/client/test/cutlass/qwp/protocol/QwpTableBufferTest.java +++ b/core/src/test/java/io/questdb/client/test/cutlass/qwp/protocol/QwpTableBufferTest.java @@ -28,6 +28,7 @@ import io.questdb.client.cutlass.line.LineSenderException; import io.questdb.client.cutlass.line.array.DoubleArray; import io.questdb.client.cutlass.line.array.LongArray; +import io.questdb.client.cutlass.qwp.client.QwpWebSocketEncoder; import io.questdb.client.cutlass.qwp.protocol.OffHeapAppendMemory; import io.questdb.client.cutlass.qwp.protocol.QwpConstants; import io.questdb.client.cutlass.qwp.protocol.QwpTableBuffer; @@ -1150,6 +1151,81 @@ public void testGetOrCreateColumnThrowsWhenExceedingMaxColumnCount() throws Exce }); } + @Test + public void testHeldRowEncodesThePrefixAndRetainsTheTailRow() throws Exception { + assertMemoryLeak(() -> { + try (QwpTableBuffer both = new QwpTableBuffer("t"); + QwpTableBuffer first = new QwpTableBuffer("t"); + QwpTableBuffer second = new QwpTableBuffer("t")) { + appendMixedRow(both, 1L, 1.5, true, "alpha", "n1", "sym", new double[]{1.0, 2.0}); + appendMixedRow(both, 2L, 2.5, false, "beta", null, "sym", new double[]{3.0, 4.0, 5.0}); + appendMixedRow(first, 1L, 1.5, true, "alpha", "n1", "sym", new double[]{1.0, 2.0}); + appendMixedRow(second, 2L, 2.5, false, "beta", null, "sym", new double[]{3.0, 4.0, 5.0}); + + assertTrue(both.holdBackLastRow()); + assertEquals(1, both.getRowCount()); + assertArrayEquals("the prefix encodes as the first row alone", encodeForTest(first), encodeForTest(both)); + + both.retainHeldRow(); + assertFalse(both.hasHeldRow()); + assertEquals(1, both.getRowCount()); + assertArrayEquals("the retained row encodes as the second row alone", encodeForTest(second), encodeForTest(both)); + } + }); + } + + @Test + public void testHoldBackLastRowGrowsPastTheInitialColumnCapacity() throws Exception { + assertMemoryLeak(() -> { + try (QwpTableBuffer both = new QwpTableBuffer("t"); + QwpTableBuffer first = new QwpTableBuffer("t"); + QwpTableBuffer second = new QwpTableBuffer("t")) { + int columnCount = 20; + for (int i = 0; i < columnCount; i++) { + both.getOrCreateColumn("c" + i, QwpConstants.TYPE_LONG, false).addLong(i); + first.getOrCreateColumn("c" + i, QwpConstants.TYPE_LONG, false).addLong(i); + } + both.nextRow(); + first.nextRow(); + for (int i = 0; i < columnCount; i++) { + both.getOrCreateColumn("c" + i, QwpConstants.TYPE_LONG, false).addLong(100 + i); + second.getOrCreateColumn("c" + i, QwpConstants.TYPE_LONG, false).addLong(100 + i); + } + both.nextRow(); + second.nextRow(); + + assertTrue(both.holdBackLastRow()); + assertTrue(both.getHeldValueCounts().length >= columnCount); + assertArrayEquals(encodeForTest(first), encodeForTest(both)); + both.retainHeldRow(); + assertArrayEquals(encodeForTest(second), encodeForTest(both)); + } + }); + } + + @Test + public void testHoldBackLastRowRefusesWithoutACommittedRowOrMidRow() throws Exception { + assertMemoryLeak(() -> { + try (QwpTableBuffer table = new QwpTableBuffer("t")) { + assertFalse("an empty buffer has no row to hold", table.holdBackLastRow()); + assertFalse(table.hasHeldRow()); + + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(1L); + table.nextRow(); + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(2L); + assertTrue(table.hasInProgressRow()); + assertFalse("a row in progress blocks the hold", table.holdBackLastRow()); + assertFalse(table.hasHeldRow()); + assertEquals(1, table.getRowCount()); + + table.nextRow(); + assertTrue(table.holdBackLastRow()); + assertFalse("a held row blocks a second hold", table.holdBackLastRow()); + assertEquals(1, table.getRowCount()); + } + }); + } + @Test public void testLongArrayMultipleRows() throws Exception { assertMemoryLeak(() -> { @@ -1388,6 +1464,130 @@ public void testNonAsciiColumnNameCaseInsensitive() throws Exception { }); } + @Test + public void testResetAndClearDropTheHeldRow() throws Exception { + assertMemoryLeak(() -> { + try (QwpTableBuffer table = new QwpTableBuffer("t")) { + appendMixedRow(table, 1L, 1.5, true, "alpha", "n1", "sym", new double[]{1.0, 2.0}); + appendMixedRow(table, 2L, 2.5, false, "beta", null, "sym", new double[]{3.0, 4.0, 5.0}); + assertTrue(table.holdBackLastRow()); + table.reset(); + assertFalse("reset() drops the hold", table.hasHeldRow()); + assertEquals(0, table.getRowCount()); + + appendMixedRow(table, 3L, 3.5, true, "gamma", "n3", "sym", new double[]{6.0}); + assertTrue(table.holdBackLastRow()); + table.clear(); + assertFalse("clear() drops the hold", table.hasHeldRow()); + assertEquals(0, table.getRowCount()); + } + }); + } + + @Test + public void testResetMovesTheSavepointToTheEmptyBuffer() throws Exception { + assertMemoryLeak(() -> { + try (QwpTableBuffer table = new QwpTableBuffer("t")) { + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(1L); + table.nextRow(); + table.markSavepoint(); + table.reset(); + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(2L); + table.nextRow(); + table.rollbackToSavepoint(); + assertEquals("reset() emptied the savepoint", 0, table.getRowCount()); + assertEquals("columns the server already knows stay", 1, table.getColumnCount()); + } + }); + } + + @Test + public void testRestoreHeldRowPutsTheRowBack() throws Exception { + assertMemoryLeak(() -> { + try (QwpTableBuffer table = new QwpTableBuffer("t")) { + appendMixedRow(table, 1L, 1.5, true, "alpha", "n1", "sym", new double[]{1.0, 2.0}); + appendMixedRow(table, 2L, 2.5, false, "beta", null, "sym", new double[]{3.0, 4.0, 5.0}); + byte[] before = encodeForTest(table); + + assertTrue(table.holdBackLastRow()); + table.restoreHeldRow(); + assertFalse(table.hasHeldRow()); + assertArrayEquals(before, encodeForTest(table)); + assertEquals(2, table.getRowCount()); + } + }); + } + + @Test + public void testRetainHeldRowKeepsGeoHashPrecisionOfANullHeldValue() throws Exception { + assertMemoryLeak(() -> { + try (QwpTableBuffer table = new QwpTableBuffer("t")) { + QwpTableBuffer.ColumnBuffer g = table.getOrCreateColumn("g", QwpConstants.TYPE_GEOHASH, true); + g.addGeoHash(123L, 20); + table.nextRow(); + table.getOrCreateColumn("g", QwpConstants.TYPE_GEOHASH, true).addNull(); + table.nextRow(); + + assertTrue(table.holdBackLastRow()); + table.retainHeldRow(); + assertEquals(1, table.getRowCount()); + assertTrue(g.isNull(0)); + assertEquals("the server locked the precision with the shipped rows", 20, g.getGeoHashPrecision()); + } + }); + } + + @Test + public void testRetainHeldRowMovesTheSavepointColumnsOnlyWhenEarlierRowsShipped() throws Exception { + assertMemoryLeak(() -> { + // rows before the held row: the caller shipped them with every column, + // including the one the held row added, so the savepoint takes it + try (QwpTableBuffer table = new QwpTableBuffer("t")) { + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(1L); + table.nextRow(); + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(2L); + table.getOrCreateColumn("z", QwpConstants.TYPE_VARCHAR, true).addString("held"); + table.nextRow(); + assertTrue(table.holdBackLastRow()); + table.retainHeldRow(); + assertEquals(1, table.getRowCount()); + assertEquals(0, table.getSavepointRowCount()); + table.rollbackToSavepoint(); + assertEquals(0, table.getRowCount()); + assertEquals("the server knows both columns", 2, table.getColumnCount()); + } + // the held row is the only row of a new buffer: nothing of the table + // shipped, so the savepoint keeps no column + try (QwpTableBuffer table = new QwpTableBuffer("t")) { + table.getOrCreateColumn("z", QwpConstants.TYPE_VARCHAR, true).addString("held"); + table.nextRow(); + assertTrue(table.holdBackLastRow()); + table.retainHeldRow(); + assertEquals(1, table.getRowCount()); + assertEquals(0, table.getSavepointRowCount()); + table.rollbackToSavepoint(); + assertEquals(0, table.getRowCount()); + assertEquals("the server has seen no column", 0, table.getColumnCount()); + } + // the held row is the only row after a flush: the savepoint keeps the + // column the flush shipped, not the one the held row added + try (QwpTableBuffer table = new QwpTableBuffer("t")) { + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(1L); + table.nextRow(); + table.reset(); + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(2L); + table.getOrCreateColumn("z", QwpConstants.TYPE_VARCHAR, true).addString("held"); + table.nextRow(); + assertTrue(table.holdBackLastRow()); + table.retainHeldRow(); + assertEquals(1, table.getRowCount()); + table.rollbackToSavepoint(); + assertEquals(0, table.getRowCount()); + assertEquals("only the column the flush shipped stays", 1, table.getColumnCount()); + } + }); + } + @Test public void testRetainInProgressRowFastClearsUnstagedNullableColumn() throws Exception { assertMemoryLeak(() -> { @@ -1451,6 +1651,145 @@ public void testRetainInProgressRowFastClearsUnstagedNullableColumn() throws Exc }); } + @Test + public void testRetainInProgressRowMovesTheSavepointToTheRetainedRow() throws Exception { + assertMemoryLeak(() -> { + try (QwpTableBuffer table = new QwpTableBuffer("t")) { + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(1L); + table.nextRow(); + table.markSavepoint(); + assertEquals(1, table.getSavepointRowCount()); + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(2L); // row 2 in progress + assertTrue(table.hasInProgressRow()); + table.retainInProgressRow(new int[]{1}, new int[]{1}, new int[]{0}, new int[]{0}); + assertEquals(0, table.getRowCount()); + assertEquals("the shipped rows are no longer a rollback target", 0, table.getSavepointRowCount()); + + table.nextRow(); + table.rollbackToSavepoint(); + assertEquals(0, table.getRowCount()); + assertEquals(1, table.getColumnCount()); + } + + try (QwpTableBuffer table = new QwpTableBuffer("t")) { + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(1L); + table.nextRow(); + table.markSavepoint(); + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(2L); + table.getOrCreateColumn("s", QwpConstants.TYPE_VARCHAR, false).addString("x"); + // "s" was created by the in-progress row: sizeBefore -1 clears it to empty + table.retainInProgressRow(new int[]{1, -1}, new int[]{1, -1}, new int[]{0, 0}, new int[]{0, 0}); + table.nextRow(); + table.rollbackToSavepoint(); + assertEquals(0, table.getRowCount()); + assertEquals("the column the server now knows is not dropped", 2, table.getColumnCount()); + } + }); + } + + @Test + public void testRollbackToSavepointDiscardsAnInProgressRow() throws Exception { + assertMemoryLeak(() -> { + try (QwpTableBuffer table = new QwpTableBuffer("t")) { + QwpTableBuffer.ColumnBuffer v = table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false); + v.addLong(1L); + table.nextRow(); + table.markSavepoint(); + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(2L); // row 2 started, not committed + assertTrue(table.hasInProgressRow()); + table.rollbackToSavepoint(); + assertEquals(1, table.getRowCount()); + assertFalse(table.hasInProgressRow()); + assertEquals(1, v.getSize()); + } + }); + } + + @Test + public void testRollbackToSavepointKeepsEarlierRowsAndDropsLaterColumns() throws Exception { + assertMemoryLeak(() -> { + try (QwpTableBuffer table = new QwpTableBuffer("t")) { + QwpTableBuffer.ColumnBuffer v = table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false); + v.addLong(1L); + long bytesAtMark = table.nextRow(); + table.markSavepoint(); + assertEquals(1, table.getSavepointRowCount()); + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(2L); + table.getOrCreateColumn("s", QwpConstants.TYPE_VARCHAR, false).addString("later"); + table.nextRow(); + assertEquals(2, table.getRowCount()); + assertEquals(2, table.getColumnCount()); + + table.rollbackToSavepoint(); + + assertEquals(1, table.getRowCount()); + assertEquals("the column added after the mark is gone", 1, table.getColumnCount()); + assertEquals(1, v.getSize()); + assertEquals(bytesAtMark, table.getBufferedBytes()); + assertFalse(table.hasInProgressRow()); + // still usable: the dropped column can come back + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(3L); + table.getOrCreateColumn("s", QwpConstants.TYPE_VARCHAR, false).addString("again"); + table.nextRow(); + assertEquals(2, table.getRowCount()); + assertEquals(2, table.getColumnCount()); + assertEquals(2, v.getSize()); + } + }); + } + + @Test + public void testRollbackToSavepointKeepsGeoHashPrecisionOfARetainedColumn() throws Exception { + assertMemoryLeak(() -> { + try (QwpTableBuffer table = new QwpTableBuffer("t")) { + QwpTableBuffer.ColumnBuffer g = table.getOrCreateColumn("g", QwpConstants.TYPE_GEOHASH, true); + g.addGeoHash(123L, 20); + table.nextRow(); + table.reset(); + table.getOrCreateColumn("g", QwpConstants.TYPE_GEOHASH, true).addGeoHash(456L, 20); + table.nextRow(); + table.rollbackToSavepoint(); + assertEquals(0, table.getRowCount()); + assertEquals(20, g.getGeoHashPrecision()); + } + }); + } + + @Test + public void testRollbackToSavepointOnANeverMarkedBufferRemovesEveryColumn() throws Exception { + assertMemoryLeak(() -> { + try (QwpTableBuffer table = new QwpTableBuffer("t")) { + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(1L); + table.nextRow(); + table.rollbackToSavepoint(); + assertEquals(0, table.getRowCount()); + assertEquals(0, table.getColumnCount()); + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(2L); + table.nextRow(); + assertEquals(1, table.getRowCount()); + assertEquals(1, table.getColumnCount()); + } + }); + } + + @Test + public void testRollbackToSavepointWithoutMarkDropsEveryRowButKeepsColumns() throws Exception { + assertMemoryLeak(() -> { + try (QwpTableBuffer table = new QwpTableBuffer("t")) { + QwpTableBuffer.ColumnBuffer v = table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false); + v.addLong(1L); + table.nextRow(); + table.reset(); + table.getOrCreateColumn("v", QwpConstants.TYPE_LONG, false).addLong(2L); + table.nextRow(); + table.rollbackToSavepoint(); + assertEquals(0, table.getRowCount()); + assertEquals(0, v.getSize()); + assertEquals("no mark: behaves like reset(), definitions stay", 1, table.getColumnCount()); + } + }); + } + private static void addSymbolUtf8(QwpTableBuffer.ColumnBuffer col, String value) { byte[] bytes = value.getBytes(StandardCharsets.UTF_8); long ptr = copyToNative(bytes); @@ -1461,6 +1800,31 @@ private static void addSymbolUtf8(QwpTableBuffer.ColumnBuffer col, String value) } } + private static void appendMixedRow( + QwpTableBuffer table, + long a, + double d, + boolean b, + String s, + String n, + String y, + double[] arr + ) { + table.getOrCreateColumn("a", QwpConstants.TYPE_LONG, false).addLong(a); + table.getOrCreateColumn("d", QwpConstants.TYPE_DOUBLE, true).addDouble(d); + table.getOrCreateColumn("b", QwpConstants.TYPE_BOOLEAN, false).addBoolean(b); + table.getOrCreateColumn("s", QwpConstants.TYPE_VARCHAR, true).addString(s); + QwpTableBuffer.ColumnBuffer nColumn = table.getOrCreateColumn("n", QwpConstants.TYPE_VARCHAR, true); + if (n == null) { + nColumn.addNull(); + } else { + nColumn.addString(n); + } + table.getOrCreateColumn("y", QwpConstants.TYPE_SYMBOL, true).addSymbol(y); + table.getOrCreateColumn("arr", QwpConstants.TYPE_DOUBLE_ARRAY, true).addDoubleArray(arr); + table.nextRow(); + } + private static long copyToNative(byte[] bytes) { long ptr = Unsafe.malloc(bytes.length, MemoryTag.NATIVE_DEFAULT); for (int i = 0; i < bytes.length; i++) { @@ -1469,6 +1833,18 @@ private static long copyToNative(byte[] bytes) { return ptr; } + private static byte[] encodeForTest(QwpTableBuffer table) { + try (QwpWebSocketEncoder encoder = new QwpWebSocketEncoder()) { + int length = encoder.encode(table); + long ptr = encoder.getBuffer().getBufferPtr(); + byte[] bytes = new byte[length]; + for (int i = 0; i < length; i++) { + bytes[i] = Unsafe.getUnsafe().getByte(ptr + i); + } + return bytes; + } + } + /** * Simulates the encoder's walk over array data — the same logic as * QwpWebSocketEncoder.writeDoubleArrayColumn(). Returns the flat diff --git a/core/src/test/java/io/questdb/client/test/impl/PoolConfigHonoredTest.java b/core/src/test/java/io/questdb/client/test/impl/PoolConfigHonoredTest.java index a8c94e40d..1cb6fb2ed 100644 --- a/core/src/test/java/io/questdb/client/test/impl/PoolConfigHonoredTest.java +++ b/core/src/test/java/io/questdb/client/test/impl/PoolConfigHonoredTest.java @@ -60,6 +60,9 @@ public void testEveryPoolKeyIsHonored() throws Exception { expected.put("idle_timeout_ms", 4321L); expected.put("max_lifetime_ms", 98765L); expected.put("housekeeper_interval_ms", 222L); + expected.put("flush_on_close", Boolean.TRUE); + expected.put("commit_rows", 500); + expected.put("commit_interval_ms", 250L); StringBuilder cfg = new StringBuilder("ws::addr=127.0.0.1:1;"); for (Map.Entry e : expected.entrySet()) { diff --git a/core/src/test/java/io/questdb/client/test/impl/RecordingSender.java b/core/src/test/java/io/questdb/client/test/impl/RecordingSender.java new file mode 100644 index 000000000..ef71caef5 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/impl/RecordingSender.java @@ -0,0 +1,282 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + + +package io.questdb.client.test.impl; + +import io.questdb.client.Sender; +import io.questdb.client.cutlass.line.array.DoubleArray; +import io.questdb.client.cutlass.line.array.LongArray; +import io.questdb.client.std.bytes.DirectByteSlice; + +import java.time.Instant; +import java.time.temporal.ChronoUnit; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * In-memory {@link Sender} double for pool tests. Records calls, models the + * buffer state the pool asks about, and can be told to fail or to block. + * Never touches the network. Not final, so a test can override a single method. + */ +class RecordingSender implements Sender { + final AtomicInteger cancelCount = new AtomicInteger(); + final AtomicInteger closeCount = new AtomicInteger(); + final AtomicInteger deferredFlushCount = new AtomicInteger(); + final AtomicInteger flushCount = new AtomicInteger(); + volatile RuntimeException cancelError; // thrown by cancelRow() when set + volatile RuntimeException deferredFlushError; // thrown by flushDeferred() when set + volatile boolean deferredFramesOpen; + volatile RuntimeException flushError; // thrown by flush() when set + volatile Error flushFatal; // thrown by flush() when set, before flushError + volatile CountDownLatch flushGate; // flush() blocks on it when set + volatile RuntimeException latchedError; // thrown by checkError() when set + volatile CountDownLatch markSavepointGate; // markSavepoint() blocks on it when set + volatile int pendingRows; + volatile boolean rowInProgress; + volatile long rowsAppended; + volatile int savepointRows; + + @Override + public void at(long timestamp, ChronoUnit unit) { + rowInProgress = false; + pendingRows++; + rowsAppended++; + } + + @Override + public void at(Instant timestamp) { + at(0L, ChronoUnit.MICROS); + } + + @Override + public void atNow() { + at(0L, ChronoUnit.MICROS); + } + + @Override + public Sender boolColumn(CharSequence name, boolean value) { + return this; + } + + @Override + public DirectByteSlice bufferView() { + return null; + } + + @Override + public void cancelRow() { + cancelCount.incrementAndGet(); + RuntimeException e = cancelError; + if (e != null) { + throw e; + } + rowInProgress = false; + } + + @Override + public void checkError() { + RuntimeException e = latchedError; + if (e != null) { + throw e; + } + } + + @Override + public void close() { + closeCount.incrementAndGet(); + } + + @Override + public Sender doubleArray(CharSequence name, double[] values) { + return this; + } + + @Override + public Sender doubleArray(CharSequence name, double[][] values) { + return this; + } + + @Override + public Sender doubleArray(CharSequence name, double[][][] values) { + return this; + } + + @Override + public Sender doubleArray(CharSequence name, DoubleArray values) { + return this; + } + + @Override + public Sender doubleColumn(CharSequence name, double value) { + return this; + } + + @Override + public boolean drain(long timeoutMillis) { + flush(); + return true; + } + + @Override + public void flush() { + awaitGate(flushGate, "flushGate"); + Error fatal = flushFatal; + if (fatal != null) { + throw fatal; + } + RuntimeException e = flushError; + if (e != null) { + throw e; + } + // Like the real sender: publishes nothing when there is nothing to + // ship and no open group to commit, so count only real work. + if (pendingRows > 0 || deferredFramesOpen) { + flushCount.incrementAndGet(); + } + pendingRows = 0; + savepointRows = 0; + deferredFramesOpen = false; + } + + @Override + public long flushDeferred() { + awaitGate(flushGate, "flushGate"); + RuntimeException deferredError = deferredFlushError; + if (deferredError != null) { + throw deferredError; + } + boolean hadRows = pendingRows > 0; + if (!hadRows) { + return -1L; // nothing pending: the real sender publishes nothing + } + pendingRows = 0; + savepointRows = 0; + deferredFramesOpen = true; + return deferredFlushCount.incrementAndGet(); + } + + @Override + public boolean hasDeferredFrames() { + return deferredFramesOpen; + } + + @Override + public boolean hasRowInProgress() { + return rowInProgress; + } + + @Override + public Sender longArray(CharSequence name, long[] values) { + return this; + } + + @Override + public Sender longArray(CharSequence name, long[][] values) { + return this; + } + + @Override + public Sender longArray(CharSequence name, long[][][] values) { + return this; + } + + @Override + public Sender longArray(CharSequence name, LongArray values) { + return this; + } + + @Override + public Sender longColumn(CharSequence name, long value) { + return this; + } + + @Override + public void markSavepoint() { + awaitGate(markSavepointGate, "markSavepointGate"); + savepointRows = pendingRows; + } + + @Override + public int pendingRowCount() { + return pendingRows; + } + + @Override + public void reset() { + pendingRows = 0; + savepointRows = 0; + rowInProgress = false; + } + + @Override + public void resetToSavepoint() { + pendingRows = savepointRows; + rowInProgress = false; + } + + @Override + public long rowsAppended() { + return rowsAppended; + } + + @Override + public Sender stringColumn(CharSequence name, CharSequence value) { + return this; + } + + @Override + public Sender symbol(CharSequence name, CharSequence value) { + return this; + } + + @Override + public Sender table(CharSequence table) { + rowInProgress = true; + return this; + } + + @Override + public Sender timestampColumn(CharSequence name, long value, ChronoUnit unit) { + return this; + } + + @Override + public Sender timestampColumn(CharSequence name, Instant value) { + return this; + } + + private static void awaitGate(CountDownLatch gate, String name) { + if (gate != null) { + try { + if (!gate.await(10, TimeUnit.SECONDS)) { + throw new AssertionError(name + " was never released"); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/impl/SenderPoolCloseSemanticsTest.java b/core/src/test/java/io/questdb/client/test/impl/SenderPoolCloseSemanticsTest.java new file mode 100644 index 000000000..9816b5649 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/impl/SenderPoolCloseSemanticsTest.java @@ -0,0 +1,456 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + + +package io.questdb.client.test.impl; + +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.spi.ILoggingEvent; +import ch.qos.logback.core.read.ListAppender; +import io.questdb.client.LineSenderServerException; +import io.questdb.client.Sender; +import io.questdb.client.SenderError; +import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.impl.PooledSender; +import io.questdb.client.impl.SenderPool; +import org.junit.Assert; +import org.junit.Test; +import org.slf4j.LoggerFactory; + +import java.lang.reflect.Field; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.atomic.AtomicReference; + +/** + * The pooled close() contract after the release-mode change: close releases, + * flush ships, drain waits. Drives {@link SenderPool} through its + * {@code senderFactory} seam with {@link RecordingSender} doubles, so nothing + * touches the network. The config string is parsed for store-and-forward + * settings only. + */ +public class SenderPoolCloseSemanticsTest { + private static final String CONFIG = "http::addr=127.0.0.1:1;protocol_version=2;auto_flush=off;"; + + @Test + public void testCancelRowThrowingAtCloseRetiresSlot() { + List built = new ArrayList<>(); + try (SenderPool pool = newPool(1, 1, built)) { + Sender first = pool.borrow(); + RecordingSender delegate = built.get(0); + // A terminal latched between the probe and the cancel: model it as + // cancelRow() itself failing. The lease must still come home. + delegate.rowInProgress = true; + LineSenderException original = new LineSenderException("latched during cancel"); + delegate.cancelError = original; + try { + first.close(); + Assert.fail("close() must propagate the cancel failure"); + } catch (LineSenderException e) { + Assert.assertSame(original, e.getCause()); + Assert.assertTrue(e.getMessage().contains("latched during cancel")); + } + Assert.assertEquals("slot must be retired, not stranded", 0, pool.totalSize()); + Assert.assertEquals(1, delegate.closeCount.get()); + } + } + + @Test + public void testCloseDoesNotFlushWithFlushOnCloseOff() { + List built = new ArrayList<>(); + try (SenderPool pool = newPool(1, 1, built)) { + Assert.assertFalse(pool.isFlushOnClose()); + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 1L).atNow(); + } + RecordingSender delegate = built.get(0); + Assert.assertEquals("close() must not ship rows", 0, delegate.flushCount.get()); + Assert.assertEquals("rows stay in the slot's buffer", 1, delegate.pendingRowCount()); + Assert.assertEquals("slot must be back on the free list", 1, pool.availableSize()); + } + } + + @Test + public void testCloseWrapperKeepsTheDelegatesRetryableFlag() { + List built = new ArrayList<>(); + try (SenderPool pool = newPool(1, 1, built)) { + Sender first = pool.borrow(); + built.get(0).latchedError = new LineSenderException("transient wire", true); + try { + first.close(); + Assert.fail("close() must rethrow the latched error"); + } catch (LineSenderException e) { + Assert.assertTrue("a retryable cause makes a retryable wrapper", e.isRetryable()); + } + Sender second = pool.borrow(); + built.get(1).latchedError = new LineSenderException("permanent", false); + try { + second.close(); + Assert.fail("close() must rethrow the latched error"); + } catch (LineSenderException e) { + Assert.assertFalse("a non-retryable cause makes a non-retryable wrapper", e.isRetryable()); + } + } + } + + @Test + public void testDirectPoolFlushesOnCloseByDefault() { + List built = new ArrayList<>(); + try (SenderPool pool = newPoolWithDefaults(1, 1, Long.MAX_VALUE, built)) { + Assert.assertTrue("a direct pool has no housekeeper to settle released rows", pool.isFlushOnClose()); + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 1L).atNow(); + } + RecordingSender delegate = built.get(0); + Assert.assertEquals(1, delegate.flushCount.get()); + Assert.assertEquals(0, delegate.pendingRowCount()); + Assert.assertEquals(1, pool.availableSize()); + } + } + + @Test + public void testFlushOnCloseRestoresShippingAtClose() { + List built = new ArrayList<>(); + try (SenderPool pool = newPool(1, 1, built)) { + pool.setFlushOnClose(true); + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 1L).atNow(); + } + RecordingSender delegate = built.get(0); + Assert.assertEquals(1, delegate.flushCount.get()); + Assert.assertEquals(0, delegate.pendingRowCount()); + Assert.assertEquals(1, pool.availableSize()); + } + } + + @Test + public void testFreeListIsLifo() { + List built = new ArrayList<>(); + try (SenderPool pool = newPool(2, 2, built)) { + Sender a = pool.borrow(); + Sender b = pool.borrow(); + Object slotB = slotOf(b); + a.close(); + b.close(); + try (Sender next = pool.borrow()) { + Assert.assertSame("most recently returned slot is borrowed first", slotB, slotOf(next)); + } + } + } + + @Test + public void testLatchedErrorAtCloseRetiresSlotAndThrows() { + List built = new ArrayList<>(); + try (SenderPool pool = newPool(1, 1, built)) { + Sender first = pool.borrow(); + RecordingSender delegate = built.get(0); + LineSenderException original = new LineSenderException("latched terminal"); + delegate.latchedError = original; + try { + first.close(); + Assert.fail("close() must rethrow the latched terminal"); + } catch (LineSenderException e) { + Assert.assertSame(original, e.getCause()); + Assert.assertTrue(e.getMessage().contains("latched terminal")); + } + Assert.assertEquals("dead slot must be retired", 0, pool.totalSize()); + Assert.assertEquals("retired delegate must be closed", 1, delegate.closeCount.get()); + try (Sender second = pool.borrow()) { + Assert.assertEquals("next borrow must build a fresh delegate", 2, built.size()); + Assert.assertNotSame(delegate, ((PooledSender) second).getDelegateForTesting()); + } + } + } + + @Test + public void testLeaseReturnedAfterPoolCloseClosesDelegate() { + List built = new ArrayList<>(); + SenderPool pool = newPool(1, 1, built); + Sender s = pool.borrow(); + s.table("t").longColumn("v", 1L).atNow(); + pool.close(); // bounded wait, lease outstanding + Assert.assertEquals("borrowed slot is left for the returning lease", 1, pool.totalSize()); + s.close(); + RecordingSender delegate = built.get(0); + Assert.assertEquals("returning lease must close its delegate", 1, delegate.closeCount.get()); + Assert.assertEquals("a lease closing into a shutting-down pool flushes so ship failures propagate", 1, delegate.flushCount.get()); + Assert.assertEquals(0, pool.totalSize()); + } + + @Test + public void testPartialRowAtCloseIsCancelledAndSlotKept() { + List built = new ArrayList<>(); + try (SenderPool pool = newPool(1, 1, built)) { + Sender first = pool.borrow(); + first.table("t").longColumn("v", 1L); // no at(): row in progress + try { + first.close(); + Assert.fail("close() with a row in progress must throw after releasing"); + } catch (LineSenderException e) { + Assert.assertTrue(e.getMessage(), e.getMessage().contains("row in progress")); + } + RecordingSender delegate = built.get(0); + Assert.assertEquals(1, delegate.cancelCount.get()); + Assert.assertEquals("slot stays in service", 1, pool.availableSize()); + try (Sender second = pool.borrow()) { + Assert.assertSame("same slot is reused", delegate, ((PooledSender) second).getDelegateForTesting()); + } + } + } + + @Test + public void testFlushDeferredThrowingAtCloseRetiresSlot() { + List built = new ArrayList<>(); + try (SenderPool pool = newPool(1, 1, built)) { + pool.setFlushOnClose(true); + pool.setCommitWindow(1000, 60_000L); + Sender first = pool.borrow(); + RecordingSender delegate = built.get(0); + LineSenderException original = new LineSenderException("deferred ship failed"); + delegate.deferredFlushError = original; + first.table("t").longColumn("v", 1L).atNow(); + try { + first.close(); + Assert.fail("close() must propagate the deferred ship failure"); + } catch (LineSenderException e) { + Assert.assertSame(original, e.getCause()); + } + Assert.assertEquals("slot must be retired, not stranded", 0, pool.totalSize()); + Assert.assertEquals(1, delegate.closeCount.get()); + } + } + + @Test + public void testLatchedServerErrorAtCloseKeepsItsType() { + List built = new ArrayList<>(); + try (SenderPool pool = newPool(1, 1, built)) { + Sender first = pool.borrow(); + RecordingSender delegate = built.get(0); + SenderError error = SenderError.dataLoss("rejected", "/tmp/q"); + LineSenderServerException original = new LineSenderServerException(error); + delegate.latchedError = original; + try { + first.close(); + Assert.fail("close() must propagate the latched error"); + } catch (LineSenderServerException e) { + Assert.assertNotSame(original, e); + Assert.assertSame(error, e.getServerError()); + Assert.assertSame(original, e.getCause()); + } + Assert.assertEquals(0, pool.totalSize()); + } + } + + @Test + public void testPoolCloseLogsRowsAFailedIdleCloseDrops() { + Logger poolLogger = (Logger) LoggerFactory.getLogger(SenderPool.class); + ListAppender appender = new ListAppender<>(); + appender.start(); + Level savedLevel = poolLogger.getLevel(); + poolLogger.setLevel(Level.ALL); + poolLogger.addAppender(appender); + try { + List built = new ArrayList<>(); + SenderPool pool = new SenderPool(CONFIG, 1, 1, 150L, Long.MAX_VALUE, Long.MAX_VALUE, idx -> { + RecordingSender s = new RecordingSender() { + @Override + public void close() { + super.close(); + throw new LineSenderException("cursor SF append failed"); + } + }; + built.add(s); + return s; + }); + pool.setFlushOnClose(false); + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 1L).atNow(); + s.table("t").longColumn("v", 2L).atNow(); + } + pool.close(); + Assert.assertEquals(1, built.get(0).closeCount.get()); + boolean logged = false; + for (ILoggingEvent e : appender.list) { + String msg = e.getFormattedMessage(); + if (e.getLevel() == Level.ERROR && msg.contains("2 unshipped row(s)") + && msg.contains("cursor SF append failed")) { + logged = true; + } + } + Assert.assertTrue("a failed idle close must log the rows it drops", logged); + } finally { + poolLogger.detachAppender(appender); + poolLogger.setLevel(savedLevel); + } + } + + @Test + public void testResetAfterAFlushInsideTheLeaseDropsEveryPendingRow() { + List built = new ArrayList<>(); + try (SenderPool pool = newPool(1, 1, built)) { + try (Sender a = pool.borrow()) { + a.table("t").longColumn("v", 1L).atNow(); + } + RecordingSender delegate = built.get(0); + try (Sender b = pool.borrow()) { + b.table("t").longColumn("v", 2L).atNow(); + b.flush(); + Assert.assertEquals("the flush shipped both rows", 1, delegate.flushCount.get()); + b.table("t").longColumn("v", 3L).atNow(); + b.reset(); + Assert.assertEquals("the flush emptied the savepoint, so reset drops every pending row", 0, delegate.pendingRowCount()); + } + } + } + + @Test + public void testResetDiscardsOnlyTheResettingLeasesRows() { + List built = new ArrayList<>(); + try (SenderPool pool = newPool(1, 1, built)) { + try (Sender a = pool.borrow()) { + a.table("t").longColumn("v", 1L).atNow(); + } + RecordingSender delegate = built.get(0); + Assert.assertEquals("close() shipped nothing", 1, delegate.pendingRowCount()); + try (Sender b = pool.borrow()) { + Assert.assertEquals("same slot", 1, built.size()); + b.table("t").longColumn("v", 2L).atNow(); + Assert.assertEquals(2, delegate.pendingRowCount()); + b.reset(); + Assert.assertEquals("A's row survives B's reset", 1, delegate.pendingRowCount()); + } + Assert.assertEquals(1, pool.settleIdle()); + Assert.assertEquals(1, delegate.flushCount.get()); + Assert.assertEquals(0, delegate.pendingRowCount()); + } + } + + @Test + public void testStaleCloseDoesNotTouchReborrowedSlot() { + List built = new ArrayList<>(); + try (SenderPool pool = newPool(1, 1, built)) { + Sender first = pool.borrow(); + first.close(); + Sender second = pool.borrow(); + second.table("t").longColumn("v", 1L); // second's row in progress + RecordingSender delegate = built.get(0); + delegate.latchedError = new LineSenderException("must not be probed by a stale close"); + first.close(); // stale: must be a no-op, must not probe + Assert.assertEquals("stale close must not retire the live slot", 1, pool.totalSize()); + delegate.latchedError = null; + Assert.assertEquals("stale close must not cancel the live borrower's row", 0, delegate.cancelCount.get()); + Assert.assertTrue(delegate.hasRowInProgress()); + Assert.assertEquals("stale close must not re-enqueue the slot", 0, pool.availableSize()); + second.atNow(); + second.close(); + Assert.assertEquals(1, pool.availableSize()); + } + } + + @Test + public void testMarkSavepointRunsOutsideThePoolLock() throws Exception { + List built = new ArrayList<>(); + try (SenderPool pool = newPool(2, 2, built)) { + Sender a = pool.borrow(); + Sender b = pool.borrow(); + a.close(); + b.close(); + CountDownLatch gate = new CountDownLatch(1); + for (RecordingSender rs : built) { + rs.markSavepointGate = gate; + } + AtomicReference first = new AtomicReference<>(); + AtomicReference second = new AtomicReference<>(); + Thread t1 = new Thread(() -> first.set(pool.borrow()), "first"); + Thread t2 = new Thread(() -> second.set(pool.borrow()), "second"); + try { + t1.start(); + awaitAvailable(pool, 1); + t2.start(); + awaitAvailable(pool, 0); + Assert.assertEquals("a second borrower takes the other slot while the first is still marking", 0, pool.availableSize()); + } finally { + gate.countDown(); + t1.join(2_000); + t2.join(2_000); + } + first.get().close(); + second.get().close(); + } + } + + @Test + public void testStaleResetDoesNotTouchReborrowedSlot() { + List built = new ArrayList<>(); + try (SenderPool pool = newPool(1, 1, built)) { + Sender first = pool.borrow(); + first.close(); + try (Sender second = pool.borrow()) { + second.table("t").longColumn("v", 1L).atNow(); + try { + first.reset(); + Assert.fail("a stale lease must not reset the slot"); + } catch (IllegalStateException expected) { + // "sender handle is closed (returned to the pool)" + } + Assert.assertEquals(1, built.get(0).pendingRowCount()); + } + } + } + + // Release-only close, the QuestDB facade's default. A directly built pool + // flushes on close unless told otherwise. + static SenderPool newPool(int min, int max, List built) { + SenderPool pool = newPoolWithDefaults(min, max, Long.MAX_VALUE, built); + pool.setFlushOnClose(false); + return pool; + } + + static SenderPool newPoolWithDefaults(int min, int max, long idleTimeoutMillis, List built) { + return new SenderPool(CONFIG, min, max, 150L, idleTimeoutMillis, Long.MAX_VALUE, idx -> { + RecordingSender s = new RecordingSender(); + built.add(s); + return s; + }); + } + + static Object slotOf(Sender pooledWrapper) { + try { + Field f = PooledSender.class.getDeclaredField("slot"); + f.setAccessible(true); + return f.get(pooledWrapper); + } catch (ReflectiveOperationException e) { + throw new AssertionError(e); + } + } + + private static void awaitAvailable(SenderPool pool, int target) throws InterruptedException { + long deadline = System.nanoTime() + 2_000_000_000L; + while (pool.availableSize() != target && System.nanoTime() < deadline) { + Thread.sleep(5); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/impl/SenderPoolCommitWindowTest.java b/core/src/test/java/io/questdb/client/test/impl/SenderPoolCommitWindowTest.java new file mode 100644 index 000000000..dba77ffce --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/impl/SenderPoolCommitWindowTest.java @@ -0,0 +1,267 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.impl; + +import io.questdb.client.Sender; +import io.questdb.client.impl.SenderPool; +import org.junit.Assert; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; + +/** + * The pool's commit window with {@code flush_on_close=on}: each close ships a + * deferred frame, the close that fills the window ships a committing frame + * instead, and the settle step commits an open group left idle. + */ +public class SenderPoolCommitWindowTest { + + @Test + public void testAgeClockRestartsAfterACommit() throws Exception { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(1, 1, built)) { + pool.setFlushOnClose(true); + pool.setCommitWindow(1_000, 30L); + borrowAppendClose(pool, 1L); // deferred: the window opens + Thread.sleep(60); + borrowAppendClose(pool, 2L); // older than 30 ms: commits + RecordingSender d = built.get(0); + Assert.assertEquals(1, d.flushCount.get()); + Assert.assertEquals(1, d.deferredFlushCount.get()); + borrowAppendClose(pool, 3L); // a fresh window: this close must defer + Assert.assertEquals("a clock that was not restarted at the commit would commit every close", 1, d.flushCount.get()); + Assert.assertEquals(2, d.deferredFlushCount.get()); + } + } + + @Test + public void testAgeWindowClosesAtNextRelease() throws Exception { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(1, 1, built)) { + pool.setFlushOnClose(true); + pool.setCommitWindow(1_000, 30L); + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 1L).atNow(); + } + RecordingSender d = built.get(0); + Assert.assertEquals(1, d.deferredFlushCount.get()); + Thread.sleep(60); + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 2L).atNow(); + } + Assert.assertEquals("group older than the interval: this close commits", 1, d.flushCount.get()); + Assert.assertEquals(1, d.deferredFlushCount.get()); + } + } + + @Test + public void testDrainResetsTheWindow() { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(1, 1, built)) { + pool.setFlushOnClose(true); + pool.setCommitWindow(3, 60_000L); + borrowAppendClose(pool, 1L); // deferred: the window is open with 1 row + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 2L).atNow(); + Assert.assertTrue(s.drain(0)); // app commits at 2 rows: the window restarts here + } + RecordingSender d = built.get(0); + Assert.assertEquals(1, d.deferredFlushCount.get()); + Assert.assertEquals(1, d.flushCount.get()); + // 3 rows in total: a counter that ignored the drain would commit here + borrowAppendClose(pool, 3L); + borrowAppendClose(pool, 4L); + Assert.assertEquals(3, d.deferredFlushCount.get()); + Assert.assertEquals(1, d.flushCount.get()); + // 3 rows since the drain: the window closes with a committing frame + borrowAppendClose(pool, 5L); + Assert.assertEquals(3, d.deferredFlushCount.get()); + Assert.assertEquals(2, d.flushCount.get()); + } + } + + @Test + public void testLeaseFlushAndGetSequenceResetsTheWindow() { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(1, 1, built)) { + pool.setFlushOnClose(true); + pool.setCommitWindow(3, 60_000L); + borrowAppendClose(pool, 1L); // deferred: the window is open with 1 row + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 2L).atNow(); + s.flushAndGetSequence(); // app commits at 2 rows: the window restarts here + } + RecordingSender d = built.get(0); + Assert.assertEquals(1, d.deferredFlushCount.get()); + Assert.assertEquals(1, d.flushCount.get()); + borrowAppendClose(pool, 3L); // 3 rows in total: a counter that ignored the flush would commit here + borrowAppendClose(pool, 4L); + Assert.assertEquals(3, d.deferredFlushCount.get()); + Assert.assertEquals(1, d.flushCount.get()); + borrowAppendClose(pool, 5L); // 3 rows since the flush: the window closes + Assert.assertEquals(3, d.deferredFlushCount.get()); + Assert.assertEquals(2, d.flushCount.get()); + } + } + + @Test + public void testLeaseFlushResetsTheWindow() { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(1, 1, built)) { + pool.setFlushOnClose(true); + pool.setCommitWindow(3, 60_000L); + borrowAppendClose(pool, 1L); // deferred: the window is open with 1 row + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 2L).atNow(); + s.flush(); // app commits at 2 rows: the window restarts here + } + RecordingSender d = built.get(0); + Assert.assertEquals(1, d.deferredFlushCount.get()); + Assert.assertEquals(1, d.flushCount.get()); + // 3 rows in total: a counter that ignored the flush would commit here + borrowAppendClose(pool, 3L); + borrowAppendClose(pool, 4L); + Assert.assertEquals(3, d.deferredFlushCount.get()); + Assert.assertEquals(1, d.flushCount.get()); + // 3 rows since the flush: the window closes with a committing frame + borrowAppendClose(pool, 5L); + Assert.assertEquals(3, d.deferredFlushCount.get()); + Assert.assertEquals(2, d.flushCount.get()); + } + } + + @Test + public void testReleaseWithoutRowsDoesNotOpenAWindow() throws Exception { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(1, 1, built)) { + pool.setFlushOnClose(true); + pool.setCommitWindow(1_000, 30L); + try (Sender s = pool.borrow()) { + Assert.assertNotNull(s); // no rows + } + Thread.sleep(60); + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 1L).atNow(); + } + RecordingSender d = built.get(0); + Assert.assertEquals("the empty release must not have started the clock", 1, d.deferredFlushCount.get()); + Assert.assertEquals(0, d.flushCount.get()); + } + } + + @Test + public void testRowsWindowClosesWithCommittingFlush() { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(1, 1, built)) { + pool.setFlushOnClose(true); + pool.setCommitWindow(3, 60_000L); + RecordingSender d = null; + for (int i = 0; i < 3; i++) { + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", i).atNow(); + } + d = built.get(0); + } + Assert.assertEquals("first two closes ship deferred", 2, d.deferredFlushCount.get()); + Assert.assertEquals("third close fills the window and commits", 1, d.flushCount.get()); + Assert.assertFalse(d.hasDeferredFrames()); + // the window restarts: a fourth close is deferred again + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 4L).atNow(); + } + Assert.assertEquals(3, d.deferredFlushCount.get()); + Assert.assertEquals(1, d.flushCount.get()); + } + } + + @Test + public void testSetCommitWindowRejectsNegativeOrMixedValues() { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(1, 1, built)) { + int[][] bad = {{-1, -1}, {0, -5}, {5, 0}, {0, 5}}; + for (int[] b : bad) { + try { + pool.setCommitWindow(b[0], b[1]); + Assert.fail("expected IllegalArgumentException for " + b[0] + ", " + b[1]); + } catch (IllegalArgumentException expected) { + Assert.assertFalse(pool.hasCommitWindow()); + } + } + pool.setCommitWindow(0, 0); + Assert.assertFalse(pool.hasCommitWindow()); + } + } + + @Test + public void testSettleCommitsIdleOpenGroup() { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(1, 1, built)) { + pool.setFlushOnClose(true); + pool.setCommitWindow(2, 60_000L); + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 1L).atNow(); + } + RecordingSender d = built.get(0); + Assert.assertTrue(d.hasDeferredFrames()); + Assert.assertEquals(0, d.pendingRowCount()); + Assert.assertEquals("an open group is unshipped work for the settle", 1, pool.settleIdle()); + Assert.assertEquals(1, d.flushCount.get()); + Assert.assertFalse(d.hasDeferredFrames()); + Assert.assertEquals("window restarted by the settle", 0, pool.settleIdle()); + // the settle marked the commit at 1 row, so the next close (2 rows + // appended in total, 1 in the window) defers; had the mark been + // missed, the window would count 2 rows and this close would commit + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 2L).atNow(); + } + Assert.assertEquals(2, d.deferredFlushCount.get()); + Assert.assertEquals(1, d.flushCount.get()); + } + } + + @Test + public void testWindowOffMeansEveryCloseCommits() { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(1, 1, built)) { + pool.setFlushOnClose(true); + Assert.assertFalse(pool.hasCommitWindow()); + for (int i = 0; i < 2; i++) { + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", i).atNow(); + } + } + RecordingSender d = built.get(0); + Assert.assertEquals(2, d.flushCount.get()); + Assert.assertEquals(0, d.deferredFlushCount.get()); + } + } + + private static void borrowAppendClose(SenderPool pool, long v) { + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", v).atNow(); + } + } +} diff --git a/core/src/test/java/io/questdb/client/test/impl/SenderPoolSettleTest.java b/core/src/test/java/io/questdb/client/test/impl/SenderPoolSettleTest.java new file mode 100644 index 000000000..24ee8eb90 --- /dev/null +++ b/core/src/test/java/io/questdb/client/test/impl/SenderPoolSettleTest.java @@ -0,0 +1,396 @@ +/*+***************************************************************************** + * ___ _ ____ ____ + * / _ \ _ _ ___ ___| |_| _ \| __ ) + * | | | | | | |/ _ \/ __| __| | | | _ \ + * | |_| | |_| | __/\__ \ |_| |_| | |_) | + * \__\_\\__,_|\___||___/\__|____/|____/ + * + * Copyright (c) 2014-2019 Appsicle + * Copyright (c) 2019-2026 QuestDB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + ******************************************************************************/ + +package io.questdb.client.test.impl; + +import io.questdb.client.Sender; +import io.questdb.client.cutlass.line.LineSenderException; +import io.questdb.client.cutlass.qwp.client.BatchTooLargeForCapException; +import io.questdb.client.impl.PooledSender; +import io.questdb.client.impl.SenderPool; +import org.junit.Assert; +import org.junit.Test; + +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; + +/** + * {@link SenderPool#settleIdle()}: ships rows left in idle slots through an + * internal lease, retires dead delegates, and never shares a slot with a + * concurrent borrower. + */ +public class SenderPoolSettleTest { + + @Test + public void testReapSkipsSlotUntilSettleShipsItsRows() throws Exception { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPoolWithDefaults(0, 1, 1L, built)) { + pool.setFlushOnClose(false); + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 1L).atNow(); + } + RecordingSender delegate = built.get(0); + Thread.sleep(20); + pool.reapIdle(); + Assert.assertEquals("an idle-expired slot holding rows is not reaped", 1, pool.totalSize()); + Assert.assertEquals(0, delegate.closeCount.get()); + Assert.assertEquals(1, delegate.pendingRowCount()); + + delegate.deferredFramesOpen = true; + delegate.pendingRows = 0; + pool.reapIdle(); + Assert.assertEquals("an open commit group is unshipped work too", 1, pool.totalSize()); + + delegate.deferredFramesOpen = false; + delegate.pendingRows = 1; + Assert.assertEquals(1, pool.settleIdle()); + pool.reapIdle(); + Assert.assertEquals("the reap closes the slot once the settle shipped it", 0, pool.totalSize()); + Assert.assertEquals(1, delegate.closeCount.get()); + } + } + + @Test + public void testSettleRetiresSlotHoldingAnOverCapBatch() { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(1, 1, built)) { + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 1L).atNow(); + } + RecordingSender delegate = built.get(0); + delegate.flushError = new BatchTooLargeForCapException("single table batch too large for server batch cap"); + Assert.assertEquals(0, pool.settleIdle()); + Assert.assertEquals("a batch the cap rejects never ships: the slot is retired", 0, pool.totalSize()); + Assert.assertEquals(1, delegate.closeCount.get()); + try (Sender next = pool.borrow()) { + Assert.assertEquals("the next borrow builds a fresh delegate", 2, built.size()); + Assert.assertNotSame(delegate, ((PooledSender) next).getDelegateForTesting()); + } + } + } + + @Test + public void testSettleRetiresSlotWhenFlushThrowsAnError() { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(1, 1, built)) { + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 1L).atNow(); + } + RecordingSender delegate = built.get(0); + delegate.flushFatal = new AssertionError("buffer invariant broken"); + Assert.assertEquals(0, pool.settleIdle()); + Assert.assertEquals("an Error is not transient: the slot is retired", 0, pool.totalSize()); + Assert.assertEquals(1, delegate.closeCount.get()); + } + } + + @Test + public void testSettleShipsRowsLeftInIdleSlot() throws Exception { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(1, 1, built)) { + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 1L).atNow(); + } + RecordingSender delegate = built.get(0); + long idleSince = idleSinceOf(pool); + Assert.assertEquals(0, delegate.flushCount.get()); + Thread.sleep(20); + Assert.assertEquals(1, pool.settleIdle()); + Assert.assertEquals(1, delegate.flushCount.get()); + Assert.assertEquals(0, delegate.pendingRowCount()); + Assert.assertEquals("slot must be back on the free list", 1, pool.availableSize()); + Assert.assertEquals("settling must not reset the idle clock", idleSince, idleSinceOf(pool)); + Assert.assertEquals("nothing left to settle", 0, pool.settleIdle()); + } + } + + @Test + public void testSettleSkipsSlotsWithoutRows() { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(2, 2, built)) { + Assert.assertEquals(0, pool.settleIdle()); + Assert.assertEquals(0, built.get(0).flushCount.get()); + Assert.assertEquals(0, built.get(1).flushCount.get()); + } + } + + @Test + public void testSettleRetiresDeadDelegate() { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(1, 1, built)) { + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 1L).atNow(); + } + RecordingSender delegate = built.get(0); + delegate.flushError = new LineSenderException("wire is dead"); + delegate.latchedError = new LineSenderException("latched terminal"); + Assert.assertEquals(0, pool.settleIdle()); + Assert.assertEquals("dead delegate must be retired", 0, pool.totalSize()); + Assert.assertEquals(1, delegate.closeCount.get()); + } + } + + @Test + public void testSettleKeepsSlotAfterTransientFlushFailure() { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(1, 1, built)) { + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 1L).atNow(); + } + RecordingSender delegate = built.get(0); + delegate.flushError = new LineSenderException("ring full, try later"); + Assert.assertEquals(0, pool.settleIdle()); + Assert.assertEquals("healthy delegate stays in the pool", 1, pool.totalSize()); + Assert.assertEquals(1, pool.availableSize()); + Assert.assertEquals("rows stay for the next settle", 1, delegate.pendingRowCount()); + delegate.flushError = null; + Assert.assertEquals(1, pool.settleIdle()); + Assert.assertEquals(0, delegate.pendingRowCount()); + } + } + + @Test + public void testBorrowerNeverSharesSlotWithSettle() throws Exception { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(1, 1, built)) { + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 1L).atNow(); + } + RecordingSender delegate = built.get(0); + CountDownLatch gate = new CountDownLatch(1); + delegate.flushGate = gate; + Thread settler = new Thread(pool::settleIdle, "settler"); + settler.start(); + try { + // wait until the settle holds the slot (free list empty) + long deadline = System.currentTimeMillis() + 2_000; + while (pool.availableSize() != 0 && System.currentTimeMillis() < deadline) { + Thread.sleep(5); + } + Assert.assertEquals("settle must hold the slot as an internal lease", 0, pool.availableSize()); + // a borrower arriving now must wait (max=1), not get the slot being flushed + try { + pool.borrow(); + Assert.fail("borrow must time out while the only slot is being settled"); + } catch (LineSenderException expected) { + // acquire timeout (150 ms in newPool) + } + } finally { + gate.countDown(); + } + settler.join(2_000); + Assert.assertFalse(settler.isAlive()); + Assert.assertEquals(1, delegate.flushCount.get()); + try (Sender s = pool.borrow()) { + Assert.assertNotNull(s); + } + } + } + + @Test + public void testPoolCloseDuringSettleRetiresSlotOnSettleThread() throws Exception { + List built = new ArrayList<>(); + SenderPool pool = new SenderPool("http::addr=127.0.0.1:1;protocol_version=2;auto_flush=off;", 1, 1, 2_000L, Long.MAX_VALUE, Long.MAX_VALUE, + slotIndex -> { + RecordingSender rs = new RecordingSender(); + built.add(rs); + return rs; + }); + pool.setFlushOnClose(false); + try { + try (Sender s = pool.borrow()) { + s.table("t").longColumn("v", 1L).atNow(); + } + RecordingSender delegate = built.get(0); + CountDownLatch gate = new CountDownLatch(1); + delegate.flushGate = gate; + AtomicReference settleFailure = new AtomicReference<>(); + AtomicInteger settleResult = new AtomicInteger(-1); + Thread settler = new Thread(() -> { + try { + settleResult.set(pool.settleIdle()); + } catch (Throwable t) { + settleFailure.set(t); + } + }, "settler"); + Thread closer = new Thread(pool::close, "closer"); + settler.start(); + try { + long deadline = System.currentTimeMillis() + 2_000; + while (pool.availableSize() != 0 && System.currentTimeMillis() < deadline) { + Thread.sleep(5); + } + Assert.assertEquals(0, pool.availableSize()); + closer.start(); + // close() parks in its timed wait for the internal lease + deadline = System.currentTimeMillis() + 2_000; + while (closer.getState() != Thread.State.TIMED_WAITING && System.currentTimeMillis() < deadline) { + Thread.sleep(5); + } + Assert.assertEquals("close() must be waiting for the internal lease", + Thread.State.TIMED_WAITING, closer.getState()); + Assert.assertTrue("close() must be waiting for the internal lease", closer.isAlive()); + } finally { + gate.countDown(); + } + settler.join(2_000); + closer.join(2_000); + Assert.assertFalse(settler.isAlive()); + Assert.assertFalse(closer.isAlive()); + Assert.assertNull(settleFailure.get()); + Assert.assertEquals(1, settleResult.get()); + Assert.assertEquals("the settle thread must retire the slot once the pool is closing", 0, pool.totalSize()); + Assert.assertEquals(1, delegate.closeCount.get()); + } finally { + pool.close(); + } + } + + @Test + public void testSettleKeepsOtherSlotsBorrowableWhileOneFlushBlocks() throws Exception { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(2, 2, built)) { + Sender a = pool.borrow(); + Sender b = pool.borrow(); + a.table("t").longColumn("v", 1L).atNow(); + b.table("t").longColumn("v", 2L).atNow(); + a.close(); + b.close(); + CountDownLatch gate = new CountDownLatch(1); + for (RecordingSender rs : built) { + rs.flushGate = gate; + } + Thread settler = new Thread(pool::settleIdle, "settler"); + settler.start(); + try { + long deadline = System.currentTimeMillis() + 2_000; + while (pool.availableSize() == 2 && System.currentTimeMillis() < deadline) { + Thread.sleep(5); + } + Assert.assertEquals("settle holds one slot at a time", 1, pool.availableSize()); + try (Sender borrowed = pool.borrow()) { + RecordingSender other = (RecordingSender) ((PooledSender) borrowed).getDelegateForTesting(); + gate.countDown(); + settler.join(2_000); + Assert.assertFalse(settler.isAlive()); + Assert.assertEquals("settle never flushes a slot a borrower took mid-pass", 0, other.flushCount.get()); + } + } finally { + gate.countDown(); + } + } + } + + @Test + public void testSettleStopsTakingSlotsOnceThePoolCloses() throws Exception { + List built = new ArrayList<>(); + SenderPool pool = SenderPoolCloseSemanticsTest.newPool(2, 2, built); + try { + Sender a = pool.borrow(); + Sender b = pool.borrow(); + a.table("t").longColumn("v", 1L).atNow(); + b.table("t").longColumn("v", 2L).atNow(); + a.close(); + b.close(); + CountDownLatch gate = new CountDownLatch(1); + for (RecordingSender rs : built) { + rs.flushGate = gate; + } + AtomicReference settleFailure = new AtomicReference<>(); + Thread settler = new Thread(() -> { + try { + pool.settleIdle(); + } catch (Throwable t) { + settleFailure.set(t); + } + }, "settler"); + Thread closer = new Thread(pool::close, "closer"); + settler.start(); + try { + long deadline = System.currentTimeMillis() + 2_000; + while (pool.availableSize() == 2 && System.currentTimeMillis() < deadline) { + Thread.sleep(5); + } + Assert.assertTrue(pool.availableSize() < 2); + closer.start(); + deadline = System.currentTimeMillis() + 2_000; + while (closer.getState() != Thread.State.TIMED_WAITING && System.currentTimeMillis() < deadline) { + Thread.sleep(5); + } + Assert.assertEquals("close() must be waiting for the internal lease", + Thread.State.TIMED_WAITING, closer.getState()); + } finally { + gate.countDown(); + } + settler.join(2_000); + closer.join(2_000); + Assert.assertFalse(settler.isAlive()); + Assert.assertFalse(closer.isAlive()); + Assert.assertNull(settleFailure.get()); + Assert.assertEquals("a closing pool stops the settle after the slot in flight", 1, + built.get(0).flushCount.get() + built.get(1).flushCount.get()); + } finally { + pool.close(); + } + } + + @Test + public void testSettleHandlesSeveralCandidatesIndependently() { + List built = new ArrayList<>(); + try (SenderPool pool = SenderPoolCloseSemanticsTest.newPool(2, 2, built)) { + Sender a = pool.borrow(); + Sender b = pool.borrow(); + a.table("t").longColumn("v", 1L).atNow(); + b.table("t").longColumn("v", 2L).atNow(); + a.close(); + b.close(); + RecordingSender dead = built.get(0); + RecordingSender healthy = built.get(1); + dead.flushError = new LineSenderException("wire is dead"); + dead.latchedError = new LineSenderException("latched terminal"); + Assert.assertEquals(1, pool.settleIdle()); + Assert.assertEquals(1, healthy.flushCount.get()); + Assert.assertEquals(1, pool.availableSize()); + Assert.assertEquals(1, pool.totalSize()); + Assert.assertEquals(1, dead.closeCount.get()); + Assert.assertEquals(0, healthy.closeCount.get()); + } + } + + @SuppressWarnings("unchecked") + private static long idleSinceOf(SenderPool pool) throws Exception { + Field availableF = SenderPool.class.getDeclaredField("available"); + availableF.setAccessible(true); + Object slot = ((ArrayDeque) availableF.get(pool)).peekLast(); + Method m = slot.getClass().getDeclaredMethod("idleSinceMillis"); + m.setAccessible(true); + return (long) m.invoke(slot); + } +} diff --git a/examples/src/main/java/com/example/QuestDBExample.java b/examples/src/main/java/com/example/QuestDBExample.java index cc39e3e23..0f95f423e 100644 --- a/examples/src/main/java/com/example/QuestDBExample.java +++ b/examples/src/main/java/com/example/QuestDBExample.java @@ -33,9 +33,10 @@ public static void main(String[] args) throws InterruptedException { try (QuestDB db = QuestDB.connect("ws::addr=localhost:9000;"); Query q = db.borrowQuery()) { - // Ingest: borrow a Sender, write rows, close() flushes it and - // returns it to the pool. The WebSocket is NOT torn down here -- - // a real disconnect only happens at db.close(). + // Ingest: borrow a Sender, write rows, close() returns it to the pool. + // Close releases, flush ships, drain waits: the rows go out on the next + // auto-flush threshold or the pool's settle. The WebSocket is NOT torn + // down here -- a real disconnect only happens at db.close(). try (Sender sender = db.borrowSender()) { sender.table("trades") .symbol("symbol", "ETH-USD") diff --git a/examples/src/main/java/com/example/sender/WsConcurrentIngestExample.java b/examples/src/main/java/com/example/sender/WsConcurrentIngestExample.java index 2abcb6c87..baaa3ff9c 100644 --- a/examples/src/main/java/com/example/sender/WsConcurrentIngestExample.java +++ b/examples/src/main/java/com/example/sender/WsConcurrentIngestExample.java @@ -13,9 +13,9 @@ * thread its own. *

* Each producer thread takes its own sender with {@link QuestDB#borrowSender()} - * and reuses it for the batch (see {@link WsExample}); close() flushes and - * returns it to the pool. We size the sender pool to the producer count so no - * thread ever waits on the acquire timeout. + * and reuses it for the batch (see {@link WsExample}); flush() ships the + * batch and close() returns the sender to the pool. We size the sender pool to + * the producer count so no thread ever waits on the acquire timeout. */ public class WsConcurrentIngestExample { @@ -45,8 +45,8 @@ public static void main(String[] args) throws InterruptedException { } private static void ingestBatch(QuestDB db, int producerId) { - // Each thread borrows its own sender and reuses it for the batch; - // close() (try-with-resources) flushes and returns it to the pool. + // Each thread borrows its own sender and reuses it for the batch; flush() + // ships it and close() (try-with-resources) returns it to the pool. try (Sender sender = db.borrowSender()) { String symbol = "SYM-" + producerId; for (int i = 0; i < ROWS_PER_PRODUCER; i++) { diff --git a/examples/src/main/java/com/example/sender/WsExample.java b/examples/src/main/java/com/example/sender/WsExample.java index 9ef0604dc..60bb745d5 100644 --- a/examples/src/main/java/com/example/sender/WsExample.java +++ b/examples/src/main/java/com/example/sender/WsExample.java @@ -9,8 +9,10 @@ * The pooled {@link Sender} is the recommended way to ingest: one * {@code QuestDB} handle, created once and shared across threads, owns a pool * of senders that stay connected between uses. Take one with - * {@link QuestDB#borrowSender()} -- a lease per use: {@code close()} flushes - * pending rows and returns the sender to the pool (it does NOT disconnect). + * {@link QuestDB#borrowSender()} -- a lease per use: {@code close()} returns + * the sender to the pool (it does NOT disconnect or flush; rows ship on the + * next auto-flush threshold or the pool's settle, or on an explicit + * {@code flush()}). * Borrow one per producer thread, or once per batch, and close it when done. *

* A {@code Sender} is not thread-safe, so no two threads may share one -- the @@ -23,8 +25,8 @@ public class WsExample { public static void main(String[] args) { try (QuestDB db = QuestDB.connect("ws::addr=localhost:9000;")) { - // Borrowed: lease per use. try-with-resources close() flushes and - // returns the sender to the pool for reuse. + // Borrowed: lease per use. try-with-resources close() returns the + // sender to the pool for reuse; flush() below ships the batch. try (Sender sender = db.borrowSender()) { sender.table("trades") .symbol("symbol", "ETH-USD") @@ -35,8 +37,8 @@ public static void main(String[] args) { } // Borrow one sender and reuse it for the whole batch. close() - // (try-with-resources) flushes pending rows and returns the sender - // to the pool -- the WebSocket is NOT torn down here. + // (try-with-resources) returns the sender to the pool -- the + // WebSocket is NOT torn down here; flush() below ships the batch. try (Sender sender = db.borrowSender()) { for (int i = 0; i < 1_000; i++) { sender.table("trades") @@ -46,10 +48,9 @@ public static void main(String[] args) { .doubleColumn("amount", 0.001) .atNow(); } - // Hand the buffered rows to the send engine. Auto-flush would - // also do this on the row/time thresholds; an explicit flush() - // makes the batch boundary deterministic. close() flushes too, - // so this call is optional here. + // Hand the buffered rows to the send engine. Auto-flush would also do + // this on the row/byte/time thresholds, and the pool settles an idle + // slot; an explicit flush() makes the batch boundary deterministic. sender.flush(); }