fix(cli): read stdin lazily so open pipes don't hang the CLI by l2ysho · Pull Request #1330 · apify/apify-cli

l2ysho · 2026-08-14T10:55:39Z

Note

TL;DR — The CLI read stdin at startup for every command, so a pipe that stays open hung it forever. Now it only reads stdin when a command wants it, and a read nobody asked for gives up when the pipe goes quiet and says so on stderr.

Closes #1206.

The startup read

_shared.ts read stdin at module scope (export const cachedStdinInput = await readStdin()), so every command blocked on stdin whether or not it uses it. #1294 fixed the socket case (a spawned child's stdio: 'pipe'), but a named pipe has no wait deadline — the read never ended and the command never even ran, so 1.8.0 still hangs with no output at all:

sleep 60 | apify --version   # 1.8.0: hangs, no output

Read stdin only where it's asked for: a - arg/flag, apify run without --input, push-data/push-items without an item arg. readStdin() memoizes, since stdin can only be drained once.

Confirmations keyed off hasData, which the eager read flipped to false after draining. Without that read it stays true, so apify <confirm-cmd> < /dev/null would try to prompt on a non-TTY — gated on isTTY instead.

The implicit read

That left one path still hanging. apify run without --input falls back to stdin, and waited there for an end a named pipe never reaches. actors call and actors start share it. Two ways to hang:

apify run < /some/fifo                          # nothing ever sent
(printf '{"a":1}'; sleep 30) | apify run        # sent, then held open

A read the user never asked for now stops at the first quiet gap: each chunk restarts a 2s clock, and when stdin goes silent the CLI uses what arrived. Giving up is no longer silent — a read that comes back with nothing prints Nothing arrived on standard input, so it was ignored. Use --input-file=- to wait for a slow writer. on stderr. A spawned child's empty socket keeps the platform's existing short deadline and stays quiet, so this adds no noise to subprocess calls.

Trade-off: on run, call and start, a writer slower than ~2.5s to its first byte is ignored, with that warning. On Windows the first-byte budget is the platform's existing 100 ms pipe deadline, not 2s. Truncation mid-stream stays loud, since the JSON fails to parse. 2s is a judgement call, not a derived number.

Commands where piping is the point

actor push-data and datasets push-items document piping as their main usage (cat ./items.json | actor push-data), so an item-less call waits for the writer to close, the way cat does. An idle cutoff there dropped the data of any producer slower than 2s. An explicit - arg or flag is unchanged and also waits.

Verification

Measured against a build of master and a build of this branch. Hangs were killed at a 10s cap.

stdin	command	master	this branch
named pipe, no data	`--version`	hang, no output	407 ms
named pipe, data then held open	`--version`	hang, no output	396 ms
spawn `stdio: 'pipe'` (#1294's case)	`--version`	483 ms	413 ms
named pipe, silent	`run` in a project	hang	2628 ms, no input, warns
named pipe, sends then holds open	`run` in a project	hang	2627 ms, input used

push-data matches master in every case, including the pre-existing wait on a pipe that never closes:

stdin	master	this branch
producer sleeps 3s, then closes	3040 ms, item pushed	3039 ms, item pushed
named pipe, no data, held open	waits	waits
closed stdin	`No item was provided.`	`No item was provided.`

Four new e2e cases, five in the file, all skipped on Windows except the first (no mkfifo). Each fails on the pre-fix build and passes after:

FIFO on stdin, empty dir — the startup read
FIFO on stdin, real project, silent — the implicit read, and the warning
FIFO on stdin, real project, sends then holds open — asserts the Actor received the input
slow producer into push-data — asserts the item reaches the dataset

pnpm run test:local green (851 passed). pnpm run test:e2e:local green (48 passed). No dependency or install-size change.

Notes

Removing the eager drain changes actor set-value and key-value-stores set-value, which hand the live process.stdin to the API client. Piped input used to be drained before they saw it and they stored 0 bytes; now it reaches them. Neither file is in this diff.
fix: actor set-value KEY with no value doesn't delete the record #1336 is open and imports cachedStdinInput, which this PR deletes. The two need to be sequenced, and fix(cli): read stdin lazily so open pipes don't hang the CLI #1330 already covers the set-value half of what fix: actor set-value KEY with no value doesn't delete the record #1336 fixes.

🤖 Generated with Claude Code

Every command paid for stdin: `_shared.ts` did `await readStdin()` at module scope, so `apify --version` blocked on a stream it never reads. With a named pipe (no wait deadline, unlike the socket a spawned child gets) the read never ended and the command never even ran. Read stdin only when a command asks for it — a `-` arg/flag, `apify run` without `--input`, `push-data`/`push-items` without an item. Result is memoized, since stdin can only be drained once. Confirmations previously keyed off `hasData`, which the eager read flipped to false once stdin was drained. Without that read the flag stays true, so `apify <confirm-cmd> < /dev/null` would try to prompt on a non-TTY; gate on `isTTY` instead. Commands that genuinely consume stdin still wait for the writer to close, same as `cat`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`apify run` without `--input` falls back to stdin, and waited there for an end that a named pipe never reaches. A pipe the process merely inherited hung the command forever, and so did a writer that sent data and kept the pipe open. `actors call` and `actors start` share the path. Reads that nobody asked for now stop at the first quiet gap: each chunk restarts a 2s clock, and when stdin goes silent the CLI uses what arrived. An explicit `-` is unchanged and still waits for the writer to close. The cost is a writer slower than ~2.5s to its first byte, whose input is now ignored. Truncation mid-stream stays loud, as the JSON fails to parse. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The implicit read replaced `waitDelay` with its own 2s deadline, so a spawned child's socket stdin lost the 50ms first-byte guard. Measured on `apify run` with a silent socket: 703ms before, 2637ms after. Keep `waitDelay` as the first-byte deadline where stdin has one, and use the 2s constant for the idle gap and for stdin that has no deadline at all. `push-data` and `push-items` fall back to stdin the same way and still hung on a silent named pipe. They now use the same deadline: 2.4s to `No item was provided.` instead of waiting forever. A writer that stalls mid-payload leaves JSON that ends nowhere, and the parse error blamed the data. `getInputOverride` now says stdin went quiet and points at `--input-file=-`. `peekStdin()` fed the bug report footer from a read that had not happened yet, so it always printed `No`. Replaced with `describeStdinRead()`, which distinguishes `Not read`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…57c' into claude/apify-cli-1206-verify-37b57c

…06-verify-37b57c

…06-verify-37b57c # Conflicts: # src/lib/command-framework/apify-command.ts

> [!NOTE] > **TL;DR** > `apify init` never terminated when stdin was not a TTY. Fixed, along with three other defects on the same path. ## What was broken `apify init` with no name and a non-TTY stdin never exited. Not a blocked read — a retry loop writing ~28 MB/s to stderr. Measured 152 MB in 6 s, which also makes the process hard to kill. `stdinCheckWrapper` throws whenever stdin cannot be read. The loop caught that, printed it, and retried. The error is permanent, so the retry could never succeed. ## What changed | | Defect | Before | After | |---|---|---|---| | D1 | `--yes` did not skip the name prompt, though `interactiveNote` and the `apify init --yes` example both promise it does | infinite loop | exit 0, uses the directory name | | D2 | The retry loop treated a permanent failure as retryable | infinite loop | one error, exit 1 | | D3 | The directory confirmation turned "cannot ask" into "no" | **exit 0**, nothing done | exit 1 with a message | | D4 | The positional name was never validated | `init 'Bad Name!!'` exited 0 and wrote that name, failing later on push | exit 1 | D2 is fixed by removing the loop, not capping it. Validation moved inside inquirer's own `validate`, so a typo still re-asks interactively, while a prompt that cannot be answered fails once with a message naming the escape hatch: ``` Error: Actor name is required. Run 'apify init <name>', or pass --yes to use the current directory name. ``` Two supporting changes: the directory name is now sanitized for every project type rather than only Python (`my_project` → `my-project`), and the prompt validator is shared with `apify create`, which had its own copy. ## Behavior changes worth a reviewer's attention - A non-Node/Python directory without a TTY goes from exit 0 to exit 1. The old exit 0 did nothing useful, but a CI script treating 0 as success will start failing. - An invalid explicit name is now rejected up front. That includes two cases where the name was previously discarded anyway: an already-initialized directory, and a Scrapy project. Rejecting a name the user typed beats silently dropping it. - The default Actor name is sanitized for all project types, not just Python. ## Tests Three e2e tests through a new `runCliBounded` helper, plus four local tests. Both layers assert on the loop itself, not on a timeout. A hang and a loop are indistinguishable by deadline and cost the full deadline either way. The helper caps combined output at 256 KB and kills on breach, so a regression fails in under a second. execa's own `maxBuffer` was the obvious alternative and was measured instead of assumed: 7–20 s to trip, against 25–500 ms for the counter. In one run it blew past its own 15 s timeout and still reported `timedOut: false`. ## Not in scope - The held-open-pipe variant still hangs. That happens in `readStdin()` at entrypoint load, before any command code runs — [#1206](#1206), fixed by [#1330](#1330). Verified: with the eager read stubbed out, that case exits 0 in 1.3 s with the default name. - [#1451](#1451) (global `--non-interactive` flag) is adjacent but different. It covers pseudo-terminals, where a TTY exists; this is the plain non-TTY path. - [#1354](#1354) — the non-interactive error naming `--confirm`/`--no-confirm`, which do not exist. `init` now has its own messages; other commands still print the misleading default. - `validateActorName` says "maximum of 30 characters" while `ACTOR_NAME.MAX_LENGTH` is 63. Pre-existing; the new `--yes` failure path surfaces it more often. ## Checks `lint`, `format`, `build` clean. `test:local` 660 passed. Local e2e 44 passed. No dependency changes, so no install-size impact. `update-docs` produces no diff — flags, args, description and examples are untouched. CI's one red check is `actors info --input`, inherited from master. The same assertion fails on master's own E2E runs daily back to 5 October, and [#1473](#1473) fixes it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

apify-factory

Good work, one thing to fix before merge.

Outside the diff:

src/commands/actor/push-data.ts: Important: piping into push-data (and datasets push-items, same change) is the documented main way to use these commands (cat ./items.json | actor push-data), not a fallback nobody asked for, so the 2s idle cutoff now breaks pipelines that used to work. If the producer takes more than 2s to send its first byte, as with python scrape.py | actor push-data or curl slow-api | apify datasets push-items ds, the command fails with 'No item was provided.' and drops the data, and if it pauses for 2s mid-stream the JSON gets cut off with no hint about why (only resolve-input.ts explains a cut-short read).

🏭 Factory review — ℹ️ not auto-approved

This change was reviewed but cannot be auto-approved: a gate failed, so the review cannot end in an approval.

check	result
Static gates	❌ 14/16 — `max-lines`, `no-risky-content`
Review findings	❌ not clean
Reviewer escalation	❌ Adds a 2s stdin idle timeout that changes how stdin-driven commands (run, actors call/start, push-data, push-items) behave in production, and the PR itself calls the value a judgement call.
Juror 1 — `claude-sonnet-5-5`, adversarial	⏭️ skipped — a gate failed

Why it was not auto-approved

Failed gates:

max-lines: 291 changed lines (+244/-47), limit 200
no-risky-content: test/e2e/commands/stdin-held-open.test.ts: environment variable access (env-access)

The comments above are the review itself; this section only explains the approval decision. Push a change and request @apify-factory again to re-run it, or ask a human to review.

_{Reviewed 98d35babb · workflow run}

Piping is the documented main usage of `actor push-data` and `datasets push-items`, so an item-less call waits for EOF instead of giving up after 2s of quiet. A producer slower than that was dropping its data silently. An implicit read that gives up on a silent pipe now says so on stderr, so a slow writer's input does not vanish without a word. A spawned child's empty socket keeps its short deadline and stays quiet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

apify-factory

Looks good, nothing blocking.

🏭 Factory review — ℹ️ not auto-approved

This change was reviewed but cannot be auto-approved: a gate failed, so the review cannot end in an approval.

check	result
Static gates	❌ 14/16 — `max-lines`, `no-risky-content`
Review findings	✅ none
Reviewer escalation	❌ Changes CLI runtime behavior beyond the diff: a new 2s idle timeout on implicit stdin for run/call/start, confirmation prompts now refused on any non-TTY stdin, and set-value commands (not in the diff) now get piped stdin they previously received empty.
Juror 1 — `claude-sonnet-5-5`, adversarial	⏭️ skipped — a gate failed

Why it was not auto-approved

Failed gates:

max-lines: 356 changed lines (+283/-73), limit 200
no-risky-content: test/e2e/commands/stdin-held-open.test.ts: environment variable access (env-access)

The comments above are the review itself; this section only explains the approval decision. Push a change and request @apify-factory again to re-run it, or ask a human to review.

_{Reviewed 81860946a · workflow run}

Resolved: a newer review found nothing blocking.

DaveHanns · 2026-10-20T09:14:08Z

+				// Standard input is only waited on for as long as it keeps talking, so a writer that stalls
+				// mid-payload leaves behind JSON that ends nowhere. Say so, or the message blames the data.
+				const cutShortHint = stdinWasCutShort()
+					? '\nStandard input went quiet before it closed, so only part of it was read. Use `--input-file=-` to wait for all of it.'


Note: in this branch readCutShort only says that bytes arrived and EOF never came; it cannot tell a stalled writer from a complete payload whose writer kept the pipe open. (printf 'not json'; sleep 3) | apify run prints "only part of it was read" for data that was complete. Hedge it:

Suggested change

? '\nStandard input went quiet before it closed, so only part of it was read. Use `--input-file=-` to wait for all of it.'

? '\nStandard input was still open when the CLI stopped waiting for it. If the input is incomplete, use `--input-file=-` to wait for all of it.'

DaveHanns · 2026-10-20T09:14:08Z

+ * Reads stdin, at most once per process. Call it only when the command actually wants stdin data.
+ * The first call decides the options; later ones reuse its result.
+ */
+export async function readStdin(options: ReadStdinOptions = {}) {


Note: nothing unit-tests this module; the e2e cases need a built dist and all but the first skip on Windows. Still untested: memoization returning the first call's result, the cut-short hint after a partial payload (the talking-pipe case only checks the input was used), and the three states of describeStdinRead(). A PassThrough mocked into useStdin plus vi.resetModules() between cases would cover all of it; test/local/lib/resolve-input.test.ts already has a getInputOverride describe that could host the stdin branch.

DaveHanns · 2026-10-20T09:14:08Z

+ * to fetch or compute its next bytes, short enough that a pipe with nothing behind it does not look
+ * like a hang.
+ */
+const IMPLICIT_STDIN_IDLE_TIMEOUT_MILLIS = 2_000;


Suggestion: this one number does two jobs. Between chunks 2 s is generous, since pipe writes of a JSON blob arrive within milliseconds. For the first byte it is tight (curl to a cold API, jq over a big file); the warning in resolve-input.ts now says so, but the input is still ignored. Consider splitting it into a first-byte wait (longer, 5 s or so) and an idle wait (1 s or so), plus an env var override as an escape hatch. Either way the cutoff and the implicit fallback deserve a sentence in the --input-file help, since neither is documented today and #704 deliberately made the CLI wait for slow writers.

DaveHanns · 2026-10-27T15:34:33Z

+
+			// A socket's short first-byte guard fires for every spawned CLI, so only a read that
+			// outwaited the idle timeout counts as given up on.
+			readCutShort = bufferChunks.length > 0 || firstByteTimeout === IMPLICIT_STDIN_IDLE_TIMEOUT_MILLIS;


Note: the module logs nothing. A cliDebugPrint('readStdin', ...) on arm, abort, and give-up (with implicit, bytes read, readCutShort) would make the next stdin bug report debuggable; today describeStdinRead() in the crash footer is the only trace.

DaveHanns · 2026-10-27T15:34:35Z

+		const fifo = path.join(cwd, 'stdin.fifo');
+		await execa('mkfifo', [fifo]);
+
+		// `r+` keeps a writer attached, so the pipe never reaches EOF even after the write below.


Nit: the r+ trick is already explained at lines 43-44; the first FIFO test opens the pipe inline instead of going through runWithFifoStdin, which is why both the setup and the explanation are duplicated.

DaveHanns

Pre-approved. This comment is worth attention. Others are mostly nits.
#1330 (comment)

github-actions Bot assigned l2ysho Aug 14, 2026

github-actions Bot added this to the 147th sprint - Tooling team milestone Aug 14, 2026

github-actions Bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Aug 14, 2026

l2ysho requested a review from vladfrangu August 14, 2026 11:00

l2ysho added t-builders Issues owned by the Builders team. adhoc Ad-hoc unplanned task added during the sprint. and removed t-tooling Issues with this label are in the ownership of the tooling team. labels Aug 14, 2026

l2ysho marked this pull request as ready for review August 14, 2026 11:02

l2ysho requested a review from DaveHanns as a code owner August 14, 2026 11:02

l2ysho force-pushed the claude/apify-cli-1206-verify-37b57c branch from 29e4187 to 2b8ebae Compare September 3, 2026 12:25

l2ysho and others added 9 commits September 3, 2026 23:15

Merge branch 'master' into claude/apify-cli-1206-verify-37b57c

3bd52f4

Merge branch 'master' into claude/apify-cli-1206-verify-37b57c

fdf077c

Merge branch 'master' into claude/apify-cli-1206-verify-37b57c

f46e491

Merge remote-tracking branch 'origin/claude/apify-cli-1206-verify-37b…

882d99a

…57c' into claude/apify-cli-1206-verify-37b57c

Merge remote-tracking branch 'origin/master' into claude/apify-cli-12…

8e9494c

…06-verify-37b57c

Merge branch 'master' into claude/apify-cli-1206-verify-37b57c

b89e421

Merge branch 'master' into claude/apify-cli-1206-verify-37b57c

6218b29

l2ysho mentioned this pull request Oct 8, 2026

fix(init): stop the name prompt looping when stdin is not a TTY #1479

Merged

CyberNativeAI mentioned this pull request Oct 15, 2026

fix(key-value-stores): store the value piped into set-value #1498

Closed

patrikbraborec mentioned this pull request Oct 16, 2026

feat(init): warn when a Python project has no Dockerfile #1506

Closed

Merge remote-tracking branch 'origin/master' into claude/apify-cli-12…

23df965

…06-verify-37b57c # Conflicts: # src/lib/command-framework/apify-command.ts

l2ysho added 3 commits October 21, 2026 16:20

Merge branch 'master' into claude/apify-cli-1206-verify-37b57c

22ebb9a

Merge branch 'master' into claude/apify-cli-1206-verify-37b57c

feadd51

Merge branch 'master' into claude/apify-cli-1206-verify-37b57c

98d35ba

l2ysho removed the request for review from vladfrangu October 27, 2026 11:16

l2ysho requested review from apify-factory and removed request for DaveHanns October 27, 2026 11:16

apify-factory previously requested changes Oct 27, 2026

View reviewed changes

l2ysho requested a review from DaveHanns October 27, 2026 11:21

l2ysho requested a review from apify-factory October 27, 2026 11:56

apify-factory reviewed Oct 27, 2026

View reviewed changes

Merge branch 'master' into claude/apify-cli-1206-verify-37b57c

be243a1

DaveHanns reviewed Oct 27, 2026

View reviewed changes

DaveHanns approved these changes Oct 27, 2026

View reviewed changes

Navigation Menu

Repository navigation

fix(cli): read stdin lazily so open pipes don't hang the CLI - #1330

fix(cli): read stdin lazily so open pipes don't hang the CLI#1330
l2ysho wants to merge 16 commits into
masterfrom
claude/apify-cli-1206-verify-37b57c

l2ysho commented Aug 14, 2026 •

edited

Loading

Uh oh!

apify-factory left a comment

Uh oh!

apify-factory left a comment

Uh oh!

DaveHanns Oct 20, 2026 •

edited

Loading

Uh oh!

DaveHanns Oct 20, 2026 •

edited

Loading

Uh oh!

DaveHanns Oct 20, 2026 •

edited

Loading

Uh oh!

DaveHanns Oct 27, 2026

Uh oh!

DaveHanns Oct 27, 2026

Uh oh!

DaveHanns left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	? '\nStandard input went quiet before it closed, so only part of it was read. Use `--input-file=-` to wait for all of it.'
	? '\nStandard input was still open when the CLI stopped waiting for it. If the input is incomplete, use `--input-file=-` to wait for all of it.'

Conversation

l2ysho commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The startup read

The implicit read

Commands where piping is the point

Verification

Notes

Uh oh!

apify-factory left a comment

Choose a reason for hiding this comment

🏭 Factory review — ℹ️ not auto-approved

Uh oh!

apify-factory left a comment

Choose a reason for hiding this comment

🏭 Factory review — ℹ️ not auto-approved

Uh oh!

DaveHanns Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DaveHanns Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DaveHanns Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DaveHanns Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

DaveHanns Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

DaveHanns left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

l2ysho commented Aug 14, 2026 •

edited

Loading

DaveHanns Oct 20, 2026 •

edited

Loading

DaveHanns Oct 20, 2026 •

edited

Loading

DaveHanns Oct 20, 2026 •

edited

Loading