[DRAFT] In house JsonSchema to TS generator by JuanGalilea · Pull Request #1382 · apify/apify-cli

JuanGalilea · 2026-09-02T13:36:18Z

base generator and tons of testing added.

Install size change (`node scripts/report-install-size.mjs`)

Metric	This release	Latest (1.10.0)	Change
Tarball download	0.36 MB	0.35 MB	+4.1%
Unpacked package	1.40 MB	1.36 MB	+3.1%
Full install (with dependencies)	46.28 MB	58.84 MB	-21.3%

AI `/show-me` below

Replaces the json-schema-to-typescript dependency with an in-house schema→TypeScript compiler (src/lib/schema-to-ts/, zero runtime deps — only node:crypto), and reframes the CLI flag surface around a single --perspective.

src/lib/
├── schema-transforms.ts          # DELETED (-225): makePropertiesRequired, clearAllRequired, stripTitles…
└── schema-to-ts/                 # NEW: ~950 lines, no runtime deps
    ├── index.ts                  # narrow public surface (compile, check, normalize*, readHeader)
    ├── preprocess/               # the only Apify-aware layer
    │   ├── input.ts              #   nullable rewrite only
    │   ├── dataset.ts            #   unwrap `fields`, drop views/actorSpecification
    │   ├── kvstore.ts            #   collections → one closed lookup object
    │   └── nullable.ts           #   {type:'x', nullable:true} → {type:['x','null']}
    ├── parser.ts                 # JSON Schema → IR  + diagnostics
    ├── ir.ts                     # 9 structural node kinds, IR_VERSION = 1
    ├── emit.ts                   # IR → TS text, `type` never `interface`
    ├── canonical.ts              # sorted serialization (order-insensitive)
    ├── hash.ts                   # sha256, 16 hex chars
    ├── check.ts                  # header regex → drift detection
    ├── compile.ts                # facade: compile() / check()
    └── diagnostics.ts            # error | warning | notice codes

The pipeline

compile(schema, { types: [{name, variant}], unknownRoot? })
  jsonSchemaToIR(schema)                    parser.ts — only place IR is built
    readTypes()                             `type` is MANDATORY (no inference)
    enumToNode()                            enum ⨯ type cross-narrowing
    fromObject() → readRequired/readProps/readAdditional
    → { ir, diagnostics, notices }
  emit(ir, opts)
    header(irHash(ir, opts))                canonical(ir) → sha256 → 16 hex
    renderNode() per requested type name
  → { source, diagnostics, notices }

Two orthogonal signals come out: diagnostics = fidelity lost (something became unknown), notices = schema lint with no type impact. Neither feeds the hash.

The supplied / received split is the conceptual core — same IR, two renderings:

 renderObject(node, variant)
-  optional = !required                       // supplied: writer's obligation
+  optional = !required && !hasDefault        // received: platform materialized defaults
-  if (open) → `{...} & Record<string, unknown>`   // supplied accepts extras
+  // received is closed: unknown key = typo

Drift detection

// @generated schema-ts v1-d1bb2b62dfb769b2 — do not edit
                        │  └── sha256(canonical(IR) + options)[0..16]
                        └── IR_VERSION, compared before the hash

canonical.ts sorts props/enum members while emit.ts reproduces authored order — so reordering keys in a schema never reports drift. Header is matched anywhere in the file (gm), so a user's prepended banner is free; two headers → duplicate-header error.

CLI surface change

 apify actor generate-schema-types [path]
   -o, --output=<dir>
-  --strict            (default true)
-  --all-optional      (default false)
+  --perspective=actor|user   (default actor)

Behavior changes worth a reviewer's attention

Before	After
`export interface Input`	`export type input` — lowercase, from `const name = 'input'`
Banner with `biome-ignore-all`, `eslint-disable`, `prettier-ignore-start`	header comment only — no lint/format suppression in generated files
Empty dataset `fields` → skip with warning	writes `export type dataset = Record<string, unknown>;`
KVS: one interface per collection, concatenated	one closed object, collection name → record type
`prettier` formatting via the library	raw emitter output (4-space indent, `Array<T>`, parenthesized intersections)

apify-factory

Good work, one thing to fix before merge.

🏭 Factory review — ℹ️ not auto-approved

This change was reviewed but cannot be auto-approved: a gate failed, so the review cannot end in an approval.

check	result
Static gates	❌ 8/16 — `author-allowed`, `max-files`, `max-lines`, `file-extensions`, `deny-globs`, `patch-present`, `no-risky-content`, `pr-title`
Review findings	❌ not clean
Reviewer escalation	❌ Replaces a dependency with a new in-house compiler and breaks the public CLI and generated-file contract (removes `--strict`/`--all-optional`, renames exported types from `Input` interfaces to lowercase `input` types, changes the KVS output shape).
Juror 1 — `claude-sonnet-5-5`, adversarial	⏭️ skipped — a gate failed

Why it was not auto-approved

Failed gates:

author-allowed: JuanGalilea: not a member of apify/product-engineering
max-files: 54 changed files, limit 10
max-lines: 22661 changed lines (+21733/-928), limit 200
file-extensions: disallowed extension: .vscode/settings.json, package.json, pnpm-lock.yaml, test/setup/output-schemas/valid.json, test/local/fixtures/lib/schema-to-ts/dataset/comment-scraper.json, test/local/fixtures/lib/schema-to-ts/dataset/google-places.json, test/local/fixtures/lib/schema-to-ts/dataset/tiktok-followers-scraper.json, test/local/fixtures/lib/schema-to-ts/input/api-scraper.json, test/local/fixtures/lib/schema-to-ts/input/comment-scraper.json, test/local/fixtures/lib/schema-to-ts/input/free-amazon-product-scraper.json, test/local/fixtures/lib/schema-to-ts/input/google-places.json, test/local/fixtures/lib/schema-to-ts/kitchen-sink.json, test/local/fixtures/lib/schema-to-ts/kvstore/maps-to-polygon.json, test/tsconfig.json
deny-globs: package.json matches **/package.json; pnpm-lock.yaml matches **/pnpm-lock.yaml
patch-present: no text diff for test/local/fixtures/lib/schema-to-ts/expected/input/google-places.ts, test/local/fixtures/lib/schema-to-ts/input/google-places.json
no-risky-content: test/local/lib/schema-to-ts/hash.test.ts: credential-like assignment (credential-assignment); test/local/lib/schema-to-ts/parser.test.ts: credential-like assignment (credential-assignment); test/local/lib/schema-to-ts/preprocess/input.test.ts: credential-like assignment (credential-assignment)
pr-title: title "[DRAFT] In house JsonSchema to TS generator" must match type(scope): message

The comments above are the review itself; this section only explains the approval decision. Push a change and request @apify-factory again to re-run it, or ask a human to review.

_{Reviewed 99faebbca · workflow run}

apify-factory · 2026-10-22T20:29:07Z

+			label: 'output',
+			outputDir,
+			fileName: 'output.ts',
+			schema: outputSchema,


Important: the output schema now goes to compile raw, and the parser treats a missing type as an error that becomes unknown, so an output schema without a root "type": "object" (like the valid.json fixture before this PR had to add it) now generates export type output = unknown; and still exits 0. The deleted prepareOutputSchemaForCompilation (and the dataset/KVS preparers) defaulted type to 'object', so this is a regression for Actors that worked before; the same thing happens to dataset fields and KVS jsonSchema that leave out type.

tru that and its not required on JSONSchemas.
Added a defaulting step to normalization preprocessors of relevant schemas

…ll failing

apify-factory

Looks good, just a few notes.

🏭 Factory review — ℹ️ not auto-approved

This change was reviewed but cannot be auto-approved: a gate failed, so the review cannot end in an approval.

check	result
Static gates	❌ 7/16 — `author-allowed`, `actor-allowed`, `max-files`, `max-lines`, `file-extensions`, `deny-globs`, `patch-present`, `no-risky-content`, `pr-title`
Review findings	❌ not clean
Reviewer escalation	❌ This replaces a dependency, removes the public `--strict`/`--all-optional` flags, and renames the exported generated types (`Input` -> `input`), all of which break existing users' scripts and imports.
Juror 1 — `claude-sonnet-5-5`, adversarial	⏭️ skipped — a gate failed

Why it was not auto-approved

Failed gates:

author-allowed: JuanGalilea: not a member of apify/product-engineering
actor-allowed: JuanGalilea: not a member of apify/product-engineering
max-files: 56 changed files, limit 10
max-lines: 24084 changed lines (+22445/-1639), limit 200
file-extensions: disallowed extension: .vscode/settings.json, package.json, pnpm-lock.yaml, test/local/fixtures/lib/schema-to-ts/dataset/comment-scraper.json, test/local/fixtures/lib/schema-to-ts/dataset/google-places.json, test/local/fixtures/lib/schema-to-ts/dataset/tiktok-followers-scraper.json, test/local/fixtures/lib/schema-to-ts/input/api-scraper.json, test/local/fixtures/lib/schema-to-ts/input/comment-scraper.json, test/local/fixtures/lib/schema-to-ts/input/free-amazon-product-scraper.json, test/local/fixtures/lib/schema-to-ts/input/google-places.json, test/local/fixtures/lib/schema-to-ts/kitchen-sink.json, test/local/fixtures/lib/schema-to-ts/kvstore/maps-to-polygon.json, test/tsconfig.json
deny-globs: package.json matches **/package.json; pnpm-lock.yaml matches **/pnpm-lock.yaml
patch-present: no text diff for pnpm-lock.yaml, test/local/fixtures/lib/schema-to-ts/expected/input/google-places.ts, test/local/fixtures/lib/schema-to-ts/input/google-places.json
no-risky-content: test/local/lib/schema-to-ts/hash.test.ts: credential-like assignment (credential-assignment); test/local/lib/schema-to-ts/parser.test.ts: credential-like assignment (credential-assignment); test/local/lib/schema-to-ts/preprocess/input.test.ts: credential-like assignment (credential-assignment)
pr-title: title "[DRAFT] In house JsonSchema to TS generator" must match type(scope): message

The comments above are the review itself; this section only explains the approval decision. Push a change and request @apify-factory again to re-run it, or ask a human to review.

_{Reviewed de83ec17e · workflow run}

apify-factory · 2026-10-27T15:32:19Z

+	// Parenthesized unconditionally: `&` binds tighter than `|` so it is already correct inside
+	// a union, but `{...} & Record<...> | null` is a horrible thing to read. Keeping it
+	// unconditional means the object never has to know what context it sits in.
+	if (values !== null) return `(${literal} & Record<string, ${values}>)`;


Note: {...} & Record<string, V> forces every declared prop to also satisfy V, so when a declared prop's type differs from additionalProperties, no object literal type-checks. For example, the kitchen-sink fixture's { known: string } & Record<string, number> rejects { known: 'x' }, which makes the supplied variant impossible to write without a cast.

apify-factory · 2026-10-27T15:32:19Z

+
+	const inner = indent + INDENT;
+	const members = node.props.map((prop) => {
+		const optional = variant === 'supplied' ? !prop.required : !prop.required && !prop.hasDefault;


Note: received makes any prop with a default required, but only the input gets defaults filled in by the platform. Under --perspective user, the dataset, output, and key-value store types (outOfActor: 'received') mark defaulted fields as always present even when the Actor never wrote them.

Resolved: a newer review found nothing blocking.

JuanGalilea self-assigned this Sep 2, 2026

JuanGalilea force-pushed the json-schema-to-ts-in-house branch 2 times, most recently from 9c2467e to 1406406 Compare September 9, 2026 11:58

JuanGalilea marked this pull request as ready for review September 9, 2026 15:14

JuanGalilea requested review from DaveHanns, l2ysho and szaganek as code owners September 9, 2026 15:14

JuanGalilea force-pushed the json-schema-to-ts-in-house branch from 024469a to fd4d2f0 Compare October 20, 2026 13:51

l2ysho requested a review from apify-factory October 22, 2026 20:27

apify-factory previously requested changes Oct 22, 2026

View reviewed changes

JuanGalilea added 20 commits October 27, 2026 16:28

switch settings to the right formatter

5615def

copy all code

d5ff38d

unit tests

1bb7277

copy fixtures

c89e965

full generation tests

13c8b26

windows compat?

e6ebac0

change compilation. perspective missing and tests need improvement

aca0a24

extensive compilation coverage test. Silent section has to be dealt with

fd2f1b5

de-clankerization, enums improved, union mislocated, kitchen sink sti…

b182023

…ll failing

union where it belongs

581ce8d

rename parser to parser xd

ba7d646

compile test green

eee1ef4

rename parser test

1f1f0e5

toBe paranoia

fe53017

parser smartness

f378c9d

fix kitchen sink test

2663163

adapt to library, kvstore support missing

65b1ec5

remove stale dependency

74b4ad7

kvs support and docs update

b1b93e5

add check flag

7b6a292

default root level type: "object" for relevant schemas (and test)

de83ec1

JuanGalilea force-pushed the json-schema-to-ts-in-house branch from c67060e to de83ec1 Compare October 27, 2026 15:29

JuanGalilea requested a review from apify-factory October 27, 2026 15:30

apify-factory reviewed Oct 27, 2026

View reviewed changes

Navigation Menu

Repository navigation

[DRAFT] In house JsonSchema to TS generator - #1382

[DRAFT] In house JsonSchema to TS generator#1382
JuanGalilea wants to merge 21 commits into
masterfrom
json-schema-to-ts-in-house

JuanGalilea commented Sep 2, 2026 •

edited

Loading

Uh oh!

apify-factory left a comment

Uh oh!

apify-factory Oct 22, 2026

Uh oh!

JuanGalilea Oct 27, 2026

Uh oh!

apify-factory left a comment

Uh oh!

apify-factory Oct 27, 2026

Uh oh!

apify-factory Oct 27, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

JuanGalilea commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Install size change (node scripts/report-install-size.mjs)

AI /show-me below

The pipeline

Drift detection

CLI surface change

Behavior changes worth a reviewer's attention

Uh oh!

apify-factory left a comment

Choose a reason for hiding this comment

🏭 Factory review — ℹ️ not auto-approved

Uh oh!

apify-factory Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

JuanGalilea Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

apify-factory left a comment

Choose a reason for hiding this comment

🏭 Factory review — ℹ️ not auto-approved

Uh oh!

apify-factory Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

apify-factory Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JuanGalilea commented Sep 2, 2026 •

edited

Loading

Install size change (`node scripts/report-install-size.mjs`)

AI `/show-me` below