Repository navigation
feat: validate actor.json paths before apify push uploads anything
#1503
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
DaveHanns
wants to merge
34
commits into
master
Choose a base branch
from
feat/push-validate-actor-json-paths
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+663
−2
Open
Changes from all commits
Commits
Show all changes
34 commits
Select commit
Hold shift + click to select a range
d71b158
feat: add the dockerContextDir resolver and Docker context file colle…
DaveHanns d49e15e
feat: refuse monorepo Actors in apify push before any platform write
DaveHanns 3fa653b
feat: validate actor.json paths before apify push uploads anything
DaveHanns afa1cb7
test: cover the actor.json path checks and the push refusal
DaveHanns 2ac0be9
test: pass the monorepo file list to the Docker context case instead …
DaveHanns 8312f66
fix: resolve the Actor directory's real path and require a directory …
DaveHanns feb2cec
fix: return forward-slash paths from the Docker context file collector
DaveHanns b2ee951
fix: name the Actor folder in the monorepo refusal when actor.json ha…
DaveHanns d1fec8d
test: cover readGitProvenance and tidy the monorepo push test
DaveHanns 86d8822
Merge remote-tracking branch 'origin/feat/push-monorepo-preflight' in…
DaveHanns db2cb8f
refactor: split the Docker context resolution into none, within, outs…
DaveHanns f728886
test: cover the none and within Docker context kinds
DaveHanns 92b2679
Merge remote-tracking branch 'origin/feat/push-monorepo-preflight' in…
DaveHanns 9300a01
fix: require the Dockerfile inside a Docker context within the Actor …
DaveHanns a9f2d19
refactor: rename readGitProvenance to readGitState
DaveHanns d9c0cb2
Merge remote-tracking branch 'origin/feat/push-monorepo-preflight' in…
DaveHanns a31364e
fix: judge actor.json paths lexically, as push uploads them
DaveHanns 0b495cf
fix: check the inputSchema and outputSchema aliases
DaveHanns 3e14b8b
fix: skip empty path fields and reject backslashes
DaveHanns 775db15
test: cover the case mismatch reason and use the temp path hook
DaveHanns 703159b
refactor: compute the path check invariants once
DaveHanns cc9e1a8
docs: state the BuildKit basis of the Docker context rule
DaveHanns 39be1cb
docs: list the rules checkPathField applies, in order
DaveHanns f42dd5b
docs: name the dangling symlink and case rules in the checkPathField …
DaveHanns 40a603c
test: cover dangling and symlinked Dockerfile paths and a symlinked A…
DaveHanns e3edc79
test: use a forward-slash absolute path so the check reads the same o…
DaveHanns 50fb1b8
docs: explain why the Docker context is judged by its path as written
DaveHanns 1af9abf
test: pin the Docker context classification through symlinks
DaveHanns c1cce18
Merge remote-tracking branch 'origin/feat/push-monorepo-preflight' in…
DaveHanns 944b6ae
fix: return the Docker context path as the upload lays it out
DaveHanns 5b807db
Merge remote-tracking branch 'origin/feat/push-monorepo-preflight' in…
DaveHanns ee0eda8
fix: compare the Dockerfile with the Docker context as the upload lay…
DaveHanns d56fdf7
test: accept a Dockerfile in a Docker context reached through a symlink
DaveHanns a91bf73
Merge remote-tracking branch 'origin/master' into feat/push-validate-…
DaveHanns File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,183 @@ | ||
| import { readdirSync, realpathSync, type Stats, statSync } from 'node:fs'; | ||
| import { basename, dirname, isAbsolute, join, relative, resolve } from 'node:path'; | ||
|
|
||
| import { ACTOR_SPECIFICATION_FOLDER } from './consts.js'; | ||
| import { isOutside, toPosix } from './docker-context.js'; | ||
|
|
||
| export interface ActorPathProblem { | ||
| field: string; | ||
| value: string; | ||
| resolvedPath: string; | ||
| reason: string; | ||
| } | ||
|
|
||
| export interface ActorPathsOptions { | ||
| actorDir: string; | ||
| /** What push sends: the Actor folder, or the source root of a monorepo Actor. */ | ||
| uploadRoot: string; | ||
| /** Relative to `uploadRoot`, as the file collectors return them. */ | ||
| uploadedFiles: string[]; | ||
| config: Record<string, unknown>; | ||
| /** | ||
| * When given, `dockerfile` must lie inside it. The worker runs BuildKit with `--local dockerfile=<context dir>` | ||
| * and `filename=<Dockerfile relative to the context>`, so a Dockerfile outside the context cannot be read by the | ||
| * build. The same holds for a context within the Actor folder, which today's push already sends. Pass it as | ||
| * `resolveDockerContext` returns it: the path as written, from the real Actor folder. | ||
| */ | ||
| dockerContextRoot?: string; | ||
| } | ||
|
|
||
| /** The platform finds these by a directory listing, so their names match case-insensitively. */ | ||
| const LISTED_FILE_FIELDS = ['dockerfile', 'readme', 'changelog']; | ||
| /** The worker copies a set alias over its base key before resolving, so the alias wins and the base key is ignored. */ | ||
| const SCHEMA_FIELDS: { field: string; alias?: string }[] = [ | ||
| { field: 'input', alias: 'inputSchema' }, | ||
| { field: 'output', alias: 'outputSchema' }, | ||
| { field: 'webServerSchema' }, | ||
| ]; | ||
| const STORAGE_SCHEMA_FIELDS = ['keyValueStore', 'dataset']; | ||
|
|
||
| interface PathField { | ||
| field: string; | ||
| value: string; | ||
| caseInsensitive: boolean; | ||
| } | ||
|
|
||
| function isRecord(value: unknown): value is Record<string, unknown> { | ||
| return typeof value === 'object' && value !== null; | ||
| } | ||
|
|
||
| /** Every file-path field the platform resolves relative to `.actor/`; an inline schema object is not a path. */ | ||
| function collectPathFields(config: Record<string, unknown>): PathField[] { | ||
| const fields: PathField[] = []; | ||
| const add = (field: string, value: unknown, caseInsensitive = false) => { | ||
| // The worker treats an empty string as an unset field. | ||
| if (typeof value === 'string' && value !== '') fields.push({ field, value, caseInsensitive }); | ||
| }; | ||
|
|
||
| for (const field of LISTED_FILE_FIELDS) add(field, config[field], true); | ||
| for (const { field, alias } of SCHEMA_FIELDS) { | ||
| if (alias && config[alias]) add(alias, config[alias]); | ||
| else add(field, config[field]); | ||
| } | ||
| if (isRecord(config.storages)) { | ||
| for (const field of STORAGE_SCHEMA_FIELDS) add(`storages.${field}`, config.storages[field]); | ||
| if (isRecord(config.storages.datasets)) { | ||
| for (const [name, value] of Object.entries(config.storages.datasets)) { | ||
| add(`storages.datasets.${name}`, value); | ||
| } | ||
| } | ||
| } | ||
| return fields; | ||
| } | ||
|
|
||
| function listDirectory(dir: string): string[] { | ||
| try { | ||
| return readdirSync(dir); | ||
| } catch { | ||
| return []; | ||
| } | ||
| } | ||
|
|
||
| /** The options as every field sees them: real roots and the uploaded files as posix paths. */ | ||
| interface PathCheckScope { | ||
| actorDir: string; | ||
| uploadRoot: string; | ||
| uploadedFiles: Set<string>; | ||
| dockerContextRoot?: string; | ||
| } | ||
|
|
||
| /** | ||
| * Applies the platform's rules to one path field of `actor.json`, in order; the first failing rule wins. | ||
| * 1. No backslashes and not absolute: the platform resolves the value relative to `.actor/` on Linux. | ||
| * 2. Resolved from `.actor/`, it lies inside the upload root; the platform never sees anything else. | ||
| * 3. The file exists in its directory: by any case for `dockerfile`, `readme` and `changelog`, which the | ||
| * platform finds by listing the directory, and by the exact name for schema references. A dangling symlink | ||
| * does not exist. | ||
| * 4. It is a file, not a directory. | ||
| * 5. `dockerfile` lies inside the Docker context, as BuildKit reads it from there. | ||
| * 6. It is among the files push uploads, in the same case: a file `.gitignore` or `.actorignore` drops never | ||
| * reaches the platform. | ||
| */ | ||
| function checkPathField( | ||
| { field, value, caseInsensitive }: PathField, | ||
| { actorDir, uploadRoot, uploadedFiles, dockerContextRoot }: PathCheckScope, | ||
| ): ActorPathProblem | undefined { | ||
| const problem = (resolvedPath: string, reason: string) => ({ field, value, resolvedPath, reason }); | ||
| if (value.includes('\\')) { | ||
| return problem(value, 'uses a backslash; the platform resolves paths with forward slashes'); | ||
| } | ||
| if (isAbsolute(value)) return problem(value, `is absolute; the path is relative to ${ACTOR_SPECIFICATION_FOLDER}/`); | ||
|
|
||
| const resolvedPath = resolve(actorDir, ACTOR_SPECIFICATION_FOLDER, value); | ||
| if (isOutside(relative(uploadRoot, resolvedPath))) { | ||
| return problem(resolvedPath, `lies outside the uploaded folder ${uploadRoot}`); | ||
| } | ||
|
|
||
| const dir = dirname(resolvedPath); | ||
| const wanted = basename(resolvedPath); | ||
| const entries = listDirectory(dir); | ||
| const sameCase = entries.find((entry) => entry === wanted); | ||
| const anyCase = entries.find((entry) => entry.toLowerCase() === wanted.toLowerCase()); | ||
| const name = sameCase ?? (caseInsensitive ? anyCase : undefined); | ||
| if (!name) { | ||
| return problem( | ||
| resolvedPath, | ||
| anyCase ? `does not exist (the name must match ${anyCase} exactly)` : 'does not exist', | ||
| ); | ||
| } | ||
|
|
||
| // push reads through a symlink and uploads the content at this path, so the path is judged as it is, not by its | ||
| // target; a dangling symlink has no content to upload. | ||
| const filePath = join(dir, name); | ||
| let stats: Stats; | ||
| try { | ||
| stats = statSync(filePath); | ||
| } catch { | ||
| return problem(resolvedPath, 'does not exist'); | ||
| } | ||
| if (stats.isDirectory()) return problem(resolvedPath, 'is a directory, not a file'); | ||
| if (dockerContextRoot && field === 'dockerfile' && isOutside(relative(dockerContextRoot, filePath))) { | ||
| return problem(resolvedPath, `lies outside the Docker context ${dockerContextRoot}`); | ||
| } | ||
|
|
||
| const uploadedPath = toPosix(relative(uploadRoot, filePath)); | ||
| if (!uploadedFiles.has(uploadedPath)) { | ||
| const differentCase = [...uploadedFiles].find((file) => file.toLowerCase() === uploadedPath.toLowerCase()); | ||
| return problem( | ||
| resolvedPath, | ||
| differentCase | ||
| ? `does not match the case of the uploaded file ${differentCase}` | ||
| : 'is not among the files apify push uploads (excluded by .gitignore or .actorignore?)', | ||
| ); | ||
| } | ||
| return undefined; | ||
| } | ||
|
|
||
| /** | ||
| * Applies the platform's rules to every file path in `actor.json` before anything is uploaded: relative to | ||
| * `.actor/`, inside the upload root, an existing file, and one that push sends. A symlink counts by its own path, | ||
| * as push uploads its content there. | ||
| */ | ||
| export function checkActorPaths(options: ActorPathsOptions): ActorPathProblem[] { | ||
| // The Actor folder and the upload root are real paths, as the Docker context resolver works from them, so a | ||
| // symlinked Actor folder does not count as an escape. A referenced path, like the Docker context the resolver | ||
| // returns, is judged as written below the real Actor folder, the way push lays out the upload. | ||
| const scope: PathCheckScope = { | ||
| actorDir: realpathSync(options.actorDir), | ||
| uploadRoot: realpathSync(options.uploadRoot), | ||
| uploadedFiles: new Set(options.uploadedFiles.map(toPosix)), | ||
| dockerContextRoot: options.dockerContextRoot, | ||
| }; | ||
| return collectPathFields(options.config).flatMap((pathField) => checkPathField(pathField, scope) ?? []); | ||
| } | ||
|
|
||
| export function formatActorPathProblems(problems: ActorPathProblem[]): string { | ||
| return [ | ||
| '.actor/actor.json refers to files the Apify platform could not use:', | ||
| ...problems.map( | ||
| ({ field, value, resolvedPath, reason }) => ` "${field}": "${value}" (${resolvedPath}) ${reason}.`, | ||
| ), | ||
| `Paths are relative to ${ACTOR_SPECIFICATION_FOLDER}/ and must point to files inside the uploaded folder.`, | ||
| ].join('\n'); | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,168 @@ | ||
| import { mkdir, writeFile } from 'node:fs/promises'; | ||
| import { dirname } from 'node:path'; | ||
|
|
||
| import { ACTOR_SOURCE_TYPES } from '@apify/consts'; | ||
|
|
||
| import { testRunCommand } from '../../../src/lib/command-framework/apify-command.js'; | ||
| import { CommandExitCodes } from '../../../src/lib/consts.js'; | ||
| import { cwdCache } from '../../../src/lib/hooks/useActorConfig.js'; | ||
| import { useConsoleSpy } from '../../__setup__/hooks/useConsoleSpy.js'; | ||
| import { useTempPath } from '../../__setup__/hooks/useTempPath.js'; | ||
|
|
||
| const actName = 'push-actor-paths-actor'; | ||
| const ACTOR_ID = 'pAtH1234efGh'; | ||
|
|
||
| const actor = { | ||
| id: ACTOR_ID, | ||
| name: actName, | ||
| // Older than any local file, so the "modified on the platform" check never trips. | ||
| modifiedAt: new Date(0), | ||
| taggedBuilds: { latest: { buildId: 'build1' } }, | ||
| }; | ||
| const build = { id: 'build1', actId: ACTOR_ID, buildNumber: '0.0.1', status: 'SUCCEEDED' }; | ||
| const currentVersion = { versionNumber: '0.0', sourceType: ACTOR_SOURCE_TYPES.SOURCE_FILES }; | ||
|
|
||
| const actorGet = vitest.fn(async () => actor); | ||
| const versionUpdate = vitest.fn(async (_modifier: unknown) => ({})); | ||
| const versionCreate = vitest.fn(async (created: unknown) => created); | ||
|
|
||
| // Stands in for the platform: the Actor and its version exist, and every read and write is recorded. | ||
| vitest.mock('../../../src/lib/utils.js', async (importOriginal) => ({ | ||
| ...(await importOriginal<typeof import('../../../src/lib/utils.js')>()), | ||
| getCurrentUserInfo: vitest.fn(async () => ({ id: 'userId', username: 'user' })), | ||
| outputJobLog: vitest.fn(async () => {}), | ||
| getLoggedClientOrThrow: vitest.fn(async () => ({ | ||
| baseUrl: 'https://api.apify.com/v2', | ||
| actor: () => ({ | ||
| get: actorGet, | ||
| version: () => ({ get: async () => currentVersion, update: versionUpdate }), | ||
| versions: () => ({ create: versionCreate }), | ||
| build: async () => build, | ||
| }), | ||
| build: () => ({ get: async () => build }), | ||
| })), | ||
| })); | ||
|
|
||
| // The mocked `node:process` copy has no signal handlers; nothing here needs them. | ||
| vitest.mock('../../../src/lib/hooks/useAbortJobOnSignal.js', () => ({ | ||
| useAbortJobOnSignal: () => ({ [Symbol.dispose]() {} }), | ||
| })); | ||
|
|
||
| const { beforeAllCalls, afterAllCalls, joinPath, tmpPath } = useTempPath(actName, { | ||
| create: true, | ||
| remove: true, | ||
| cwd: true, | ||
| cwdParent: false, | ||
| }); | ||
|
|
||
| const { logMessages } = useConsoleSpy(); | ||
|
|
||
| const { ActorsPushCommand } = await import('../../../src/commands/actors/push.js'); | ||
| // The command sets the exit code on this mocked copy of `node:process`, not on the global one. | ||
| const { default: commandProcess } = await import('node:process'); | ||
|
|
||
| async function write(relativePath: string, content: string) { | ||
| await mkdir(dirname(joinPath(relativePath)), { recursive: true }); | ||
| await writeFile(joinPath(relativePath), content); | ||
| } | ||
|
|
||
| async function writeActorConfig(fields: Record<string, unknown>) { | ||
| await write( | ||
| '.actor/actor.json', | ||
| JSON.stringify({ actorSpecification: 1, name: actName, version: '0.0', buildTag: 'latest', ...fields }), | ||
| ); | ||
| } | ||
|
|
||
| // The mocked copy may share the exit code with the real process, so both are restored. | ||
| let originalExitCode: typeof process.exitCode; | ||
| let originalGlobalExitCode: typeof process.exitCode; | ||
|
|
||
| beforeEach(async () => { | ||
| originalExitCode = commandProcess.exitCode; | ||
| originalGlobalExitCode = process.exitCode; | ||
| actorGet.mockClear(); | ||
| versionUpdate.mockClear(); | ||
| versionCreate.mockClear(); | ||
| cwdCache.clear(); | ||
| await beforeAllCalls(); | ||
|
|
||
| await write('.actor/INPUT_SCHEMA.json', '{}'); | ||
| await write('.actorignore', 'Dockerfile.ci\n'); | ||
| await write('Dockerfile', 'FROM apify/actor-node:20\n'); | ||
| await write('Dockerfile.ci', 'FROM apify/actor-node:20\n'); | ||
| await write('src/main.js', 'console.log(1);\n'); | ||
| }); | ||
|
|
||
| afterEach(async () => { | ||
| commandProcess.exitCode = originalExitCode; | ||
| process.exitCode = originalGlobalExitCode; | ||
| await afterAllCalls(); | ||
| }); | ||
|
|
||
| describe('apify push with file paths in actor.json', () => { | ||
| it('stops before reading or writing the Actor, listing every wrong path', async () => { | ||
| await writeActorConfig({ | ||
| dockerfile: '../Dockerfile.prod', | ||
| readme: '../Dockerfile.ci', | ||
| input: '../../INPUT_SCHEMA.json', | ||
| }); | ||
|
|
||
| await testRunCommand(ActorsPushCommand, {}); | ||
|
|
||
| expect(commandProcess.exitCode).toBe(CommandExitCodes.InvalidActorJson); | ||
| const stderr = logMessages.error.join('\n'); | ||
| expect(stderr).toContain('.actor/actor.json refers to files the Apify platform could not use:'); | ||
| expect(stderr).toContain(`"dockerfile": "../Dockerfile.prod" (${joinPath('Dockerfile.prod')}) does not exist.`); | ||
| expect(stderr).toContain( | ||
| `"readme": "../Dockerfile.ci" (${joinPath('Dockerfile.ci')}) is not among the files apify push uploads (excluded by .gitignore or .actorignore?).`, | ||
| ); | ||
| expect(stderr).toContain( | ||
| `"input": "../../INPUT_SCHEMA.json" (${joinPath('..', 'INPUT_SCHEMA.json')}) lies outside the uploaded folder ${tmpPath}.`, | ||
| ); | ||
| expect(actorGet).not.toHaveBeenCalled(); | ||
| expect(versionCreate).not.toHaveBeenCalled(); | ||
| expect(versionUpdate).not.toHaveBeenCalled(); | ||
| }); | ||
|
|
||
| it('keeps stdout clean with --json', async () => { | ||
| await writeActorConfig({ dockerfile: '../Dockerfile.prod' }); | ||
|
|
||
| await testRunCommand(ActorsPushCommand, { flags_json: true }); | ||
|
|
||
| expect(commandProcess.exitCode).toBe(CommandExitCodes.InvalidActorJson); | ||
| expect(logMessages.log).toEqual([]); | ||
| expect(logMessages.error.join('\n')).toContain('"dockerfile": "../Dockerfile.prod"'); | ||
| expect(versionUpdate).not.toHaveBeenCalled(); | ||
| }); | ||
|
|
||
| it('requires the Dockerfile inside a Docker context within the Actor folder', async () => { | ||
| await write('docker/Dockerfile', 'FROM apify/actor-node:20\n'); | ||
| await writeActorConfig({ dockerContextDir: '../docker', dockerfile: '../Dockerfile' }); | ||
|
|
||
| await testRunCommand(ActorsPushCommand, {}); | ||
|
|
||
| expect(commandProcess.exitCode).toBe(CommandExitCodes.InvalidActorJson); | ||
| expect(logMessages.error.join('\n')).toContain( | ||
| `"dockerfile": "../Dockerfile" (${joinPath('Dockerfile')}) lies outside the Docker context ${joinPath('docker')}.`, | ||
| ); | ||
| expect(versionUpdate).not.toHaveBeenCalled(); | ||
| }); | ||
|
|
||
| it('pushes as usual when every path points to an uploaded file', async () => { | ||
| await writeActorConfig({ dockerfile: '../dockerfile', input: './INPUT_SCHEMA.json' }); | ||
|
|
||
| await testRunCommand(ActorsPushCommand, {}); | ||
|
|
||
| expect(commandProcess.exitCode).toBeFalsy(); | ||
| expect(versionUpdate).toHaveBeenCalledTimes(1); | ||
| const modifier = versionUpdate.mock.calls[0][0] as { sourceFiles: { name: string }[] }; | ||
| expect(modifier.sourceFiles.map((file) => file.name).sort()).toEqual([ | ||
| '.actor/INPUT_SCHEMA.json', | ||
| '.actor/actor.json', | ||
| '.actorignore', | ||
| 'Dockerfile', | ||
| 'src/main.js', | ||
| ]); | ||
| expect(logMessages.log.join('\n')).toContain('Apify push result: SUCCEEDED'); | ||
| }); | ||
| }); |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
apify pushchecks dockerfile only when the key is present. as@apify/json_schemas@1.0.1sets default to dockerfileapify pushsucceed and then build uses the platform’s default image or fail (depending if ../Dockerfile exists or not)Steps to reproduce
FAILED — open ../Dockerfile: no such file or directorySteps to reproduce
Dockerfile not found, using the default one.In both scenarios
pushshould fail locally