Cloudera Blueprint: Optagent

Project template for a Cloudera Machine Learning AMP. Copy it when a new prototype needs a catalog entry, a workbench runbook, and a local devenv shell that runs the same scripts.

Table of Contents

• Overview
• Demo
• Use Case
• Key Features
• Quickstart
• Architecture
• Target Audience
• Repository Structure
• Prerequisites
• Hardware Requirements
• Documentation
• Copy this template

Overview

Optagent launches as an Applied ML Prototype. The workbench installs requirements.txt, registers impala_fdw when the worker can build it, writes a status snapshot, and serves that snapshot. On a workstation, devenv runs PostgreSQL 16 on port 5599 and the same scripts. The shell is Python 3.12 with uv. CML does not read devenv.nix.

Kudu and Iceberg stay where they are. Postgres foreign tables reach them through Impala. Kudu scans and writes use kudu_scan. Iceberg tables and UNION views use impala_sql. The extension is weathership/impala_fdw, the same repository Gaius and Signals build. Product auth is Kerberos.

The ledger relation is optagent.span. Hot hours are span_tier0 on Kudu. Settled hours are span_tier1 on Iceberg. The view masks a tier1 hour that is still in tier0. Session inspection starts from the local agentsview checkout at ../../rch/oss-agentsview, wired as an editable replace in go.work and ui/go.mod.

Local devenv stays on the FDW. An AMP sets OPTAGENT_POSTURE=amp and talks to Impala HS2 only. The host, principal, and keytab for that posture live in a SOPS file, .env.cai.enc, decrypted on the workbench. config/base.conf is the HOCON source. Spark is not a dependency. OPTAGENT_CDE_ENABLED only records that Cloudera Data Engineering joins later.

flowchart LR
  install["Install dependencies"] --> prepare["Prepare project"]
  prepare --> snapshot["status.json"]
  snapshot --> app["Application"]

Loading

Demo

A Reprise walkthrough is not recorded yet. Leave reprise_link empty in METADATA.yaml until one exists.

Locally, devenv up serves the page at http://tinybox.dev.vista.zndx.org:9892, on all interfaces, next to Signals on :9889 and Gaius on :9890.

Use Case

New prototypes need the same CML shape every time: a catalog entry, a .project-metadata.yaml runbook, numbered steps, and a local environment that can run those steps before anyone opens a workbench. This repository is that shape, with a small job and application so the launch path succeeds before the real workload replaces them.

Key Features

• A CML runbook that creates two jobs, runs them, and starts the application
• Install, prepare, and serve scripts that find the project when CML leaves __file__ unset
• A devenv shell that syncs the project with uv and supervises the same prepare job and application
• Catalog files for a custom AMP source and for the blueprint website
• Tests that keep the runbook, the dependency lists, and the devenv shell aligned

Quickstart

1. Clone this repository.

2. Enter the devenv shell. With direnv, direnv allow. Otherwise run devenv shell.

3. Run the tests and serve the page:

   just test
   devenv up

   Then open http://tinybox.dev.vista.zndx.org:9892. With Postgres up, register the warehouse wrapper:

   just impala-fdw-install

   just prepare rewrites assets/data/status.json. just app serves the page without the process supervisor.

4. To launch it from a CML AMP catalog, add this repository as a catalog source and set the catalog file to catalog-entry.yaml. Site Administration steps are in Adding a catalog.

   CML reads .project-metadata.yaml and runs the tasks in order:

   Step	Type	What it does
   Install Dependencies	create_job + run_job	Installs requirements.txt into the workbench Python
   Register Impala FDW	create_job + run_job	Builds and registers impala_fdw when PostgreSQL headers and Thrift are on the worker. A stock PBJ image skips this and the launch continues
   Prepare project	create_job + run_job	Writes assets/data/status.json, including whether the FDW is loaded
   Optagent	start_application	Serves the snapshot on CDSW_APP_PORT

   The jobs stay in the project. Run Install Dependencies again from the Jobs tab when requirements.txt changes.

Architecture

The package in src/optagent holds the snapshot writer. The numbered folders are the CML steps, in order. Devenv starts the prepare job, then the application.

Component detail is in docs/architecture.md.

Target Audience

• Developers packaging an internal prototype for Cloudera Machine Learning
• Solution architects who need a catalog entry and a one-click launch path
• Platform engineers checking that a new AMP uses the job runbook and a devenv shell

Repository Structure

Path	Description
src/optagent/	Snapshot writer and Impala FDW registration SQL
components/impala_fdw/	weathership/impala_fdw submodule (trunk)
scripts/warehouse/	Build, install, and CML registration entrypoint
0_session-install-dependencies/	CML job that installs requirements.txt
1_job-prepare-project/	CML job that writes the status snapshot
2_app-serve/	CML application that serves the snapshot
assets/data/	Generated status snapshot
devenv.nix	Local Python 3.12 environment, uv sync, and the two processes
requirements.txt	Runtime dependencies installed by the CML job
docs/architecture.md	Runbook, ports, and configuration
METADATA.yaml	Catalog metadata for the Cloudera blueprint website
catalog-entry.yaml	CML AMP catalog entry
.project-metadata.yaml	CML runtime and task runbook
tests/	Runbook, devenv, snapshot, and page checks

Prerequisites

• devenv and direnv for local runs
• Python 3.12 is provided by the devenv shell
• A Cloudera Machine Learning workspace with a PBJ Workbench runtime on Python 3.12 or 3.13, or JupyterLab on Python 3.12, Standard edition, when launching the AMP
• No API keys. OPTAGENT_GREETING and OPTAGENT_STATUS_PATH have defaults

Hardware Requirements

Deployment	Minimum
Launchable / demo	1 CPU, 2 GB RAM, no GPU
Production / enterprise	Size the workload you add. The template jobs request 1 CPU and 2 GB

Documentation

• Architecture
• AMP project specification
• Catalog file specification
• Creating new AMPs

Copy this template

Keep the task order and the devenv shell. Replace the sample workload.

Change	Where
Prototype name, slug, maintainer	METADATA.yaml, catalog-entry.yaml, .project-metadata.yaml
git_url and git_ref	catalog-entry.yaml
Subdomain	.project-metadata.yaml subdomain
Python package	src/optagent/, pyproject.toml name
Runtime dependencies	requirements.txt and [project].dependencies, same names in both
Prepare job	1_job-prepare-project/job.py and processes.prepare
Application	2_app-serve/app.py and processes.app

The install task stays a job. Give each create_job a kernel: python3 and arguments: "". Point languages.python.package at pkgs.python312, and install local tools with uv.sync.