Read a flat_object field in PPL

[RyanL1997]

Self-review PR against my own fork — not for upstream. This is the first half of the flat_object work, split out so it can be read on its own: reading the field. Filtering it is the follow-up.

Why this exists

A flat_object field is invisible to PPL today: parseMapping skips the type, so describe does not list it and any reference fails with field not found (opensearch-project#1604, opensearch-project#3067).

The representation: text, like spath

A flat_object gives its leaves no mapping of their own — the index files every leaf value as a keyword term with the path folded in — so a leaf has no type either. Every value reads as text, and a path holding more than one value reads as their JSON. This is the rule spath already applies to a JSON string, and typeof on a leaf reports string.

document	reads as
{"a": {"b": 1}}	a.b: "1"
{"a.b": 1}	a.b: "1"
{"a": [1, 2]}	a: "[\"1\",\"2\"]"
[{"a": 1}, {"a": 2}]	a: "[\"1\",\"2\"]"
{"nested": [{"k": 1}, {"k": 2}]}	nested.k: "[\"1\",\"2\"]"
{"a": {"b": 1}, "a.b": 2}	a.b: "[\"1\",\"2\"]"

Every way of writing a path reaches the same leaf, because the index files one term either way. A path written more than once holds every value rather than the last: the index answers a lookup for any of them, so dropping one would make a projection disagree with a filter.

Supported

source=logs | fields attributes
source=logs | fields attributes.duration_ms
source=logs | eval ms = cast(attributes.duration_ms as double)

A cast counts as reading: a leaf is text, so converting it is the parse a caller would do anyway and the value comes from _source either way. It is how a query asks for a leaf as a number — which the field type leaves to the caller.

Rejected, while the plan is built

Filtering by a leaf, aggregating it, grouping or sorting by it, computing a value from it, expanding it. None of this is a PPL restriction: the limitations are the field type's own, and answering any of it would mean opening every record — the cost this type exists to avoid.

Cannot sort by attributes.duration_ms
  code       UNSUPPORTED_OPERATION
  suggestion A flat_object field can only be read. Its leaves have no mapping of their own, so
             nothing else can be answered without opening every record.
             See https://docs.opensearch.org/latest/mappings/supported-field-types/flat-object/

How the check works

FlatObjectScopeValidator runs from CalciteRelNodeVisitor.analyze, so the unified query pipeline — which calls the visitor directly rather than through QueryService — is held to the same rules.

It finds the field by its mapping type, not by the type it takes in the plan: a map of text to text is also what spath produces for an extracted document, so recognising by plan type broke CalcitePPLSpathTest and the opensearch-project#5718 yaml tests. It also tracks a column a projection derived from a leaf, so a command that sorts or groups by a leaf — projecting it into a column of its own first, under a generated name — is caught and reported by the leaf's name rather than $f2.

Two smaller changes riding along

• expand over anything that is not a column of the index now raises the error mvexpand already raises, instead of the ClassCastException the cast threw.
• RexStandardizer refuses to build a script over a flat_object field: it has no usable doc values, so a script could only read _source, and the field type does not support Painless for retrieving subfield values either.

Why text rather than a typed representation

An earlier attempt kept each leaf's written type with Calcite's VARIANT. Measured on a cluster, it buys only how a projection displays — every numeric operation is rejected in this scope either way — and it costs a second set of semantics that has to be kept in sync with the index by hand: the index compares text and ANY-matches multi-values, Calcite compares typed scalars. Two divergence bugs came out of that gap. It also had to reject cast, because Calcite's variant cast answers null for a leaf not written as text — so a user could not get a number at all. Text has one semantics, and the cast works.

Verified

Full core, opensearch, ppl module tests and the entire yamlRestTest suite pass, and the behaviour above was exercised end to end against a running cluster.