Repository navigation
Conversation
IRENEKO
requested review from
antonpibm,
aviv1ron1,
freunda and
yairallouche
as code owners
September 29, 2026 02:03
Collaborator
|
/gpu-test-multi |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
❌ GPU tests failed —
|
❌ GPU tests failed —
|
Port of granite-switch-staging PR generative-computing#108 (feature/switched-classifier). Adds the switched classifier: a classifier slot is an alternative occupant of an adapter slot, holding a small linear classification head instead of a LoRA. It is fired by the same control-token machinery -- one <|name|> token, one adapter index -- but where a LoRA modifies the hidden states, a classifier reads the clean base hidden state and produces per-label logits. WHAT * Compose: classifier slots declared in a YAML manifest via `kind: classifier` and `labels:`, alongside LoRA slots in one checkpoint. `type` (token placement) and `kind` (LoRA vs classifier) are independent axes. Slots are stably reordered LoRAs-first / classifiers-last so the LoRA weight transfer stays positional and the head bank pads uniformly. * Trained-head loading with a different label count per slot: heads live in one stacked bank padded to `max_classifier_labels`, mirroring the padding rank in the LoRA bank. * Label words resolve to single token ids at compose time, failing loud and listing offenders when a word is not atomic under the base tokenizer. * Verdict exit: the request's final logit row is rewritten so the winning label's token id wins the argmax, so the endpoint emits the literal label word through the standard OpenAI-compatible server with no custom client. HF and vLLM backends kept in sync. * Chat template places the control token at the end of the last user turn, before that turn's closing marker, for both the role-marker (4.0/4.1) and ChatML (4.2) template families. * MultiSwitch: an identity `split_indices` on both backends, routing the classifier stream to zeros, since a classifier slot loads only for a quick verdict. WHY Detection/classification tasks do not need a generative adapter. A linear head over the base hidden state is far cheaper than a LoRA, and riding the existing control-token and slot machinery means it composes, serves and activates exactly like any other adapter -- no separate endpoint, no custom decode path. Tutorial: tutorials/guides/classifier_slots.md Signed-off-by: IRENEKO <cyko@ibm.com>
Update the Step 3 paragraph to match GraniteSwitchConfig: `adapter_kinds` and `classifier_label_token_ids` are the stored fields, with the per-slot label counts and control-token ids derived from them. Signed-off-by: IRENEKO <cyko@ibm.com>
The read point was ``marker - 1``; it is now the marker's own position. The marker's row is always in the pass that contains the marker, so the cross-pass stash that served the one-token look-back is gone, along with its bookkeeping and the error for a missing stash entry. Signed-off-by: IRENEKO <cyko@ibm.com>
A classifier reads its verdict at the marker, so no generation prompt may follow it. vLLM's chat-completions endpoint defaults add_generation_prompt to True, so relying on callers to turn it off would silently place the assistant prompt after the marker. The composed template now forces add_generation_prompt to false when adapter_name names a classifier slot; renders with other adapters or no adapter still honor the flag. test_classifier_drops_generation_prompt pins this on granite-4.0-micro, granite-4.1-3b and granite-4.2-3b (ChatML). Signed-off-by: IRENEKO <cyko@ibm.com>
Append the classifier control token after the whole template instead of inside the last user turn. Since classifier renders already drop the generation prompt, a single append makes the marker the last token in both template families, replacing the per-format turn-close rewrites. The tutorial's raw-completions examples follow the new layout. Signed-off-by: IRENEKO <cyko@ibm.com>
With the marker as the prompt's last token, it always lands in the final prefill chunk, so the cross-pass verdict carry is unnecessary. Run the head only on a pass's marker tokens and rewrite just their logit rows; raise when a marker is not its request's last token. Drop the read-position tests that no longer apply. Signed-off-by: IRENEKO <cyko@ibm.com>
Compute the verdict only at classifier markers, require each marker to be its row's last real token, and write it to the marker's own logit row. Decode steps after the verdict now run as plain LM steps instead of raising, matching vLLM. Signed-off-by: IRENEKO <cyko@ibm.com>
Give classifier slots their own substitute instead of inheriting the LoRA probe: the base's role-open marker, which follows the last turn's close where the classifier token sits. The tutorial tells trainers to read the hidden state at that marker. Signed-off-by: IRENEKO <cyko@ibm.com>
Add per-slot classifier read-layer configuration to the manifest, composer, and model config. Score intermediate-layer classifiers as decoder layers run, in both the HF and vLLM backends, while preserving the active LoRA selection at the classifier marker. In vLLM, keep classifier scores aligned with sampled rows through the hidden-state/logits path. Add coverage for CUDA graph replay, chunked prefill, mixed classifier and plain requests, LoRA coactivation, and intermediate-layer reads. Intermediate reads require pipeline parallelism of 1 and are not supported with Shadow Residual. Speculative decoding and microbatching are unsupported with classifier slots, and classifier slot ids are limited to 256. Classifier markers must be the request's final token; vLLM misplaced-marker validation is deferred. Signed-off-by: IRENEKO <cyko@ibm.com>
…ages Compose classifier heads alongside Shadow Residual adapters, choosing SR mode from the adapting slots, and keep HF classifier routing through save/reload. SR classifier slots can now read any decoder layer: the adapted stream for active tokens and the base stream otherwise, before final normalization. Marker-only indices and accumulated scores are carried across pipeline ranks, and layer-output selection is shared through DecoderInterface. The packed slot-count guard is removed. Stack the two PP halves in SRDecoderInterface.enter_decoder_stack instead of concatenating them, so vLLM's graph splitter keeps the 2*M row relation; concatenation produced a partition cycle under compile. Test recorder callbacks move to an importable module so PP workers can receive them. Known issue, deferred: under vLLM 0.26, FULL CUDA-graph replay diverges from eager for the standard (non-SR) 3B classifier fixtures. Signed-off-by: IRENEKO <cyko@ibm.com>
Chat templates now accept classifier_name alongside adapter_name: the adapter activates at its usual location and the classifier marker is placed last. adapter_name naming a classifier keeps the classifier-only behavior. SR and assistant-boundary aLoRA keep the activation generation prompt; LoRA and user-invoked aLoRA use the transcript-only read point. Existing checkpoints need regenerated templates, not new weights. Signed-off-by: IRENEKO <cyko@ibm.com>
AlonMalach
force-pushed
the
port/108-classifier-public
branch
from
October 19, 2026 05:39
db41758 to
861b917
Compare
Collaborator
|
/gpu-test-multi |
❌ GPU tests failed —
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the switched classifier: a classifier slot is an alternative occupant of an adapter slot, holding a small linear classification head instead of a LoRA. It is fired by the same control-token machinery — one
<|name|>token, one adapter index — but where a LoRA modifies the hidden states, a classifier reads the clean base hidden state and produces per-label logits.kind: classifierandlabels:, alongside LoRA slots in one checkpoint.type(token placement) andkind(LoRA vs classifier) are independent axes. Slots are stably reordered LoRAs-first / classifiers-last so the LoRA weight transfer stays positional and the head bank pads uniformly.max_classifier_labels, mirroring the padding rank in the LoRA bank.split_indiceson both backends, routing the classifier stream to zeros, since a classifier slot loads only for a quick verdict.Tutorial:
tutorials/guides/classifier_slots.mdWhy
Detection/classification tasks do not need a generative adapter. A linear head over the base hidden state is far cheaper than a LoRA, and riding the existing control-token and slot machinery means it composes, serves and activates exactly like any other adapter — no separate endpoint, no custom decode path.
Provenance
Port of
granite-switch-stagingPR #108 (feature/switched-classifier, heade7f6bff), squashed into one commit. That commit's tree is byte-identical to the staging PR head, so this is a content-faithful port rather than a re-implementation.A second commit corrects documentation found stale while porting: the tutorial's Step 3 paragraph named two
config.jsonfields that do not exist insrc/. It now listsadapter_kindsandclassifier_label_token_idsas the stored fields, with the per-slot label counts and control-token ids derived from them. The same stale text is still present in staging and wants the same fix there.Scope note: this PR is the feature code only. Classifier training infrastructure and trained safety-guard artifacts live on a separate internal branch and are not included here.
Testing
Run on this branch; GPU suites on H100 nodes.
tests/unit/tests/composer/tests/composer/test_skinning_equivalence.py(vLLM, GPU)tests/hf/test_classifier.pytests/vllm/test_classifier_chunked_prefill.py(GPU)All 17 classifier tests pass (3 composer, 11 HF, 3 vLLM).
The 2 unit failures are
tests/unit/test_asr.py::TestMonoAndResample::test_resample_real_vllm_backend[8000-16000]and[44100-16000], bothModuleNotFoundError: No module named 'resampy'— a missing optional audio dependency in the local venv, in ASR code this change does not touch. Flagged rather than proven pre-existing: no baseline run onmainwas done to confirm they fail there too.All pre-commit hooks pass, including SPDX headers on the two new
classifier.pyfiles and the link validator on the new tutorial.