Skip to content

feat: hybrid interop tests - #4176

Open
NoelStephensUnity wants to merge 9 commits into
develop-3.x.xfrom
feat/hybrid-interop-tests
Open

NoelStephensUnity wants to merge 9 commits into
develop-3.x.xfrom
feat/hybrid-interop-tests

Conversation

@NoelStephensUnity

@NoelStephensUnity NoelStephensUnity commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Purpose of this PR

This PR adds 29 hybrid prefab test cases, and fixes 9 issues the tests found: 6 NGO issues in hybrid prefab mode, 2 NGO scene migration issues that affect every NGO project, and 1 test harness issue. The results back the hybrid mode compatibility matrix (N4E prediction, unified remotes and GhostFields used with NGO RPCs, NetworkVariables and ownership).

PR Scope:

New tests (Tests/Runtime/Unified, UnifiedHost / UnifiedServer):

Combination Result
Owner-predicted hybrid prefab Predicted and re-simulated on the owning client. The host and server never re-simulate.
NGO RPC sent from PredictionUpdate Sent again for every re-simulated tick. Sent once per tick when gated on IsFirstTimeFullyPredictingTick.
NetworkVariable written from PredictionUpdate The owner's value moves backwards when older ticks re-simulate. Only moves forward when gated.
NetworkVariable read in PredictionUpdate Not rolled back, so a re-simulated tick can read a different value than its first run.
Tick-stamped NetworkVariable (applied when the predicted tick reaches its stamp) The same value for every re-simulation of a tick.
Unified remote to NGO RPC to unified remote, and NGO RPC to unified remote to NGO RPC Both complete.
NGO ownership change The ghost's owner follows, and the new owner predicts.

CreateHybridPrefab takes an optional GhostMode so tests can create predicted and owner-predicted hybrid prefabs.

Fixes:

  • UnifiedBootstrap re-registers the worlds it created for other NetworkManagers. Every ClientServerBootstrap constructor clears N4E's ServerWorlds and ClientWorlds, and each NetworkManager creates its own bootstrap. Remote methods send through those lists, so with several NetworkManagers in one process, server-to-client remotes were dropped.
  • NetworkObjectBridge defaults a GhostObject to interpolation only when the bridge is first added (Reset). It no longer does it on every OnValidate, which reverted prediction set in the inspector.
  • NGO spawn ownership and ownership changes set GhostObject.OwnerNetworkId. Before this, an owner-predicted ghost was never given an owner, so no client predicted it.
  • NetworkManager shutdown disposes only its own world instead of calling World.DisposeAllWorlds(). This means one peer shutting down no longer breaks the others.
  • A client that disconnects itself now receives its own ClientDisconnected event. UnifiedNetcodeTransport.DisconnectLocalClient only requested the N4E disconnect, which N4E reports after NGO's shutdown has stopped listening. It now notifies immediately, as UnityTransport does.
  • A hybrid prefab instance that is part of a client's initial synchronization, but only spawns once its ghost arrives after the synchronization completed, is now moved into its server-side scene like the rest of the synchronized NetworkObjects. It stayed in the active scene.

NGO scene migration fixes (affect every NGO project, with CHANGELOG entries, and are also ported to develop-2.0.0):

  • A scene migration (SceneEventType.ObjectSceneChanged) was sent to every connected client, including clients that did not observe the migrated NetworkObject and logged "Trying to synchronize NetworkObjectId but it was not spawned". In client-server mode it is now only sent to the clients that observe at least one migrated NetworkObject, and each client is only sent the NetworkObjects it observes. Distributed authority is unchanged.
  • A NetworkObject shown with NetworkShow after it migrated into another scene, while hidden from that client, was instantiated in the client's active scene, since the client was never sent that migration. The client now moves the spawned NetworkObject into its server-side scene, as it already does during its initial synchronization.

Test harness fix:

  • The integration test hook that registers hybrid instances as pending ghosts also registered the server's own instance. On a dedicated server, which is never a connected client, that moved the instance into the DontDestroyOnLoad scene, and the server then sent every client a scene migration for an object none of them had spawned ("Trying to synchronize NetworkObjectId but it was not spawned", SceneEventData.cs:1302). The hook now skips the server, as the runtime does.

NetworkObjectDontDestroyWithOwnerTests (6 cases) and NetworkSpawnManagerTests (4 cases) now run in hybrid prefab mode. Both failed before the shutdown fix. DisconnectTests (4 cases) and PeerDisconnectCallbackTests (12 cases) also run in hybrid prefab mode. Their client-initiated cases failed before the disconnect fix.

New NetworkObjectSceneMigrationObserverTests (4 tests, Host / Server / UnifiedHost / UnifiedServer) cover both scene migration fixes and the late synchronization fix.

Out of this PR's scope:

  • GatherInput, PredictedPhysicsUpdate and prediction switching combined with NGO features (not covered yet).
  • Unified parenting (GhostObject.ParentReplication is not in N4E 7.0.0; NGO will defer parenting to it in a separate PR).
  • InstantiateAndSpawn and hybrid player prefabs not selecting the world (to be resolved with GhostObject.DelaySpawning / Spawn()).

Jira ticket

MTT-16222

Changelog

  • Fixed: Issue where moving a NetworkObject into another scene made the clients that did not observe it log "Trying to synchronize NetworkObjectId but it was not spawned". The scene migration is now only sent to the clients that observe the NetworkObject.
  • Fixed: Issue where a NetworkObject that was moved into another scene while hidden from a client spawned in that client's active scene when it was shown with NetworkShow, instead of the scene it is in on the server.

Documentation

  • No documentation changes or additions were necessary.

Testing & QA (How your changes can be verified during release Playtest)

Headless PlayMode runs on 6000.7.0b1 (N4E 7.0.0) with the unified job's scripting defines, filtered to the hybrid fixtures from #4172 plus the fixtures in this PR.

Run Passed Failed
UNIFIED_TESTS=true 286 0
UNIFIED_TESTS unset, all of Unity.Netcode.RuntimeTests 5644 96 (all in NetworkVariableTests, which fails the same 96 cases before these changes, at d4fda26e8. It only fails with the unified scripting defines and UNIFIED_TESTS unset, a combination CI does not run)
testproject scene management tests (NetworkSceneManager*, NetworkObjectSceneMigrationTests, DontDestroyOnLoad, scene event tests), UNIFIED_TESTS unset 602 0
NetworkObjectSceneMigrationObserverTests, UNIFIED_TESTS=true and unset 16 0
Changed fixtures, UNIFIED_TESTS=true, 3 runs 44 each 0
NetworkObjectDontDestroyWithOwnerTests, UNIFIED_TESTS=true, 3 runs 6 each 0
Disconnect fixtures, UNIFIED_TESTS=true, 3 runs 16 each 0
Disconnect fixtures, UNIFIED_TESTS unset 16 0

Functional Testing

Manual testing :

  • Manual testing done

Automated tests:

  • Covered by existing automated tests
  • Covered by new automated tests

Does the change require QA team to:

  • Review automated tests?
  • Execute manual tests?
  • Provide feedback about the PR?

Up-port

Not needed. Hybrid prefab mode only exists on develop-3.x.x. The two NGO scene migration fixes are ported to develop-2.0.0 in #TBD.

Backports

Not needed.

CreateHybridPrefab takes an optional GhostMode. Predicted modes are applied after NetworkObjectBridge is
added, because its editor OnValidate resets the supported ghost modes to interpolated.

HybridPredictionTests: an owner-predicted hybrid prefab predicts and re-simulates on the owning client.

HybridInteropTests combines N4E and NGO features on one hybrid prefab:
- An NGO RPC sent from PredictionUpdate repeats for re-simulated ticks, and sends each tick once when gated
  on IsFirstTimeFullyPredictingTick.
- Unified remote to NGO RPC to unified remote, and NGO RPC to unified remote to NGO RPC.
- A NetworkVariable read during prediction is not tick-aligned, while a tick-stamped value applied from its
  stamp tick is consistent across re-simulation.
…on hybrid prefabs

- A NetworkVariable written from PredictionUpdate moves backwards when older ticks re-simulate, and only
  moves forward when gated on IsFirstTimeFullyPredictingTick.
- An NGO ownership change does not change the ghost's N4E owner.
- UnifiedBootstrap registers the worlds it created for other NetworkManagers again after each bootstrap,
  since every ClientServerBootstrap constructor clears N4E's ServerWorlds and ClientWorlds. Remote methods
  send through those lists, so with several NetworkManagers in one process server-to-client remotes were
  dropped.
- NetworkObjectBridge defaults a GhostObject to interpolation only when the bridge is first added, instead
  of on every OnValidate, so prediction enabled on a hybrid prefab is kept.
- NGO spawn ownership and ownership changes set the ghost's owner, so an owner-predicted ghost is
  predicted by its NGO owner.
- NetworkManager shutdown disposes only its own world instead of every world in the process.

NetworkObjectDontDestroyWithOwnerTests and NetworkSpawnManagerTests now run in hybrid prefab mode.
NetworkShowThenClientDisconnects is ignored for hybrid prefabs: a scene migration update can reach a
client before the object's ghost has spawned there.
@NoelStephensUnity
NoelStephensUnity marked this pull request as ready for review October 1, 2026 00:53
@NoelStephensUnity
NoelStephensUnity requested a review from a team as a code owner October 1, 2026 00:53
@u-pr

u-pr Bot commented Oct 1, 2026

Copy link
Copy Markdown

⚠️ Some settings were ignored in '.pr_agent.toml (repository root)' repo settings

Check the configuration's TOML syntax, permitted settings, and size limit. See the error below.


Error message:
Ignored prohibited settings: github_app.handle_pr_actions. Permitted settings were applied.

Configuration content:
# This is the configuration for the u-pr bot
# https://github.cds.internal.unity3d.com/unity/u-pr
# For configuration of this file:
# https://developer.portal.internal.unity.com/catalog/default/component/u-pr/docs/configuration/automatic_runs/

[github_app]
handle_pr_actions = ['opened', 'ready_for_review']  # PR events that auto-run pr_commands
pr_commands       = ["/harness-review"]             # what runs on those events
reviewer_commands = ["/harness-review"]             # what runs when @u-pr is added as reviewer

@u-pr u-pr Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Harness Review

Needs changes

The change adds hybrid prediction and interop coverage and keeps ghost ownership and world disposal scoped to each network manager, but the spawn callback order exposes an inconsistent owner state and two test assertions leave intended behavior unprotected.

Reviewed commit f6fee1d

🤖 Helpful? 👍/👎

Comment thread com.unity.netcode.gameobjects/Runtime/Spawning/NetworkSpawnManager.cs Outdated
Comment thread com.unity.netcode.gameobjects/Tests/Runtime/Unified/HybridInteropTests.cs Outdated
- The ghost owner is now assigned right after SetupOnSpawn, so OnNetworkSpawn sees the same owner on the
  NetworkObject and its ghost. The interop test records the ghost owner in OnNetworkSpawn and asserts it.
- NetworkVariableReadDuringPredictionIsNotTickAligned asserts that a re-simulated tick reads a different value
  than its first prediction, rather than counting reads before the stamp tick.
- NetworkShowThenClientDisconnects is only ignored for UnifiedServer, the fixture where the scene migration
  failure occurs.
@NoelStephensUnity

Copy link
Copy Markdown
Member Author

@u-pr please review again

@u-pr

u-pr Bot commented Oct 1, 2026

Copy link
Copy Markdown

The changes addressing the previous review comments look good:

  • Setting the ghost owner immediately following SetupOnSpawn in SpawnNetworkObjectLocallyCommon ensures OnNetworkSpawn callbacks see the correct ghost ownership.
  • Asserting InconsistentRawTicks > 0 directly verifies that re-simulated ticks receive updated values when NetworkVariable updates arrive without tick alignment.
  • Limiting the ignored test in NetworkObjectDontDestroyWithOwnerTests to !m_UseHost preserves coverage for the UnifiedHost test fixture.

Summary of New Observations

  1. Disposed World Cleanup in UnifiedBootstrap:
    When a NetworkManager shuts down, NetcodeWorld.Dispose() is called, but the world reference remains in s_CreatedWorlds, ServerWorlds, and ClientWorlds until a subsequent bootstrap runs. In scenarios where a client shuts down while the server remains running (or in multi-client environments), remote invocations iterating over ClientWorlds may encounter disposed worlds and throw an ObjectDisposedException. Exposing an unregister method on UnifiedBootstrap to remove the world upon shutdown would keep these static lists consistent and avoid keeping disposed worlds referenced in memory.

  2. Null Safety in NetworkObjectBridge.UpdateGhostOwner:
    Adding a null check for networkManager.NetcodeWorld before accessing .LocalConnection prevents potential NullReferenceExceptions if UpdateGhostOwner is called while the netcode world is not initialized.

  3. Re-simulation Timing in WriteNetworkVariableFromPrediction:
    When gateOnFirstTimeTick is false, the wait condition on line 385 only checks the total number of value changes. If 20 first-time predictions complete before a snapshot triggers re-simulation, OwnerWrittenTickDecreases may still be 0 when WriteNetworkVariableFromPrediction finishes. Waiting for at least one decrease when ungated (analogous to checking ResimulatedTicks > 0 in SendRpcsFromPrediction) prevents intermittent test flakiness.

🤖 Helpful? 👍/👎

Comment thread com.unity.netcode.gameobjects/Tests/Runtime/Unified/HybridInteropTests.cs Outdated
…ation in the backwards-write test

- UpdateGhostOwner leaves the ghost unowned if a host NetworkManager has no NetcodeWorld.
- NetworkVariableWrittenFromPredictionUpdateMovesBackwards keeps writing until a backwards move has happened,
  instead of stopping after 20 changes, which could come before any re-simulation.
@NoelStephensUnity

Copy link
Copy Markdown
Member Author

@u-pr please review again

@u-pr

u-pr Bot commented Oct 1, 2026

Copy link
Copy Markdown

The latest changes in commit d4fda26 look solid and address the previous comments:

  • Host NetcodeWorld null guard: Guarding networkManager.NetcodeWorld != null in NetworkObjectBridge.UpdateGhostOwner safely prevents potential NullReferenceExceptions if ghost ownership updates occur before or during host world initialization.
  • Re-simulation decrease wait condition: Waiting for clientBehaviour.OwnerWrittenTickDecreases > 0 when ungated in WriteNetworkVariableFromPrediction ensures a re-simulation has moved the value backward before concluding the write phase, eliminating intermittent timing flakes.

Review Summary

  • Ghost ownership propagation: Moving UpdateGhostOwner to SpawnNetworkObjectLocallyCommon right after SetupOnSpawn ensures OnNetworkSpawn callbacks see the correct ghost owner network ID, and updating it during ChangeOwnership keeps N4E ghost prediction aligned with NGO object ownership.
  • Prefab configuration: Moving default interpolation configuration to Reset() preserves prediction settings configured in the inspector or in test helpers.
  • Bootstrap world restoration: Re-registering existing worlds in UnifiedBootstrap ensures multi-NetworkManager environments maintain valid lists in ServerWorlds and ClientWorlds. (As a minor note for the planned client disconnect follow-up PR, unregistering disposed worlds from UnifiedBootstrap on shutdown can be tackled there to prevent stale references).
  • Test coverage: The new tests in HybridPredictionTests and HybridInteropTests thoroughly validate ghost prediction, RPC/Remote interop, and tick-stamped NetworkVariable consistency across re-simulations.

Overall, the changes are clean, well-tested, and ready to go.

🤖 Helpful? 👍/👎

@codecov-github-com

codecov-github-com Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 90.00000% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...cts/Runtime/SceneManagement/NetworkSceneManager.cs 88.88% 5 Missing ⚠️
...eobjects/Runtime/SceneManagement/SceneEventData.cs 90.47% 2 Missing ⚠️
@@                Coverage Diff                @@
##           develop-3.x.x    #4176      +/-   ##
=================================================
+ Coverage          78.19%   78.23%   +0.03%     
=================================================
  Files                153      153              
  Lines              26272    26324      +52     
=================================================
+ Hits               20544    20595      +51     
- Misses              5728     5729       +1     
Flag Coverage Δ
NGOv2_project_testproject_ubuntu 76.68% <90.00%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...netcode.gameobjects/Runtime/Core/NetworkManager.cs 79.97% <ø> (ø)
.../Runtime/Messaging/Messages/CreateObjectMessage.cs 86.87% <100.00%> (+0.33%) ⬆️
...ameobjects/Runtime/Spawning/NetworkSpawnManager.cs 73.22% <ø> (ø)
...eobjects/Runtime/SceneManagement/SceneEventData.cs 77.99% <90.47%> (+0.50%) ⬆️
...cts/Runtime/SceneManagement/NetworkSceneManager.cs 79.83% <88.88%> (+0.33%) ⬆️
Components Coverage Δ
com.unity.netcode.gameobjects 78.23% <90.00%> (+0.03%) ⬆️

ℹ️ Need help interpreting these results?

…connects

UnifiedNetcodeTransport.DisconnectLocalClient only requested the N4E disconnect. N4E reports it later, after NGO's shutdown has stopped listening, so the disconnecting client never received its ClientDisconnected event. It now notifies immediately, as UnityTransport does, and ignores N4E's later event. A server-initiated disconnect removes the connection before notifying, so the shutdown it triggers does not notify a second time.

DisconnectTests and PeerDisconnectCallbackTests now run in hybrid prefab mode.
…ntegration tests

The integration test hook that replaces GhostSpawnManager.RegisterGhostBridge also registered the server's own instance. On a dedicated server, which is never a connected client, that moved the instance into the DontDestroyOnLoad scene. The server then sent a scene migration for it to every client, and none of them had spawned it. The hook now skips the server, as the runtime does.

NetworkShowThenClientDisconnects no longer needs to be ignored for UnifiedServer.
…brid spawns into their scene

ObjectSceneChanged was sent to every connected client, including clients that did not observe the migrated NetworkObjects and had never spawned them, which logged "Trying to synchronize NetworkObjectId but it was not spawned". In client-server mode it is now only sent to clients that observe at least one migrated NetworkObject, and each client is only sent the NetworkObjects it observes. Distributed authority is unchanged.

A hybrid prefab instance that is part of a client's initial synchronization can spawn after the synchronization has completed, once its ghost arrives. It missed the step that migrates synchronized NetworkObjects into their server-side scene, and stayed in the active scene. It now runs that step when it spawns.
…r's scene

A client that starts observing a NetworkObject with NetworkShow after the NetworkObject migrated into another scene was never sent that migration, and instantiated the NetworkObject in its active scene. The client now moves a spawned NetworkObject into its server-side scene, as it already does for NetworkObjects that are part of its initial synchronization.

Adds CHANGELOG entries for this and for scene migrations being sent to clients that do not observe the NetworkObject.
@NoelStephensUnity

Copy link
Copy Markdown
Member Author

@u-pr please review again

@u-pr

u-pr Bot commented Oct 2, 2026

Copy link
Copy Markdown

Review Summary

The latest commits (5e8e7e7, b60f7a1, 2897835, and 423c9e2) make substantial improvements to scene migration handling and transport disconnect signaling:

  • Observer-filtered scene migrations: Filtering ObjectSceneChanged targets and payload objects so only observing clients receive scene migrations prevents clients from logging unspawned-object warnings.
  • Scene synchronization for late/shown objects: Calling SynchronizeNetworkObjectScene in CreateObjectMessage (when objects are revealed via NetworkShow) and in GhostSpawnManager (when hybrid instances complete ghost spawning) ensures objects land in their server-side scenes even if they missed the initial migration event.
  • Dedicated server test registration: Skipping pending ghost registration for servers in NetcodeIntegrationTest.RegisterPendingGhost resolves unwanted DontDestroyOnLoad migrations on dedicated servers.
  • Client disconnect handling: Notifying NetworkEvent.Disconnect immediately in UnifiedNetcodeTransport.DisconnectLocalClient aligns behavior with UnityTransport and ensures disconnect callbacks fire before shutdown tears down listeners.
  • Integration test coverage: The new tests in NetworkObjectSceneMigrationObserverTests thoroughly validate observer filtering and scene synchronization across various show/join scenarios.

Findings

  1. Scene Management Guard in GhostSpawnManager (see inline comment):
    In GhostSpawnManager.ProcessGhostPendingSynchronization, m_NetworkManager.SceneManager.SynchronizeNetworkObjectScene(networkObject) is called without verifying m_NetworkManager.NetworkConfig.EnableSceneManagement. In NGO, SceneManager is null when scene management is disabled. Adding the guard (consistent with CreateObjectMessage.cs) prevents a potential NullReferenceException.

  2. Connection ID Cleanup on Server Disconnect (see inline comment):
    In UnifiedNetcodeTransport.OnClientDisconnectFromServer, m_Connections.Remove(connectionEvent.Id.Value) is invoked to prevent DisconnectLocalClient from re-notifying upon shutdown. Because DisconnectLocalClient checks (int)ServerClientId, also removing (int)ServerClientId ensures wasConnected evaluates to false even if connectionEvent.Id.Value differs from ServerClientId.

🤖 Helpful? 👍/👎

var networkObject = NetworkObject.DeserializeAndSpawnObject(serializedObject, reader, m_NetworkManager);

// The synchronization moved the other NetworkObjects into their server-side scenes before this one spawned
if (networkObject != null)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If NetworkConfig.EnableSceneManagement is false, m_NetworkManager.SceneManager is null. Guarding with EnableSceneManagement (consistent with CreateObjectMessage.cs) prevents a potential NullReferenceException.

Suggested change
if (networkObject != null)
if (networkObject != null && m_NetworkManager.NetworkConfig.EnableSceneManagement)

🤖 Helpful? 👍/👎

GetDisconnectMessageFromNetworkStreamDisconnectReason(connectionEvent.DisconnectReason)
);
// Removed before notifying, so the shutdown this triggers does not notify again from DisconnectLocalClient.
m_Connections.Remove(connectionEvent.Id.Value);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In DisconnectLocalClient, wasConnected is determined by removing (int)ServerClientId. If connectionEvent.Id.Value is not equal to (int)ServerClientId, DisconnectLocalClient could still find (int)ServerClientId in m_Connections and invoke a duplicate disconnect event. Have you considered removing (int)ServerClientId here as well?

Suggested change
m_Connections.Remove(connectionEvent.Id.Value);
m_Connections.Remove(connectionEvent.Id.Value);
m_Connections.Remove((int)ServerClientId);

🤖 Helpful? 👍/👎

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant