Skip to content

KVM: Support multi vlan trunk NICs - #14166

Draft
Pearl1594 wants to merge 27 commits into
mainfrom
support-multi-vlan-guestnet
Draft

Pearl1594 wants to merge 27 commits into
mainfrom
support-multi-vlan-guestnet

Conversation

@Pearl1594

@Pearl1594 Pearl1594 commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Description

This PR adds support for multi-vlan trunk NICs

doc PR: apache/cloudstack-documentation#689

https://cwiki.apache.org/confluence/spaces/CLOUDSTACK/pages/451972290/Multi+VLAN+trunk+NICs+support+in+KVM

  • Added multi-VLAN trunk NIC support for KVM: a single VM NIC can now be associated with multiple existing, ordinary single-VLAN guest networks simultaneously, delivered via libvirt's native XML (libvirt ≥11.0.0) or a manual bridge vlan add/del fallback on older libvirt.
  • New nic_network_map table and nics.multi_network flag to track a NIC's additional network associations, fully additive, zero behavior change for any NIC with no associations.
  • Host capability tracking: new vlan.filtering.enabled and vlan.trunk.xml.supported host details, surfaced via HostResponse, gating placement, deployment, and migration decisions.
  • New admin-only APIs: associateNetworkToNic and disassociateNetworkFromNic, with per-association IP support and duplicate-VLAN/CIDR-overlap validation.
  • VM deploy and VNF appliance support via a new nicnetworkslist parameter, letting a trunk NIC be requested directly at deploy time; VNF management NICs are explicitly blocked from being trunked.
  • DHCP delivery to every associated network, not just the primary, including correct behavior for VPC-tier associations sharing one virtual router.
  • Guest-side VLAN discovery via a new nic-vlan-mapping metadata entry, letting a VNF or guest script learn its associated networks' VLAN tags without hardcoding.
  • Static NAT, Port Forwarding, and Load Balancing rules widened to target an address on any of a NIC's associated networks, not just its primary.
  • Usage and billing events extended to bill every network a trunk NIC is associated with, not only its primary; a related pre-existing billing bug found and queued as a separate fix. (Usage: Fix usage_network_offering row mismatches on nic removal and default-nic swap #14341)
  • Migration support: destination-capability-driven interface XML regeneration so a VM can move between hosts with different bridge models or libvirt capabilities; one real cross-boundary migration bug found and fixed.
  • Dropped the original same-VPC-only restriction on trunk associations; a NIC can now be associated with tiers of different VPCs, since per-network access checks and VPC ACL enforcement already apply correctly regardless of VPC membership.
  • UI support: NICs tab associate/disassociate with a loop-risk warning, deploy wizard NIC grouping, VNF wizard multi-select, host and zone readiness indicators, and migration dialog unsuitability reasons.
  • Fixed a pre-existing, unrelated UI bug where a previously selected network was lost when paging through the network list in the deploy wizard.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Screenshots (if appropriate):

How Has This Been Tested?

Core trunk mechanism (KVM delivery)

  • Trunk NIC wire-level delivery confirmed via bridge vlan show: primary network native/untagged, each associated network tagged-only, on both the native-libvirt-XML path and the manual bridge vlan add fallback path.
  • Real cross-host and same-host guest connectivity confirmed (ARP + ping) on associated networks, including a VM with two independently-trunked NICs on different network sets simultaneously.
  • Host bridge-readiness and libvirt trunk-XML capability detection confirmed via host details and the Hosts list UI badge.

Association lifecycle

  • Live associateNetworkToNic / disassociateNetworkFromNic on a Running VM, including re-association after a prior disassociation.
  • changeNicPrimaryNetwork on a stopped VM, confirmed via console that the guest correctly reacquired the new primary's address via DHCP, plus a real ping round-trip after restart.
  • nics.multi_network flag correctly set and cleared across the full associate/disassociate lifecycle.

Cross-VPC trunk associations

  • created a second VPC, associated one of its tiers onto a NIC whose primary was a tier of a separate VPC, confirmed via API (trunked:true, correct associated network) and via bridge vlan show on the host.
  • Real cross-host guest-to-guest ping (0% loss) between the trunk NIC's associated-network VLAN sub-interface and a probe VM on the second VPC's tier.

VM deploy and VNF

  • Deploy-time nicnetworkslist trunk creation, including a VM mixing trunk and ordinary NICs in one deploy.
  • VNF management-NIC-cannot-be-trunked rejection, both at deploy time and via live associate.
  • A 3-NIC VNF template with two independently-trunked data-plane NICs, deploy + wire-level + DHCP + ping all confirmed for both.
  • Static NAT, Port Forwarding, and Load Balancing rules created and confirmed reachable against a trunk NIC's associated-network address.
  • Plain/legacy non-trunk VM regression pass: deploy, stop/start, PF/LB/Static NAT, DHCP entries and NIC state confirmed byte-identical before and after.

Network type coverage

  • L2, Shared (no security groups), and VPC-tier trunk associations all deploy- and wire-level confirmed on one VM covering all three in a single test pass.
  • CIDR-overlap rejection confirmed at both the live-associate and deploy-time paths.

DHCP correctness

  • trunk NIC's associated networks receive a DHCP entry
  • Confirmed DHCP correctness for VPC-tier trunk associations, where a primary and association share one VPC virtual router: both leases obtained simultaneously on the same NIC, verified live. Also regression-checked against L2 and Shared network types to confirm they're unaffected.

Usage and billing

  • Full usage/billing test pass run against the real nightly usage job, hour-level figures hand-verified against raw event timestamps: network-offering-change propagation, deploy-time trunk billing, changeNicPrimaryNetwork role swap, and live associate/disassociate billing immediately rather than waiting for next VM start, all confirmed passing.
  • Found and fixed a real, pre-existing (non-trunk-specific) bug in default-NIC-swap billing that silently wiped billing for the nic becoming default.

Migration

  • Full matrix run live across mixed-capability clusters: intra-cluster boundary crossing in both directions, trunk NICs correctly hard-blocked from crossing a non-filtering-enabled boundary, cross-cluster migration forcing storage motion, a multi-NIC VM (two independently trunked NICs) migrated cross-cluster with each NIC's wire configuration verified independently, host-readiness gating confirmed via both the migration host list and fully-automatic planner-driven migration, and a migration crossing both the libvirt-capability and bridge-readiness boundaries at once.
  • legacy-to-shared-bridge migration for ordinary (non-trunk) multi-NIC VMs in both directions, single-NIC and multi-NIC.
  • Real post-migration connectivity confirmed (0% loss) on both primary and associated VLANs.

AssignVirtualMachine to another account

  • tested moving a VM from admin to a new account having a trunked nic (1 primary network and 1 associated network)

UI

  • NICs tab: trunked badge, associated-networks list with disassociate action, admin-gated Associate Network modal.
  • Deploy wizard and VNF deploy wizard: multi-network grouping control with loop-risk warning, confirmed not to regress a non-grouped deploy.
  • Migration dialog: unsuitable-host tooltip confirmed showing the correct reason.
  • Host and zone readiness badges/rollup confirmed live.

How did you try to break this feature and the system with this change?

shwstppr and others added 2 commits September 14, 2026 15:29
Adds a 4.23.0 to 24.0.0 upgrade path (squashed from sb/upgradepath-424:
engine-schema: upgrade path for 24.0.0, fix CS version, fix upgrade
unit tests for cutover, fix imports).
@github-actions

Copy link
Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@codecov

codecov Bot commented Sep 14, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 45.06274% with 1007 lines in your changes missing coverage. Please review.
✅ Project coverage is 19.99%. Comparing base (1a48a87) to head (6da1e9a).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
...ain/java/com/cloud/network/NetworkServiceImpl.java 77.94% 55 Missing and 31 partials ⚠️
...ommand/admin/network/AssociateNetworkToNicCmd.java 0.00% 61 Missing ⚠️
...cloudstack/api/response/NicNetworkMapResponse.java 0.00% 55 Missing ⚠️
...cloud/hypervisor/kvm/resource/BridgeVifDriver.java 64.42% 45 Missing and 8 partials ⚠️
.../src/main/java/com/cloud/vm/UserVmManagerImpl.java 44.21% 46 Missing and 7 partials ⚠️
...mand/admin/network/ChangeNicPrimaryNetworkCmd.java 0.00% 43 Missing ⚠️
...d/admin/network/DisassociateNetworkFromNicCmd.java 0.00% 43 Missing ⚠️
...rc/main/java/com/cloud/vm/dao/NicNetworkMapVO.java 29.31% 41 Missing ⚠️
...loudstack/api/command/user/vm/BaseDeployVMCmd.java 61.00% 27 Missing and 12 partials ⚠️
...in/java/com/cloud/vm/dao/NicNetworkMapDaoImpl.java 0.00% 38 Missing ⚠️
... and 49 more
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #14166      +/-   ##
============================================
+ Coverage     19.91%   19.99%   +0.08%     
- Complexity    20199    20424     +225     
============================================
  Files          6373     6385      +12     
  Lines        577230   579197    +1967     
  Branches      70696    71037     +341     
============================================
+ Hits         114950   115810     +860     
- Misses       449713   450718    +1005     
- Partials      12567    12669     +102     
Flag Coverage Δ
uitests 3.69% <ø> (-0.02%) ⬇️
unittests 21.27% <45.06%> (+0.09%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Sep 14, 2026 •

Copy link
Copy Markdown

🔴 Test Coverage Grade: D — Marginal

Metric Value
Line coverage 24.94%
Branch coverage 19.23%

Grade Scale

Grade Line Coverage Meaning
🟢 A ≥ 80% Excellent - this code sleeps well at night 😴
🟡 B 60-79% Good - almost there, don't stop now 😉
🟠 C 40-59% Acceptable - your code is wearing a seatbelt, but no airbags 😬
🔴 D 20-39% Marginal - boldly shipping where no test has gone before 🖖
⛔ F < 20% Failing - tests? what tests? 🔥

Branch coverage is shown as a secondary signal. Grade is determined by line coverage.
View full Actions run

@github-actions

Copy link
Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

The (nic_id, network_id) unique key rejected re-associating a network
after a prior disassociate, since the old row's uniqueness survives as
a soft-deleted (removed) row. Switch to a plain index so re-association
after disassociation works, relying on application-level checks (not
the DB) to reject a true duplicate live association.
…fallback

Trunk NIC VLAN membership is delivered by reusing the existing guest
bridge with vlan_filtering=1: the primary VLAN gets native/untagged
membership, associated VLANs get tagged-only membership, via libvirt's
<vlan trunk='yes'> element where supported, or manual `bridge vlan add`
on older libvirt. Adds a live membership update path
(UpdateNicVlanMembershipCommand/Answer + its KVM wrapper) so an
association change on a running VM is pushed to the host without a
restart, and a non-blocking diagnostic warning when the physical
uplink is missing a VLAN a trunk NIC needs (CloudStack no longer
manages the uplink's own VLAN membership - that is the operator's
responsibility).
- New UsageEventUtils.publishNicNetworkOfferingUsageEvents() emits one
  usage event per network a nic bills against (primary + any trunk
  associations), replacing the old single-event publishUsageEvent at
  every NIC-billing call site
- Ordinary non-trunk nics still produce exactly one event, unchanged
- New nullable usage_network_offering.network_id column disambiguates
  rows when a trunk nic's networks share one network offering
- Backfills network_id on a nic's own still-open usage row(s) the
  moment it first converts to a trunk nic; historic rows otherwise
  left alone
- New NicNetworkMapResponse + trunked/associatednetworks fields on
  NicResponse
- Populated in listNics, listVirtualMachines, and the
  associate/disassociate/change-primary command responses
- Trunk status derived from nic_network_map rows rather than a new
  column on the wide UserVmJoinVO view
- ASSOCIATED_NETWORKS/TRUNKED, needed by the previous commit's
  NicResponse fields
Core capability:
- New associateNetworkToNic/disassociateNetworkFromNic/changeNicPrimaryNetwork
  admin APIs and associateNetworksToNic (programmatic, used by deploy)
- A nic keeps one primary network plus any number of additional
  associated networks (nic_network_map), delivered as a VLAN trunk
- changeNicPrimaryNetwork only allowed while the Instance is stopped
- Rejects associating a network already reachable via another of the
  VM's nics, or with an overlapping CIDR to an existing association
- Disassociating a network is blocked while an active PF/Static
  NAT/LB rule targets its allocated IP

PF/Static NAT/LB support for associated networks:
- NetworkModel.getNicAndIpInNetwork resolves a nic/guest-ip pair via
  the nic's primary match or a trunk association, reused by
  RulesManagerImpl and LoadBalancingRulesManagerImpl so these rules
  can target a trunk nic's associated networks, not just its primary

Deployment planning:
- Hosts without VLAN filtering enabled are excluded up front for any
  VM with a multi-VLAN trunk nic, with a reactive fallback exclusion
  during start as a backstop
- Removing a nic cleans up any trunk associations it held as primary;
  deleting a network is blocked while still associated as a secondary

Metadata:
- A trunk nic's associated-network VLAN tags are exposed to the guest
  via the metadata service (nic-vlan-mapping file), gated by a new
  account-scoped config key, off by default
- EVENT_NIC_NETWORK_ASSOCIATE/DISASSOCIATE/PRIMARY_NETWORK_CHANGE,
  needed by the previous commit's @actionevent annotations
- New nicnetworkslist deploy param (nicnetworkslist[N].networkids,
  first id primary) lets a VM be deployed with trunk nics directly,
  mutually exclusive with networkids/iptonetworklist/vApp nicnetworklist
- Also carries per-entry ip4address/ip6address for the primary and
  ip4addresses/ip6addresses (comma-separated) for its associated
  networks - a network with no requested IP auto-allocates
- VNF: rejects a management-device nic from being requested as a
  trunk, both at deploy time and via the live associate API
- checkNoActiveRulesOnAssociation used findByIpAndNetworkId, which
  matches on a public IP's own address - the association's IP is a
  guest IP, so this never matched and the guard silently let
  disassociation through even with an active static NAT rule
- Use findByAssociatedVmIdAndVmIp instead, which matches the actual
  static NAT target mapping
- validateVnfApplianceTrunkNics only ran at deploy time
  (nicnetworkslist) - the live associateNetworkToNic/associateNetworksToNic
  path had no equivalent check, so a VNF's management nic could be
  trunked after deploy
- New single-nic validateVnfApplianceTrunkNic, called from
  associateNetworksInternal so both entry points are covered
- No-op for any non-VNF template
- changeNicPrimaryNetwork updated networkId and the IPv4/IPv6 address
  but left gateway, netmask, broadcast/isolation uri, and IPv6
  gateway/cidr pointing at the old primary network - wrong
  indefinitely, since nothing else ever recomputes them
- New applyNetworkAddressingToNic refreshes them from the new primary
  network, mirroring the field derivation used when a nic is first
  created
- disassociateNetworkFromNic now clears nics.multi_network once
  nic_network_map has no remaining rows for that nic
- Without this, listNics/listVirtualMachines kept reporting trunked=true
  and the deployment planner kept restricting the VM to VLAN-filtering-
  capable hosts even after all associations were removed
- New CommandSetupHelper.createDhcpEntryCommand overload takes addressing
  explicitly instead of reading it off a NicVO, since a nic's own DB row
  only ever carries its primary network's IP/gateway
- NetworkServiceImpl sends the entry directly to the associated network's
  router(s) on live associate/disassociate, bypassing the
  DhcpServiceProvider pipeline (which re-resolves the nic from the DB and
  so can never see anything but the primary IP)
- UserVmManagerImpl.finalizeStart converges a nic's associated-network
  DHCP entries with its current nic_network_map rows on every VM start,
  covering associations made while the VM was stopped and cleaning up
  entries for associations removed while stopped
- New NicNetworkMapDao.listRemovedByNicId to support that cleanup
- Add missing VlanTrunkMigrationHelper import in VirtualMachineManagerImpl
- Fix AssignLoadBalancerTest's NetworkModelImpl spy to stub the new
  getNicAndIpInNetwork method used by LoadBalancingRulesManagerImpl,
  matching the existing getNics() stub it replaced
…ommand

replaceVlanTrunkInterfaces iterates VirtualMachineTO.getNics(), which
the test's bare mock returned null for (unstubbed), unlike every real
VirtualMachineTO built via toVmTO, which always sets nics. Stub it the
same way getDisks() already was.
…ion rule/listing support

- VM deploy/VNF UI: nicnetworkslist grouping in DeployVM/DeployVnfAppliance,
  zone-flag gating in NicsTab/NicsTable/host list, MigrateWizard unsuitable-
  reason tooltip
- Migration: expose getMultiNetworkNicUnsuitableReason via
  findHostsForMigration/HostForMigrationResponse; destination-capability-
  driven bridge/VLAN-XML rewrite in LibvirtMigrateCommandWrapper now also
  covers a destination-only vlan_filtering boundary, not just the source's
- DHCP: VR full-rebuild (createDhcpEntryCommandsForVMs) now includes a nic's
  trunk associations, not just its primary network
- Billing: live associate/disassociate on a Running Instance now bills
  immediately instead of waiting for the next VM start
- Association API hardening: admin-only authorization on associate/change-
  primary/disassociate, access check on requested networks, CIDR-overlap
  error includes network names
- listNics/listVirtualMachines now also match a nic via its trunk
  associations, not just its primary network, so Static NAT/Port Forwarding/
  Load Balancing pickers can target an associated network's IP, labeled
  distinctly from primary/secondary IPs
@weizhouapache

Copy link
Copy Markdown
Member

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 19456

@Pearl1594

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@Pearl1594 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19457

@Pearl1594

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@Pearl1594 Pearl1594 changed the title [WIP] KVM: Support multi vlan trunk NICs KVM: Support multi vlan trunk NICs Oct 7, 2026
@sonarqubecloud

sonarqubecloud Bot commented Oct 7, 2026

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
38.0% Coverage on New Code (required ≥ 40%)
C Reliability Rating on New Code (required ≥ B)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@Pearl1594

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@Pearl1594 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19458

@Pearl1594

Copy link
Copy Markdown
Contributor Author

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@Pearl1594 a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants