Repository navigation
snapshot as Abnormal residue on rbd #12002
Description
Activity
Thanks for opening your first issue here! Be sure to follow the issue template!
@lubxun @DaanHoogland @abh1sar Ok, just reproduced and investigated this issue.. so, basically, when snapshot.backup.to.secondary=false, snapshots live only on primary as RBD snapshots.
And as per Ceph docs, an RBD image can't be removed until its snapshots are removed; CloudStack handles this correctly by purging all RBD snapshots before deleting the image. So no orphans on the Ceph side BUT the DB retains stale snapshots and snapshot_store_ref records for these primary-only snapshots even after the data is gone from Ceph.
Since there are no copies elsewhere, these entries become undeletable in the UI (fails with "Problem with condition: state" in DefaultSnapshotStrategy.java#L438).
Proposed fix:
Imo, it's best to clean up snapshot DB records for primary-only snapshots before the volume is deleted as they'll be destroyed with the image anyway.
problem
when use ceph rbd store to primary store. and turn off [snapshot.backup.to.secondary]
take Instance volume snapshot, and then Destroy Instance and expunge,
the snapshot will Abnormal residue.
because Destroy Instance and expunge, will remove rbd image and its snap. after this the cloudstack job go [ StorpoolSnapshotStrategy] to delete snap, however ,this time the snap is not exit
so cloudstack will hung,and the snapshot whill Abnormal residue in MySQL and UI. and can not delete!
versions
CloudStack 4.21.0.0
The steps to reproduce the bug
1.use ceph rbd store to primary store.
2.turn off [snapshot.backup.to.secondary]
3.take a Instance volume snapshot
4.Destroy Instance and expunge,
5. check the snapshot
What to do about it?
to fix this ,you can fix the msyql snapshot ‘status’ ，fix ‘backedup' to 'CreatedOnPrimary'，it may work