Ceph cluster recovery
Ceph can continue serving some data while OSDs are down or placement groups are degraded. Automatic remapping and recovery are normal availability features, but they also change where object copies live and can complicate evidence after multiple failures.
Understand the symptom
An OSD can be down but still in the cluster, after which monitors may mark it out and remap placement groups. Degraded PGs may remain readable, while incomplete or unfound objects require surviving histories and replicas. Network faults, full ratios, failed devices, stale maps, and simultaneous node loss must be separated before replacing or reinitializing OSDs.
What you may see
The daemon or host is unavailable while CRUSH still assigns data to that OSD.
The cluster has fewer available replicas than the pool policy expects.
Ceph cannot establish an authoritative object history from the currently available set.
Expected object versions are not present on reachable OSDs.
Protect the source
Large-scale recovery writes can change surviving replicas and increase load during an unstable incident.
Zap, purge, or redeploy operations can erase BlueStore metadata and object data.
Lost-object commands can remove references when the missing OSD or history might still be recoverable.
Do this now
Pause automation, upgrades, balancing, scrubbing, and OSD replacement while the incident is recorded.
Save ceph status, health detail, osd tree, df, pg states, maps, crash reports, and relevant logs.
Keep failed drives, DB or WAL devices, encryption keys, labels, host assignments, and original servers.
Record power, network, fullness, maintenance, replacements, noout flags, and commands that preceded the failure.
NEPFOSYS process
NEPFOSYS correlates maps, PG histories, CRUSH placement, OSD metadata, device health, and business priorities.
Relevant OSD, DB, WAL, and host storage are acquired or held unchanged before destructive remediation.
Recoverable pools, RBD images, filesystems, or priority objects are reconstructed from protected sources.
A clear next step
Start a case with the exact symptoms, incident timeline, prior attempts, encryption details, and the files that matter most. NEPFOSYS will route the case to the appropriate DFW intake or nationwide mail-in path.
Start with a case reviewQuestions, answered
No. Many degraded PGs remain active and readable, but redundancy is reduced and the condition should be stabilized before more failures occur.
Not automatically during a multi-failure recovery. Marking out initiates remapping and should follow a review of capacity, replicas, device condition, and the incident timeline.
Keep the data devices, BlueStore DB and WAL devices, keys, labels, OSD IDs, logs, maps, configuration, and host-to-device assignments.
DFW local · Nationwide mail-in