Ceph monitor recovery

A corrupt Ceph monitor store can hide cluster maps even when OSD data remains on disk.

Ceph monitors maintain consensus about cluster maps, authentication and membership. Corruption, missing monitor data or loss of enough monitors can prevent quorum and make management unavailable while OSDs still contain object data.

Understand the symptom

Monitor recovery must restore authoritative cluster identity, not create a look-alike empty quorum.

WHY

What is happening

A monitor store contains maps and consensus history tied to the cluster FSID. Ceph documents rebuilding monitor state from surviving OSDs when stores are lost, but that procedure depends on the original cluster identity and available OSD maps. Mixing FSIDs or initializing fresh monitor state can orphan the data-bearing devices.

What you may see

Symptoms that change the recovery path.

01

No monitor quorum

Clients and administrative commands cannot obtain an authoritative cluster map.

02

Monitor store I/O or checksum errors

The local monitor database cannot be read consistently.

03

Monitors disagree on epochs

Members have divergent or incomplete consensus histories.

04

OSDs remain present but unavailable

Data devices exist while control-plane maps are missing.

Protect the source

Actions that can make the situation worse.

01

New cluster initialization changes identity

Existing OSDs and authentication state may no longer match.

02

Purging monitor data removes maps

The most complete surviving epoch may be lost.

03

Restart loops can compact damaged stores

Repeated operations may change the evidence needed for reconstruction.

Do this now

Safe steps before evaluation.

01

Stop destructive orchestration

Pause redeploy, purge and cluster-reset actions.

02

Preserve all monitor stores

Copy monitor directories, logs, configs, keys and maps from every node.

03

Inventory data-bearing services

Record OSD IDs, devices, hosts, pools, CephFS and RBD priorities.

04

Protect OSD media

Do not zap, reformat or adopt devices into another cluster.

NEPFOSYS process

How a data-first evaluation proceeds.

01

Cluster identity mapping

NEPFOSYS correlates FSID, monitor epochs, keyrings, OSD maps and hosts.

02

Isolated monitor reconstruction

Surviving monitor data and OSD-derived maps are tested in a controlled environment.

03

Pool and object validation

Priority RBD, CephFS and object data are checked after quorum is restored.

A clear next step

Preserve the original device before another repair or recovery attempt.

Start a case with the exact symptoms, incident timeline, prior attempts, encryption details, and the files that matter most. NEPFOSYS will route the case to the appropriate DFW intake or nationwide mail-in path.

Start with a case review

Questions, answered

What customers ask before starting.

Are Ceph OSDs useless without monitors?

No. OSDs hold object data, but the cluster needs valid maps and identity to locate and serve it correctly.

Can monitor state be rebuilt from OSDs?

Ceph documents an advanced recovery path when monitor stores are lost, but preserve all stores and cluster identity before attempting it.

Should I create a new Ceph cluster with the same disks?

No. New initialization can overwrite metadata and create an incompatible FSID and map history.

DFW local · Nationwide mail-in

Protect the device. Start with the facts.

Start my recovery case