Ceph BlueStore recovery

A missing Ceph BlueStore DB or WAL device can strand an OSD even when its main data device still holds objects.

BlueStore stores object data on the main block device and can place RocksDB metadata or WAL on faster auxiliary devices. Losing block.db, block.wal, LVM metadata or device labels can prevent OSD activation and leave placement groups degraded.

Understand the symptom

BlueStore needs object data and its metadata database to agree at the same OSD epoch.

WHY

What is happening

Recovery maps the OSD ID and FSID, block labels, BlueFS extents, RocksDB files, LVM tags and cluster maps. The main device can contain data blobs that are difficult to identify without the missing metadata. Another replica may be safer, but only after confirming placement-group state and avoiding a destructive recovery wave.

What you may see

Symptoms that change the recovery path.

01

OSD will not activate

The service cannot locate or open block, block.db or block.wal.

02

BlueFS or RocksDB errors

Metadata files, extents or transactions are missing or corrupt.

03

LVM symlink points nowhere

A logical volume or device identity changed after hardware or host work.

04

PGs are degraded or incomplete

The cluster lacks enough usable replicas for one or more object histories.

Protect the source

Actions that can make the situation worse.

01

ceph-volume zap erases labels and metadata

Reusing the device can destroy the OSD identity needed for recovery.

02

Marking lost can discard object history

The cluster may accept other replicas even when they are stale or incomplete.

03

BlueStore repair writes metadata

Repair tools can change the only copy and should be tested on images.

Do this now

Safe steps before evaluation.

01

Freeze OSD replacement

Stop zap, purge, redeploy, rebalance and automated orchestration for affected devices.

02

Map every device

Record OSD ID, FSID, serials, LVM tags, block links, host and bay positions.

03

Preserve cluster evidence

Save OSD and monitor maps, PG queries, crash logs, keyrings and crush configuration.

04

Image all OSD components

Acquire main, DB and WAL devices before repair or activation attempts.

NEPFOSYS process

How a data-first evaluation proceeds.

01

OSD component mapping

NEPFOSYS correlates labels, BlueFS extents, RocksDB state, LVM tags and cluster history.

02

Replica and object analysis

Other OSD copies and placement-group histories are compared before source modification.

03

Protected extraction

Viable OSD activation or object-level recovery is tested on device copies and validated against priorities.

A clear next step

Preserve the original device before another repair or recovery attempt.

Start a case with the exact symptoms, incident timeline, prior attempts, encryption details, and the files that matter most. NEPFOSYS will route the case to the appropriate DFW intake or nationwide mail-in path.

Start with a case review

Questions, answered

What customers ask before starting.

Is block.db optional?

BlueStore can colocate metadata on the main device, but when an OSD was created with a separate DB device, losing it can remove essential metadata.

Can I create a new DB device for the old OSD?

A blank device does not contain the old RocksDB state. Preserve and analyze the original layout before migration or repair.

Should I let Ceph rebuild from replicas?

Only after confirming enough healthy, current replicas exist and preserving the failed OSDs. Rebalancing can increase load and make older copies harder to analyze.

DFW local · Nationwide mail-in

Protect the device. Start with the facts.

Start my recovery case