Proxmox cluster recovery

A Proxmox cluster without quorum protects its configuration by becoming read-only; forcing votes before preservation can create split-brain state.

Proxmox uses corosync for membership and pmxcfs for a replicated configuration filesystem. Failed nodes, changed networks, lost corosync.conf or divergent cluster databases can make /etc/pve read-only and hide VM definitions.

Understand the symptom

Recovery must establish one authoritative cluster configuration without overwriting unique VM definitions or storage mappings.

WHY

What is happening

Quorum prevents conflicting writes when nodes cannot agree on membership. Temporary expected-vote changes can enable repair on a known surviving node, but using them on divergent nodes can create competing histories. VM disks may remain intact while pmxcfs metadata is unavailable.

What you may see

Symptoms that change the recovery path.

01

/etc/pve is read-only

The node lacks quorum and refuses cluster configuration writes.

02

corosync.conf is missing or wrong

Membership or ring addresses no longer represent the surviving cluster.

03

VMs disappear from the GUI

Configuration files are absent even though disks remain on storage.

04

Nodes form separate views

Network or database divergence creates split-brain risk.

Protect the source

Actions that can make the situation worse.

01

Forcing quorum on multiple partitions creates conflicts

Different sides can assign duplicate VM IDs or change shared storage.

02

Rejoining can overwrite local cluster state

Unique configurations on the joining node may be replaced.

03

Creating new VMs can reuse existing disks or IDs

Hidden definitions do not mean the underlying data is free.

Do this now

Safe steps before evaluation.

01

Stop cluster reconfiguration

Pause joins, removals, expected-vote changes and new VM creation.

02

Preserve every node

Copy pmxcfs database, /etc/pve, corosync files, logs and VM configs from all members.

03

Inventory storage and IDs

Record VM IDs, disk paths, Ceph or shared-storage mappings and node ownership.

04

Choose a trusted source

Compare timestamps and configuration completeness before any node becomes authoritative.

NEPFOSYS process

How a data-first evaluation proceeds.

01

Cluster-history comparison

NEPFOSYS maps memberships, config versions, pmxcfs state and VM definitions.

02

Storage-to-config reconciliation

Existing virtual disks and containers are matched to preserved configuration copies.

03

Protected cluster rebuild

A single authoritative state is tested before nodes and workloads are reintroduced.

A clear next step

Preserve the original device before another repair or recovery attempt.

Start a case with the exact symptoms, incident timeline, prior attempts, encryption details, and the files that matter most. NEPFOSYS will route the case to the appropriate DFW intake or nationwide mail-in path.

Start with a case review

Questions, answered

What customers ask before starting.

What does pvecm expected 1 do?

It temporarily changes the expected vote count so one surviving node can become quorate; it should not be used casually on multiple divergent nodes.

Are VM disks lost when /etc/pve is empty?

Not necessarily. Configuration may be missing while disks remain on local, LVM, ZFS, Ceph or shared storage.

Can I remove and re-add every node?

Only after preserving cluster and VM configuration. Rejoining changes node identity and can overwrite local cluster state.

DFW local · Nationwide mail-in

Protect the device. Start with the facts.

Start my recovery case