Safely Decommission a Node From a K3s Cluster With Longhorn or Local-Path Storage

Safely Decommission a Node From a K3s Cluster With Longhorn or Local-Path Storage

Posted
Safely Decommission a Node From a K3s Cluster With Longhorn or Local-Path Storage
Safely Decommission a Node From a K3s Cluster With Longhorn or Local-Path Storage
By Yevhen Vasylenko
18 min read
Safely Decommission a Node From a K3s Cluster With Longhorn or Local-Path Storage

Removing a node from a K3s cluster is simple when the node only runs stateless workloads. It becomes more dangerous when the node hosts volumes, because Kubernetes can move pods but it cannot magically move local data unless the storage layer supports replication or migration.

This runbook describes a safe approach for decommissioning a K3s node when workloads use either Longhorn or the default local-path provisioner.

Before You Start

Set the node name once and reuse it in all commands.

Let’s assume you have a two-node cluster: one master node and one worker node. Longhorn storage has two replicas configured in the StorageClass, and now you need to remove the worker node from the cluster for decommissioning or replacement.

Migrate Longhorn Storage

Take Backups First

Even though you currently have two replicas, after this migration you will end up with only one replica. Take Longhorn backups for important PVCs before starting.

In Longhorn UI:

Volume -> Backup -> Create Backup

Change Longhorn Default Replica Count to 1

This affects new volumes, not necessarily existing ones.

In Longhorn UI:

Settings -> General -> Default Replica Count -> 1

If you installed Longhorn by Helm, also update your Helm values so it does not revert later:

1
2
3
4
5

defaultSettings:
  defaultReplicaCount: 1

persistence:
  defaultClassReplicaCount: 1

Then upgrade Longhorn with your chart values.

Change Existing Longhorn Volumes to 1 Replica

Check current volumes:

1

kubectl -n longhorn-system get volumes.longhorn.io

For each volume, patch replica count to 1:

1
2
3

kubectl -n longhorn-system patch volumes.longhorn.io <volume-name> \
  --type=merge \
  -p '{"spec":{"numberOfReplicas":1}}'

Or for all volumes:

1
2
3
4
5

for v in $(kubectl -n longhorn-system get volumes.longhorn.io -o jsonpath='{.items[*].metadata.name}'); do
  kubectl -n longhorn-system patch volumes.longhorn.io "$v" \
    --type=merge \
    -p '{"spec":{"numberOfReplicas":1}}'
done

Then watch:

1
2

kubectl -n longhorn-system get volumes.longhorn.io
kubectl -n longhorn-system get replicas.longhorn.io -o wide

You want each volume to have one healthy replica, preferably on the master node.

Disable Longhorn Scheduling on the Slave

In Longhorn UI:

Node -> slave node -> Edit Node and Disks

Set:

1
2

Scheduling: Disable
Eviction Requested: true

Also disable scheduling on the disk under that node.

CLI option:

1
2
3

kubectl -n longhorn-system patch nodes.longhorn.io <slave-node-name> \
  --type=merge \
  -p '{"spec":{"allowScheduling":false,"evictionRequested":true}}'

Then check replicas again:

1

kubectl -n longhorn-system get replicas.longhorn.io -o wide

If you already changed volumes to numberOfReplicas: 1, Longhorn should remove the extra replica on the slave or leave only the selected healthy replica. If a volume’s only remaining replica is still on the slave, do not remove the node yet. In that case, use Longhorn UI to verify the volume replica placement and rebuild or move it to the master first.

Cordon the Slave Node 

1

kubectl cordon <slave-node-name>

Migrate Local-Path Storage

Local-path volumes are different from Longhorn: they are node-local directories, so Kubernetes cannot automatically move the data. The safe approach is to identify those PVCs, stop the pods, copy the data from the slave node to the master node, then recreate or retarget the PV so it points to the new node/path.

Find Workloads Using Local-Path 

1

kubectl get pvc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SC:.spec.storageClassName,VOLUME:.spec.volumeName

Filter local-path:

1

kubectl get pvc -A -o jsonpath='{range .items[?(@.spec.storageClassName=="local-path")]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.spec.volumeName}{"\n"}{end}'

Then check where each PV is located:

1

kubectl get pv <pv-name> -o yaml

Look for:

1
2
3

spec:
  local:
    path: /opt/local-path-provisioner/...

or:

1
2
3

spec:
  hostPath:
    path: /opt/local-path-provisioner/...

Also check node affinity:

1

kubectl get pv <pv-name> -o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].values}{"\n"}'

That tells you which node owns the volume.

Decide Which PVCs Need Migration

You only need to migrate PVCs whose PV is on the slave node.

Example command:

1

kubectl get pv -o custom-columns=PV:.metadata.name,SC:.spec.storageClassName,PATH:.spec.hostPath.path,LOCALPATH:.spec.local.path,NODE:.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].values

If the NODE column shows the slave node, that data must be copied before removing the node.

Stop the Workload Using the PVC

For a Deployment:

1

kubectl -n <namespace> scale deployment <name> --replicas=0

For a StatefulSet:

1

kubectl -n <namespace> scale statefulset <name> --replicas=0

Then verify no pod is using the PVC:

1

kubectl get pods -A -o wide | grep <pvc-name>

For databases, stop the application cleanly before copying data.

Copy the Data From Slave to Master

On the slave node, find the path from the PV:

1

kubectl get pv <pv-name> -o yaml

Example source path:

1

/opt/local-path-provisioner/pvc-xxxx_<namespace>_<pvc-name>

Copy it to the master node.

Example with rsync:

1
2
3

sudo rsync -aHAX --numeric-ids \
  /opt/local-path-provisioner/pvc-xxxx_<namespace>_<pvc-name>/ \
  root@<master-ip>:/opt/local-path-provisioner/pvc-xxxx_<namespace>_<pvc-name>/

If SSH root login is not allowed:

1
2
3

sudo rsync -aHAX --numeric-ids \
  /opt/local-path-provisioner/pvc-xxxx_<namespace>_<pvc-name>/ \
  <user>@<master-ip>:/tmp/<pvc-name>/

Then on master:

1
2

sudo mkdir -p /opt/local-path-provisioner/pvc-xxxx_<namespace>_<pvc-name>
sudo rsync -aHAX --numeric-ids /tmp/<pvc-name>/ /opt/local-path-provisioner/pvc-xxxx_<namespace>_<pvc-name>/

Preserving ownership is important, especially for PostgreSQL, MySQL, Redis, MinIO, etc.

Recreate the PV for the Master Node

This is the cleanest manual method. First, save current PV YAML:

1

kubectl get pv <pv-name> -o yaml > pv-old.yaml

Edit it into a new static PV bound to the same PVC, but with master node affinity. Important fields to keep/change:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

apiVersion: v1
kind: PersistentVolume
metadata:
  name: <new-pv-name>
spec:
  capacity:
    storage: <same-size>
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-path
  claimRef:
    namespace: <namespace>
    name: <pvc-name>
  hostPath:
    path: /opt/local-path-provisioner/pvc-xxxx_<namespace>_<pvc-name>
    type: DirectoryOrCreate
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - <master-node-name>

If the old PV uses local.path instead of hostPath.path, use the same style:

1
2

local:
  path: /opt/local-path-provisioner/pvc-xxxx_<namespace>_<pvc-name>

Do not copy these fields from the old PV:

1
2
3
4
5
6

resourceVersion
uid
creationTimestamp
status
finalizers
managedFields

Delete Old PVC/PV Carefully, Keeping Data

Because the PVC is bound to the old PV, you usually need to recreate the PVC or PV binding. First, make sure the old PV reclaim policy is Retain:

1

kubectl patch pv <old-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Delete the workload first, then delete the PVC:

1

kubectl -n <namespace> delete pvc <pvc-name>

The PV should move to Released, not delete the actual directory. Then delete the old PV object:

1

kubectl delete pv <old-pv-name>

Create the new PV on master:

1

kubectl apply -f pv-new.yaml

Then recreate the PVC with the same name. Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: <pvc-name>
  namespace: <namespace>
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  volumeName: <new-pv-name>
  resources:
    requests:
      storage: <same-size>

Apply it:

1

kubectl apply -f pvc.yaml

Check binding:

1
2

kubectl -n <namespace> get pvc <pvc-name>
kubectl get pv <new-pv-name>

You want:

1

STATUS: Bound

Start the Workload on Master

Start the workload:

1

kubectl -n <namespace> scale deployment <name> --replicas=1

or:

1

kubectl -n <namespace> scale statefulset <name> --replicas=1

Verify:

1
2

kubectl -n <namespace> get pods -o wide
kubectl -n <namespace> logs <pod-name>

Drain Node and Uninstall K3s

Cordon and drain the slave node:

1
2
3

kubectl drain <slave-node-name> \
  --ignore-daemonsets \
  --delete-emptydir-data

After drain succeeds:

On the slave machine:

1
2

sudo systemctl stop k3s-agent
sudo systemctl disable k3s-agent

Then delete the node from the Kubernetes cluster:

1

kubectl delete node <slave-node-name>

Then remove it from Longhorn UI:

Longhorn UI -> Node -> slave node -> Delete

k3s kubernetes longhorn storage