How Upgrade to Upper version

Pre-Upgrade Requirements

Before upgrading a Ceph cluster, make sure of the following:

The cluster health must be in HEALTH_OK state.

The upgrade order of services must strictly follow this sequence:
mgr -> mon -> crash -> osd -> mds -> rgw -> rbd-mirror -> cephfs-mirror -> ceph-exporter -> iscsi -> nfs -> nvmeof

The upgrade process may take some time.

The cluster health status might temporarily change to HEALTH_WARNING during the upgrade.

Before the Upgrade

Take a backup of the current ceph.conf configuration file.

You can view and back up the cluster configuration with:

ceph config dump

For additional safety, back up RocksDB, Mon Map, OSD Map, Crush Map, and PG Map.

Verify the health of physical disks.

Download the new image and, preferably, store it in an internal repository.

Upgrade Steps

Set the following flags:

ceph osd pool set noautoscale

ceph osd set noout

ceph osd set noscrub

ceph osd set nodeep-scrub

Set the container_image parameter:

ceph config set mgr container_image IMAGE_NAME

Note: Although cephadm allows upgrading all cluster services with a single command, it is recommended to upgrade one daemon type at a time. Start the next upgrade only after confirming the cluster is healthy.

Upgrade mgr Service

Each time you run the following command, only one mgr daemon will upgrade.

Start with standby nodes and upgrade the active node last.

ceph orch upgrade start --image IMAGE_NAME --daemon-types mgr --limit 1

Upgrade mon Service

ceph orch upgrade start --image IMAGE_NAME --daemon-types mon --limit 1

Use the same procedure for the crash service.

Upgrade osd Service

ceph orch upgrade start --image IMAGE_NAME --daemon-types osd --hosts HOST_NAME --limit 1

Monitoring the Upgrade

You can monitor the upgrade process with:

ceph -W cephadm

ceph orch upgrade status

Expected output during upgrade:

"target_image": "IMAGE_NAME",

"in_progress": true,

"which": "Upgrading all daemon types on all hosts",

"services_complete": [],

"progress": "",

"message": "",

"is_paused": false

Expected output when upgrade is paused/stopped:

"target_image": null,

"in_progress": false,

"which": "<unknown>",

"services_complete": [],

"progress": null,

"message": "",

"is_paused": false

To check versions of all services:

ceph versions

To pause (not stop) the upgrade process:

ceph orch upgrade stop

Post-Upgrade Steps

Unset the previously set flags:

ceph osd pool unset noautoscale

ceph osd unset noout

ceph osd unset noscrub

ceph osd unset nodeep-scrub

Apply the new OSD release version:

ceph osd require-osd-release CEPH_VERSION_NAME

Upgrade the cephadm and ceph-common packages to match the cluster version.

To check the current package versions:

ceph -v

Thanks to Samira Darvishnia for testing and preparing this document.

linkedin.com/in/samira-darvishnia-486a97140