Why ceph

Ceph provides massive storage capacity, and it supports numerous use cases, such as:

• The Ceph Block Device client is a leading storage backend for cloud platforms that provides limitless storage for volumes and images with high performance features like copy-on-write cloning.

• The Ceph Object Gateway client is a leading storage backend for cloud platforms that provides a RESTful S3-compliant and Swift-compliant object storage for objects like audio, bitmap, video, and other data.

• The Ceph File System for traditional file storage.

A CRUSH map describes a topography of cluster resources, and the map exists both on client hosts and Ceph Monitor hosts within the cluster. Ceph clients and Ceph OSDs both use the CRUSH map and the CRUSH algorithm. Ceph clients communicate directly with OSDs, eliminating a centralized object lookup and a potential performance bottleneck. With awareness of the CRUSH map and communication with their peers, OSDs can handle replication, backfilling, and recovery, allowing for dynamic failure recovery. 

Scenario and Deployment

ceph01 192.168.11.21

ceph02 192.168.11.22

ceph03 192.168.11.23

cephOSD01 192.168.11.31

cephOSD02 192.168.11.32

cephOSD03 192.168.11.33

cephOSD04 192.168.11.34

• Disable SELINUX and FIREWALLD

setenforce 0

grubby --update-kernel ALL --args selinux=0 

systemctl stop firewalld.service

systemctl disable firewalld.service

systemctl mask firewalld.service

• set your NIC interfaces to MTU 9000 in system-connections or in Netplan, for example 
  edit /etc/NetworkManager/system-connections/ens192.nmconnection add add :

[ethernet]

mtu=9000

• Or set it by nmcli 

~]# nmcli connection show "ens192" | grep 802-3-ethernet.mtu

802-3-ethernet.mtu:                     auto
====================================================================

sudo nmcli connection modify ens192 802-3-ethernet.mtu 9000
sudo nmcli connection down ens192 && sudo nmcli connection up ens192

====================================================================

~]# nmcli connection show "ens192" | grep 802-3-ethernet.mtu

802-3-ethernet.mtu:                     9000

VERY IMPORTANT

• Install chrony on all nodes ! 

dnf install chrony -y 

IF YOU NEED ADD YOUR NTP SERVER TO CHRONY CONFIG

/etc/chrony.conf

server ntp.example.com iburst prefer

AFTER ALL BE CAREFUL TO WATCH THE TIMEDATECTL System clock synchronized STATUS to BE YESSSS ! 

• INSTALLING  CEPH

dnf install epel-release -y

dnf install nano wget bash-completion bind-utils podman -y

CEPH_RELEASE=19.2.2

curl -o /usr/local/bin/cephadm --silent --remote-name --location https://download.ceph.com/rpm-${CEPH_RELEASE}/el9/noarch/cephadm

chmod 755 /usr/local/bin/cephadm

~]# cephadm version

cephadm version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)

cephadm add-repo --release 19.2.2

cephadm install ceph-common

~]# ceph -v

ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)

• Install Ceph Crush tool:

~]#  dnf install ceph-base -y

RUN BELOW COMMANDS ON FIRST NODE OR SELECT A NODE FOR INITIAL INSTALLATION 

cephadm bootstrap --mon-ip --cluster-network --skip-monitoring-stack --with-centralized-logging

cephadm bootstrap --mon-ip 192.168.11.21 --skip-monitoring-stack 

Ceph Dashboard is now available at:

             URL: https://ceph01:8443/

            User: admin

        Password: 0npoabe73p

• Go to Dashboard and change the initial password 

• Ceph mgr dashboard automatically redirect you to Active mgr web dashboard

~]# cat /etc/ceph/ceph.pub

add this key to all other hosts /root/.ssh/authorized_keys

adding other mon and mgr (control plain nodes) nodes

ceph orch host add <HOSTNAME> <HOST_IP> --labels _admin

ceph orch host add ceph02 192.168.11.22 --labels _admin

ceph orch host add ceph03 192.168.11.23 --labels _admin

ceph orch apply mon --placement="3 label:_admin"

ceph orch apply mgr --placement="3 label:_admin"

~]# ceph orch ls --service-type mon

NAME  PORTS  RUNNING  REFRESHED  AGE  PLACEMENT

mon              3/3  5m ago     67s  count:3;label:_admin

~]# ceph orch ls --service-type mgr

NAME  PORTS  RUNNING  REFRESHED  AGE   PLACEMENT

mgr              3/3  5m ago     113s  count:3;label:_admin

• َِِADD OSD NODES 

ceph orch host add cephosd01 192.168.11.31

ceph orch host add cephosd02 192.168.11.32

ceph orch host add cephosd03 192.168.11.33

ceph orch host add cephosd04 192.168.11.34

• see ceph -s output 

• if some nodes has not been synchronized in clock by clock skew warning restart it, for example:

~]# ceph orch daemon restart mon.ceph02

• add Disk to VMs by nvme bus 

~]# ceph orch device ls | grep -i nvme

cephosd01  /dev/nvme0n1  ssd   VMware_Virtual_NVMe_Disk_VMware_NVME_0000            5120M  Yes        0s ago

ceph orch daemon add osd cephosd01:/dev/nvme0n1

• add Diks to vm by bigger size than nvme to VMs by SATA bus

~]# ceph orch device ls | grep -i hdd

cephosd01  /dev/sdb      hdd   ATA_VMware_Virtual_SATA_Hard_Drive_00000000000000000001  10.0G  Yes        6s ago

• add OSD to cluster :

ceph orch daemon add osd cephosd01:/dev/sdb

• If you want to change the class of an OSD for any reason, you can do so with these commands:

ceph osd crush rm-device-class 4 5 6 7
ceph osd crush set-device-class  hdd 4 5 6 7

• Upeer Commande make OSD 4 5 6 7 to hdd class

• create proper crush rule for SSD and HDD rule, be notice this is OK for test and lab env but in a production and a real datacenter, you need to define more precise and granular type like PDU or RACK in CRUSH rule map

ceph osd crush rule create-replicated replicated_hdd default host hdd

ceph osd crush rule create-replicated replicated_ssd default host ssd

• For production act like this : 

get crush map on binery file

ceph osd getcrushmap -o crushmap

crushtool -d crushmap -o crushmap.txt

• Edit CRUSH map file by an editor:

nano crushmap.txt 

• Decompile txt CRUSH map file to binery  and inject it to ceph cluster map rule

crushtool -c crushmap-modified.txt -o crushmap-new.bin
ceph osd setcrushmap -i crushmap-new.bin

~]# ceph osd crush rule ls

replicated_rule

replicated_ssd

replicated_hdd

ssd_rack_rule

hdd_rack_rule

• Now its time to create POOLs

ceph osd pool create ssd_pool 64 64 replicated ssd_rack_rule --autoscale-mode=warn
ceph osd pool set ssd_pool size 2

ceph osd pool create hdd_pool 64 64 replicated hdd_rack_rule --autoscale-mode=warn
ceph osd pool set hdd_pool size 2

ceph osd pool set hdd_pool pg_num 16

• IF you need to delete pool, you should set a allow delete pool flag 

ceph config set mon mon_allow_pool_delete true 

ceph osd pool ls detail

pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 17 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89

pool 4 'ssd_pool' replicated size 2 min_size 1 crush_rule 3 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 89 flags hashpspool stripe_width 0 read_balance_score 1.25

pool 5 'hdd_pool' replicated size 2 min_size 1 crush_rule 4 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 96 flags hashpspool stripe_width 0 read_balance_score 1.19

• MAKE POOLs ready for RBD 

rbd pool init hdd_pool

rbd pool init ssd_pool

~]# ceph osd pool application get ssd_pool

{

    "rbd": {}

}

• Be Notice  that if your Crush rule  mismatch your real world rack numbers, for example you have 2 rack and your fail domain is rack and your Pool size is 3 !  you may get pg_undersized warning in you ceph cluster until you adding other rack to make you topolgy like your crush

now you can connect your images manually to a clinet by native RBD client which is using librbd in in its heart,

Now you can use this pool for OpenStack Cinder or NOVA or even use it directly in qemu/KVM by mount image volume in host and add it by its path to VM, For OpenStack Refer to the document. For Manual mouting is like this and use it KVM by Virt-Manager :

RBD

RGW