Content

RADOS GateWay OR Ceph Object Gateway

According to the ceph website documentation:
Ceph Object Gateway is an object storage interface built on top of librados. It provides a RESTful gateway between applications and Ceph Storage Clusters. Ceph Object Storage supports two interfaces:

S3-compatible: Provides object storage functionality with an interface that is compatible with a large subset of the Amazon S3 RESTful API.

Swift-compatible: Provides object storage functionality with an interface that is compatible with a large subset of the OpenStack Swift API.

The Ceph Object Gateway supports two embedded HTTP frontend libraries that can be configured with rgw_frontends. 
New one is BEAST(Like Beauty and Beast or MR Beast 😄 ) and old one is CIVETWEB which has been deprecated ! We will talk about this later, as well as use SSL termination in forntend.

This service embodies concepts like Realm, Zonegroup and Zone, which is often mean in Multisite Deployment. in a Single Site deployment they are just bunch of names

Look at this pic

Realm: It consists of one or more zone groups, including a primary zone group and other secondary zone groups. All Rados gateways in the domain pull configuration from the Rados gateway in the primary zone group and the primary zone.

Zonegroup: Consists of one or more zones, including a master zone. Other zones are called secondary zones. All zones in the zonegroup synchronize data.
Typically the name of the Zone Group is also the name of the S3 realm like “us” or “eu” and the zone will have a name like “us-east-1” or “us-west-1” to borrow from some common AWS zone names.

Zone: A ceph cluster can contain multiple zones. A zone belongs to only one cluster. A zone can have multiple RGWs.

Look to another Pic :

Realm: A realm represents a globally unique namespace consisting of one or more zonegroups containing one or more zones, and zones containing buckets, which in turn contain objects. A realm enables the Ceph Object Gateway to support multiple namespaces and their configuration on the same hardware.

A realm contains the notion of periods. Each period represents the state of the zone group and zone configuration in time. Each time you make a change to a zonegroup or zone, update the period and commit it.

ZoneGroup: Zone groups define the geographic location of one or more Ceph Object Gateway instances within one or more zones.

Zone: Ceph Object Gateway supports the notion of zones. A zone defines a logical group consisting of one or more Ceph Object Gateway instances.

Ceph object storage clusters consist of two primary storage pools, one for metadata and one for data.   

The metadata pool stores the index of all the objects for every bucket and contains “rgw.bucket.index” in the name. Essentially the bucket index pool is a collection of databases, one for each bucket which contains the list of every object in that bucket and information on the location of each chunk of data (RADOS object) that makes up each S3 object.
The metadata in the bucket index pool needs to be on fast storage that’s great for small reads and writes (IOPS) as it is essentially a collection of databases.  As such (and for various technical reasons beyond this article) this pool must be configured with a replica layout and ideally should be stored on all-flash storage media.  Flash storage for the bucket index pool is also important as buckets must resize their bucket index databases (RocksDB based) periodically to make it larger to make more room for more object metadata as a bucket grows.   This process is called “resharding” and it all happens automatically behind the scenes

Data pools typically contain “rgw.buckets.data” in their name and they store all the actual data blocks (RADOS objects) that make up each S3 object in your cluster. 
the data pool (eg default.rgw.buckets.data) is typically storing large chunks of data that can be written efficiently to HDDs.  This is where erasure coding layouts shine and provide one with a big boost in usable capacity (usually 66% or more vs 33% usable with replica=3).

Erasure coding also has great write performance when you’re working with objects that are large enough (generally anything 4MB and larger but ideally 64MB and larger) as there’s much less network write amplification when using erasure coding (~125% of client N/S traffic) vs a replica based layout (300% of client N/S traffic).

Storage classes provide us with a way to tag objects to go into the data pool of our choosing.  When you set up a cluster with a single data pool like you see above you’ll have a single storage class mapped to it called “STANDARD” and your cluster will look like this. 

https://blog.osnexus.com/2024/02/26/auto-tiering-ceph-object-storage-part-one/

As you can see, in each zone you can have different types of Storage Classes, each pointing to a pool in your Ceph Cluster. But the important thing is that in each zone you only have one pool for indexing all the buckets that are on your different Storage Classes.

One of the most important concepts of Object Storage is that object storage is a flat model of storing data i.e., there is no hierarchy (directory tree) in the stored items, as is seen in other data storage methods. For example, in a file system, each folder is able to potentially contain its own set of subfolders and files. In object storage, data is stored as discrete units called objects in a flat layout. 

Each object typically contains:

• The data payload (the actual content, such as an image, video, or document).

• Metadata (key–value pairs describing attributes of the object, like content type, creation date, or custom tags).

• A unique identifier or key that allows the object to be accessed efficiently. This means duplicate keys (or identifiers) aren’t allowed in object stores.

This lack of a hierachy is known as a flat namespace, meaning that all objects (data items) are stored at the same level without a nested structure like folders and subdirectories.

This model is actually similar to a hash map, where each value is indexed by a unique key. The presence of this unique key in both hash maps and object stores allows for basically instantaneous data retrieval as we don’t need to traverse down a nested directory tree, even when dealing with billions of objects.

Many Object Storage providers, such as Amazon S3, uses interfaces that mimic the “look and feel” of folders.
However, these “folders” are simply an illusion and do not actually contain the objects within it. These so-called “folders” are simply used as part of the unique names of each object in the store.  They Simply just are Prefixes for  an object name like in Linux, in  Linux HFS  (Hierarchical File System) All files name start with / (the root directory) and end to file name like /root/home/ubuntu/mt_file.txt  .
So in S3 Object Storage , Folders is just prefixes !  The folder name, in combination with an object key, provides a way for users to directly access a specific file, And hence every single Bucket name has to be globally unique in a Realm (Because of Multisite notion, else it's just enough in Zone).

Indexless buckets are a feature in  Ceph Storage that disable the maintenance of a traditional object index within a bucket, increasing performance by eliminating metadata operations during writes. This configuration is beneficial for applications that handle their own indexing or deal with a very large number of objects in a single bucket, as it removes resource contention and reduces the complexity of metadata management. However, a limitation of indexless buckets is that they cannot be listed, as there is no index to query.
This feature is designed for scenarios where the application itself maintains the necessary indexing for its data.
It is ideal for workloads where individual buckets need to store a massive quantity of objects. Like Billion object in a Single Bucket without any needs to Resharding 😃
But these buckets are not suitable for multisite environments because their index is not exist to used for replication ! 🙄

When writing an object, its head object is written last. This acts as an atomic ‘commit’ to make it visible to read requests.

https://www.ibm.com/docs/en/storage-ceph/7.1.0?topic=gateway-creating-indexless-buckets

Deploy

Concept is Enough, ✋ Let's go to Deploy Service !

To deploy RGW Service, Firstly we need to Create Realm, Zonegroup and Zone, 

radosgw-admin realm create --rgw-realm=realm1 --default

radosgw-admin zonegroup create --rgw-zonegroup=zg1  --master --default

radosgw-admin zone create --rgw-zonegroup=zg1 --rgw-zone=zn1 --master --default

radosgw-admin period update --rgw-realm=realm1 --commit

zn1 -> zg1 -> realm1

Now it's better to create a rgw spec yaml file to pass it to ceph orch for applying and deploying service: 

cat << EOF > rgw_spec.yaml

service_type: rgw

service_id: anisa

placement:

  hosts:

  - node01

  - node02

  count_per_host: 1

spec:

  rgw_realm: realm1

  rgw_zone: zn1

  rgw_zonegroup: zg1

  rgw_frontend_port: 9000

extra_container_args:

  - "--stop-timeout=120"

#networks:

#  -  NETWORK_CIDR

EOF

ceph orch apply -i  rgw_spec.yaml

In this confguration you may want to change the frontend port to your desired port number or use another Network to connect your RGW Endpoint for put and get object or deploy this service (read it web server!) on a dedicated hosts which added to your ceph cluster. but note that these dedicated nodes must have access to ceph Public (Client) network.
we can add SSL certificate to rgw service at this moment but still wait! 😉

After running service, you may see somthing like this in your cli 

ceph orch ps  --daemon_type=rgw

NAME                     HOST    PORTS    STATUS        REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID

rgw.anisa.node01.nhdzte  node01  *:6000   running (29h)     1m ago  4m     186M        -  19.2.3   aade1b12b8e6  5750b88bdc5c

rgw.anisa.node02.ixnboq  node02  *:6000   running (29h)     1m ago  4m    95.5M        -  19.2.3   aade1b12b8e6  5d0ccf2a8b44

now you rgw Service and Beast server is ready to response to your request for out and get object in Buckets, but you dont have any Bucket or User to authenticate within !

At this moment, the rgw service is deployed, but the pools created are all replicas with size 3 and the crush rule replicated_rule is the default and will probably be a mix of different disk classes. We will talk a little later about how to create the service with our own custom pools. pools are:

znfirst.rgw.log
znfirst.rgw.control
znfirst.rgw.meta
znfirst.rgw.buckets.index
znfirst.rgw.buckets.data
znfirst.rgw.buckets.non-ec

Three of the most important pools are:

znfirst.rgw.buckets.index
znfirst.rgw.buckets.data
znfirst.rgw.buckets.non-ec

The "buckets.index" pool, as its name suggests, will store the data containing the index objects in the buckets and must be of the replica type and SSD disks. Before you start pouring any data into the buckets, change its crash rule type.

The "buckets.data" pool is the pool that stores data uploaded to buckets by users and can be EC or replica. In most environments, the main pool considers data to be EC, but considerations such as having a lifecycle or having different storage classes to change the storage cost can change this, which we will discuss in this article.

And finally, the "buckets.non-ec" pool, which is absolutely screaming that it shouldn't be of type EC! This pool is used to temporarily store the Multipart Upload data pieces created by S3 clients and is filled and emptied as users use multipart, which is very common. Multipart upload exists to improve upload operations and causes large files to be converted into smaller pieces on the user side and started uploading, but on the server side these pieces are first stored in this pool and after all the parts of that file (object) are assembled, they will then be emptied into the relevant pool (for example, here buckets.data).

Custom Pool

As we have seen, when we deploy an RGW service, many pools are created with names that are quite predictable, so one way to have a service with arbitrary crash rules is to create pools with names that should exist before you deploy the service.

Another method is that after creating the services, change crash rule of the pools you want to change. Of course, if you want to change the pool type, you must delete it and create the exact same application with the same name.

But the most interesting method is to inject the desired settings into the zone and zone group after we created it!
We will first use the two commands radosgw-admin zonegroup get and radosgw-admin zonegroup get to see and save the current settings somewhere, and in the desired sections, after making changes, we will set them with radosgw-admin zonegroup get and radosgw-admin zonegroup get. First save tu current zone and zonegroup config.

radosgw-admin zone get > zone.json

radosgw-admin zonegroup get > zonegroup.json

Using a precise query with the jq tool, we can change the desired section and save it in a new file.

radosgw-admin zone get | jq '.placement_pools[].val.storage_classes.STANDARD.data_pool = "znfirst.rgw.buckets.MYECPOOL"' > new_zone.json

This command changes the data_pool value in the placement_pools section to znfirst.rgw.buckets.MYECPOOL.
Remember that when you run this command, you must have previously created the znfirst.rgw.buckets.MYECPOOL pool with your desired type and crash rule.

Now, to apply this new configuration, we proceed as follows:

radosgw-admin zone set < new_zone.json

radosgw-admin period update --rgw-realm=realm1 --commit

If we look at the placement_pools section, we can see the different pools that RGW uses in this zone. As we did above, the rest of these pools are all interchangeable.

radosgw-admin zone get | jq .placement_pools

[

  {

    "key": "default-placement",

    "val": {

      "index_pool": "znfirst.rgw.buckets.index",

      "storage_classes": {

        "STANDARD": {

          "data_pool": "znfirst.rgw.buckets.data"

        }

      },

      "data_extra_pool": "znfirst.rgw.buckets.non-ec",

      "index_type": 0,

      "inline_data": true

    }

  }

]

And as you saw above, don't forget to commit to the realm after making changes to the zone and zonegroup settings.

User Creation and Deletion

So we first going to make an User. Creating Users has two mandatory part, uid (User ID) and display name : 

radosgw-admin user create --uid user1 --display-name user1

if your command issue succses you see the long output of user info and access and secret key among like this : 

{

    "user_id": "user1",

    "display_name": "user1",

    "email": "",

    "suspended": 0,

    "max_buckets": 1000,

    "subusers": [],

    "keys": [

        {

            "user": "user1",

            "access_key": "E36AC5P8NLSSRXO0AVVO",

            "secret_key": "Mr1aAiMbY8QtHfZ9i1hSBj4REyx7GuW9meKD2bMw",

            "active": true,

            "create_date": "2025-09-16T15:45:56.562768Z"

        }

    ],

    "swift_keys": [],

    "caps": [],

    "op_mask": "read, write, delete",

    "default_placement": "",

    "default_storage_class": "",

    "placement_tags": [],

    "bucket_quota": {

        "enabled": false,

        "check_on_raw": false,

        "max_size": -1,

        "max_size_kb": 0,

        "max_objects": -1

    },

    "user_quota": {

        "enabled": false,

        "check_on_raw": false,

        "max_size": -1,

        "max_size_kb": 0,

        "max_objects": -1

    },

    "temp_url_keys": [],

    "type": "rgw",

    "mfa_ids": [],

    "account_id": "",

    "path": "/",

    "create_date": "2025-09-16T15:45:56.562323Z",

    "tags": [],

    "group_ids": []

}

But we dont need thme all data, Just need Access and Secret, you can see it shorter like below, remember to install jq before use it : 

radosgw-admin user info --uid=user1 | jq .keys

[

  {

    "user": "user1 ",

    "access_key": "E36AC5P8NLSSRXO0AVVO",

    "secret_key": "Mr1aAiMbY8QtHfZ9i1hSBj4REyx7GuW9meKD2bMw",

    "active": true,

    "create_date": "2025-09-16T15:45:56.562768Z"

  }

]

To delete a user in radosgw-admin, use the command

radosgw-admin user rm --uid=USER_ID

For a complete removal, add the options --purge-data to delete all data or --purge-keys to delete all access keys associated with the user.

Create Bucket and Install S3 Clients

Unbelievably ceph radosgw-admin cli has not any command to create bucket as admin for a user ! you need to create bucket by your Access and Secret Key in any AWS S3 compatilbe cli like AWS itself cli or S3CMD, Be notice that radosgw-admin cli has another Bucket operation cli, like Move, delete, change owner or ... but dosent have creating capability, of you see Creating bucket in Ceph WebGUI dont wonder to how can it my be ! Ceph WebGUI, actually use Users Acess and Secret in place by and API in web front to create bucket. It's as simple as that, that's why you have the ability to create buckets in the dashboard but not in the CLI. So with all these explanations, we should use one of the packages I mentioned for ease in creating buckets. I personally prefer S3CMD commands which have a more convenient syntax.

Simplly you can install it by your Linux standard package managet like apt or dnf or if yout wnat to live on edge you can install really latest version from github 😁 

After installing you need to configure your s3cmd to use you Access and Secret and introduce your RGW Endpoint (gateways) , s3cmd command line config is more complicated than aws cli but after config its simpller to use it !

s3cmd --configure

give your Access and Secret key
leave Default Region
type your rgw IP:PORT as S3 Endpoint
type you IP+PORT as DNS-style bucket+hostname:port template again
leave other question and say No to Use HTTPS protocol, beacuse we dosent have tls confguration for our rgw gateway
and finally it shows you all config and say Y  to save this config, 

If you want to use aws standard cli you nned to install aws cli from its official ways: 

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

unzip awscliv2.zip

sudo ./aws/install

now aws cli is install on you bash :

aws configure

AWS Access Key ID [None]:

AWS Secret Access Key [None]:

Default region name [None]:

Default output format [None]:

aws cli at config stage does not ask your endpoint , you need call it in cli directly or add it in its config file : 

aws --profile default configure set endpoint_url http://192.168.11.21:6000

Introduce one of the rgw endpoints as desired in the command. or you an call endpoint each time in cli directly like this :

aws s3 ls --endpoint http://192.168.11.21:6000

Filnally we can communicate to our rgw gateway endpoints ! 

s3cmd mb s3://my-first-bucket

 we reated bucket and now, we can put (upload) and object ( a file) to it.

s3cmd put file.img s3://my-first-bucket

its take a while depending your network connectivity to your rgw endpoint and also your ceph storage backends to upload it.

Please read the cheat sheet below for more instructions.

https://www.filestash.app/aws-s3-cli.html

https://dft.wiki/?p=2108

Find Bucket owner and vice versa

radosgw-admin bucket stats --bucket=buck2 | jq .owner

"anisa"

radosgw-admin bucket list --uid=anisa

[

    "anisabuck",

    "buck2"

]

Delete Bucket

Note that you cannot delete non-empty buckets or buckets with files and content, and you must empty the bucket and then delete them, whether with radosgw-admin or with aws cli

Of course, there is a way to do this regardless of whether the bucket is full or empty, which is not a very nice thing! Because then you will make the rgw garbage collector more busy!
What!? What is the garbage collector!?
Oh! It's okay, we will talk more about this a little later😌

radosgw-admin bucket rm --bucket=BUCKET_NAME--purge-objects --bypass-gc

or
remove an entire bucket with its content:

aws s3 rb --force s3://my-first-bucket

Be careful when using the --bypass-gc switch and be sure to consider things like off-peak hours and the number of objects in that bucket, because using this option will cause object deletion to start regardless of the garbage collector, which may increase system I/O and severely impact cluster performance. We will talk about the garbage collector in detail a little later.

NFS

As a very nice advantage, you can also have the buckets as an NFS share and mount that bucket with NFS in a path. Like CephFS you first need to create and NFS cluster and create an NFS export, For creating nfs cluster, see the cephfs documentation. And when the nfs service came up, create nfs export share for a Bucket: 

ceph nfs export create rgw <cluster_id> <pseudo_path> [<bucket>] [<user_id>]

ceph nfs export create rgw nfs-rgw /mortybuck mortybuck morty

{

  "bind": "/mortybuck",

  "cluster": "nfs-rgw",

  "mode": "RW",

  "path": "mortybuck",

  "squash": "none"

}

ceph nfs export ls nfs-rgw

[

  "/mortybuck"

]

you can mount this nfs share as a mount point in any cluster which has access to nfs-rgw cluster node.

ceph nfs cluster info nfs-rgw

{

  "nfs-rgw": {

    "backend": [

      {

        "hostname": "node01",

        "ip": "192.168.11.21",

        "port": 2049

      }

    ],

    "virtual_ip": null

  }

}

on on client

mount -t nfs 192.168.11.21:/mortybuck /mnt

df -h  /mnt

Filesystem                Size  Used Avail Use% Mounted on

192.168.11.21:/mortybuck   68G  3.4G   65G   5% /mnt

touch /mnt/file

ls /mnt/ -ltrh

total 0

-rw-r--r-- 1 root root 0 Jan  1  1970 file

LC

Now we are here to explain what is LC (LifeCycle) and GC (Garbage Collector), and What are they and what are their uses. 🥹

According to AWS explanation , S3 Lifecycle helps you store objects cost effectively throughout their lifecycle by transitioning them to lower-cost storage classes, or, deleting expired objects on your behalf.

This means that you can use this policy to move your objects to slower storage that is slower in terms of speed and IOPS after they have not been used for a while (i.e. read or write, practically any type of IO) because there is no reason for them to occupy your fast and expensive disks (OSD) !

In AWS, because each Storage Class of the S3 service has a different cost, this LifeCycle Policy helps manage costs, but in a place like Ceph, because the arrangement and distribution of your disks can be on a mixed cluster, and depending on this, they will have different prices for you, so here, too, you can practically say that you are managing cost and user experience, because the user wants the data he deals with the most to be available to him faster, but less important data that he has not accessed for a long time can be stored on the disks more slowly to make room for more important and accessible data.

I think we now understand what the lifecycle is for! 🧐

GC

But in order to understand why GC exists, we need to understand how RGW handles data deletion:

When you delete an object, it is not immediately erased from the disk. Instead, only its reference is removed from the main index. However, the actual object data stored in Ceph’s underlying RADOS pools might still remain. and why it does act like this?

In some cases (like multipart uploads or overwrites), parts of data fragments can remain in storage as dangling or orphaned fragments.
in a distributed system immediately deleting such data can cause inconsistency between metadata and the underlying data. for example a reference to an object might be temporarily lost due to network latency or other issues. Immediate deletion could lead to the loss of data that is still in use.
As well as, deleting data from a disk is a time-consuming and resource-intensive operation. and may hurt system performance under heavy load. By marking objects for future deletion, the system can respond quickly to user requests and perform the actual cleanup during periods of low traffic.
And another important point is that If versioning is enabled, older versions of an object must be retained for a period, even after a new version is uploaded or the current version is deleted.
And alos in a distributed system, a reference to an object might be temporarily lost due to network latency or other issues. Immediate deletion could lead to the loss of data that is still in use.

Therefore, RGW includes a Garbage Collection mechanism that removes these “leftover” or “stale” data fragments asynchronously — in the background and in a controlled manner. Its primary goal is to free up storage space and maintain the health and performance of the cluster.

Now that we understand the main purpose and function of the Garbage Collector better, let's see how it does its job:

The GC periodically scans logs and indexes to find objects that no longer have any valid references. These objects include deleted items, replaced older versions, or residual data from failed uploads.
The identified objects are placed in a queue for final deletion. In a GC cycle, RGW scans queue of objects marked for deletion — those whose minimum wait time has elapsed.
Finally, the GC physically erases these objects from the relevant RADOS pools and the storage disks (OSDs), freeing up the occupied space.
Once deletion is complete, those entries are removed from the GC queue.

GC/LC Challenges and Considerations

If many objects are deleted quickly, the GC queue might grow large — meaning GC may lag behind, delaying space reclamation.
Deleting large batches of objects at once can put pressure on OSD I/O, so concurrency limits are important.
Large GC metadata maps (omap objects) can slow down GC operations if not tuned properly.
“Orphaned” RADOS objects — data without corresponding RGW metadata — may accumulate over time and must be cleaned by GC to free resources.

And about LC, The biggest risk is that the scheduled LC process does not have enough time to move all targeted data from the Hot Pool to the Cold Pool within the allotted off-peak window.
When the migration fails to complete, "dirty data" (data that should have been moved) remains on the expensive, high-performance tier. This reduces the available space for new, incoming data that genuinely requires fast I/O, leading to performance degradation.
If the LC process runs longer than expected and spills into peak business hours, the heavy I/O operations of moving data can compete with user requests, slowing down the entire system. 

So you should definitely keep two things in mind:

Right-Size Your Tiers: The Hot Pool must be large enough to handle the daily influx of new data plus a buffer. The Cold Pool's write speed is the bottleneck; you must ensure it can ingest data fast enough during the LC window. The size difference is key—the small Hot Pool needs to be cleared out efficiently to make room for the next day's workload.

Calculate the Off-Peak Window: You need to accurately estimate how long it takes to move a typical amount of data from the Hot Pool to the Cold Pool. The LC schedule must be set for a time with minimal user activity (e.g., overnight) and be long enough for the transfer to complete successfully.

StorageClass

We got to know storage_classes a little in the Custom pools section and saw that it is where we introduce data pools and other pools for use. Before dive into Lifecycle's Policy, we need to define our Storage Classes in our Zone and ZoneGroup.
Each storage class is defined in the form of a placement.

radosgw-admin zonegroup placement add \

--rgw-zonegroup zgfirst \

--placement-id default-placement \

--storage-class COLD

radosgw-admin zone placement add \

--rgw-zone znfirst \

--placement-id default-placement \

--storage-class COLD \

--data-pool COLD.pool \

--compression zstd

There are several things to talk about with these two commands

First are data pool and its name!
In the radosgw-admin zone placement add command, there are two parameters named storage-class and data-pool.
storage-class is simply a label that is referenced in lifecycle rules to specify the storage location where objects should be stored according to the defined policy.
data pool is the RADOS pool where the data is actually stored.

RGW supports server-side compression of uploaded objects. Note that compression is completely optional and can be on or off, because of the compression algorithms, if you have a relatively good CPU, it's fine to enable it. Compression may have a very small impact on your performance, but in return it will use your space more efficiently.
Supported compression plugins include the following:

• lz4

• snappy

• zlib

• zstd

all algorithms used to make data smaller, but they trade-off between speed and how small the final file is (compression ratio)

lz4: This is the fastest of the group for both compression and decompression. Its main goal is raw speed, which means the final compressed file size is larger than the others.

snappy: Developed by Google, this algorithm is also focused on very high speed, almost as fast as lz4. It offers a good balance between speed and a reasonable compression ratio.

zlib: This is one of the oldest and most widely used algorithms. It provides a very good compression ratio (small file sizes) but is much slower than the others.

zstd: Developed by Facebook, this is a modern algorithm that offers the best overall balance. It achieves a compression ratio almost as good as zlib while being nearly as fast as lz4. It is often considered the best choice for general-purpose use.

now if we look at the zone and zonegorup config you'll see the new storage class:

radosgw-admin zone get | jq .placement_pools

[

  {

    "key": "default-placement",

    "val": {

      "index_pool": "znfirst.rgw.buckets.index",

      "storage_classes": {

        "COLD": {

          "data_pool": "COLD.pool",

          "compression_type": "zstd"

        },

        "STANDARD": {

          "data_pool": "znfirst.rgw.buckets.data"

        }

      },

      "data_extra_pool": "znfirst.rgw.buckets.non-ec",

      "index_type": 0,

      "inline_data": true

    }

  }

]

radosgw-admin zonegroup get | jq .placement_targets

[

  {

    "name": "default-placement",

    "tags": [],

    "storage_classes": [

      "COLD",

      "STANDARD"

    ]

  }

]

LifeCycle create, assign, delete

Okay! Now, after all this explanation and discussion, let's write an LC Policy together that will do some work for us. Let's not forget that we can only manually tell LC what to do, but GC will do its work in the background, and we will only get acquainted with its parameters, how we can make it faster or run it at a specific time, like LC.

Typically, we can call two types of operations in LC, which called Rule that include Transition and Expire, respectively which refer to the transition between different Storage Classes and the deletion of data after a certain time.

Each policy can include one or both of these two items.
Remember that LC is applied as a policy on the bucket.
You can set Lifecycle by AWS S3 client or even S3CMD.
LifeCycly Rules policy may exist in JSON or XML format.

Looks at the examples: 

XML

<LifecycleConfiguration>

  <Rule>

    <ID>Transition then Expire Rule</ID>

    <Filter>

       <Prefix></Prefix>

    </Filter>

    <Status>Enabled</Status>

    <Transition>

      <Days>7</Days>

      <StorageClass>COLD</StorageClass>

    </Transition>

    <Expiration>

      <Days>30</Days>

    </Expiration>

  </Rule>

</LifecycleConfiguration>

Upper rule Transit data to COLD storage class and delete them after 1 month.

Optionally you can pass the prefix to run transition or expiration on a specific folder for example: 

<LifecycleConfiguration>

    <Rule>

        <ID>ExampleRule</ID>

        <Filter>

           <Prefix>documents/</Prefix>

        </Filter>

        <Status>Enabled</Status>

        <Transition>        

           <Days>365</Days>        

           <StorageClass>COLD</StorageClass>

        </Transition>    

        <Expiration>

             <Days>3650</Days>

        </Expiration>

    </Rule>

</LifecycleConfiguration>

Same LC in JSON

{

    "Rules": [

        {

            "Filter": {

                "Prefix": "documents/"

            },

            "Status": "Enabled",

            "Transitions": [

                {

                    "Days": 365,

                    "StorageClass": "COLD"

                }

            ],

            "Expiration": {

                "Days": 3650

            },

            "ID": "ExampleRule"

        }

    ]

}

Now we need to apply some of these rule on a bucket as a owner of bucket with proper access: 

Save the JSON Lifecycle configuration in a file (for example, lifecycle.json or lifecycle.xml ).

Run the following AWS CLI  or s3cmd to set the Lifecycle configuration on your bucket.

aws s3api put-bucket-lifecycle-configuration  \

--bucket mortybuck  \

--lifecycle-configuration file://lifecycle.json 

s3cmd  setlifecycle lifecycle.json  s3://mortybuck

To verify, retrieve the S3 Lifecycle configuration

aws s3api get-bucket-lifecycle-configuration  \

--bucket mortybuck

s3cmd  getlifecycle s3://mortybuck

<?xml version="1.0" ?>

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">

        <Rule>

                <ID>Transition then Expire Rule</ID>

                <Prefix/>

                <Status>Enabled</Status>

                <Expiration>

                        <Days>60</Days>

                </Expiration>

                <Transition>

                        <Days>1</Days>

                        <StorageClass>HOT</StorageClass>

                </Transition>

        </Rule>

</LifecycleConfiguration>

Or as an RGW admin :

radosgw-admin lc get --bucket=mortybuck | jq .rule_map

[

  {

    "id": "Transition then Expire Rule",

    "rule": {

      "id": "Transition then Expire Rule",

      "prefix": "",

      "status": "Enabled",

      "expiration": {

        "days": "60",

        "date": ""

      },

      "noncur_expiration": {

        "days": "",

        "date": ""

      },

      "mp_expiration": {

        "days": "",

        "date": ""

      },

      "filter": {

        "prefix": "",

        "obj_tags": {

          "tagset": {}

        }

      },

      "transitions": {

        "HOT": {

          "days": "1",

          "date": "",

          "storage_class": "HOT"

        }

      },

      "noncur_transitions": {},

      "dm_expiration": false

    }

  }

]

To delete the S3 Lifecycle configuration, use the delete-bucket-lifecycle AWS CLI command as follows:

aws s3api delete-bucket-lifecycle \

--bucket amzn-s3-demo-bucket

LC Debug

In some cases, you may need to test and debug your LifeCycles rule and you cannot wait days and month! 

RGW has an option to turn on debug mode for more control on time! This allows a set period of time (typically 60 seconds) will correlate to 24 hours. This allows fast testing of rules to ensure they are doing exactly what is expected, and gives the ability to iterate quickly.

ceph config get client.rgw rgw_lc_debug_interval

-1

ceph config set client.rgw rgw_lc_debug_interval 60

This sets the lifecycle interval to correlate every 60 seconds to 24 hours.
Once the config file is saved, restart the RGW’s. 

ceph orch  restart rgw.first
Scheduled to restart rgw.first.node01.lbzrcs on host 'node01'

Scheduled to restart rgw.first.node02.xtdzno on host 'node02'

Replace the name rgw.first with the name of your RGW service.

If you are having trouble with your lifecycle rules, it is most likely because the rules are written incorrectly, or the RGW debug interval is not working.

In order to check the status of your RGW lifecycle rules, run the following command on a node with an admin keyring on it.

radosgw-admin lc list

This lists the lifecycle rules you have in place. If they are not set to “complete” you can force Ceph to start the rule by running the following command:

radosgw-admin lc process