Skip to content

RADOS gateway S3

Access to S3 storage

Under the settings tab in the HyperCloud dashboard, create an attribute for the user:

RADOSGW=true

RGW settings

Wait five to ten minutes for HyperCloud to engage and populate the following attributes:

RADOSGW_ACCESS_KEY
RADOSGW_SECRET_KEY

These attributes behave the same as the S3/AWS values of a similar name.

S3 configuration

The S3 command line tool s3cmd will look for a configuration file to set up the settings for the utility. The default location it will browse is: ~/.s3cfg; however, it can be placed anywhere and can simply be specified with: s3cmd --config /path/to/s3cfg after installation.

Here is a sample S3 configuration needed for s3cmd:

# cat example.s3cfg
[default]
access_key = ${value of RADOSGW_ACCESS_KEY}
host_base = hypercloud.softiron.com:7443
host_bucket = s3.hypercloud.com:7443
secret_key = ${value of RADOSGW_SECRET_KEY}

Ports

  • 7480 / 7443 are load-balanced to the entire cluster.
  • 8080 / 8443 are only on the dashboard (and intended to be the destination port for multi-site since RGW needs to be routable). These ports can be used; however, it will be less resilient and less performant.

Certificates

SSL certificates may be required for the s3cmd to work properly. These certificates can be stored somewhere within the cluster so that they can be fetched for any VMs that might need for s3cmd or otherwise.

Buckets

New S3 buckets can be made by running the following command:

s3cmd mb s3://<bucket_name>

The existing buckets can be listed by running the following command:

s3cmd ls
2022-05-26 22:52  s3://my-new-bucket
2022-05-25 21:17  s3://bucket-test
2022-09-07 17:12  s3://foobar
2022-09-07 21:45  s3://yoda

Removing the buckets can be running the following command:

s3cmd rb s3://<bucket_name>

Files

Upload

Files can be uploaded via the following command:

s3cmd put ca.crt s3://<bucket_name>/ca.crt

# Set the permissions to allow users with URL to fetch it
s3cmd setacl s3://<bucket_name>/ca.crt --acl-public

# From another host, the file can be fetched, e.g. with curl:
curl -k https://hypercloud.softiron.com:7443/<bucket_name>/ca.crt > /usr/local/share/ca-certificates/ca.crt && update-ca-certificates

{{< callout context="note" title="Note" icon="info-circle" >}} The curl "insecure" flag is used because of the nature of fetching the certificate which will be used for future signing. {{< /callout >}}

Fetch

Files can be retrieved with get command:

s3cmd get s3://${RADOS_BUCKET}/somefile.tar.gz

List

Files stored can be listed with various granularity, such as:

s3cmd -c simpleonetool/s3cfg ls s3://<bucket_name>/
2022-05-26 22:59          814  s3://<bucket_name>/ca.crt
2022-07-06 18:47         1078  s3://<bucket_name>/hook.sh
2022-07-19 21:42      4463626  s3://<bucket_name>/overlay.tar.gz
2023-02-17 22:45      7856098  s3://<bucket_name>/simpleonetool-0.1.0-darwin-arm64
2023-02-17 22:46      8268962  s3://<bucket_name>/simpleonetool-0.1.0-linux-amd64
2023-02-17 22:46      7917639  s3://<bucket_name>/simpleonetool-0.1.0-linux-arm64
2023-03-14 21:35      7856690  s3://<bucket_name>/simpleonetool-0.2.0-darwin-arm64
2023-03-14 21:35      8276596  s3://<bucket_name>/simpleonetool-0.2.0-linux-amd64
2023-03-14 21:35      7919057  s3://<bucket_name>/simpleonetool-0.2.0-linux-arm64
2023-09-18 21:10      7878946  s3://<bucket_name>/simpleonetool-0.2.1-darwin-arm64
2023-09-18 21:10      8234642  s3://<bucket_name>/simpleonetool-0.2.1-linux-amd64
2023-09-18 21:10      7833758  s3://<bucket_name>/simpleonetool-0.2.1-linux-arm64

Note

s3cmd requires trailing slashes for listing directories.

Delete

Files can be deleted by running the following command:

s3cmd del s3://<bucket_name>/<file_path>

Confirm the deletion when prompted.

Examples

A specific file: 

s3cmd del s3://my-new-bucket/example.txt
All files in a specific folder: 
s3cmd del s3://my-new-bucket/my-folder/*