Skip to content

Object storage with MinIO

Introduction

MinIO is an open source S3 compatible object storage software solution capable of scalable high performance. This document describes the use of MinIO in a SoftIron HyperCloud environment as well as creating a base image that can be used as a template and/or converted to a Marketplace application for building MinIO environments.

MinIO Installation options

The MinIO template can be used to create base VMs for any of the installation types. Storage is added to suit the mode and the MinIO configuration file is used to set up stand-alone or distributed modes.

Single-Node Single-Drive (SNSD or "Stand-alone")

Local development and evaluation with no/limited reliability

Single-Node Multi-Drive (SNMD or "Stand- alone Multi-Drive")

  • Workloads with lower performance, scale, and capacity requirements
  • Drive-level reliability with configurable tolerance for loss of up to 1/2 all drives
  • Evaluation of multi-drive topologies and fail over behavior.

Multi-Node Multi-Drive (MNMD or "Distributed")

  • Enterprise-grade high-performance object storage
  • Multi Node/Drive level reliability with configurable tolerance for loss of up to 1/2 all nodes/drives
  • Primary storage for AI/ML, Distributed Query, Analytics, and other Data Lake components
  • Scalable for Petabyte+ workloads - both storage capacity and performance

Deploying MinIO

Storage Options

In a physical environment, MinIO recommends the use of directly attached storage devices or JBODs. Use of network storage solutions is discouraged. In a HyperCloud environment, storage is virtualized so some consideration should be used in an optimal solution. MinIO makes use of Erasure Coding (EC) to provide resilience based on the use of physical drives.

In HyperCloud, attached "drive" images are inherently protected using the underlying Ceph storage. Hence even an apparent single drive mapped to the MinIO VMs has resilience already. The use of EC by MinIO implies a potential increase in resilience by layering EC on top of the underlying Ceph but this can be potentially inefficient in terms of raw capacity usage.

If a MinIO multi-drive setup is used, it may be useful to configure underlying Ceph pools specifically as data stores for MinIO storage with the assumption that EC will be used on top of the Ceph storage.

Instantiate VM(s)

  • Templates / VMs (Select) -> Instantiate (non-persistent)
    • Edit RAM, CPUs, networks, disk capacity etc. as required
    • "Instantiate"

Once deployed, one can add a second disk image (or more) for data. The new VM will boot straight away once instantiated. Either use existing "Empty disk" App or create a new blank disk:

  • Apps (Search for Empty Disk)
  • Download to datastore (default or customized one if it has been created)

or:

  • Image -> Create -> Type= Generic storage Datablock -> persistent=no -> Empty disk image (Select capacity). Image mapping raw (qcow2 fails - it will not allow the VM to be created)

then:

  • Template / VMs -> (select) -> Update ->
    • Storage -> Add (+) -> select new Datablock
    • Network -> select network and security group
      • The network security group needs to allow the ports defined for MinIO (/etc/default/MinIO) - typically 9000 and 9001.

Configuring Multi–Node

MinIO reference

Although MinIO expects multi-node setups to use multiple drives, due to the inherent data resilience of Ceph, it may be valid to use single drives per node. However, as Erasure Coding is used by Ceph with an expected +2 parity configuration, small numbers of servers would be inefficient. For example, 4 nodes with single drives would effectively be 50% capacity efficient (EC 2+2) at best before considering the underlying Ceph resilience overheads.

When setting up multi-node (or multi-drive) systems, MinIO requires that nodes (and/or drives) are sequentially numbered as it makes use of {a…b} notation (note 3 dots!) to describe groups of devices. For example in a multi-node system, an example /etc/default/minio file could be:

MINIO_VOLUMES="https://minio{1...4}.example.net:9000/mnt/disk{1...4}/minio"
MINIO_OPTS="--console-address :9001"
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minio-secret-key-CHANGE-ME
MINIO_SERVER_URL="https://minio.example.net:9000"

It is highly recommended to use a load balancer with a multi-node configuration. Nginx or HAProxy are suggested by MinIO.

Managing MinIO

Access the MinIO console with http://<IP>:9000 to get redirected to a random console port. Use of --console-address :9001 in the config file fixes a specific console port.

User and Policy Management

Create users in the GUI and manage policies to allow for controlled access.

Create User

Create a bucket for the user

Create Bucket

Create a policy to allow the user access to their own specific bucket hierarchy

Create Policy

Assign the new policy to the user

Assign Policy

Create Access Key for user. This will create an Access Key and Secret - the secret will only be shown once during this creation process but there is an option to download it. It can not be recovered if it is not recorded or saved at this point. The Access and Secret Keys can now be loaded into the client's ~/.s3cfg file.

Create Access key 1

Create Access key 2

Create Access key 3

Network Encryption

See the MinIO docs for details of enabling network encryption (TLS).

For an internal / test / lab system, a self-signed certificate can be generated using the MinIO certgen utility: https://github.com/MinIO/certgen (Example setup below)

wget https://github.com/MinIO/certgen/releases/latest/download/certgen-linux-amd64
mv certgen-linux-amd64 certgen
chmod +x certgen
install certgen /usr/bin
certgen -host "127.0.0.1,localhost,10.10.100.38"
cp public.crt /home/MinIO-user/.MinIO/certs/
cp private.key /home/MinIO-user/.MinIO/certs/
systemctl restart MinIO

With the private and public keys installed now, MinIO will now require secure (HTTPS etc.) operation for S3 I/O and accessing the GUI. Check the system is working by accessing the GUI at https://<MinIO-SERVER>:9001. The default user / password is minioadmin defined in /etc/default/MinIO

Client Install (s3cmd)

e.g. Debian:

apt install s3cmd
Edit ~/.s3cfg for the user who will access the system. Sample file:

host_base = 10.127.4.116:9000
host_bucket = 10.127.4.116:9000
use_https = False

access_key = EWdedjmedRkNrhOL
secret_key = eYkM99PgNDHGIpRVSk9dkrg6bHwn2DCd

signature_v2 = False

If secure TLS mode is used, put the public.crt file from the server in /usr/local/share/ca-certificates/ and run update-ca-certificates

If s3cmd is used with self-generated certificates and there is no internal certificate authority configured, the command should be run with --no-check-certificate or add the following to ~/.s3cfg

check_ssl_certificate = False
check_ssl_hostname = False

Some common s3cmd operation examples:

s3cmd ls
s3cmd la
s3cmd mb s3://test
s3cmd put /boot/initrd.img-4.19.0-16-amd64 s3://test
s3cmd get s3://test/datafile /tmp/
s3cmd du

Build MinIO Base VM

On Debian 11, there is a .deb installation available from MinIO or a direct "generic" Linux install option.

Create a new VM to be used as a base to build the MinIO template / application (e.g. MinIO-Build).

Node specifications (RAM, CPU etc.) should be adjusted during the specific VM instantiation to meet required needs but for now, a minimal template can be built. For example, 2 GB RAM, 1 physical CPU, 2 virtual CPUs, 10 GB system disk.

Check OS is up to date:

apt update
apt upgrade

Clear out any old kernels if the upgrade has added a new one. This will save ~300 MB. Take care to only unused versions.

dpkg -l | grep linux-image
apt remove --purge linux-image-5.10.0-20-amd64
update-grub

It is useful to install to create the recommended xfs storage:

apt install xfsprogs

Download the latest .deb from the MinIO release pages.

This can be automated by (for example), 

ARCHIVE=https://dl.min.io/server/MinIO/release/linux-amd64/
DEB=`curl $ARCHIVE 2>/dev/null | grep "amd64.deb<" | cut -d '"' -f 2 | sort | tail -n 1`
wget $ARCHIVE$DEB
dpkg -i $DEB

Identify a directory for the MinIO data. Typically this is a mount of a block storage device and xfs is the recommended file system. MinIO discourages shared file systems but suggests that NFSv4 is used if NAS is required.

For now we just define where the data mount point will be as the final storage will be determined when the MinIO VM is instantiated.

Create MinIO-user and define access rights for the MinIO storage. The user MinIO runs as (default = minio-user) is defined in /etc/systemd/system/MinIO.service

groupadd -r MinIO-user
useradd -M -r -g MinIO-user MinIO-user
mkdir /mnt/MinIO
chown -R MinIO-user:MinIO-user /mnt/MinIO
MinIO config file
Create this - example for a default:
~# cat /etc/default/MinIO
MinIO_VOLUMES="/mnt/MinIO"
MinIO_OPTS="--address :9000 --console-address :9001"
MinIO_ACCESS_KEY="MinIOadmin"
MinIO_SECRET_KEY="MinIOadmin"

Enable, start and check MinIO service:

systemctl enable MinIO.service

Shut down the instance

poweroff

At this point, the VM is suitable for saving as a template.

Create Template from VM

If a Template is meant to be consumed by end-users, its Images should not be persistent. A persistent Image can only be used by one VM simultaneously, and the next user will find the changes made by the previous user.

If the users need persistent storage, they can use the "instantiate to persistent" functionality.

Select the VM and the "Save to Template" icon:

The Network options can be set on the template or decided on instantiation.

The template can now be instantiated with required modifications to parameters such as CPU, RAM, storage etc.

Add App to the Marketplace

A template needs to be created from the VM which will have the MinIO image. Now you can upload the image to the new store for sharing.

For example:

  • Browse to 'Storage' -> 'Images'
  • Click the image you want to upload
  • Click the "Marketplace" button
  • Enter a name
  • Select the Marketplace
  • Add App and VM template parameters (see below)
  • Click "Create"
  • The image will upload using ssh. It may take some time.

You should now be able to view the image in 'Storage' -> 'Apps' and download it for use locally.

To allow the app image to be used as a template, select the App and add a VM template. For an example, it could be helpful to simply copy an existing template form the OS that the app was created from. In the MinIO setup where Debian 11 was used, the Debian 11 market place app uses the following templates:

App template:

DEV_PREFIX="vd"
DRIVER="qcow2"
TYPE="OS"

VM template: 
CONTEXT = [ NETWORK  ="YES",SSH_PUBLIC_KEY  ="$USER[SSH_PUBLIC_KEY]"]

CPU = "1"
GRAPHICS = [ LISTEN  ="0.0.0.0",TYPE  ="vnc"]

MEMORY = "768"
OS = [ ARCH  ="x86_64"]

LOGO = "images/logos/debian.png"
LXD_SECURITY_PRIVILEGED = "true"

Use App from Marketplace

  • Select Storage -> Apps
  • Select MinIO S3 Server (search if needed)
  • Choose "Import into Datastore" and select datastore (usually "default")

There should now be a Template VM ready to deploy.

Create a Marketplace server

Based on Debian 11 with HTTP method.

This works with the HyperCloud dashboard machine uploading images for created apps via SSH to the Nginx data directory. This is then presented to the dashboard via HTTP so the important rights needed on the market place server are:

  • The dashboard server must be able to log in to the marketplace server, usually using oneadmin, and oneadmin rights must allow writing to the Nginx data directory.
  • Nginx must have rights to read the assigned data directory on the marketplace server.

Follow the instructions as in HyperCloud User Guide

  • Create a Debian 11 VM, update it and install Nginx:
apt update
apt upgrade
apt install nginx
  • Create User:
useradd -m oneadmin
passwd oneadmin
  • Log in as that user (oneadmin) and edit the file called ~/.ssh/authorized_keys
  • Get the ssh key for the oneadmin user from the HyperCloud dashboard (while logged in as oneadmin, click the username at the top right of the screen, select settings, select the auth tab and copy the Public SSH key)
  • Paste that into the authorized_keys file on the new host
  • Create a directory for the images - for example /srv/my_market
  • Make sure the oneadmin user has write permissions to that directory
  • Create an Nginx config for the service in /etc/nginx/sites-enabled/my_market, the bare minimum for this is something like:
server {
  listen *:8080;
  server_name my_market.demo;                   
  root /srv/my_market;
  autoindex on;
}
  • Start (or restart) the Nginx service (systemctl restart nginx)

Create a Private Marketplace

  • In the HyperCloud management GUI browse to 'Storage' -> 'Marketplaces'
  • Click the green plus ( + ) sign
  • Add a name and fill in the details - replacing [NEW_VM_IP] with the correct details.
  • Storage backend must be "HTTP server" for this new marketplace
  • Base URL is IP / name with HTTP and :8080 to match the Nginx configuration
  • Directory path is used to upload images to the server by the oneadmin user. It is not an HTTP based reference.
  • Storage bridge list is IP addresses (or URLs) of the server(s) that the system will ssh image files to - i.e. the new marketplace server's IP. If it is left blank, it will assume that the market place is located on the local host (itself).

Marketplace