Skip to content

Reset to Factory

If you are reusing nodes from an existing HyperCloud cluster or plan to repurpose nodes from a legacy HyperDrive cluster then these steps must be performed before starting the HyperCloud conversion.

Warning

Any hardware component faults or failures (HDDs, SSDs, NICs, RAM, etc.) should be resolved prior to starting a HyperCloud conversion. Upgrading a cluster with failed hardware is not a supported operation.

Check the firmware

Check the BMC firmware has been upgraded to BMC v8.20 or later. IPMI must be enabled

ipmitool mc info | grep "Firmware Revision"

Updating the BMC firmware involves copying the new binary onto the target machine using the flash_over_ssh.sh script. Once the binary is copied the script will then trigger the system to reboot at which point the new firmware will be applied.

Equipment required

  1. Linux or macOS "host" computer connected to the target node

Prerequisites

  1. The BMC admin account needs to be enabled for all target machines requiring this upgrade. To enable the BMC admin account type bmc_password on the node console and enter a password.

  2. The BMC IP address is required for all target machines requiring this upgrade. To set a static IP address use the following procedure

  3. The new firmware tarball must be unpacked and executed on a host machine that is connected to the BMC network of all the target machines

BMC upgrade process

  1. On the host machine: Download both BMC package and extract tarball: 
    https://cdn.softiron.com/bmc/bmc-v8.20.tar.gz
  2. On all target machine(s): Enable the BMC admin account bmc_password (and enter the BMC admin password)
  3. On all target machine(s): Shutdown the OS shutdown -h now
  4. On the host machine: Install the new BMC firmware on all target machines. You will be prompted for the BMC admin password 
    ./flash_over_ssh.sh rootfs.ubifs <target-BMC-IP-addr> --factory-reset
  5. On all target machine(s): Wait for the OS to reboot

Check the hardware clock

It is recommended to set the hardware clock on all nodes to a time and date as close together as possible.

If the hardware clocks are not in sync it will take the cluster longer to converge. If the clocks are too far apart, such as if the CMOS battery has failed, then cluster convergence may take several hours.

To set the hardware clock use the following commands, where YY-mm-dd HH:MM:SS denotes the UTC year, month, day, hour, minute and second

date -s YY-mm-dd HH:MM:SS
hwclock -w

Wipe data drives

If the node(s) are from an existing HyperCloud or HyperDrive environment, it is necessary to wipe the data prior to reuse.

Stop bcache

If the device was part of a bcache SSD/HDD cache set then this should be stopped first:

for dev in /sys/block/bcache*/bcache/stop; do echo 1 > ${dev}; done

Wipe the data

On every data drive run the following command to wipe file system signatures:

wipefs -a -f /dev/XXXX

where XXXX is the disk that will have its file system wiped.

Nuking a node

If the node was part of a previous cluster build, HyperCloud may refuse to wipe an existing data disk and ingest it into the cluster. If you are unable to boot another OS to wipe the data disk(s), then you can create a cluster control fact to nuke the disks.

Run the following command:

touch /var/run/cluster-control/facts/clusternode_NODENAME_nuke_disks

Replace NODENAME with the HyperCloud hostname of the machine in question. This will create a file in the /cluster-control/facts/ directory which will be seen by HyperCloud's routine file scans and trigger the system to demolish the named node.

Install HyperCloud boot image

Install the latest HyperCloud OS image onto the boot drive of the three (3) static nodes

For a legacy HyperDrive cluster conversion SoftIron can provide replacement HyperCloud static node boot drives, imaged with the latest HyperCloud release.

Equipment required

  • Linux or macOS computer
  • USB to m.2 SSD adapter
    • Ideally, an adapter that can handle both SATA and NVMe m.2 drives, for example: Orico m.2 Enclosure.

Software required

  • Latest disk image of HyperCloud from HyperCloud Repo; for example, https://git.softiron.com/jenkins/cloud/v2.0.4/hypercloud-2.0.4-aarch64-diskimage.tar.gz

    • This will extract (tar zxvf <FILE>) into a full disk image (e.g. hypercloud-aarch64-diskimage) and a bmap file (e.g. hypercloud-aarch64-diskimage.bmap), which is a human readable description of the data in the image.

    • This is used with the bmaptool which is a disk image reader / writer that cleverly maps only the used data areas on a disk. A target disk only needs the used data areas to be written rather than simply writing the whole image including blocks of zeroes, such as a tool like dd would perform.

    • If a disk image only has 5% of its capacity used, only 5% of the disk needs to be written, which considerably speeds up the image copy compared with dd. For example, the hypercloud-aarch64-diskimage has ~1.5% usage when new.

  • Install bmap-tools, which may already be a part of the Linux distribution (e.g. apt install bmap-tools) or obtain from GitHub.

    • On macOS, download using curl -Lo bmaptool https://github.com/01org/bmap-tools/releases/download/v3.4/bmaptool; for example, version 3.4 has been tested and works (August 2023).

    • Python 3 is also needed and can be installed using Homebrew (e.g. brew install python).

      • There needs to be a link to python3 from python (e.g. sudo ln -s python3 python, where python3 is stored) or by some other way to define python environments.

Flash process

  1. Carefully remove the m.2 SSD from the SoftIron node.

  2. Connect the m.2 SSD via the m.2 to USB adapter.

    Danger

    Be absolutely certain which disk ID has been assigned to the m.2 so that the computer's OS disk is not overwritten.

    Tip

    Check dmesg, or on Linux: fdisk -l, or on macOS, diskutil list, to show storage devices. The size should help identify the m.2, or run utilities before and after inserting the SSD to see the new m.2 designation.

  3. Use bmaptool to flash the image to the m.2 working in the directory where the image and the bmap files.

    • Linux: sudo ./bmaptool copy hypercloud-aarch64-diskimage /dev/sdd
    • macOS: sudo ./bmaptool copy hypercloud-aarch64-diskimage /dev/rdisk64

    Tip

    On macOS, there is a diskX and an rdiskX ID for devices, use the rdiskX version.

  4. When process is complete, carefully remove the m.2 SSD from the adapter and place it back in its original position within the SoftIron static node.

Remove boot drives

  1. Remove all m.2 boot drives except for those installed in the three (3) HyperCloud static nodes. This is particularly applicable to legacy HyperDrive nodes.