Interconnect Setup

Before the HyperCloud nodes are powered the interconnects need to be configured. This is principally to:

• Configure the user uplink VLAN
• Configure the storage & compute ports
  • The example defaults to a 50:50 split, 18 storage node ports, 18 compute node ports.
  • This only needs to be changed if the cluster contains more than 18 of one type of node.

Preparation

1. Download and untar the SI SW Interconnect Configurations Tarball

2. The download contains the configuration scripts and example configurations.

   The scripts oob.sh and high-speed-switch.sh use the configuration files to create a list of SONiC commands that can be executed on each interconnect. It is best practice to modify the configuration files and generate the list of commands on a local machine. The commands can then be copied over to each interconnect and executed.

   Tarball contents

   ./
   ./opt/
   ./opt/softiron/
   ./opt/softiron/switch-config/
   ./opt/softiron/switch-config/docs/
   ./opt/softiron/switch-config/docs/README.md
   ./opt/softiron/switch-config/EXAMPLE/
   ./opt/softiron/switch-config/EXAMPLE/cluster.vars
   ./opt/softiron/switch-config/EXAMPLE/mgmt-switch-oob.vars
   ./opt/softiron/switch-config/EXAMPLE/high-speed-switch.vars
   ./opt/softiron/switch-config/edgecore/
   ./opt/softiron/switch-config/edgecore/4630/
   ./opt/softiron/switch-config/edgecore/4630/bin/
   ./opt/softiron/switch-config/edgecore/4630/bin/oob.sh
   ./opt/softiron/switch-config/edgecore/7326/
   ./opt/softiron/switch-config/edgecore/7326/bin/
   ./opt/softiron/switch-config/edgecore/7326/bin/high-speed-switch.sh
   

3. Copy the scripts (oob.sh and high-speed-switch.sh) and configuration files (cluster.vars, mgmt-switch-oob.vars, high-speed-switch.vars) into the same directory

4. Edit the Management interconnect configuration file (mgmt-switch-oob.vars)

   • In most cases the Management interconnect configuration does not need to be changed

5. Edit the High Speed interconnect configuration file (high-speed-switch.vars)

   • Select the uplink port:

     • 10/25 GbE: port 47
     • 40/100 GbE: port 48

   • Specify the uplink port speed:

     • 10 GbE: CUSTOMER_PORT_SLOW_SFP="TRUE"
     • 25 GbE: CUSTOMER_PORT_SLOW_SFP="FALSE"
     • 40 GbE: CUSTOMER_PORT_SLOW_QSFP="TRUE"
     • 100 GbE: CUSTOMER_PORT_SLOW_QSFP="FALSE"

   Warning

   Only one uplink port should be chosen, port 47 or port 48

6. Execute the oob.sh script to generate the Management Interconnect configuration commands:

   ./oob.sh up > up.sh
   

7. Execute the high-speed-switch.sh script to generate the primary High Speed Interconnect configuration commands:

   ./high-speed-switch.sh up primary > primary.sh
   

8. Execute the high-speed-switch.sh script to generate the secondary High Speed Interconnect configuration commands:

   ./high-speed-switch.sh up secondary > secondary.sh
   

   Example

   sudo sh -c 'cat /etc/rc.local | head -n -1 > /etc/rc.local.new'
   echo "/usr/sbin/ifconfig eth0 hw ether \$(/usr/sbin/ifconfig eth0 | grep ether | awk '{print \$2}' | awk -F: '{if (\$5==\"ff\") {\$5=\"00\"} else {\$5=sprintf(\"%02x\",(\"0x\"\$5)+1)} ; print \$1\":\"\$2\":\"\$3\":\"\$4\":\"\$5\":\"\$6\"\"}')" > /tmp/hwaddr-tmp-replace
   sudo sh -c 'cat /tmp/hwaddr-tmp-replace >> /etc/rc.local.new'
   sudo sh -c 'echo "exit 0" >> /etc/rc.local.new'
   sudo mv /etc/rc.local.new /etc/rc.local
   sudo chmod 755 /etc/rc.local
   rm -f /tmp/hwaddr-tmp-replace
   
   sudo config vlan add 1
   sudo config interface ip add eth0 10.1.2.23/24 10.1.2.1
   sudo config portchannel add PortChannel0 --fallback=true --lacp-key=1
   sudo config portchannel member add PortChannel0 Ethernet50
   sudo config portchannel member add PortChannel0 Ethernet51
   sudo config vlan member add -u 1 PortChannel0
   sudo config vlan member add -u 1 Ethernet0
   sudo config vlan member add -u 1 Ethernet1
   .
   .
   .
   sudo config vlan member add -u 1 Ethernet45
   sudo config vlan member add -u 1 Ethernet46
   sudo config vlan member add -u 1 Ethernet47
   sudo config save -y
   

Recommended

It is recommended that the interconnect configuration command scripts: up.sh, primary.sh and secondary.sh are saved for future reference

Management Interconnect

1. Connect to the Management Interconnect via either the USB A or Serial port on the left side of the device

   • Serial baud rate of 115200 baud.
   • The default credentials are: admin:YourPaSsWoRd.

2. Verify the switch boots into SONiC console.

3. Verify the interconnect is running the latest approved SONiC firmware

   • login and run show version.

   

   Warning

   If the interconnect is not running SONiC (boots to ONIE) OR if the interconnect is running an alternative release of SONiC, install the firmware image on the interconnect:

   • Enterprise SONiC Installation Image Instructions

4. On the local machine generate the SHA-256 hash of the interconnect configuration script:

   sha256sum up.sh
   

5. Compress and BASE64 encode the up.sh script:

   cat up.sh | gzip -9c | openssl base64
   

6. Copy the output into the keyboard buffer (ctrl-cor ⌘-c)

7. On the Management Interconnect create a new file up.b64 and paste the string from above (ctrl-vor ⌘-v)
8. Decode and uncompress the up.b64 file:

   cat up.b64 | base64 -d | gzip -d - > up.sh
   

9. Generate the SHA-256 hash of the up.sh script on the interconnect:

   sha256sum up.sh
   

10. Verify that the checksums from both the source and destination match EXACTLY

11. Set the command file to executable:

    sudo chmod 755 up.sh
    

12. Execute the command script:

    sudo ./up.sh
    

    The console may output failure of name resolution when configuring the interconnects, this can be ignored as there is no DNS in place to resolve these targets.

13. Verify that the script ran with no errors with the command:

    echo $?
    

    and confirm a result of 0

14. Finally sync and reboot of the interconnect:

    sync && sync && sync && sudo reboot
    

The Management Interconnect is now configured

High Speed Interconnects

The High Speed Interconnect configuration follows the same steps as the Management Interconnect.

1. Configure the primary (eth0) interconnect:

   • Follow the Management interconnect configuration procedure substituting up.sh with the primary interconnect command file primary.sh
   • After the interconnect has rebooted check the configuration:

     show mclag brief
     show vlan brief
     show vlan config
     

   • The Primary High Speed Interconnect is now configured

2. Configure the secondary (eth1) interconnect:

   • Follow the Management interconnect configuration procedure substituting up.sh with the secondary interconnect command file secondary.sh
   • After the interconnect has rebooted check the configuration:

     show mclag brief
     show vlan brief
     show vlan config
     

   • The Secondary High Speed Interconnect is now configured

Info

If any issues occur during the configuration of the interconnects, follow the link below to reset the interconnects to their default state and restart the configuration procedure:

[Enterprise SONiC] Reset default configuration

To check for errors, run echo $? after the running the configuration scripts. In the event of a non-zero exit status, the following steps will wipe the interconnect and the process can be restarted:

sudo rm /etc/sonic/config_db.json
sudo config-setup factory
sudo reboot