Skip to content

Racking Guidelines

Racking Layout

Suggested layout scheme (from top, working downward)

  • Management switch at top
  • Interconnects
  • Compute
  • Three Static Nodes
  • Storage

Ensure that only the three (3) static nodes have M.2 drives installed and have been flashed with the latest HyperCloud disk image.

  • Double-check that all other nodes' boot drives are removed
  • Static nodes are three "fastest" storage nodes in the cluster, and typically have the most RAM.

Tip

If a cluster is made up of 8 - HD11120s and 3 - HD21216s, the 3 - HD21216s should be selected as the "static" nodes. A reasonable rule of thumb is that the third and fourth digit represents the node class, with the higher number representing the "faster" node.

Cabling

The [last job], before verifying the setup's success, is to connect the uplinks to the customer infrastructure.

Port Tables

Information

Physical Port Number Logical Port Number (in SONiC) Breakout Membership
1 Ethernet0 Ethernet5
2 Ethernet1 Ethernet5
3 Ethernet2 Ethernet5
4 Ethernet3 Ethernet6
5 Ethernet4 Ethernet6
6 Ethernet5 Ethernet5
7 Ethernet6 Ethernet6
8 Ethernet7 Ethernet10
9 Ethernet8 Ethernet6
10 Ethernet9 Ethernet10
11 Ethernet10 Ethernet10
12 Ethernet11 Ethernet10
13 Ethernet12 Ethernet17
14 Ethernet13 Ethernet17
15 Ethernet14 Ethernet17
16 Ethernet15 Ethernet18
17 Ethernet16 Ethernet18
18 Ethernet17 Ethernet17
19 Ethernet18 Ethernet18
20 Ethernet19 Ethernet20
21 Ethernet20 Ethernet18
22 Ethernet21 Ethernet20
23 Ethernet22 Ethernet20
24 Ethernet23 Ethernet20
25 Ethernet24 Ethernet29
26 Ethernet25 Ethernet29
27 Ethernet26 Ethernet29
28 Ethernet27 Ethernet30
29 Ethernet28 Ethernet30
30 Ethernet29 Ethernet29
31 Ethernet30 Ethernet30
32 Ethernet31 Ethernet34
33 Ethernet32 Ethernet30
34 Ethernet33 Ethernet34
35 Ethernet34 Ethernet34
36 Ethernet35 Ethernet34
37 Ethernet36 Ethernet41
38 Ethernet37 Ethernet41
39 Ethernet38 Ethernet41
40 Ethernet39 Ethernet39
41 Ethernet40 Ethernet39
42 Ethernet41 Ethernet41
43 Ethernet42 Ethernet39
44 Ethernet43 Ethernet43
45 Ethernet44 Ethernet39
46 Ethernet45 Ethernet43
47 Ethernet46 Ethernet43
48 Ethernet47 Ethernet43
49 Ethernet48 Ethernet48
50 Ethernet52 Ethernet52
51 Ethernet56 Ethernet56
52 Ethernet60 Ethernet60
53 Ethernet64 Ethernet64
54 Ethernet68 Ethernet68
55 Ethernet72 Ethernet72
56 Ethernet76 Ethernet76
Physical Port Number Logical Port Number (in SONiC) Breakout Membership
1 Ethernet0 Ethernet0
2 Ethernet1 Ethernet1
3 Ethernet2 Ethernet2
4 Ethernet3 Ethernet3
5 Ethernet4 Ethernet4
6 Ethernet5 Ethernet5
7 Ethernet6 Ethernet6
8 Ethernet7 Ethernet7
9 Ethernet8 Ethernet8
10 Ethernet9 Ethernet9
11 Ethernet10 Ethernet10
12 Ethernet11 Ethernet11
13 Ethernet12 Ethernet12
14 Ethernet13 Ethernet13
15 Ethernet14 Ethernet14
16 Ethernet15 Ethernet15
17 Ethernet16 Ethernet16
18 Ethernet17 Ethernet17
19 Ethernet18 Ethernet18
20 Ethernet19 Ethernet19
21 Ethernet20 Ethernet20
22 Ethernet21 Ethernet21
23 Ethernet22 Ethernet22
24 Ethernet23 Ethernet23
25 Ethernet24 Ethernet24
26 Ethernet25 Ethernet25
27 Ethernet26 Ethernet26
28 Ethernet27 Ethernet27
29 Ethernet28 Ethernet28
30 Ethernet29 Ethernet29
31 Ethernet30 Ethernet30
32 Ethernet31 Ethernet31
33 Ethernet32 Ethernet32
34 Ethernet33 Ethernet33
35 Ethernet34 Ethernet34
36 Ethernet35 Ethernet35
37 Ethernet36 Ethernet36
38 Ethernet37 Ethernet37
39 Ethernet38 Ethernet38
40 Ethernet39 Ethernet39
41 Ethernet40 Ethernet40
42 Ethernet41 Ethernet41
43 Ethernet42 Ethernet42
44 Ethernet43 Ethernet43
45 Ethernet44 Ethernet44
46 Ethernet45 Ethernet45
47 Ethernet46 Ethernet46
48 Ethernet47 Ethernet47
49 (25Gb) Ethernet48
50 (25Gb) Ethernet49
51 (25Gb) Ethernet50
52 (25Gb) Ethernet51
53 (100Gb) Ethernet52
54 (100Gb) Ethernet56

Procedure

  1. Bundling the cables into groups of four will make it easier to manage later (use the velcro ties from the DACs, etc.)

    Large Image

  2. Power Supply Units (PSUs) can be connected at any time and status lights can be verified when connections are made.

    Note

    For the ports:

    • The default example assumes up to 18 compute nodes and 18 storage nodes. Depending on cluster topology deployed, port selection may need to be modified to account for node counts.

  3. The switch ports 1 - 18 (0 - 17 logical port mapping) are reserved for storage nodes.

    Compute & Storage Nodes

  4. Ports 19 - 36 (18 - 35 logical port mapping) are reserved for the compute nodes.

  5. Port 46 is the default port for the uplink to the Management switch. Plug into ports 50 and 51 of the Management switch.

  6. Ports 72 and 76 are the 100 Gb/s cross-links between the High Speed switches, as defined in the software; however, the physical ports are 55 and 56. Plug Port 55 on High Speed switch 1 into Port 55 on High Speed switch 2. Do the same for Port 56.

    Info

    The SONiC switch mapping starts the port count at 0, not 1 as on the physical label, and it counts each of the 100 Gb ports as 4, which is why the port numbers are 72 and 76, even though they are physically adjacent.

  7. Finally, the uplink(s) to the customer infrastructure will use either the last 25 Gb port (48), or the first 100 Gb port (49).

  8. The interconnect management ports can be routed to ports 46, 47, and 48 on the Management interconnect. (This includes the management port on the management interconnect itself)