So, having got some instances going… I thought I better sort out the networking issues proper. While it was working, I wanted to do a few things:
- Bring a dedicated link down from my room into the rack directly for redundancy
- Define some more VLANs
- Sort out the intermittent faults being reported by Ceph
I decided to tackle (1) first. I have two 8-port Cisco SG-200 switches linked via a length of Cat5E that snakes its way from our study, through the ceiling cavity then comes up through a small hole in the floor of my room, near where two brush-tail possums call home.
I drilled a new hole next to where the existing cable entered, then came the fun of trying to feed the new cable along side the old one. First attempt had the cable nearly coil itself just inside the cavity. I tried to make a tool to grab the end of it, but it was well and truly out of reach. I ended up getting the job done by taping the cable to a section of fibreglass tubing, feeding that in, taping another section of tubing to that, feed that in, etc… but then I ran out of tubing.
Luckily, a rummage around, and I found some rigid plastic that I was able to tape to the tubing, and that got me within a half-metre of my target. Brilliant, except I forgot to put a leader cable through for next time didn’t I?
So more rummaging around for a length of suitable nylon rope, tape the rope to the Cat5E, haul the Cat5E out, then grab another length of rope and tape that to the end and use the nylon rope to haul everything back in.
The rope should be handy for when I come to install the solar panels.
I had one 16-way patch panel, so wound up terminating the rack-end with that, and just putting a RJ-45 on the end in my room and plugging that directly into the switch. So on the shopping list will be some RJ-45 wall jacks.
The cable tester tells me I possibly have brown and white-brown switched, but never mind, I’ll be re-terminating it properly when I get the parts, and that pair isn’t used anyway.
The upshot: I now have a nice 1Gbps ring loop between the two SG-200s and the LGS326 in the rack. No animals were harmed in the formation of this ring, although two possums were mildly inconvenienced. (I call that payback for the times they’ve held the Marsupial Olympics at 2AM when I’m trying to sleep!)
Having gotten the physical layer sorted out, I was able to introduce the upstairs SG-200 to the new switch, then remove the single-port LAG I had defined on the downstairs SG-200. A bit more tinkering going, and I had a nice redundant set-up: setting my laptop to ping one of the instances in the cluster over WiFi, I could unplug my upstairs trunk, wait a few seconds, plug it back in, wait some more, unplug the downstairs trunk, wait some more again, then plug in back in again, and not lose a single ICMP packet.
I moved my two switches and my AP over to the new management VLAN I had set up, along side the IPMI interfaces on the nodes. The SG-200s were easy, aside from them insisting on one port being configured with a PVID equal to the management VLAN (I guess they want to ensure you don’t get locked out), it all went smoothly.
The AP though, a Cisco WAP4410N… not so easy. In their wisdom, and unlike the SG-200s, the management VLAN settings page is separate from the IP interface page, so you can’t change both at the same time. I wound up changing the VLAN, only to find I had locked myself out of it. Much swearing at the cantankerous AP and wondering how could someone overlook such a fundamental requirement! That, and the switch where the AP plugs in, helpfully didn’t add the management VLAN to the right port like I asked of it.
Once that was sorted out, I was able to configure an IP on the old subnet and move the AP across.
That just left dealing with the intermittent issues with Ceph. My original intention with the cluster was to use 802.3AD so each node had two 2Gbps links. Except: the LGS326-AU only supports 4 LAGs. For me to do this, I need 10!
Thankfully, the bonding support in the Linux kernel has several other options available. Switching from 802.3ad to balance-tlb, resolved the issue.
slaves_bond0="enp0s20f0 enp0s20f1"
slaves_bond1="enp0s20f2 enp0s20f3"
config_bond0="null"
config_bond1="null"
config_enp0s20f0="null"
config_enp0s20f1="null"
config_enp0s20f2="null"
config_enp0s20f3="null"
rc_net_bond0_need="net.enp0s20f0 net.enp0s20f1"
rc_net_bond1_need="net.enp0s20f2 net.enp0s20f3"
mode_bond0="balance-tlb"
mode_bond1="balance-tlb"
I am now currently setting up a core router instance (with OpenBSD 6.1) and a OpenNebula instance (with Gentoo AMD64/musl libc).
Recent Comments