Here’s to those who provide closure
I don’t even need to ask if you’ve been there, I know you have, we all have. So, let’s take a moment and give thanks to those amazing human beings that, even though they solved their own issue, took the time to provide the solution and especially those who can explain how they arrived there! We solute you: the Heroes of Forums!
Our Complete Network Overhaul
We had been having issues with our network: some spotty WiFi, laggy connections, etc. When the ethernet cable that runs from the server rack, around the basement, through the office ceiling, around the living room baseboard to the 8-port switch under our television – providing the oh-so-necessary bandwidth for 150Mbps of uncompressed mindblowing video and 7.2 Atmos surround sound – had an embolism. Yes, the ethernet cord just croaked. We tested the wall jacks, replaced them, tested the terminal ends, tried new last-mile cables… the cable tester showed a short between pins 1 and 2, confirmed with the multimeter. This is not a cable you can replace with a fish tape – disappearing into finished ceilings, running through walls. Let’s just say, we were pretty pissed.
We had new network gear sitting around for about a month now, procrastinating the installation as we knew it was going to be frustration, or at minimum a full day’s undertaking. With the network issues and our most precious of Cat6 runs dead, there really wasn’t much excuse not to. So, yesterday, we yanked the EdgeSwitch 48, EdgeSwitch 16 POE, the pFSense box, the who-knows-how-old dell connect switch, and similarly aged Linksys switch and racked up our new UniFi Switch 48, UniFi Switch 24, UniFi Switch 8 POEs, and the neat (in theory) UniFi Dream Machine Pro. While my partner was running the new Cat6 cables in the cabinet, I set to running 1/2″ raceway from the server rack out to the living room on the ceiling so we can get our 4K fix – when I hear cursing from the basement.
Turns out one of our storage nodes decided to report degraded disks. I’m not 100% sure on what the issue is or how he resolved it – I know very little of CephFS and didn’t want to distract from his repair work. So I cleaned up a bit around the house until the issue was resolved… but wouldn’t you know it – yeah, yesterday was a game of whack-a-mole-tech-problems – now that the storage array was back online, none of the machines could mount the volumes. Exhausted, pissed, frustrated, and pretty much falling asleep – he decides to give up for the night and wants to watch a movie. After mixing up some delicious Moscow Mules, he passes out 4 minutes into the movie but I’m wide awake.
Enter Teagan, P.I.
There were a million possible causes for the volumes failing to mount. My first hunch was the firewall, though before he gave up my partner listed numerous networking services (that I’d never heard of before) that the UniFi Dream Machine might be blocking. Our storage array consists of 4 nodes, 3 of which are fairly new, 12 bay, 16 core beasts but the other one, the same one that reported errors earlier, is a tiny little 1U box that is attached via SAS to a dumb disk shelf (CEPH01) – in fact, it only has 2Gbps ethernet while the others have 4 Gbps. Knowing this is important to understanding the first hypothesis: journalctl reported connecting to CEPH02 but losing connection and attempting CEPH01 followed by the connection timing out completely. Seems reasonable to assume CEPH01 was causing the timeout. So, I did the cabling, redid the LAGG assignments on the switches, reset the router – nothing.
Ok, so having little knowledge of CephFS – I needed to know what might be causing timeouts when mounting the remote volumes. To the Google! Here’s the thing though: the UniFi Dream Machine is fairly new, CephFS is a little niche, and combining the two? Forget it! From around 5am to almost 9am I searched for something, anything! Sure I got a few hits that seemed possible – but ended up going down rabbit holes. Then, at 8:48am (had to check my browser history), I stumble onto this post:
https://forum.proxmox.com/threads/laggy-ceph-status-and-got-timeout-in-proxmox-gui.50118
ftrojahn‘s description sounded nearly identical – except his stack is different; ProxMox not too long ago added native CephFS support to their software – if you want some experience with a decent piece of virtualization software and enterprise-level storage solution I would definitely recommend you check it out. ftrojahn not only explained the setup, issues, and attempts to diagnose the cause very well, did what few out there dare (care):
The Heroes We Need, and Deserve
It was an issue with mismatched MTU size! Do you know why this never crossed my mind? Because of this little toggle right here on the UDMPRO’s web interface:
First, why do we care about MTU size – the default for most systems is 1500. By increasing the size of the frames, we reduce the number of packets being sent over the wire, as well as drastically reducing the ridiculous amounts of handshakes that happen between transmit and receive (if you are unfamiliar, here’s a link describing TCP handshakes). This is especially beneficial when you have terabytes of data flying around your network 24/7 like we do.
Ok, so Jumbo Frames are enabled, which should be 9000 – every host on our network has the MTU set to 9000 by default. Why is there still this timeout issue?
Well, I had luckily glanced at this post many hours earlier – notice the last comment, here’s the key piece:
Unfortunately, on newer Gen 2 devices, Jumbo Frames appear to be only 8184 bytes
https://community.ui.com/questions/When-you-enable-jumbo-frames-on-UDM-Pro-what-MTU-value-is-it-setting/04ceb4ec-aa5f-434d-abb3-2a14f3f6e1ed
Now, this little tidbit seems to be missing from any of the documentation I could find, so phastier you are a hero, we deserve more heroes in forums! The final challenge came down to the question: what the fuck do I do now? I love my partner, he has taught me so much about Linux, networking, DevOps – I wanted to show him all that knowledge has not gone to waste.
Making the UDMPRO My Bitch
It was time to learn what the hell MTUs really were and if any of the options on the web interface could help me. I found one: MSS Clamping – this sets the maximum segment size for TCP packets, maybe? HAHAHA NOPE! MSS tops out at 1452 – a little shy of the necessary 9000 (minus headers). Ok… time to get my hands dirty. The web interface isn’t the only way to configure this hunk of metal; in the past, my partner has made changes via SSH that are not available via the user interface. Since this device is a router and then some, I found it had 45 network interfaces – VLANs, bridges, loops, etc. While setting the MTU I found setting the MTU for the network interface is actually fairly easy: ip link set mtu 4096 dev eth0 I wasn’t about to run that command 45 times. Thankfully, /sys/class/net has an easily parsable list of the interface names.
ls -1 /sys/class/net | while read line ; do ip link set mtu 9000 dev $line ; doneWith that one line, there was peace in the world… Ok not really but I was so proud of finding this solution I just had to wake him up to share the good news…