4 ms·

> Not what I was expecting. I somehow associated "find console" and "board bringup" with notions of signal analysis and 23 SMA connectors and "BUG: unable to ha

> Not what I was expecting. I somehow associated "find console" and "board bringup" with notions of signal analysis and 23 SMA connectors and "BUG: unable to handle kernel paging request" and "ARGH it was a hairline solder bridge this entire time". Need to despecialize my definitions a bit. Oh it can be a bit like that sometimes. Kernel crashes, general instability, overheating, broken (or shorted) PMICs, miswired signals (turns out it's very easy to get TX and RX pairs between CPU & peripheral mixed up), wrong voltage levels, wrong clocks, peripherals missing firmware.. I've seen it all, and I've had plenty of use for the oscilloscope recently. Hopefully we'll get the next prototype soon-ish but there's still stuff that needs to be debugged :) I'm kinda spread thin between board bringup & debugging and writing new features for the customer. > How does the redundant connectivity stuff work? Just basic failover, or WAN bonding? There's more than one mechanism in place. The basic route failover stuff in Linux doesn't work very well (afaict it only detects the gateway going down, which doesn't help if the gateway is another on-prem router which stays alive when the link to ISP goes belly up). So we've got periodic ping probes (to a configurable set of IPs) which promote/demote the corresponding route when a few probes succeed/fail. We've also got a daemon which maintains multiple connections (also multiple protocols) to multiple servers through all WAN interfaces and all connections do regular keepalives; packets are rerouted to another connection when one fails. It can also duplicate packets to multiple connections simultaneously (it's often necessary to get alarms delivered to more than one place), among other things. It's sort of an overlay network that performs VPN-like functions (and more) on and off the internet, and can tunnel anything from SMS to serial ports to raw ethernet to individual TCP & UDP ports or specific protocols like FTP. Bonding as such isn't implemented but it's on the radar.

exikyut 5y ago
> Oh it can be a bit like that sometimes. Kernel crashes, general instability, overheating, broken (or shorted) PMICs, miswired signals (turns out it's very easy to get TX and RX pairs between CPU & peripheral mixed up), wrong voltage levels, wrong clocks, peripherals missing firmware.. I've seen it all, and I've had plenty of use for the oscilloscope recently. Hopefully we'll get the next prototype soon-ish but there's still stuff that needs to be debugged :) On the one hand, yikes; on the other hand, coool :) > I'm kinda spread thin between board bringup & debugging and writing new features for the customer. [Cries in long distance] Shipping hardware to AU would probably be a bit... unsustainable ;_; > There's more than one mechanism in place. The basic route failover stuff in Linux doesn't work very well (afaict it only detects the gateway going down, which doesn't help if the gateway is another on-prem router which stays alive when the link to ISP goes belly up). Oh yay. > So we've got periodic ping probes (to a configurable set of IPs) which promote/demote the corresponding route when a few probes succeed/fail. *Takes notes* I'm switching to sorta-fiber soon and the connection comes with a 4G backup. Using a modem without SMA (which no Consumer™ would ever need, right?) in a "what's service?" reception area. So I'm planning to reimplement the 4G failover with my own dongle (and antenna) :3 At least that's the theory. Not sure if it'll work. Waiting for my old DSL connection to die first so I can record the click, because I'm... weird. :P > We've also got a daemon which maintains multiple connections (also multiple protocols) to multiple servers through all WAN interfaces and all connections do regular keepalives; packets are rerouted to another connection when one fails. Cool. > It can also duplicate packets to multiple connections simultaneously (it's often necessary to get alarms delivered to more than one place), among other things. Does this use a confidence scoring system or does it just copy everything Nx -> 1:N? (...admittedly I'm now thinking of the bandwidth management in multi-SIM video broadcast systems. Not quite the same thing.) > It's sort of an overlay network that performs VPN-like functions (and more) on and off the internet, and can tunnel anything from SMS to serial ports to raw ethernet to individual TCP & UDP ports or specific protocols like FTP. O.o neat > Bonding as such isn't implemented but it's on the radar. Interesting, I can see how it's simultaneously not-so-relevant and yet would probably simplify things here and there.

exikyut 5y ago

> Oh it can be a bit like that sometimes. Kernel crashes, general instability, overheating, broken (or shorted) PMICs, miswired signals (turns out it's very easy to get TX and RX pairs between CPU & peripheral mixed up), wrong voltage levels, wrong clocks, peripherals missing firmware.. I've seen it all, and I've had plenty of use for the oscilloscope recently. Hopefully we'll get the next prototype soon-ish but there's still stuff that needs to be debugged :) On the one hand, yikes; on the other hand, coool :) > I'm kinda spread thin between board bringup & debugging and writing new features for the customer. [Cries in long distance] Shipping hardware to AU would probably be a bit... unsustainable ;_; > There's more than one mechanism in place. The basic route failover stuff in Linux doesn't work very well (afaict it only detects the gateway going down, which doesn't help if the gateway is another on-prem router which stays alive when the link to ISP goes belly up). Oh yay. > So we've got periodic ping probes (to a configurable set of IPs) which promote/demote the corresponding route when a few probes succeed/fail. *Takes notes* I'm switching to sorta-fiber soon and the connection comes with a 4G backup. Using a modem without SMA (which no Consumer™ would ever need, right?) in a "what's service?" reception area. So I'm planning to reimplement the 4G failover with my own dongle (and antenna) :3 At least that's the theory. Not sure if it'll work. Waiting for my old DSL connection to die first so I can record the click, because I'm... weird. :P > We've also got a daemon which maintains multiple connections (also multiple protocols) to multiple servers through all WAN interfaces and all connections do regular keepalives; packets are rerouted to another connection when one fails. Cool. > It can also duplicate packets to multiple connections simultaneously (it's often necessary to get alarms delivered to more than one place), among other things. Does this use a confidence scoring system or does it just copy everything Nx -> 1:N? (...admittedly I'm now thinking of the bandwidth management in multi-SIM video broadcast systems. Not quite the same thing.) > It's sort of an overlay network that performs VPN-like functions (and more) on and off the internet, and can tunnel anything from SMS to serial ports to raw ethernet to individual TCP & UDP ports or specific protocols like FTP. O.o neat > Bonding as such isn't implemented but it's on the radar. Interesting, I can see how it's simultaneously not-so-relevant and yet would probably simplify things here and there.