16 ms·
Nobody said anything about not having NTP in the loop! Here's the deal: RPi<5 runs at the speed of the voltage coming in. The OS creates a fake software clock
by peteforde 3y ago
Nobody said anything about not having NTP in the loop!
Here's the deal: RPi<5 runs at the speed of the voltage coming in. The OS creates a fake software clock that assumes it's a perfect 5v, but in reality not only does voltage fluctuate, it drops over the length of a cable run. If you have, I dunno, 70 Pis connected, the 5v on the first one is very different than the 5v on the last one, no matter how frequently you ask NTP to broadcast.
The first thing we did was deliver 12V down the wire with a high quality power supply. Then we used UBEC buck converter modules (intended for RC quadcopters) to deliver power directly into the Pi via its pins.
However, even now you're just chasing physics. There's no way to send a command to all of the Pis to do something right now. Instead, we tell all of them to do something at a very precise moment in about three seconds. In our case, this meant writing a very low-level C program to listen for raw UDP messages and trigger the camera module at the hardware register level.
The only way to verify that it was working as well as we believed it did was to create a reference clock. We had a friend build a 20 LED timer that would count 10ms and 1ms increments.
We achieved < 1ms syncronization across 70 Pis in this manner.
- uxp8u61q 3y agoI may be completely off base here, but maybe the rpi wasn't the best tool for the project then? What was it, if I may ask?
- peteforde 3y agoI mean, I'd be curious what you'd have suggested! We needed it to be small, low power, low heat, provide hardware level access to a camera module, be ethernet accessible. Oh, and under $100 including a fixed M-mount lens. At the time, it didn't seem like rolling our own hardware would have been faster. I would argue - given we succeeded at sub 1ms syncronization - that the Pi was close to the perfect tool. It just would have been even more perfect if it had an RTC.
- Fnoord 3y agoYou already solved your issue, but still I'll mention it: For my Pi's I buy a separate RTC (uses GPIO pins). There's various available, the DS3231 is the most expensive but also the best. If you buy it in bulk (70+) I suppose you could shave some money off. If you're OK with a cheaper one (less than half the price) which functions less well (but still does the job) then PCF8523. I got mine from a local Adafruit reseller.
- peteforde 3y agoFWIW, the cheapest RTC module we could find that had kernel support was about $31 at the time. Things have gotten a lot better and cheaper since then. Adafruit didn't launch that DS3231 until four years after we did this. We couldn't justify adding ~$2500 to the bottom line, but you can be confident that we sure did argue about it.
- Goz3rr 3y agoMissed this comment when asking why you didn't add a $2 RTC, but given that you mention Adafruit I assume making your own hardware to stick onto a Pi wasn't an option? The DS3231 you mention is a chip with mainline support since 2009. Cheaper options like a DS1307 cost less than a $0.50 a piece when you buy 100 of them, and they've been supported in Linux since 2006.
- peteforde 3y agoTL;DR we were programmers, not engineers. I'd say 50/50 that I didn't know what I2C was in 2014. Still, if we had gone and designed our own RTC daughterboard, it would have replaced the UBEC in our architecture diagram. We were buying those for $3-4 in 1000 unit quantities. So, had you been on hand a decade ago, you might have won the argument although if I put on my board member hat, fabbing our own PCB sounds a lot scarier than buying boxes of buck converters. It's hard to express how wild it is that they just pop right on and guarantee close to perfect voltage to each device. https://www.alibaba.com/product-detail/UBEC-5V-3A-5A-7A-15A_1600691062848.html https://www.alibaba.com/product-detail/UBEC-5V-3A-5A-7A-15A_...
- pomatic 3y agoAn ESP32 seems a better fit to me, and would also be cheaper and more power efficient.
- peteforde 3y agoThe ESP32 didn't launch until we were already in market. Still, I would politely suggest that you're underestimating the part where we needed low level access to a camera module that came with the RPi. Even if the ESP32 had been out, we didn't have anyone handy to spend six months reverse engineering CMOS registers. Everything is super easy a) in hindsight and b) when you're commenting on Hacker News and not actually doing it.
- pomatic 3y agoGiven that you can't share the unique details of your use case, what could have been an interesting discussion regarding design tradeoffs is moot. (And I would politely ask that you don't make assumptions about what I might or might not have actually accomplished in similar spheres, you have zero idea.)
- peteforde 3y agoIf you read the whole thread, I've gone into significant detail. Happy to answer any further questions you might have to the degree that I'm able.
- numpad0 3y agoThis is an armchair hindsight, but sounds like that needed a way to broadcast trigger GPIO interrupts, which are like JavaScript event handler for hardware. You can attach a function to a pin, and CPU force-jump into a preprogrammed address for a handler routine, which leads into your code, then return(return(return())) back to where it came from. The trigger can be delivered by, say, a Xenon flash pointed at the ceiling and a light sensor, or a dedicated pair of wires from a master Arduino, or a intentional glitch in power line, a drone receiver per each pi all on a same channel, etc. > fixed M-mount lens I assume this is meant to say metric threaded C/CS mount for surveillance cameras, as M-mount is boutique full-frame Leica mount.
- peteforde 3y agoIf I knew then what I know now, I would have spent more time on interrupts as a solution, it’s true.
- uxp8u61q 3y agoWell, if you absolutely needed to use that specific rpi camera module (if I understood your other comment correctly) and you didn't have time to reverse engineer it, then rpi was probably the best option, sure. It just feels weird to assume that, or that every single module needed to be connected by ethernet. Rpi and low power in the same sentence also feels weird. I'd have tried to build little devices out of STM32. But again, I barely know anything about your project.
- peteforde 3y agoEthernet was handy for moving the images off the Pis once they were captured. The RP2 was indeed low powered compared to the Intel/AMD powered systems we were otherwise familiar with. Outside of the bubble of people with EE skills, however big the bubble is, stopping to learn expert-level mcu programming from scratch is rarely the way things work in a software startup. Despite the obvious focus of this thread, the hardware we assembled and configured was all in support of our software.
- uxp8u61q 3y agoI assume that the rather hostile reaction you've received in this thread stems from the combination of your insistence that you tried everything to "bend the law of physics", but dismissal of any suggestion of simpler methods because you're a software person and couldn't have possibly thought/known of everything and we should just all assume that you found the optimal solution. Said differently, it's a strange combination of "we were a team of geniuses so trust me our solution was the best one" and "we were all software people so stop suggesting hardware solutions, it was too complicated for us to learn".
- peteforde 3y agoI didn't perceive a hostile reaction, beyond one dude who declared that something hard would be easy. On the contrary, I've had a number of solid back-and-forths with folks about GPS, GPIO and homebrew RTC daughterboards that we could likely find success with today - almost a decade after we built what we built. I'm honestly not sure what else you expect from me. Should I apologize for our relative ignorance at the time? Are you upset that I'm proud we managed to succeed despite doing things differently than others might have?
- eurekin 3y agoOh my.. Please, can you share more about it? Some NDAs about the function of that pi fleet?
- peteforde 3y agoUnfortunately, I probably can't share much more than the above. However, if you have more specific technical questions, I'll try to do my best.
- zh3 3y agoWe've synchronised multiple Pi's to fractions of a millisecond, with flash systems with accuracy in the microseconds for specialised motion capture. Happy to discuss if you're interested - I'm on <hn username> at 'tessierlabs . com' (without spaces).
- xorcist 3y agoWhat's with the patching? It sounds like pretty much spot on for what NTP was designed for. Each clock is monotonically increasing, with its individual drift. NTP will pretty soon find out exactly what the right polling interval is. Even if you run it in broadcast mode (why would you though, with only 70 devices).
- peteforde 3y agoIt's been eight years, but if memory serves, the primary need was to force it to broadcast as frequently as possible. Everything you said is totally legit unless you're addressing devices with software clocks that are running at speeds dictated by subtly shifting voltage. Remember, according to the Pi itself, it's 100% in sync. No problem hereeeeeeee (robot voice as voltage dips to 4.9887 volts).
- mytailorisrich 3y agoAt some point, and considering how cheap they are these days, the simplest option is to add a GPS module if your use-case allows for satellites in decent view.
- peteforde 3y agoDoes this work inside? How cheap is cheap?
- mytailorisrich 3y ago> Does this work inside It depends. Hence my previous comment. > How cheap is cheap? Google says unit retail price for module with PPS output can be under $5.
- peteforde 3y agoOur deployment environments were occasionally in places where daylight wasn't a thing. However, if we could have used GPS at a marginal cost of $5 at the time, it's very likely we would have been excited to hear about this. Thanks for the lightbulb moment.
- stavros 3y agoWould it have worked to add something like a single GPS module connected to an ESP8266 that sends a pulse every minute to all the Pis' GPIOs so they can sync?
- Pwngu 3y ago[dead]