10 ms·
Time is not a synchronization primitive
- hayley-patton 3y agoDid a devops elective last semester; I was surprised to hear that sleeping to wait for the system to come up was "common in the industry". 30 seconds was the magic number, which was somehow always either mostly a waste or not enough when I tried to use it. For similar reasons, we similarly can't keep making sleep-sort faster by scaling the sleeps down.
- drdaeman 3y agoSleeping on error is a valid pattern when combined with retries. Consider if that would be a long-starting external process that sets up the listener (or even a remote server spinning up). I.e. something we have no control over and that has no ability to signal us it's ready. In such case, sleeping a little then trying to connect (and repeating) would be simple, effective and economical solution.
- mindcrime 3y agoSleeping on error is a valid pattern when combined with retries. Agreed. Just note that it can sometimes (often?) be a good idea to add a small stochastic element to the sleep time, and/or do something similar to the exponential back-off[1] approach that Ethernet (among other things) uses. [1]: https://en.wikipedia.org/wiki/Exponential_backoff https://en.wikipedia.org/wiki/Exponential_backoff
- tylerhou 3y agoYou ideally want both exponential back off plus some random delay (to prevent thundering herd).
- chrsig 3y agoI've never personaly seen any benefit from exponential backoff... constant backoff + jitters + short timeouts has always worked well for me. If anything, people underestimating the ramp up of exponential back off(or not capping it) has left me waiting much longer than necessary
- justcool393 3y agothe amount of times something (Discord is the first example that comes to mind) starts waiting 30 seconds between retries because I wasn't connected to the network has been super absurd.
- hayley-patton 3y agoThat was "more sophisticated" and I was given a sample script without a loop. (I got Terraform to wait in a retry loop, which was apparently too complex.)
- xena 3y agoYeah for the record I'm not talking about delay loops, I'm talking about hardcoded sleeps in test code that cause random failures when unspoken invariants change. DevOps is a unique blend of hellfire that is almost unfair to compare to everything else in the industry. I'm glad I don't work in DevOps anymore.
- bheadmaster 3y ago> we similarly can't keep making sleep-sort faster by scaling the sleeps down. Technically, we can, at least to a certain limit - as long as the total time needed to schedule the sleeps is less than the shortest sleep duration, sleepsort will still work. Though we're just offloading the sorting to the scheduler, which has to implement the sorting algorithm anyway...
- FireBeyond 3y agoI ended up writing a function that would sleep and retry according to the Fibonacci sequence up to a given total elapsed time, so you could set it to a max timeout of 60s, for example, but if the system came up in 8 seconds, you'd be waiting not much longer than that (11 seconds, technically).
- bandrami 3y agoThis bleeds into the Init Wars (sorrows, prayers), but there was this idea for a while that every service was going to be directly queryable by the run control system so that there would never be a need for magic sleep numbers. The result fell rather short of the ideal (particularly on the teardown side of things) but it did at least help admins get a firmer handle on exactly where we were (and still are, for the most part) using magic waits of "long enough".
- murphyslaw 3y agoWay back I worked on a project to used virtualized routers inside labs for a bunch of use cases. The idea was to remove the need for building after building filled with equipment that just sat there idling 99% of the time. The biggest headache we had was to figure out when everything was up. There simply was no way to detect this programmatically, so the only way would have been to alter the code to give some kind of signal on the console like "System is up." similar to a linux box showing the login prompt. So I resorted to wait loops too. We ended up canceling the project.
- bandrami 3y agoThere have been several developments in the past decade or so that center on the philosophy of "make it the client's problem" (hello, Wayland) which has simultaneously led to much more robust and maintainable centralized systems with much more fragmented and unpredictable peripheral clients. I'm still not sure whether I regret this or not.
- paranoidrobot 3y agoI've been doing "devops" stuff for a long time, but only got into Kubernetes in the last 18 months or so. I've been endlessly frustrated by how prevalent "just add a <x> second sleep" is in the Kubernetes world. Examples: If you're using IRSA to provide pods an AWS IAM Role, it can take some time on first-run for the role to become usable. So your app has to just wait until it can successfully talk to AWS. If you're using the AWS Elastic Load Balancer controller, you need to add sleeps during pre-stop for a pod. The controller has to see the shutdown event, then start target deregistration on the ELB Target Group. That's a minimum of 30 seconds, during which you can be getting new connections. Service meshes on Kubernetes add some fun on both startup AND shutdown. Linkerd, for instance, will only inject it's linkerd-proxy container into your pod after startup. So it's entirely possible for your main application container to start, begin talking to the world and then a moment later all your connections get reset when Linkerd-proxy takes over the pod's networking configuration. You can fix this by injecting into all of your containers a linkerd-await utility that replaces the container entrypoint. It will wait until linkerd starts before starting your regular entrypoint. It's basically a more advanced version of sleep. Shutdown is just as fun - linkerd-proxy will immediately terminate, dropping all network connectivity for all the containers in your pod. There's a fix for this, too, which involves the above linkerd-await and another option.
- mikepurvis 3y agoWith systemd having standardized so much stuff on sd_notify for a basic liveness check, it's frustrating that container setups can't piggy-back more on that.
- arianvanp 3y agoSystemd's socket activation mechanism also got born from the "Sleep is not a synchronisation primitive; talking to a socket is!" and the container world just ignored it whilst it's a great fit! Actually it's a generalisation of what OP is doing here. Move the listener _out_ of the program and you never have to know if a service is up or not. Connecting to a service just blocks until it's up.
- jabradoodle 3y ago
- Aloha 3y agoI think the sleep N > check for availability paradigm is pretty useful. The alternatives have a ton more complexity for not much benefit.
- pmontra 3y agoI agree that it is no synchronization primitive and that even if it was, it's very fragile. However I found it in nearly every project I worked on, in backend tests, frontend tests and even in integration tests where the library driving the browser should take care of waiting for elements to appear on screen. Modals and browser dialogs are particularly good at breaking stuff. The dynamic is: developers write the the test and it usually works on their machines. If it does not there is a chance that they figure out what's wrong and rewrite it in the proper way. Tests often work locally, where the CPU and disk are nearly 100% available to run them. Sometimes they fail on CI systems. Developers scratch their heads and attempt the easy fix of adding a sleep of 1 second. That almosts always work and it took 5 minutes, code, commit, test run. They know that it stinks but they have stuff to do. Every few months somebody attempts to remove some of those sleeps with mixed and often unsuccessful results. Integration tests are particularly nasty.
- lionkor 3y agoand thus you end up with slow software
- pmontra 3y agoSlow tests.
- DougBTX 3y agoYes, though is suspect it is cultural. A team that writes slow flakey tests probably writes slow flakey production code too, for the same reasons.
- sebzim4500 3y agoI'm sure that's a factor, but having slow tests makes it much harder to notice when you have made the actual code slower.
- closeparen 3y agoA team that insists on high-performance tests probably ships lots of bugs, because the tests are so local/isolated/mock-driven that they don’t actually exercise interesting interactions.
- drdaeman 3y agoIt's a weird example, because it moves listener creation elsewhere, which is not always possible (e.g. consider if the goroutine would be spawning an external program instead of setting up the listener of its own). Go has excellent primitive for this, a channel: // Setup a channel. We'll send one and only one messsage over it, then close it. // The message will be either an error (if something went wrong) // or nil, if everything is fine and we can proceed. ready := make(chan error) go func() { // No matter how this goroutine exits, when it exits the channel will be closed defer close(ready) // Do whatever we must: open a socket, run a subprocess, or something else lis, err := net.Listen(...) if err != nil { // Let it be known we've failed ready <- err return } defer lis.Close() // Announce the "parent" goroutine we're ready, without errors ready <- nil ... }() // Here, we block and wait for our "child" goroutine to tell // us something or at least die and close the socket. // You can extend this and e.g. use `select` if you want to implement a timeout err, ok := <-ready if err != nil { return fmt.Errorf("failed to start listener goroutine: %w", err) } else if !ok { return errors.New("listener goroutine died unexpectedly") } If there are multiple tasks and the case is not complex, then use sync.WaitGroup to count the children and a shared variable to convey the error status.
- lloydatkinson 3y agoMultiple languages have channels, I’m not sure why people only discuss this in terms of Go. .NET and Rust are the first two I can name off the top of my head.
- lloydatkinson 3y ago[flagged]
- becquerel 3y agoConversely, it makes me trust the author's expertise deeply. Our industry would be nowhere without furries.
- sk0g 3y agoI don't know about the industry as a whole, but at least the (early) Rust community evangelism seemed to have a lot of them involved.
- burnished 3y agoYeah its weird, but the author consistently produces well polished and effective documents. You see that sort of conversational technique used in other places to great effect so the odd bit really is just the furry stickers part. And I guess I feel that we should be accepting of a little weirdness as an audience, especially considering the quality? I don't have that idea all figured out but the two concepts feel related.
- bowsamic 3y ago[flagged]
- stefncb 3y agoI somewhat agree with the sentiment, in the sense that I also don't enjoy tech in that way. But we're all different, and the fact that we enjoy things in different ways shouldn't mean that we can't listen to and learn from each other. The author is a very knowledgeable person that consistently writes really good articles. As an aside, don't put everyone in the same bucket, it's really not great for e.g. trans people that don't want to be associated with that kind of tech enthusiasm. This is like the new variant of strict gender roles, not fun.
- 3y ago
- sakras 3y agoThe implementers of Google Spanner would like to have a word!
- iso8859-1 3y agoWhat happens when two 802.11 devices use the frequency at the same time? I was under the impression that they'd both re-transmit after a arbitrarily chosen pause. If that is true, time would be a de-synchronization primitive. Then, it is surely also a sync primitive. How do you arrange a date with your friend in the city? I usually say "let's meet at 17:00 at the main train station, under the clock".[0] You have to know your counter-party's accuracy. But if you synchronize on any other metric, surely that could also be inaccurate? So how is time any different? [0]: Note: in Danish: https://www.berlingske.dk/under-uret-uret-skiftes-ud https://www.berlingske.dk/under-uret-uret-skiftes-ud
- CyberDildonics 3y agoIf that is true, time would be a de-synchronization primitive That's not desynchronization or synchronization. The synchronization is realizing they are both trying to connect at the same time. How do you arrange a date with your friend in the city? I usually say "let's meet at 17:00 at the main train station, under the clock".[0] The conversation is synchronization here. Synchronization is unambiguous communication.
- xyzzy_plugh 3y agoSynchronization is inherently temporal. The "chron" in the word even means time. Synchronization is unambiguous communication with respect to some time coordinates. It's certainly possible to communicate in a way that appears to be unambiguous without explicitly referencing time, but the time is always implicit. Practically, all communication is otherwise ambiguous.
- CyberDildonics 3y agoSynchronization is inherently temporal. Everything happens over time Synchronization is unambiguous communication with respect to some time coordinates. No, it could mean ordering but that doesn't mean 'time coordinates'. It's certainly possible to communicate in a way that appears to be unambiguous without explicitly referencing time That's because time isn't the most fundamental aspect, reading shared information and ordered writing are. If cars need to merge on the highway into one lane, the space between them isn't the most important aspect, deciding on the order is. There could be a little or a lot of space in between and it still works.
- samsquire 3y agoI think existing programming languages have lack of general purpose listening mechanism for state changes or events to objects in memory. People write their own adhoc observer pattern or hook pattern. I think people are forced into polling because of the lack of mechanism. Webhooks are an example of solution as an integration pattern between separate services but I'm talking about within a process, page or program. I am inspired by Smalltalk but with a twist: it is the pattern and sequence of interactions (such as events, states or messages) between MULTIPLE objects - all interacting - that are interesting. Golang's and occams select operator is powerful for switching between events from different sources. My idea: I want to be able to efficiently epoll_wait/uring a sequence of steps and state changes between multiple objects. The idea is complex event processing for POJOs or objects and states. Epoll for general purpose objects has usecases in efficient and scalable change detection in microservices, graphical user interfaces dirtychecking, job systems and workflow systems. If you have a pull or push driven system and you want to adapt it, your hand might be forced into a substandard solution to add reactivity to an event or state or event sequence. In integration testing, I see this problem. You can fire events in Javascript that your test runner can listen to, this is one solution to this problem but you have to build it yourself. You could hook into Redux state management. I have only created APIs for pages for interrogation for use by Selenium. But what I feel you want is state communication that isn't dependent on timing but efficient notification. (I don't want to use Selenium ExpectedConditions Wait for which is what I generally used to fix tests, there's a far more elegant solution)
- gpderetta 3y agoWhat do you need more complex than a pipe and your favourite unix fd multiplexer?
- samsquire 3y agoImagine you want to wait for events B(key=value) and C(key=value) all caused by common cause A as a single atomic event. You can have a descriptor that represents the combination of all that state. https://en.wikipedia.org/wiki/Complex_event_processing https://en.wikipedia.org/wiki/Complex_event_processing I think it would be helpful for microservices and graphical user interfaces for complicated user behaviour.
- Smaug123 3y agoA neat special case of "hope is not a strategy" from SRE-land!
- mberning 3y ago[flagged]
- dang 3y agoPlease don't cross into personal attack. If you know more than someone else, that's great, but then please share some of what you know so the rest of us can learn. If you don't want to do that or don't have time, another fine option is not to post anything—but swipes and putdowns only poison the ecosystem, so please don't do those. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- mgaunard 3y agoRCU would like to disagree.
- gpderetta 3y agoRCU uses epochs for synchronization. That's very different than winging it and waiting an arbitrary amount of time. In fact RCU exists exactly because the previously common method of waiting an arbitrary time before deleting a potentially shared reference and hoping for the best was no longer considered acceptable.
- mgaunard 3y agoThat's still time as a synchronization primitive.
- facu17y 3y agoTruTime is, in Google Spanner, using atomic clocks
- Nocturium 3y ago[flagged]
- platz 3y agoIt would've been nice if the author had described the change that they made instead of just writing "consider the following code". They do say "you generally want to use something that synchronizes that state", but then fail to describe exactly which method they selected in this particular example. A single sentence explaining the particular change would've been sufficient. This is a little thing but would greatly improve the communicability of the post and practices greater empathy for the reader.
- kazinator 3y ago"synchronize" literally has the word "time" in it as a greek root (χρόνο/chrónos). Time is fine for synchronizing when all components are actually reading and acting in accordance with a clock, and agree on using the same one (or a well-characterized facsimile of the same one). Time is used at the bottom layers of hardware. Certain components generate a signal, and expect the response signal to be settled within a certain time. Even in situations in which there is robust signaling (like a device can indicate for indefinite periods that it is "not ready" and another device will wait), the underlying signals on which that is based have to meet timing constraints. You have rules like "when pin X is driven high, the data is expected to be present on lines Y_0 through Y_32 within time T_x." If not, there will be garbage. All notion order in the machine rests on arbitrary timing at the lowest electronic design level.
- marcosdumay 3y ago> "synchronize" literally has the word "time" in it as a greek root (χρόνο/chrónos). Oh, yeah, the simple times before everybody learned that time is a local and observer-dependent measure. Modern computers do almost none time-based at-distance synchronization. About 2 decades ago, when I was studying asynchronous computation, hacking the chip-design process to make it possible to do time-based synchronization of nearby transistors on the same silicon crystal was a very trendy idea. It was very hard, because everything is extremely noisy, and I imagine the idea faded away, because everything is much more noisy nowadays. Doing that on any large scale is asking for everything to break. For all of the computer-era, synchronization has being done by signaling. The millenia-old tech of doing it by time is just deprecated.
- kazinator 3y agoHow does signaling work without time?
- marcosdumay 3y agoI'm having a really hard time understanding your question. If it's on the line of "signals only exist because we have time", then yeah, duh; and it's such an outrageous misconstruction of my comment that I imagine you meant something else.
- chrsig 3y agoSome thoughts: - The general point is correct -- time is not a synchronization mechanism - Flakey tests are bad for all the reasons listed - Shoddy synchronization makes for flakey tests The example chosen -- waiting for a test server to be available is probably a huge source of the issues. However, I don't think the example -- really demonstrates how to do synchronization. My core objection is that it's really relying on the operating system to take care of the synchronization, and that winds up hiding it entirely: - the dial will not happen until the operating system has competed the `listen` call - the operating system is selecting a random port at the time of the `listen` call, and reports the chosen port to the user This is the correct way to approach this problem, but it doesn't really show a proper synchronization. It shows the operating system synchronizing on behalf of the program. It's also an example that probably only shows up in test -- client and server in the same process space. In practice, many clients and servers may start at the same time in different processes or on different hosts, meaning that the client has no signal at all that the server has become ready. The best thing that can really be done is quickly polling with low timeouts until the server reports healthy. I'd have liked to see use of a `sync.Cond`, or channel to signal that the operation has completed, and the waiting goroutine being safe to continue. It wouldn't fit well in this example, because, well, the operating system provides those facilities. Over, because it does come up a lot with the example used, I'm happy there is a blog posts addresses it. Hopefully I'll see less of the problem in the world. I suppose I hope the author writes a part 2, showing other ways sleeps creep in and how to remove them.
- a-dub 3y agoon the flip side i've sometimes wondered how useful a syscall time manipulator that injects delays in order to explore the space of concurrent execution sequences would be. either something systematic or a just a simple stochastic scheduler.
- Brian_K_White 3y agoLikewise, if you think you think you need a sync(), you're wrong. idk how common that is these days, but I saw it all over, used basically blindly to "fix" all manner of mystery reliability problems and never not one of those countless examples was that ever actually the problem or the fix. It was always simply failing to close a file before trying to do something else with it from some other context like a system() command. It was used mystic-dead-chicken style even worse that sticking sleeps everywhere for no actually understood reason.