kentonv

56 ms·

871.
by kentonv 4y ago
Thanks for catching this. Not sure what happened exactly but jiggling the handle (republishing the post) seems to have fixed it.
872.
by kentonv 4y ago
Yeah but it involves several extra syscalls, two file descriptors, and a kernel buffer that never has any actual data in it... poll() + longjmp() out of the signal handler is actually surprisingly portable and much more efficient (but break
873.
by kentonv 4y ago
A network filesystem is still expected to respond to reads and writes in a short finite time, assuming the network isn't broken. For an arbitrary network protocol, the amount of time you wait for an event is arbitrary and application-d
874.
by kentonv 4y ago
The engine is open source: https://github.com/cloudflare/workerd We did not create our own engine to create "lock-in". On the contrary, it would be a huge win for us if we could simply run FastCGI or Node or
875.
by kentonv 4y ago
Sure, fair enough. To be clear I do not believe that async disk I/O is never useful, I just think that it's not as useful as people at first imagine when they learn about async I/O. Yes, it may be 1000 times slower than mem
876.
by kentonv 4y ago
I'm handwaving a little because I haven't actually built something like this yet. But I imagine you'd add the per-job epoll to the global epoll with EPOLLONESHOT, so that once an event is reported, it is unregistered with the
877.
by kentonv 4y ago
> And if dlopen() is called in another thread, needing to (re)allocate TLS space and/or rebuild the index? You're relying on implementation details. This could also happen when I'm not in a signal handler, and the exact
878.
by kentonv 4y ago
> removes the risk of doing too much in the signal handler Not a concern here, since the signal handler is restricted from running at any time other than during epoll_pwait(), so the usual async-signal-safety concerns don't apply. I
879.
by kentonv 4y ago
OK but whether or not this is a proper definition of "blocking" doesn't really change my point. Alternatively, maybe my point is that disk I/O is not "blocking" either, it just "takes some measurable amoun
880.
by kentonv 4y ago
> distinction about disk vs network latency Ah but that's not the distinction I'm making at all. This has nothing to do with the physical latency of the network, which can easily be less than that of a disk. It's about the
881.
by kentonv 4y ago
> Unfortunately, thread-local storage is not async-signal safe. Doesn't matter, because the signal handler in this case is strictly called "during" invocation of epoll_pwait, so there's no risk of it interrupting the
882.
by kentonv 4y ago
> This is not a useful definition of "block". I think what I'm saying is that calling file I/O "blocking" is also not a useful definition of "block". Because I don't really see the fundamental
883.
by kentonv 4y ago
None of this seems to contradict my point? My argument is that disk I/O is more like memory I/O than it is like network I/O, and so for concurrency purposes it may make more sense to treat it like you would memory I/O (u
884.
by kentonv 4y ago
Another use case for this: Say you have a set of "jobs" each composed of many "tasks" (each waiting for some event). The "jobs" are able to run concurrently on different threads, but the "tasks" must
885.
by kentonv 4y ago
> Direct read/write from memory for all intents and purposes doesn't block. Sure it does! Main memory is much slower than cache so on a cache miss the CPU has to stop and wait for main memory to respond. The CPU may even switch
886.
by kentonv 4y ago
Hmm, I don't think that's it says (unless they edited it since your post?). It mentions socketpair explicitly as something that is epoll-friendly, and which you can use to communicate with another thread, in the case where you m
887.
by kentonv 4y ago
In my case I use a thread_local pointer that I initialize right before epoll_pwait and set back to null immediately after. The pointer points to the same data structures that I would otherwise use to handle signalfd events. Yeah it's a
888.
by kentonv 4y ago
Yes, file I/O can block. However, there is an assumption that file I/O will never block "indefinitely" -- unless something is severely broken, the kernel will always finish the operation in finite time probably measured
889.
by kentonv 4y ago
I highly recommend that you do NOT use signalfd to get notification of signals through epoll. Instead, block (mask) the signal, set a signal handler, and use epoll_pwait() to atomically unblock it while you wait for events. Note that in thi
890.
by kentonv 4y ago
> Cap'n'Proto dared to dream a little more multi-service, to follow somewhat after E-language, but never materialized 3-plus-way communication. I still intend to implement this! It just has yet to become the most pressing issue
891.
by kentonv 4y ago
> If the GNU folks want to remove the wrapper scripts, they can. It's their software and they write the best practices for it. I'm not making any argument about the maintainer's rights. I am making an argument about whethe
892.
by kentonv 4y ago
Some context: egrep and fgrap are implemented as wrapper scripts that look like this: #!/bin/sh exec grep -E "$@" 28 bytes. Now consider the cost/benefit between two options: 1. Maintain these wrapper s
893.
by kentonv 4y ago
My fault. The install script refuses to install a release more than 30 days old in order to prevent downgrade attacks, but I failed to push a new release in the last 30 days. Yeah, that provision of the install script is absurdly paranoid.
894.
by kentonv 4y ago
The parts we released are the parts you'd need to host your app on other typical cloud hosting services in a typical way (e.g. using kubernetes), which should address lock-in concerns. The parts we did not release are the parts that wo
895.
by kentonv 4y ago
This does imply a hard limit on transactions per second on a particular object, since those transactions must be serialized in a thread. But, this is true of all databases: there is a limit on transactions per second operating on the same &
896.
by kentonv 4y ago
You can implement synchronization across DOs, but the application has to coordinate it. E.g. you could implement two-phase commit to get strongly consistent semantics (with the possibility of temporary unavailability if the network between
897.
by kentonv 4y ago
This is really cool as a demonstration of how distributed systems can be built on top of Durable Objects. In particular I was really happy to see this: > What about replication? What about leadership election? > There is none, at leas
898.
by kentonv 4y ago
vlovich123 built Cloudflare's R2, I think he has some experience with distributed systems. The RPC debate is confused because the two sides of the debate have different ideas of what RPC is. The anti-RPC crowd is arguing about RPC syst
899.
by kentonv 4y ago
Durable Object storage writes are not confirmed until they are on disk in three geographically separate datacenters. So, pretty reliable. The fact that there is one unique instance of the object live at a time (and therefore only one client
900.
by kentonv 4y ago
There's a bit of a catch 22 problem with these other language runtimes in that because they haven't been battle-tested in security-hostile environments, it's harder to choose them for security-critical use cases. But that in

More ›