3 ms·
Yeah, I had the same instinct - this feels very much like a "nice idea" but the execution falls short. I mean - busily banging on sqlite like this? Shit at that
by d1l 5mo ago
Yeah, I had the same instinct - this feels very much like a "nice idea" but the execution falls short. I mean - busily banging on sqlite like this? Shit at that point just use Redis.
- andai 5mo agoWhat's the CPU usage? Like 2%? I had a manual fs polling thing a while back. It was ugly (low time budget, didn't wanna mess with the native watchers), just scanned the whole thing once per second. It averaged out to like 0.3% CPU. Not elegant, but acceptable for my purposes! (Small-ish directory, and "ping me within a second or two" was realtime enough for this use case.)
- booi 5mo agoi mean, technically this is once per millisecond, so this would happen 1000x more. In your case due to the kernel overhead you would likely not even be able to do it (300% CPU?). Either way this does seem like a very large overhead due to the fact that there's just no other way to do it without a deeper kernel integration which might be outside the scope of what sqlite is trying to do.
- nine_k 5mo agoIf the fs tree scanned once per second had 1000 files, it would be once per millisecond for a file.
- kimixa 5mo agoIf this stops the core being able to drop to a lower power state it can be whole multiples of power use on some devices. Wake ups are death for mobile form factors, even if not really doing much work.
- tptacek 5mo agoI'm not even saying it's unworkable, just, my intuition is not that the "lightweight per-millisecond select" is an optimal design.
- giraffe_lady 5mo agoReally might be in sqlite. I've learned to never trust my intuition about performance with that thing. So many times I've gone to "optimize" something and discovered that the naive hack way I had been doing it was faster anyway. It's built for this sort of bullshit.
- tptacek 5mo agoMaybe, I'm really writing about the language on this page, not about the design (I responded about this upthread).
- giraffe_lady 5mo agoOh, yes, I see what you mean now.
- koito17 5mo agoFor what it's worth, Kine (software that k3s uses to replace etcd with SQL databases) implements etcd watches on SQLite through polling[1]. The reason being that SQLite does not offer NOTIFY/LISTEN like MySQL and Postgres do. Ironically, Honkey attempts implementing NOTIFY/LISTEN through polling. k3s has been running on my home server for about three years now (using the default SQLite backend), and there doesn't seem to be excessive CPU usage despite dozens of watches existing in the simulated etcd. Of course, this doesn't say much about Honker, but it's nonetheless worth pointing out that sometimes the choice of database forces one towards a certain design. [1] https://github.com/k3s-io/kine/blob/648a2daa/pkg/logstructured/sqllog/sql.go#L452 https://github.com/k3s-io/kine/blob/648a2daa/pkg/logstructur...