benbjohnson

16 ms·

61.
by benbjohnson 3y ago
You're correct that Litestream supports SFTP, however, the underlying tool here is LiteFS. I'm the author of both. Litestream works great if you're running a single node. The SFTP support works but can be slow. LiteFS builds
62.
by benbjohnson 3y ago
Thanks, Michael! Before LiteFS Cloud, I would have said that LiteFS doesn't add much value over vanilla SQLite for single-node deployments. But with the new streaming backup feature, I think it makes it easier to run single-node applic
63.
by benbjohnson 3y ago
I haven't personally but typically people use a CRDT-based approach like vlcn[1] or Mycelite[2]. [1]: https://github.com/vlcn-io/cr-sqlite [2]: https://github.com/mycelial/mycelite
64.
by benbjohnson 3y ago
Author here. I love single-instance deployments. I think they make a lot of sense when you don't need high-availability. We have quite a number of people that run Litestream on a single instance with a volume attached. We run Litestrea
65.
by benbjohnson 3y ago
Author here. We take a daily snapshot and then store incremental changes in multiple time intervals to optimize the storage usage and optimize the time to restore. Your app just sends us the incremental changes every second and we handle al
66.
by benbjohnson 3y ago
Let me know how it works for you if you get a chance to try it out. We learned a lot since making Litestream so hopefully it's a big step up! The underlying format of how we're storing snapshots and incremental backups should make
67.
by benbjohnson 3y ago
LiteFS author here. LiteFS works sorta like that. It provides read replicas on all your application servers so you can use it just like vanilla SQLite for queries. Write transactions have to occur on the primary node but that's mostly
68.
by benbjohnson 3y ago
The SQLite Session extension[1] is one option for shipping row-level change sets between databases. It may be more efficient than rsync and you can do it in batches. Can you talk more about your use case? I'm the author of LiteFS which
69.
by benbjohnson 3y ago
Litestream/LiteFS author here. It's been a while since I looked at Mycelial but IIRC it does eventually consistent replication using CRDTs. That's great for systems that need multi-master but there are consistency tradeoffs t
70.
by benbjohnson 3y ago
LiteFS author here. I’d love to compare and contrast D1 with how LiteFS works but there’s not enough technical details at the moment. I’m looking forward to future technical blog posts from Cloudflare or perhaps they could open source their
71.
by benbjohnson 3y ago
If you're just using the sync'd version as read-only or as a backup then I suspect that's fine. It seems kinda complicated but you should be ok. :)
72.
by benbjohnson 3y ago
Litestream author here. I haven't personally tried Syncthing on SQLite databases but I suspect it wouldn't work well. Syncthing doesn't have any concept of SQLite's locks so even in the best case scenario where you only
73.
by benbjohnson 4y ago
Thanks! These challenges aren't part of our hiring process. We do an at-home code challenge and then we do a short "workday" on our Slack instead. We wrote up some details about the process a while back: https://fl
74.
by benbjohnson 4y ago
I pushed up the "go mod tidy" instruction so hopefully that helps other folks that run across that issue in the future. Thanks again for catching that. As for Docker, that's probably a good idea for bundling Maestrom but it&#
75.
by benbjohnson 4y ago
I tried reproducing the error and I think I know what happened. There's an instruction to run "go mod tidy" in Go's "go mod init" command but that could be more clear in the instructions on our site. I'll
76.
by benbjohnson 4y ago
hey John, long time! Sorry that the post came off as thought leadership. We thought it'd be fun to build something like Cryptopals but for distributed systems. Building projects like this helps give awareness to Fly.io but I'd lik
77.
by benbjohnson 4y ago
Ah, you solved the first challenge! Noticing my typos. :) I fixed up and validated all the JSON. The fixed version will be up on the web site in a couple minutes. Thanks for letting me know.
78.
by benbjohnson 4y ago
Thanks, Phil! Good question. Yes, there is some overlap with the Maelstrom docs but most of the challenges either have different goals or they are new (Kafka, Totally-Available Transactions). These challenges also are aimed at people who ar
79.
by benbjohnson 4y ago
> Seems there is a bit of vendor lock-in here with flyio, the way he's throwing custom http error codes for their infra. Can I just run LiteFS on a few virtual machines and forward writes back to master? LiteFS author here. You can
80.
by benbjohnson 4y ago
> So LiteFS turns sqlite into a client server database. But SQLite being embedded is what gives it its low latency in the first place. So, I kind of have to ask - what's point? Just prefering the way sqlite does SQL over postgres? L
81.
by benbjohnson 4y ago
> I’m fairly interested in LiteFS as a distributed cache. We've been finding a lot of use cases for this at Fly.io. We'll use the static leasing in LiteFS for this as it's simpler for that use case.
82.
by benbjohnson 4y ago
> Does it support snapshots? We'll be adding an "export" feature soon to do snapshots. We'll also be adding support for streaming backups similar to what Litestream does (although it'll be cluster aware). > Fr
83.
by benbjohnson 4y ago
If you're running a single-node SQLite application then you can handle reactivity within your application code itself and use either websockets or SSE to propagate it to the client. It definitely gets more complicated when you have a d
84.
by benbjohnson 4y ago
LiteFS author here. I agree with you that most applications don't need to have all the distributed bells and whistles. Most projects that I run start out with just SQLite on a single node with Litestream and/or a cron backup. Howe
85.
by benbjohnson 4y ago
Not yet. Litestream doesn't support moving the primary node around. However, we'll be adding support for writing to a streaming back up directly in LiteFS soon. You might be able to get away with using a static primary node and ru
86.
by benbjohnson 4y ago
> I think there are some filesystems in which a crash during append can end up with the file enlarged The SQLite WAL file itself has a running checksum that starts from the beginning so if you had an enlarged file or even a block of zero
87.
by benbjohnson 4y ago
> Have you compared LZ4 to other compression algorithms, zstd for example? I looked at zstd as well. IIRC, LZ4 compresses faster and zstd has a better compression ratio. A lot of the files being compressed are short-lived so it's ty
88.
by benbjohnson 4y ago
> If a LiteFS node is way behind on replication, can it let the app know when it's caught up? LiteFS exposes the replication position via a file[1] suffixed with "-pos" for each database. That has a monotonically increasin
89.
by benbjohnson 4y ago
You'll probably hit file descriptor limits so you'll need to up those. There's also some memory overhead per connection. I'd recommend limiting the number of concurrent connections on a server if you're running on a
90.
by benbjohnson 4y ago
> What are some reasons you reckon that the current setup won't scale beyond 10GB? It's more of an arbitrary threshold right now. A lot of testing that we do right now is chaos testing where we frequently kill nodes to ensure t

More ›