ryanworl

8 ms·

31.
Husky: Exactly-Once Ingestion and Multi-Tenancy at Scale at Datadog(datadoghq.com)
20 points by ryanworl 4y ago|0 comments
32.
by ryanworl 4y ago
FoundationDB was never open source until 2018. They had a free binary version you could download from the website until they were acquired by Apple, though.
33.
by ryanworl 4y ago
Storage layout is not the primary issue here because IO throughput on commodity hardware has increased significantly in the last 10 years. DuckDB is significantly faster than SQLite because it has a vectorized execution engine which is more
34.
by ryanworl 4y ago
Star Schema Benchmark https://www.cs.umb.edu/~poneil/StarSchemaB.PDF
35.
by ryanworl 4y ago
"Conditional update" requires consensus to be fault tolerant, and regular Cassandra operations don't use consensus. Cassandra has a feature called light-weight transactions which can sort of get you there, and they use Paxos
36.
by ryanworl 4y ago
I'm not as familiar with the literature on replication for file systems as I am with state machine replication, so perhaps the usage of those terms have diverged since then. Regardless, I think my analysis is correct for state machine
37.
by ryanworl 4y ago
Do you attempt to guarantee linearizability of read-only operations? The scenario I'm concerned about is when a partitioned compute node is processing a read-only transaction from a partitioned client, and neither has noticed the parti
38.
by ryanworl 4y ago
"Witness" comes from Frugal Paxos [1] (AFAIK, and not cited in the Megastore paper directly from what I saw while skimming it a few minutes ago) and indeed means an acceptor who does not contain a state machine replica, but does s
39.
by ryanworl 4y ago
You're probably aware of this, but for the sake of others reading: Crashing after an fsync failure isn't sufficient if you're using buffered IO either. Dirty pages in the page cache could cause your consensus implementation t
40.
by ryanworl 4y ago
How are you ensuring the durability of the data?
41.
by ryanworl 4y ago
What is your strategy for storage? It seems that because you're offering this as SaaS, your customers both set a high bar for durability of their data and your COGS would be very high if you just kept a sheet this big in memory all the
42.
by ryanworl 4y ago
The amount is in a screenshot in the article itself with the subtitle "The amount of money that changed hands is public, because ZSF is a nonprofit."
43.
by ryanworl 4y ago
This is ultimately an unimportant distinction. Multi-Paxos and Raft have only minor differences which are only important internally. From the perspective of designing a larger database system like Spanner or CockroachDB, the differences are
44.
by ryanworl 4y ago
This is the first reference to Antithesis being used I've seen since it was announced. Can you describe what their tools are doing for you and how you're integrating with them?
45.
by ryanworl 5y ago
Sounds good! The SNIA presentation was very interesting.
46.
by ryanworl 5y ago
I watched the SNIA presentation from SDC2020 on EFS and it described each extent in the file system as a state machine replicated via multi-paxos. It seems possible to implement this feature via a time-based leader lease on the extent where
47.
by ryanworl 5y ago
> Cool, our hypothesis has been validated. This should be easy to fix: if we sync the log when we open it, we should guarantee that no unsynced reads are ever served: I'm not sure this is accurate in general. If fsync fails (either
48.
by ryanworl 5y ago
I think we're still talking about different things, but that is a good move on their part regardless. :) I mean the optional called `FreelistType` has a new option called `FreelistMapType` and the default is `FreelistArrayType`. There
49.
by ryanworl 5y ago
Is this option enabled by default? I don't this it is and I don't think they actually set it manually anywhere. EDIT: I think we're talking about two different options. I meant the ability to leave sync turned on but change t
50.
by ryanworl 5y ago
It seems that Consul does not have the ability to use the newer hashmap implementation of freelist that Alibaba implemented for etcd. I cannot find any reference to setting this option in Consul's configuration. Unfortunate, given it h
51.
by ryanworl 5y ago
Notably, the Dynamo paper is _not_ a description of how DynamoDB (the product available from AWS today) works. They are fundamentally different, and it is not possible to implement notable features like conditional updates using the algorit
52.
by ryanworl 5y ago
I think you'd still need to change the core of the database to avoid stale reads when an old primary and client are partitioned away from the new primary, or force all client communication through a proxy smart enough to contact a quor
53.
by ryanworl 5y ago
I like this feature. It has saved me from annoying bugs multiple times in Go during refactoring.
54.
by ryanworl 5y ago
Am I correct that this orders and makes SQL statements durable via Raft and then executes them serially against a local SQLite database? How would you use this to perform a transaction, or at minimum a compare-and-swap operation against a r
55.
by ryanworl 5y ago
https://mwhittaker.github.io/publications/compartmentalized_... This is both recent enough that you've probably not seen it and extremely relevant to what you're interested in!
56.
by ryanworl 5y ago
Could you provide some more details on the storage system here? Is it built on Ceph's S3 compatibility? Your durability numbers imply erasure coding. Is that the case?
57.
by ryanworl 5y ago
This is currently in progress right now. https://github.com/apple/foundationdb/blob/e7d7b39f12afa8ea2...
58.
by ryanworl 5y ago
Two quotes from the paper that I think will motivate people to read it: "Rigorous correctness testing via simulation makes FDB extremely reliable. In the past several years, CloudKit [59] has deployed FDB for more than 0.5M disk years
59.
by ryanworl 5y ago
Atomic rename is a mechanism to provide compare-and-swap, upon which you can build higher level things like transaction commit in a database built on top of the distributed file system.
60.
by ryanworl 5y ago
What does the metadata structure look like?

More ›