3 ms·

Hi, I'm on the team that works on Pebble. Partially synced WAL records are easier to detect, as they would just appear as corrupt records and we can stop WAL re

Hi, I'm on the team that works on Pebble. Partially synced WAL records are easier to detect, as they would just appear as corrupt records and we can stop WAL replay at that point. Non-WAL writes are even easier to handle as SSTable files are immutable once fully written and synced. We rely pretty heavily on fsync/fdatasync calls to guarantee that "all" the data in a given range made it. In addition to randomized crash tests, we have a suite of end-to-end integration tests on top of Cockroach, called Roachtests, that put clusters under a combination of node crash/restart scenarios and confirm data consistency.

ignoramous 6y ago
> We rely pretty heavily on fsync/fdatasync calls to guarantee that "all" the data in a given range made it. Reminds me of fsync gate: https://news.ycombinator.com/item?id=20491965 https://news.ycombinator.com/item?id=20491965 and https://news.ycombinator.com/item?id=19119991 https://news.ycombinator.com/item?id=19119991 (not implying Pebble uses fysnc incorrectly).
diehunde 6y ago
Hi! Awesome work. One of the cool things about the LevelDB codebase is the `Repository contents` section. A brief description of the most relevant modules so people can get familiarized with the code base quicker. As someone very interested in storage engines, I would love to see something similar here. Are you guys planning on adding some extra documentation to the project?