3 ms·

> All of that means you have to code to it differently. > That requires new coding, we're on it. Can you give a rough example to guide my thinking? I understa

by bch 10y ago

> All of that means you have to code to it differently. > That requires new coding, we're on it. Can you give a rough example to guide my thinking? I understand OSes are considering ways that this has both traditional disk and RAM properties, and are sorting out storage subsystems of drivers, but I assume you're talking about something closer to user-level structures and algorithms?

wtallis 10y ago
Databases have traditionally gone to great lengths to bypass whatever heuristics the operating system is using for I/O scheduling and caching so that they can implement their own optimizations while maintaining their data consistency guarantees. Any changes to OS behavior you can imagine are also potentially applicable to databases that take this approach.
bbulkow 10y ago
Yes, I think there are clear user-level approaches you can ( and must ) take. There are some interesting talks out there about NAND, how it works, and how to optimize - I saw something here on HN a few days ago about writing your own time-series database, which got a variety of the facts wrong but was an example of how to choose data structures that are NAND-reasonable. You can look up some of my YouTube talks and slideshare, for example - I've been talking this for a while. At a high level, NAND has more IOPs than god, because they don't seek. An old enterprise spindle littlerally does 200 to 250 seeks per second. And Flash can read from 500,000 different random locations per second. That's so far apart that different user level approaches are called for. In terms of XPoint, let me give you one detail. What does a "commit" look like in XPoint? What do the different kinds of memory barriers look like? What's the best way to validate this kind of persistence on restart, which you don't have to do with DRAM? Does that change your "malloc" free list structure, because you need to validate? Is it a good idea to chop up all the space available, so you can validate different parts independently, or does that mean you end up with the multi-record transaction problem? These are the kinds of things we consider in database design on new hardware ( obligatory: we are hiring ).

by bch 10y ago

wtallis 10y ago

Databases have traditionally gone to great lengths to bypass whatever heuristics the operating system is using for I/O scheduling and caching so that they can implement their own optimizations while maintaining their data consistency guarantees. Any changes to OS behavior you can imagine are also potentially applicable to databases that take this approach.

bbulkow 10y ago

Yes, I think there are clear user-level approaches you can ( and must ) take. There are some interesting talks out there about NAND, how it works, and how to optimize - I saw something here on HN a few days ago about writing your own time-series database, which got a variety of the facts wrong but was an example of how to choose data structures that are NAND-reasonable. You can look up some of my YouTube talks and slideshare, for example - I've been talking this for a while. At a high level, NAND has more IOPs than god, because they don't seek. An old enterprise spindle littlerally does 200 to 250 seeks per second. And Flash can read from 500,000 different random locations per second. That's so far apart that different user level approaches are called for. In terms of XPoint, let me give you one detail. What does a "commit" look like in XPoint? What do the different kinds of memory barriers look like? What's the best way to validate this kind of persistence on restart, which you don't have to do with DRAM? Does that change your "malloc" free list structure, because you need to validate? Is it a good idea to chop up all the space available, so you can validate different parts independently, or does that mean you end up with the multi-record transaction problem? These are the kinds of things we consider in database design on new hardware ( obligatory: we are hiring ).