4 ms·

My point was that database performance goes far beyond "waiting for the disk." The data is laid out in specific ways with specific widths, there are decades of

by timf 18y ago

My point was that database performance goes far beyond "waiting for the disk." The data is laid out in specific ways with specific widths, there are decades of implementation experience behind making queries and inserts fast because of this. Among many other things, controlling which part of the disk based data to mmap in. These are things that a higher level language's runtime can't just optimize for you.

tptacek 18y ago
What part of laying things out in specific widths prohibits an implementation in a high level language? From 2001-2005, at Arbor Networks, I worked on a codebase that basically had to bucket the entire Internet, as seen from (say) the core routers at British Telecom, on varying timescales. Obviously, we wrote it in C. It was not optimal. What's worse, changing it was a huge nightmare. Most of the performance improvements we came up with (at least in my first year) were design issues (like messing with precision, storing runs of zeroes, things like that). Most of the bugs we has were textbook C issues, like failing to address shared memory properly. I'm totally not sold that high speed disk I/O demands C.
- timf 18y ago
  "What part of laying things out in specific widths prohibits an implementation in a high level language?" Nothing, mechanically. I was more referring to the efficiency of doing it (OK, one thing I'm learning on HN is that I can't just think of it as a casual chatroom, need to flesh comments out more). And note the original question was an actual question and also I'm sympathetic to what you're saying. I was taught that with raw pointers and memory control, the database implementation would chunk all the right stuff efficiently in and out from the disk and it could get directly to any row/column naturally with pointer arithmetic. I thought DBs even managed the disk directly and had their own buffer cache like in the kernel. This part of things doesn't look like a high level language task to me, especially because of garbage collection. And I'm sure, like with most things where I lack implementation experience, this is only the most cursory view of the field. My preferred mode of work is to use Python and if there's a true bottleneck/limitation from the language features then break it out into a C module. Maybe something like this would be more appropriate here instead.

by timf 18y ago

tptacek 18y ago

What part of laying things out in specific widths prohibits an implementation in a high level language? From 2001-2005, at Arbor Networks, I worked on a codebase that basically had to bucket the entire Internet, as seen from (say) the core routers at British Telecom, on varying timescales. Obviously, we wrote it in C. It was not optimal. What's worse, changing it was a huge nightmare. Most of the performance improvements we came up with (at least in my first year) were design issues (like messing with precision, storing runs of zeroes, things like that). Most of the bugs we has were textbook C issues, like failing to address shared memory properly. I'm totally not sold that high speed disk I/O demands C.

timf 18y ago
"What part of laying things out in specific widths prohibits an implementation in a high level language?" Nothing, mechanically. I was more referring to the efficiency of doing it (OK, one thing I'm learning on HN is that I can't just think of it as a casual chatroom, need to flesh comments out more). And note the original question was an actual question and also I'm sympathetic to what you're saying. I was taught that with raw pointers and memory control, the database implementation would chunk all the right stuff efficiently in and out from the disk and it could get directly to any row/column naturally with pointer arithmetic. I thought DBs even managed the disk directly and had their own buffer cache like in the kernel. This part of things doesn't look like a high level language task to me, especially because of garbage collection. And I'm sure, like with most things where I lack implementation experience, this is only the most cursory view of the field. My preferred mode of work is to use Python and if there's a true bottleneck/limitation from the language features then break it out into a C module. Maybe something like this would be more appropriate here instead.