4 ms·

This has not been my experience of using technology effectively. Without an understanding of the implementation details, you inevitably use something inefficien

This has not been my experience of using technology effectively. Without an understanding of the implementation details, you inevitably use something inefficiently or for not quite the right purpose. I cannot imagine anyone using a database effectively on any significant amount of data without understanding indexes, how different joins work, why join order is important, what effect join orders have on performance, etc. Get to a certain scale and it's not enough to know about indexes; you need to understand the structure of b-trees, disk I/O performance, how CPU cache performance affects b-tree navigation even when index is cached in memory, how to use compound indexes effectively to reduce random access through the index, etc. The constraints of CPU and memory never go away, and if you're trying to scale something, you're going to be limited on either or both of those resources. That in turn forces you to understand execution and memory behaviour of the abstractions you're working with. All abstractions leak when pushed.

yakshaving_jgt 7y ago
> I cannot imagine anyone using a database effectively on any significant amount of data without understanding… You'd be surprised just what proportion of systems running in the market operate on amounts of data you would not deem "significant". And as I said in another comment, Haskell doesn't try to pretend that computations run with no hardware constraints. > All abstractions leak when pushed Yeah. But we might have wildly different ideas for where that boundary is.
- barrkel 7y ago
  I wouldn't be surprised because I seek employment in areas where my skills are valuable.
tome 7y ago
> I cannot imagine anyone using a database effectively on any significant amount of data without understanding indexes, how different joins work, why join order is important, what effect join orders have on performance, etc. But conversely you probably didn't have to understand what filesystem the database runs on, whether it is in a RAID array, whether the network connection was over Ethernet or T1, etc. All abstractions leak. The question is how leaky they are. In my experience Haskell abstractions are much less leaky than most.
- barrkel 7y ago
  I did and do; when performance at the database level isn't right, you need to look into OS stats, I/O stats, disk stats (the SAN is an area ripe for unexpected contention, RAID5/6 act like a single spindle and inhibit random access in ways that RAID1 or RAID10 don't, stripe length in RAID5/6 bloats the size of small writes, etc.), but I had to stop somewhere :)
  - yakshaving_jgt 7y ago
    > but I had to stop somewhere Why? You seem unsatisfied with any of the previous layers of abstraction, so where does it end? When can I truly call myself a user of a database?
    - ukj 7y ago
      When you don't have to fix it when it stops servicing requests, or worry about scaling workloads. When you treat it as a black-to-grey box, not a grey-to-white box.
  - bryanrasmussen 7y ago
    So I'm confused - how many years did you have to study before you felt confident enough in your grasp of all the underlying concepts to write your first "Hello World"?
    - barrkel 7y ago
      Hello World doesn't push much to the limits.
- vnorilo 7y ago
  I’m not much of a haskeller, but friends’ war stories indicate that laziness leaks quite a bit.
  - tome 7y ago
    Aha, yes! You are right. But then so does strictness. See https://augustss.blogspot.com/2011/05/more-points-for-lazy-evaluation-in.html https://augustss.blogspot.com/2011/05/more-points-for-lazy-e... for some examples. Laziness is a definitely a double-edged sword, with sharp edges.
StreamBright 7y ago
This interesting, I work as a data engineer but never had to really understand how joins or indexes work because I work 99% of the time with denormalized data. I also do not know much about b-trees. I think you come from the database developer point of you rather than the database user point of view. There is also other aspect of this, even if I do not understand joins or b-trees I can measure performance so I can figure out which combintion of joins are the faster. The reason that I prefer performance testing over getting to know the theorethical background for certain things is because in many cases your performance also depend on the implementation details, not only on which tree data structure backs your implementation.
- barrkel 7y ago
  It's exactly because the performance depends on the implementation details that when you understand implementation details, it can guide what you test and help find an optimization peak faster - or come up with a theory of another peak more distant, and journey through the performance valley to find it. Implementation details follow the contours of the fundamental algorithms and data structures. You understand the outline of the implementation details - the bones, if you will - from that. And you can dive into the source (or disassembler - I've done that on Windows) to fine tune knowledge of other specifics, with e.g. gdb stack traces on the database engine when it is abnormally slow to focus attention. Without having gone to battle together over some performance problems, or me writing significantly longer battle-story posts, we're probably not going to be able to communicate effectively here.
NextHendrix 7y ago
Do you struggle to use a computer due to not having an expertise in semiconductor physics?
falcolas 7y ago
<rant> I work with a team that has a 16 TB database, who doesn't understand how it works. They seem really surprised when their queries run slow. </rant>

yakshaving_jgt 7y ago

> I cannot imagine anyone using a database effectively on any significant amount of data without understanding… You'd be surprised just what proportion of systems running in the market operate on amounts of data you would not deem "significant". And as I said in another comment, Haskell doesn't try to pretend that computations run with no hardware constraints. > All abstractions leak when pushed Yeah. But we might have wildly different ideas for where that boundary is.

barrkel 7y ago
I wouldn't be surprised because I seek employment in areas where my skills are valuable.

tome 7y ago

> I cannot imagine anyone using a database effectively on any significant amount of data without understanding indexes, how different joins work, why join order is important, what effect join orders have on performance, etc. But conversely you probably didn't have to understand what filesystem the database runs on, whether it is in a RAID array, whether the network connection was over Ethernet or T1, etc. All abstractions leak. The question is how leaky they are. In my experience Haskell abstractions are much less leaky than most.

barrkel 7y ago
I did and do; when performance at the database level isn't right, you need to look into OS stats, I/O stats, disk stats (the SAN is an area ripe for unexpected contention, RAID5/6 act like a single spindle and inhibit random access in ways that RAID1 or RAID10 don't, stripe length in RAID5/6 bloats the size of small writes, etc.), but I had to stop somewhere :)
- yakshaving_jgt 7y ago
  > but I had to stop somewhere Why? You seem unsatisfied with any of the previous layers of abstraction, so where does it end? When can I truly call myself a user of a database?
  - ukj 7y ago
    When you don't have to fix it when it stops servicing requests, or worry about scaling workloads. When you treat it as a black-to-grey box, not a grey-to-white box.
- bryanrasmussen 7y ago
  So I'm confused - how many years did you have to study before you felt confident enough in your grasp of all the underlying concepts to write your first "Hello World"?
  - barrkel 7y ago
    Hello World doesn't push much to the limits.
vnorilo 7y ago
I’m not much of a haskeller, but friends’ war stories indicate that laziness leaks quite a bit.
- tome 7y ago
  Aha, yes! You are right. But then so does strictness. See https://augustss.blogspot.com/2011/05/more-points-for-lazy-evaluation-in.html https://augustss.blogspot.com/2011/05/more-points-for-lazy-e... for some examples. Laziness is a definitely a double-edged sword, with sharp edges.

StreamBright 7y ago

This interesting, I work as a data engineer but never had to really understand how joins or indexes work because I work 99% of the time with denormalized data. I also do not know much about b-trees. I think you come from the database developer point of you rather than the database user point of view. There is also other aspect of this, even if I do not understand joins or b-trees I can measure performance so I can figure out which combintion of joins are the faster. The reason that I prefer performance testing over getting to know the theorethical background for certain things is because in many cases your performance also depend on the implementation details, not only on which tree data structure backs your implementation.

barrkel 7y ago
It's exactly because the performance depends on the implementation details that when you understand implementation details, it can guide what you test and help find an optimization peak faster - or come up with a theory of another peak more distant, and journey through the performance valley to find it. Implementation details follow the contours of the fundamental algorithms and data structures. You understand the outline of the implementation details - the bones, if you will - from that. And you can dive into the source (or disassembler - I've done that on Windows) to fine tune knowledge of other specifics, with e.g. gdb stack traces on the database engine when it is abnormally slow to focus attention. Without having gone to battle together over some performance problems, or me writing significantly longer battle-story posts, we're probably not going to be able to communicate effectively here.

NextHendrix 7y ago

Do you struggle to use a computer due to not having an expertise in semiconductor physics?

falcolas 7y ago

<rant> I work with a team that has a 16 TB database, who doesn't understand how it works. They seem really surprised when their queries run slow. </rant>