5 ms·

> I can’t help but look at it and see immense complexity, because I see behind the simple and intuitive userland, and that makes me run for vim and some nice b

by zvmaz 3y ago

> I can’t help but look at it and see immense complexity, because I see behind the simple and intuitive userland, and that makes me run for vim and some nice bare metal where I know what my hardware is doing. But is it humanly possible to know details in every level of abstraction? There should be a balance between abstractions and knowing some details beneath some layers of abstraction. What do you think?

spacecadet 3y ago
Why not? It's really not that many principles- to me, it is entities that add complexity by deviating standards or introducing esoteric nomenclature on-top of existing concepts.
sgarland 3y ago
Maybe not to an expert level (for some, undoubtedly it is), but it’s eminently possible to have a working knowledge of the stack, from hardware to the nth abstraction. I’ve recently become interested in how data makes it from {INSERT,SELECT} to the disk, and so set about understanding that. You can read Postgres or MySQL’s source code and use a debugger to understand how it goes through that layer, then strace to follow the base syscalls, then blkparse to watch it go to the device driver. From there, you can experiment with forcing OS page cache drops to see when a “disk read” isn’t really a disk read, and also observe read-ahead behavior. I’ve skipped the device driver layer for now, but beyond that you can also use dd to observe changes to the raw block device. The latter is easier on a simplistic level with a few bytes at a time. You have to be genuinely interested in this stuff though, because there are precious few companies who are going to pay you to play around like this all day.
- knome 3y ago
  sqlite has a fantastic codebase for understanding how databases work. there's a pretty comfortable parser generator in there as well (lemon), that I found nicer than bison when I was still messing around with them instead of just defaulting to recursive descent as I do these days.
- madaxe_again 3y ago
  Oh, it isn’t always play - sometimes you discover that you are IO bound when you shouldn’t be, and end up tracing your woes through a similar route to that which you describe to a bug in the hypervisor scheduler where network and disk IO are blocking each other due to a shared interrupt specific to the combination of disk controller and silicon that’s running the kernel. You blow a day of engineering time, but you improve performance of your product by an order of magnitude on the same hardware.
  - sgarland 3y ago
    That would be amazingly fun for me! Did you happen to file a bug report somewhere publicly? I’d love to read more about it.
    - madaxe_again 3y ago
      No, as it was a configuration issue down to the VPS provider, who are now huge but at the time were just starting out, and a known issue - we coached them through fixing it. I do have bug reports and patches floating around for all sorts of other arcane stuff, from drivers to the Linux kernel to memcached to nginx - most of which are from 10-15 years ago now, as I ended up bogged down running the business. These days The Understanding is instead deployed for living off grid and making infrastructure work.

by zvmaz 3y ago

spacecadet 3y ago

Why not? It's really not that many principles- to me, it is entities that add complexity by deviating standards or introducing esoteric nomenclature on-top of existing concepts.

sgarland 3y ago

Maybe not to an expert level (for some, undoubtedly it is), but it’s eminently possible to have a working knowledge of the stack, from hardware to the nth abstraction. I’ve recently become interested in how data makes it from {INSERT,SELECT} to the disk, and so set about understanding that. You can read Postgres or MySQL’s source code and use a debugger to understand how it goes through that layer, then strace to follow the base syscalls, then blkparse to watch it go to the device driver. From there, you can experiment with forcing OS page cache drops to see when a “disk read” isn’t really a disk read, and also observe read-ahead behavior. I’ve skipped the device driver layer for now, but beyond that you can also use dd to observe changes to the raw block device. The latter is easier on a simplistic level with a few bytes at a time. You have to be genuinely interested in this stuff though, because there are precious few companies who are going to pay you to play around like this all day.

knome 3y ago
sqlite has a fantastic codebase for understanding how databases work. there's a pretty comfortable parser generator in there as well (lemon), that I found nicer than bison when I was still messing around with them instead of just defaulting to recursive descent as I do these days.
madaxe_again 3y ago
Oh, it isn’t always play - sometimes you discover that you are IO bound when you shouldn’t be, and end up tracing your woes through a similar route to that which you describe to a bug in the hypervisor scheduler where network and disk IO are blocking each other due to a shared interrupt specific to the combination of disk controller and silicon that’s running the kernel. You blow a day of engineering time, but you improve performance of your product by an order of magnitude on the same hardware.
- sgarland 3y ago
  That would be amazingly fun for me! Did you happen to file a bug report somewhere publicly? I’d love to read more about it.
  - madaxe_again 3y ago
    No, as it was a configuration issue down to the VPS provider, who are now huge but at the time were just starting out, and a known issue - we coached them through fixing it. I do have bug reports and patches floating around for all sorts of other arcane stuff, from drivers to the Linux kernel to memcached to nginx - most of which are from 10-15 years ago now, as I ended up bogged down running the business. These days The Understanding is instead deployed for living off grid and making infrastructure work.