6 ms·

> and then it's probably better to commit and learn the ins and outs of that specific problem domain. Even more, it's probably better to start off with a full

by derefr 5y ago

> and then it's probably better to commit and learn the ins and outs of that specific problem domain. Even more, it's probably better to start off with a full understanding of these systems, so you can recognize when you're Greenspunning one of these abstractions, and just switch to using a robust off-the-shelf implementation of the same abstraction. (E.g. switch from writing a hand-rolled parser, to using a real parser-combinator library, or writing a formal grammar for a PEG compiler.) Emphasis on "off-the-shelf": the most important thing to get from experience with these abstractions, is the understanding that you it will be less work to use the abstraction that has more power; and, in fact, that much of the tooling around these abstractions is built to assume that you want the full power of the tool by default, such that you have to add configuration to get less. So the workflow is actually "start off ordering 'one with everything', and then streamline your system over time, removing bits and pieces, as you notice properties of your particular situation that mean that you can do with something less powerful at particular stages."

deleted 5y ago
[deleted]
joatmon-snoo 5y ago
The problem, of course, is that it's impossible to know everything :) IME the best compromise I've found is to learn enough to recognize the problems, so that when I implement a bastardized compiler, I'm aware that that's what I'm doing and that at some point in the future I'll need to decide how deep the implementation should go or switch to actually using some library.
- derefr 5y ago
  I somewhat disagree — there really is a finite amount of Computer Science to know. The more you dive deep into different domains, the more you find yourself ending up in the same places, learning the same things with different names. What is a SQL query planner but a compiler? Is an OS scheduler really that different from an HPC workload manager, or your GPU's computer-shader arbitrator? Is RDMA against an NVMe storage pool fundamentally new to you if you've already written hierarchical chunk persistence+caching for a distributed key-value store? Etc. At some point, it's all just Computer Science. And not too much of it, either! An amount that could easily be learned in a (good) 4-year undergraduate program.
  - leeoniya 5y ago
    > An amount that could easily be learned in a (good) 4-year undergraduate program. i think any kind of undergrad program can only ever scratch the surface of some of these topics. people spend decades becoming proficient in just a single vertical (microchip architecture, distributed systems, asics/fpgas, signal processing, gpu shaders, network engineering, jit compilers, language design, color spaces, database engines, etc). sure, it's all "just" CS/EE :D look at how much knowledge/skill was required to make an even faster fizbuzz: https://news.ycombinator.com/item?id=29031488 https://news.ycombinator.com/item?id=29031488
  - joatmon-snoo 5y ago
    Yes, but also no- I see what you're saying from a high level, but my point is that the devil's in the details and that you can't know all the details of each domain, the best you can do is be aware that those details are there. Just being exposed to even a subset of the high-level is pushing the limits of a 4-year undergrad curriculum. > What is a SQL query planner but a compiler? Kind of? Lexing, sure, and to some extent IR, but an execution plan is not built from an instruction set, especially when you start talking distributed. Even if you ignore the distributed scenario, there's still a huge difference between understanding indices/columnar-storage/row-storage and implementing PGO or LTO. Plus, there's also a lot of domain complexity here- are you aiming for just ANSI compat, or are you going deeper and trying to be compatible with specific SQL Server / Oracle / Postgres features? > Is an OS scheduler really that different from an HPC workload manager, or your GPU's computer-shader arbitrator? Knowing little about HPC or GPUs, all I can say is that yes, there's absolutely a difference between the kinds of issues a kernel, hypervisor, and distributed scheduler have to deal with. Hypervisors need to integrate with higher-level RBACs, for one; distributed schedulers also need to be able to evict bad machines from the same system. Telemetry also changes massively: if you're dealing with a kernel/hypervisor, stats from things like strace etc are useful. If you're distributed? You now have multiple cardinality problems (hostname, network devices, microservice release ID). > Is RDMA against an NVMe storage pool fundamentally new to you if you've already written hierarchical chunk persistence+caching for a distributed key-value store? Again, yes (at least I assume so, I'm not super sure exactly what "NVMe storage pool" is, I assume you're talking about the protocol that SSDs speak) - e.g. what's your resilience story against a network failure (someone cutting a fiber line, or losing a subgraph of a clos fabric)?
imoverclocked 5y ago
> more power eg: log4j -- so much power you can enable random bitcoin miners with your infrastructure (tongue in cheek) ie: Sometimes it's not about more power but about the smallest/least powerful implementation that accomplishes the job. I agree that using off-the-shelf is often important but it also comes with the need to vet the thing in the first place. That's not always trivial if you care about packaged licenses from dependencies or a lack of hidden features or actively developed or a plethora of other dimensions to what the "shelf" provides.
- derefr 5y ago
  I think "tools that build tools" is a very different ecosystem from most others. When you pull in something like e.g. LLVM, you're not really having to deal with any of that. It's a clean bolt-in API, freely-licensed, no weird integration points, with many active users who've sanded off any weird expectations. I think the difference is that something like log4j has no pressure to be "the least power it can while doing the biggest job it can" — it has mostly enterprise customers who want the power, but don't care about the scope creep. Something like LLVM, on the other hand, has both big enterprise use-cases, and tiny embedded use-cases, with big players invested in both. So its size and scope have been much more well-considered, with any components that do something other than "compile these data structures representing inputs, into these other data structures representing outputs" having been long factored out into additional "plugin" sort of downstream-dependency libs.
  - imoverclocked 5y ago
    In the specific case of LLVM, I mostly agree but for different reasons. However, Node would be a good counterpoint to “tools that build tools”. There are plenty of large interests there, though maybe not so much emphasis on the embedded space. More importantly, a bug or vulnerability in Node is generally far less impactful than one in LLVM. Owning a compiler that compiles everything in an OS >> owning Node.

by derefr 5y ago

deleted 5y ago

[deleted]

joatmon-snoo 5y ago

The problem, of course, is that it's impossible to know everything :) IME the best compromise I've found is to learn enough to recognize the problems, so that when I implement a bastardized compiler, I'm aware that that's what I'm doing and that at some point in the future I'll need to decide how deep the implementation should go or switch to actually using some library.

derefr 5y ago
I somewhat disagree — there really is a finite amount of Computer Science to know. The more you dive deep into different domains, the more you find yourself ending up in the same places, learning the same things with different names. What is a SQL query planner but a compiler? Is an OS scheduler really that different from an HPC workload manager, or your GPU's computer-shader arbitrator? Is RDMA against an NVMe storage pool fundamentally new to you if you've already written hierarchical chunk persistence+caching for a distributed key-value store? Etc. At some point, it's all just Computer Science. And not too much of it, either! An amount that could easily be learned in a (good) 4-year undergraduate program.
- leeoniya 5y ago
  > An amount that could easily be learned in a (good) 4-year undergraduate program. i think any kind of undergrad program can only ever scratch the surface of some of these topics. people spend decades becoming proficient in just a single vertical (microchip architecture, distributed systems, asics/fpgas, signal processing, gpu shaders, network engineering, jit compilers, language design, color spaces, database engines, etc). sure, it's all "just" CS/EE :D look at how much knowledge/skill was required to make an even faster fizbuzz: https://news.ycombinator.com/item?id=29031488 https://news.ycombinator.com/item?id=29031488
- joatmon-snoo 5y ago
  Yes, but also no- I see what you're saying from a high level, but my point is that the devil's in the details and that you can't know all the details of each domain, the best you can do is be aware that those details are there. Just being exposed to even a subset of the high-level is pushing the limits of a 4-year undergrad curriculum. > What is a SQL query planner but a compiler? Kind of? Lexing, sure, and to some extent IR, but an execution plan is not built from an instruction set, especially when you start talking distributed. Even if you ignore the distributed scenario, there's still a huge difference between understanding indices/columnar-storage/row-storage and implementing PGO or LTO. Plus, there's also a lot of domain complexity here- are you aiming for just ANSI compat, or are you going deeper and trying to be compatible with specific SQL Server / Oracle / Postgres features? > Is an OS scheduler really that different from an HPC workload manager, or your GPU's computer-shader arbitrator? Knowing little about HPC or GPUs, all I can say is that yes, there's absolutely a difference between the kinds of issues a kernel, hypervisor, and distributed scheduler have to deal with. Hypervisors need to integrate with higher-level RBACs, for one; distributed schedulers also need to be able to evict bad machines from the same system. Telemetry also changes massively: if you're dealing with a kernel/hypervisor, stats from things like strace etc are useful. If you're distributed? You now have multiple cardinality problems (hostname, network devices, microservice release ID). > Is RDMA against an NVMe storage pool fundamentally new to you if you've already written hierarchical chunk persistence+caching for a distributed key-value store? Again, yes (at least I assume so, I'm not super sure exactly what "NVMe storage pool" is, I assume you're talking about the protocol that SSDs speak) - e.g. what's your resilience story against a network failure (someone cutting a fiber line, or losing a subgraph of a clos fabric)?

imoverclocked 5y ago

> more power eg: log4j -- so much power you can enable random bitcoin miners with your infrastructure (tongue in cheek) ie: Sometimes it's not about more power but about the smallest/least powerful implementation that accomplishes the job. I agree that using off-the-shelf is often important but it also comes with the need to vet the thing in the first place. That's not always trivial if you care about packaged licenses from dependencies or a lack of hidden features or actively developed or a plethora of other dimensions to what the "shelf" provides.

derefr 5y ago
I think "tools that build tools" is a very different ecosystem from most others. When you pull in something like e.g. LLVM, you're not really having to deal with any of that. It's a clean bolt-in API, freely-licensed, no weird integration points, with many active users who've sanded off any weird expectations. I think the difference is that something like log4j has no pressure to be "the least power it can while doing the biggest job it can" — it has mostly enterprise customers who want the power, but don't care about the scope creep. Something like LLVM, on the other hand, has both big enterprise use-cases, and tiny embedded use-cases, with big players invested in both. So its size and scope have been much more well-considered, with any components that do something other than "compile these data structures representing inputs, into these other data structures representing outputs" having been long factored out into additional "plugin" sort of downstream-dependency libs.
- imoverclocked 5y ago
  In the specific case of LLVM, I mostly agree but for different reasons. However, Node would be a good counterpoint to “tools that build tools”. There are plenty of large interests there, though maybe not so much emphasis on the embedded space. More importantly, a bug or vulnerability in Node is generally far less impactful than one in LLVM. Owning a compiler that compiles everything in an OS >> owning Node.