3 ms·

I share the sentiment. While the tech is getting faster and faster, our typical ways of handling data are getting worse and worse, eliminating all the gains. De

by dpc_pw 8y ago

I share the sentiment. While the tech is getting faster and faster, our typical ways of handling data are getting worse and worse, eliminating all the gains. Devs are just tolerating slower languages, tools, architectures, methodologies etc. until a wall of some kind is being hit. "With Rust in Lambda, each 1GB file takes about 23 seconds to download and parse. That’s about a 78x speedup compared to each Python Glue worker." There should be no separate "download" and "processing". The whole thing should take exactly as much time as it takes to download the file via stream processing. Also: https://github.com/facebook/zstd https://github.com/facebook/zstd ; better ratios, better speed.

heavenlyblue 8y ago
When our industry hits the ceiling of scaling, then we'll start thinking about optimisation. Otherwise there are other low-hanging fruit lying around. On that matter - does anyone want to start a group that would lobby for making the computation more efficient exactly the same way the greens are lobbying to get rid of single-use plastics?
- dpc_pw 8y ago
  Exactly. And because of that basic business incentives no matter how fast the hardware gets, we will always have software that is just fast enough to be barely bearable, but not a notch faster. That's why I'm depressed ;).
  - heavenlyblue 8y ago
    You could also argue that computation has the same tendency as building new/better roads in the cities: the more efficient and more available it is - the more of it we will be using. E.G. as they say about the CGI - computers became more powerful, but the minutes per frame have not generally changed year on year - they've just become more detailed.
    - HereBeBeasties 8y ago
      The difference is that the roads are pushing way more traffic and the frames of CGI are getting more and more detailed / realistic / whatever. A more accurate analogy would be building a 400mph maglev train line and discovering people are riding bicycles down the tracks. Or that you've built a 400 lane highway but everyone's still stuck in traffic because they each made their cars one hundred times wider.
      - CyberDildonics 8y ago
        Artists also don't have to optimize their renders as much, leading to less time spent worry about technical details and decreased complexity, so there is some overlap with CGI. In general though, your analogy is much more true than not, since renderers are mostly optimized as much as possible.
        heavenlyblue 8y ago
        You could argue that the only reason renderers are optimised is because what they're rendering is the sole selling point of the movie. If they are not optimised, they'll not be able to deliver the novel picture that is expected to be loved by the audience.
      - heavenlyblue 8y ago
        The sole reason the road problem is known is because when we build new highways going to the city centre, we start using the roads for less important things, thus reducing the efficiency of using roads. It's a well-known problem in city planning. E.g. Instead of only using a car when you need to get there "now", you start using a car to get some coffee in the central area. So there is no difference and that's why I mentioned it.
deleted 8y ago
[deleted]

by dpc_pw 8y ago

heavenlyblue 8y ago

When our industry hits the ceiling of scaling, then we'll start thinking about optimisation. Otherwise there are other low-hanging fruit lying around. On that matter - does anyone want to start a group that would lobby for making the computation more efficient exactly the same way the greens are lobbying to get rid of single-use plastics?

dpc_pw 8y ago
Exactly. And because of that basic business incentives no matter how fast the hardware gets, we will always have software that is just fast enough to be barely bearable, but not a notch faster. That's why I'm depressed ;).
- heavenlyblue 8y ago
  You could also argue that computation has the same tendency as building new/better roads in the cities: the more efficient and more available it is - the more of it we will be using. E.G. as they say about the CGI - computers became more powerful, but the minutes per frame have not generally changed year on year - they've just become more detailed.
  - HereBeBeasties 8y ago
    The difference is that the roads are pushing way more traffic and the frames of CGI are getting more and more detailed / realistic / whatever. A more accurate analogy would be building a 400mph maglev train line and discovering people are riding bicycles down the tracks. Or that you've built a 400 lane highway but everyone's still stuck in traffic because they each made their cars one hundred times wider.
    - CyberDildonics 8y ago
      Artists also don't have to optimize their renders as much, leading to less time spent worry about technical details and decreased complexity, so there is some overlap with CGI. In general though, your analogy is much more true than not, since renderers are mostly optimized as much as possible.
      - heavenlyblue 8y ago
        You could argue that the only reason renderers are optimised is because what they're rendering is the sole selling point of the movie. If they are not optimised, they'll not be able to deliver the novel picture that is expected to be loved by the audience.
    - heavenlyblue 8y ago
      The sole reason the road problem is known is because when we build new highways going to the city centre, we start using the roads for less important things, thus reducing the efficiency of using roads. It's a well-known problem in city planning. E.g. Instead of only using a car when you need to get there "now", you start using a car to get some coffee in the central area. So there is no difference and that's why I mentioned it.

deleted 8y ago

[deleted]