14 ms·

Several years ago, I was chatting with another engineer from a close competitor. He told me about how they'd set up a system to run hundreds of data processing

Several years ago, I was chatting with another engineer from a close competitor. He told me about how they'd set up a system to run hundreds of data processing jobs a day over a dozen machines, using docker, load balancing, a bunch of AWS stuff. I knew these jobs very well, they were basically identical for any company in the space. He then mentioned that he'd noticed that somehow my employer had been processing thousands of jobs, much faster than his, and asked how many machines we were using. I didn't have the heart to tell him we were running everything manually on my two-year-old macbook air.

cobookman 6y ago
I once joked with a colleague that my sqlite3 install is faster than his Hadoop cluster for running a report across a multi-gig file. We benchmarked it, i was much, much faster. Technically though once that multi-gig file becomes many hundreds of gigs, my computer would loose by a huge margin.
- movedx 6y ago
  It's all about the right tool for the job('s scale.)
- Aeolun 6y ago
  At that point you can just get a bigger computer though.
  - ramraj07 6y ago
    No you cannot, you cannot infinitely scale SQLite, you can’t load 100 G of data into a single SQLite file in any meaningful amount of time. Then try creating an index on it and cry. I have tried this, I literally wanted to create a simple web app that is powered by the cheapest solution possible, but it had to serve from a database that cannot be smaller than 150GB. SQLite failed. Even Postgres by itself was very hard! In the end I now launch redshift for a couple days, process all the data, then pipe it to Postgres running on a lightsail vps via dblink. Haven’t found a better solution.
    - unchar1 6y ago
      Curious if you tried this on an EC2 instance in AWS? The IOPS for EBS volume are notoriously low, and possibly why a lot of self-hosted DB instances feel very slow vs similarly priced AWS services. Personal anecdote, but moving to a a dedicated server from EC2 increased the max throughput by a factor of 80 for us.
      - trashtester 6y ago
        Did you try to compare that to EC2 instances with ephemeral nvme drives? I'm seeing hdfs throughput of up to several GB/node using such instances.
      - ramraj07 6y ago
        Got burned there for sure! Speed is one thing but the cost is outrageous for io heavy apps! Anyways I moved to lightsail which doesn’t have io costs paradoxically so while io is slow at least the cost is predictable!
      - karmakaze 6y ago
        You can use locally attached SSD instances. Then you're responsible for its reliability so not getting all the 'cloud' benefits. Used them for provisioning own CI cluster with raid-0 btrfs running PostgreSQL. Only backed up the provisioning and CI scripts.
    - mschuster91 6y ago
      > but it had to serve from a database that cannot be smaller than 150GB. SQLite failed. Even Postgres by itself was very hard! The PostgreSQL database for a CMS project I work on weighs about 250GB (all assets are binary in the database), and we have no problem at all serving a boatload of requests (with the replicated database and the serving CMS running on each live server, with 8GB of RAM). To me, it smells like you've lacked some indices or ran on a rpi?
      - christophilus 6y ago
        It sounds like the op is trying to provision and load 150GB in a reasonably fast manner. Once loaded, presumably any of the usual suspects will be fast enough. It’s the up front loading costs which are the problem. Anyway, I’m curious what kind of data the op is trying to process.
- bartimus 6y ago
  Does hundreds of gigs introduce a general performance hit or could it still be further optimized using some smart indexing strategy?
  - z3t4 6y ago
    Everything is fast if it fits in memory and caches.
    - yarcob 6y ago
      Unless it's accidentally quadratic. Then all the RAM in the world isn't going to help you.
    - Silhouette 6y ago
      And in 2021, almost everything* does. *Obviously not really. But very very many things do, even doing useful jobs in production, as long as you have high enough specs.
- deleted 6y ago
  [deleted]
- StreamBright 6y ago
  You can skip Hadoop and go from SQLite to something like S3 + Presto that scales to extremely high volumes with low latency and better than linear financial scaling.
- pmlnr 6y ago
  "Command-line Tools can be 235x Faster than your Hadoop Cluster" https://web.archive.org/web/20200414235857/https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://web.archive.org/web/20200414235857/https://adamdrake...
  - Frost1x 6y ago
    I recently did some data processing on a single (albeit beefy node) someone had been using a cluster for. I composed and ran ETL in a day what took them weeks in their infrastructure (they were actually still in the process of fixing it).
movedx 6y ago
F- me. I love this. This is a really important message. It's like Jonathon Blow asks: why does Photoshop take longer to load today than it did in the 90s despite the (insane) advances in hardware? I believe it's due to a bunch of things, but over complicating the entire process is one of the big issues. If people (developers/engineers) would only sit back and realise just how much computing power they have available to them, and then realised that if they kept things simple and efficient, they could build blazing fast solutions over night. I cringe thinking about the wasted opportunities out there.
- spondyl 6y ago
  If you haven't come across it, you might appreciate this post: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
  - movedx 6y ago
    I'll give it a read. Thank you. (I like anything that echos well in this chamber of mine... just kidding ;)
- adflux 6y ago
  Compare photoshop between the 90s and now. 10x on features. Size of photos grown exponentially as well.
  - e_proxus 6y ago
    Perhaps, but size of photos should affect load times when starting the app (and in my opinion, so shouldn’t most feature either, but that depends on your architecture I suppose).
  - thefz 6y ago
    Yeah Jonathan Blow isn't exactly a luminary in computer science. I once read him going on a meltdown over linux ports because "programming is hard". This is the kind of minds Apple enable, i.e. "why isn't this easy?"
darkwater 6y ago
And what would happen the day you were hit by a bus?
- krab 6y ago
  On that day, most probably nothing with regards to this task. Then, later, probably someone would check out the scripts from a shared repo. Then, read an outdated README, try it out, swear a bit, check for correctness with someone dependent on the results, and finally learn how to do the task. There is a lot of business processes that can tolerate days or weeks of delay in case of such a tragic (and hopefully improbable) event. The trick is to know which of them can't.
  - nicbou 6y ago
    That second paragraph is very relatable.
  - darkwater 6y ago
    > There is a lot of business processes that can tolerate days or weeks of delay in case of such a tragic (and hopefully improbable) event. The trick is to know which of them can't. This is really true BUT that kind of problems are OK - nobody cares - until somebody starts caring and then all of a sudden it is urgent (exactly because they were undetected for weeks/months due to their periodicity)
    - krab 6y ago
      I meant it's fine to take calculated risks. E. g. we have less than 1% chance per year that a given person leaves us on bad terms or suffers a bad accident or illness. In case it really happens, it will cost us X in delays and extra work. To lower the probability of this risk to Y% would cost us Z (money, delay, etc). If you do this math, you can tell if it's a good idea to optimize here, or if you have more pressing issues. In my experience, this sort of one-man jobs gets automated or at least well described and checked for fear of mistakes and/or employee fraud rather than "downtime".
littlestymaar 6y ago
Reminded me of Fabrice Bellard's Pi digits record[1] The previous Pi computation record of about 2577 billion decimal digits was published by Daisuke Takahashi on August 17th 2009. The main computation lasted 29 hours and used 640 nodes of a T2K Open Supercomputer (Appro Xtreme-X3 Server). Each node contains 4 Opteron Quad Core CPUs at 2.3 GHz, giving a peak processing power of 94.2 Tflops (trillion floating point operations per second). My computation used a single Core i7 Quad Core CPU at 2.93 GHz giving a peak processing power of 46.9 Gflops. So the supercomputer is about 2000 times faster than my computer. However, my computation lasted 116 days, which is 96 times slower than the supercomputer for about the same number of digits. So my computation is roughly 20 times more efficient. [1]: https://bellard.org/pi/pi2700e9/faq.html https://bellard.org/pi/pi2700e9/faq.html
joshxyz 6y ago
God I love stories like this
tootie 6y ago
I've had similar experiences. Sometimes we'll have a dataset with tens of thousands records and it will give rise to the belief that it's a problem that requires a highly scalable solution because "tens of thousands" is more than a human can hold in their head. In reality, if the records are just a few columns of data, the whole set can be serialized to a single file and consumed in one gulp into a single object in memory on commodity hardware no sweat. Then process it with a for loop. Very few enterprises actually have big big data.
- Cthulhu_ 6y ago
  Said enterprise WISH they had big data. Or maybe it's fear, as in, 'what IF we get big data?'
  - SketchySeaBeast 6y ago
    I'm aware of a couple of companies who behave like that - "well, we could increase our user based by an order of magnitude at any point here, so better spring for the order of magnitude more expensive database, just in case we need it." Feels like toxic optimism.
    - dvfjsdhgfv 6y ago
      It's not just about scaling databases, some people are simply unable to assess reasonable limits on any system. A few years ago certain Scandinavian publisher decided to replace their standard industry tools by a single "Digital Experience Platform" that was expected to do everything. After a couple of years they understood it's a stupid idea and gave up. Then later someone in the management thought that since they already spent some millions of euros they should continue anyway. This behemoth is so slow and buggy the end users work at 1/4th speed but everyone is afraid to say anything as the ones who did have been fired. The current PM is sending weekly success messages. It's hilarious. And all because someone once had a fantasy of having one huge system that does everything.
    - spratzt 6y ago
      I love the phrase 'toxic optimism'.
api 6y ago
I wonder what percentage of data center power is wasted running totally unnecessary trendy abstractions...
medium_burrito 6y ago
My favorite part of conducting design interviews is when a candidate has pulled some complex distributed system out of their ass, and I ask them what the actual throughput/memory usage looks like.
d_burfoot 6y ago
Unfortunately, this kind of behavior will be rewarded by the job market, because he's now got a bunch more tech buzzwords on his resume than you. Call it the Resume Industrial Complex: engineers build systems with as many bells and whistles as possible, because they want to learn all the hot new tech stacks so they can show off their extensive "skills" to potential employers.

cobookman 6y ago

I once joked with a colleague that my sqlite3 install is faster than his Hadoop cluster for running a report across a multi-gig file. We benchmarked it, i was much, much faster. Technically though once that multi-gig file becomes many hundreds of gigs, my computer would loose by a huge margin.

movedx 6y ago
It's all about the right tool for the job('s scale.)
Aeolun 6y ago
At that point you can just get a bigger computer though.
- ramraj07 6y ago
  No you cannot, you cannot infinitely scale SQLite, you can’t load 100 G of data into a single SQLite file in any meaningful amount of time. Then try creating an index on it and cry. I have tried this, I literally wanted to create a simple web app that is powered by the cheapest solution possible, but it had to serve from a database that cannot be smaller than 150GB. SQLite failed. Even Postgres by itself was very hard! In the end I now launch redshift for a couple days, process all the data, then pipe it to Postgres running on a lightsail vps via dblink. Haven’t found a better solution.
  - unchar1 6y ago
    Curious if you tried this on an EC2 instance in AWS? The IOPS for EBS volume are notoriously low, and possibly why a lot of self-hosted DB instances feel very slow vs similarly priced AWS services. Personal anecdote, but moving to a a dedicated server from EC2 increased the max throughput by a factor of 80 for us.
    - trashtester 6y ago
      Did you try to compare that to EC2 instances with ephemeral nvme drives? I'm seeing hdfs throughput of up to several GB/node using such instances.
    - ramraj07 6y ago
      Got burned there for sure! Speed is one thing but the cost is outrageous for io heavy apps! Anyways I moved to lightsail which doesn’t have io costs paradoxically so while io is slow at least the cost is predictable!
    - karmakaze 6y ago
      You can use locally attached SSD instances. Then you're responsible for its reliability so not getting all the 'cloud' benefits. Used them for provisioning own CI cluster with raid-0 btrfs running PostgreSQL. Only backed up the provisioning and CI scripts.
  - mschuster91 6y ago
    > but it had to serve from a database that cannot be smaller than 150GB. SQLite failed. Even Postgres by itself was very hard! The PostgreSQL database for a CMS project I work on weighs about 250GB (all assets are binary in the database), and we have no problem at all serving a boatload of requests (with the replicated database and the serving CMS running on each live server, with 8GB of RAM). To me, it smells like you've lacked some indices or ran on a rpi?
    - christophilus 6y ago
      It sounds like the op is trying to provision and load 150GB in a reasonably fast manner. Once loaded, presumably any of the usual suspects will be fast enough. It’s the up front loading costs which are the problem. Anyway, I’m curious what kind of data the op is trying to process.
bartimus 6y ago
Does hundreds of gigs introduce a general performance hit or could it still be further optimized using some smart indexing strategy?
- z3t4 6y ago
  Everything is fast if it fits in memory and caches.
  - yarcob 6y ago
    Unless it's accidentally quadratic. Then all the RAM in the world isn't going to help you.
  - Silhouette 6y ago
    And in 2021, almost everything* does. *Obviously not really. But very very many things do, even doing useful jobs in production, as long as you have high enough specs.
deleted 6y ago
[deleted]
StreamBright 6y ago
You can skip Hadoop and go from SQLite to something like S3 + Presto that scales to extremely high volumes with low latency and better than linear financial scaling.
pmlnr 6y ago
"Command-line Tools can be 235x Faster than your Hadoop Cluster" https://web.archive.org/web/20200414235857/https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://web.archive.org/web/20200414235857/https://adamdrake...
- Frost1x 6y ago
  I recently did some data processing on a single (albeit beefy node) someone had been using a cluster for. I composed and ran ETL in a day what took them weeks in their infrastructure (they were actually still in the process of fixing it).

movedx 6y ago

F- me. I love this. This is a really important message. It's like Jonathon Blow asks: why does Photoshop take longer to load today than it did in the 90s despite the (insane) advances in hardware? I believe it's due to a bunch of things, but over complicating the entire process is one of the big issues. If people (developers/engineers) would only sit back and realise just how much computing power they have available to them, and then realised that if they kept things simple and efficient, they could build blazing fast solutions over night. I cringe thinking about the wasted opportunities out there.

spondyl 6y ago
If you haven't come across it, you might appreciate this post: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
- movedx 6y ago
  I'll give it a read. Thank you. (I like anything that echos well in this chamber of mine... just kidding ;)
adflux 6y ago
Compare photoshop between the 90s and now. 10x on features. Size of photos grown exponentially as well.
- e_proxus 6y ago
  Perhaps, but size of photos should affect load times when starting the app (and in my opinion, so shouldn’t most feature either, but that depends on your architecture I suppose).
- thefz 6y ago
  Yeah Jonathan Blow isn't exactly a luminary in computer science. I once read him going on a meltdown over linux ports because "programming is hard". This is the kind of minds Apple enable, i.e. "why isn't this easy?"

darkwater 6y ago

And what would happen the day you were hit by a bus?

krab 6y ago
On that day, most probably nothing with regards to this task. Then, later, probably someone would check out the scripts from a shared repo. Then, read an outdated README, try it out, swear a bit, check for correctness with someone dependent on the results, and finally learn how to do the task. There is a lot of business processes that can tolerate days or weeks of delay in case of such a tragic (and hopefully improbable) event. The trick is to know which of them can't.
- nicbou 6y ago
  That second paragraph is very relatable.
- darkwater 6y ago
  > There is a lot of business processes that can tolerate days or weeks of delay in case of such a tragic (and hopefully improbable) event. The trick is to know which of them can't. This is really true BUT that kind of problems are OK - nobody cares - until somebody starts caring and then all of a sudden it is urgent (exactly because they were undetected for weeks/months due to their periodicity)
  - krab 6y ago
    I meant it's fine to take calculated risks. E. g. we have less than 1% chance per year that a given person leaves us on bad terms or suffers a bad accident or illness. In case it really happens, it will cost us X in delays and extra work. To lower the probability of this risk to Y% would cost us Z (money, delay, etc). If you do this math, you can tell if it's a good idea to optimize here, or if you have more pressing issues. In my experience, this sort of one-man jobs gets automated or at least well described and checked for fear of mistakes and/or employee fraud rather than "downtime".

littlestymaar 6y ago

Reminded me of Fabrice Bellard's Pi digits record[1] The previous Pi computation record of about 2577 billion decimal digits was published by Daisuke Takahashi on August 17th 2009. The main computation lasted 29 hours and used 640 nodes of a T2K Open Supercomputer (Appro Xtreme-X3 Server). Each node contains 4 Opteron Quad Core CPUs at 2.3 GHz, giving a peak processing power of 94.2 Tflops (trillion floating point operations per second). My computation used a single Core i7 Quad Core CPU at 2.93 GHz giving a peak processing power of 46.9 Gflops. So the supercomputer is about 2000 times faster than my computer. However, my computation lasted 116 days, which is 96 times slower than the supercomputer for about the same number of digits. So my computation is roughly 20 times more efficient. [1]: https://bellard.org/pi/pi2700e9/faq.html https://bellard.org/pi/pi2700e9/faq.html

joshxyz 6y ago

God I love stories like this

tootie 6y ago

I've had similar experiences. Sometimes we'll have a dataset with tens of thousands records and it will give rise to the belief that it's a problem that requires a highly scalable solution because "tens of thousands" is more than a human can hold in their head. In reality, if the records are just a few columns of data, the whole set can be serialized to a single file and consumed in one gulp into a single object in memory on commodity hardware no sweat. Then process it with a for loop. Very few enterprises actually have big big data.

Cthulhu_ 6y ago
Said enterprise WISH they had big data. Or maybe it's fear, as in, 'what IF we get big data?'
- SketchySeaBeast 6y ago
  I'm aware of a couple of companies who behave like that - "well, we could increase our user based by an order of magnitude at any point here, so better spring for the order of magnitude more expensive database, just in case we need it." Feels like toxic optimism.
  - dvfjsdhgfv 6y ago
    It's not just about scaling databases, some people are simply unable to assess reasonable limits on any system. A few years ago certain Scandinavian publisher decided to replace their standard industry tools by a single "Digital Experience Platform" that was expected to do everything. After a couple of years they understood it's a stupid idea and gave up. Then later someone in the management thought that since they already spent some millions of euros they should continue anyway. This behemoth is so slow and buggy the end users work at 1/4th speed but everyone is afraid to say anything as the ones who did have been fired. The current PM is sending weekly success messages. It's hilarious. And all because someone once had a fantasy of having one huge system that does everything.
  - spratzt 6y ago
    I love the phrase 'toxic optimism'.

api 6y ago

I wonder what percentage of data center power is wasted running totally unnecessary trendy abstractions...

medium_burrito 6y ago

My favorite part of conducting design interviews is when a candidate has pulled some complex distributed system out of their ass, and I ask them what the actual throughput/memory usage looks like.

d_burfoot 6y ago

Unfortunately, this kind of behavior will be rewarded by the job market, because he's now got a bunch more tech buzzwords on his resume than you. Call it the Resume Industrial Complex: engineers build systems with as many bells and whistles as possible, because they want to learn all the hot new tech stacks so they can show off their extensive "skills" to potential employers.