8 ms·

What Andy giveth, Bill taketh away.[0] I'm more than a little annoyed that so much data engineering is still done in Scala Spark or PySpark. Both suffer from p

What Andy giveth, Bill taketh away.[0] I'm more than a little annoyed that so much data engineering is still done in Scala Spark or PySpark. Both suffer from pretty high memory overhead, which leads to suboptimal resource utilization. I've worked with a few different systems that compile their queries into C/C++ (which is transparent to the developer). Those tend to be significantly faster or can use fewer nodes to process. I get that quick & dirty scripts for exploration don't need to be super optimized, and that throwing more hardware at the problem _can_ be cheaper than engineering time, but in my experience, the latter ends up costing my org tens of millions of dollars annually -- just write some code and allocate a ton of resources to make it work in a reasonable amount of time. I'm hopeful that Ballista[1], for example, will see uptake and improve this. [0] https://en.wikipedia.org/wiki/Andy_and_Bill%27s_law https://en.wikipedia.org/wiki/Andy_and_Bill%27s_law [1] https://github.com/apache/arrow-datafusion/tree/master/ballista https://github.com/apache/arrow-datafusion/tree/master/balli...

b9a2cab5 5y ago
Abstractions almost always end up leaky. Spark SQL, for example, does whole-stage codegen which collapses multiple project/filter stages into a single compiled stage, but your underlying data format still needs to be memory friendly (i.e. linear accesses, low branching, etc.). The codegen is very naive and the JVM JIT can only do so much. What I've seen is that you need people who deeply understand the system (e.g. Spark) to be able to tune for these edge cases (e.g. see [1] for examples of some of the tradeoffs between different processing schemes). Those people are expensive (think $500k+ annual salaries) and are really only cost effective when your compute spend is in the tens of millions or higher annually. Everyone else is using open source and throwing more compute at the problem or relying on their data scientists/data engineers to figure out what magic knob to turn. [1]: https://www.vldb.org/pvldb/vol11/p2209-kersten.pdf https://www.vldb.org/pvldb/vol11/p2209-kersten.pdf
- paulluuk 5y ago
  What's wrong with relying on data engineers for data engineering?
  - disgruntledphd2 5y ago
    Spark is very very odd to tune. Like, it seems (from my limited experience) to have the problems common to distributed data processing (skew, it's almost always skew) but because it's lazy, people end up really confused as to what actually drives the performance problems. That being said, Spark is literally the only (relatively) easy way to run distributed ML that's open source. The competitors are GPU's (if you have a GPU friendly problem) and running multiple Python processes across the network. (I'm really hoping that people will now school me, and I'll discover a much better way in the comments).
  - b9a2cab5 5y ago
    Data engineers should be building pipelines and delivering business value, not fidgeting with some JVM or Spark parameter that saves them runtime on a join (or for that matter, from what I've seen at a certain bigco, building their own custom join algorithms). That's why I said it's only economical for big companies to run efficient abstractions and everyone else just throws more compute at the issue.
Waterluvian 5y ago
What reminded me of this the other day is how MacOS will grow your cursor if you “shake” it to help you find it on a big screen. I was thinking about how they must have a routine that’s constantly taking mouse input, buffering history, and running some algorithm to determine when user input is a mouse “shake”. And how many features like this add up to eat up a nontrivial amount of resources.
- nostrademons 5y ago
  That particular example seems like something that's probably a lot cheaper than you'd initially think. The OS has to constantly take mouse input anyway to move the pointer and dispatch events to userspace. It also needs to record the current and new position of the mouse pointer to dispatch the events. Detecting whether the mouse is being "shaken" can be done with a ring buffer of mouse velocities over the last second or two of ticks. At 60 fps, that's about 120 ints = 480 bytes. Since you don't need to be precise, you can take Manhattan distance (x + y) rather than Euclidean distance (sqrt(x^2 + y^2)), which is a basically negligible computation. Add up the running total of the ring buffer - and you don't even need to visit each element, just keep a running total in a variable, add the new velocity, subtract the velocity that's about to be overwritten - and if this passes a threshold that's say 1-2 screen widths, the mouse is being "shaken" and the pointer should enlarge. In total you're looking at < 500 bytes and a few dozen CPU cycles per tick for this feature.
  - Waterluvian 5y ago
    Thanks for the thoughtful analysis and napkin math. You may very well be right. I wonder if this is true in practice or if they suffer from any interface abstractions and whatnot.
    - rollcat 5y ago
      On every modern (past few decades) platform, the mouse cursor is a hardware sprite with a dedicated, optimized, *fast* path thru every layer of the stack, just to shave off a few ms of user-perceived latency. Grab a window and shake it violently, you'll notice it lags a few pixels behind the cursor - that's the magic in action. In some places there's no room left for unnecessary abstractions, I can imagine most of the code touching mouse / cursor handling is in that category.
iaw 5y ago
Could you elaborate on other systems besides Ballista? (which looks great btw, thank you for sharing)
- mynameisash 5y ago
  U-SQL on Azure Data Lake Analytics[0] and ECL from HPCC Systems[1] are two I have fairly extensive experience with. There may be other such platforms, but those and Ballista are the three on my radar. [0] https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/data-lake-analytics/data-lake-analytics-u-sql-sdk.md#working-directory https://github.com/MicrosoftDocs/azure-docs/blob/master/arti... [1] https://wiki.hpccsystems.com/pages/viewpage.action?pageId=28803509 https://wiki.hpccsystems.com/pages/viewpage.action?pageId=28...
  - disgruntledphd2 5y ago
    ECL is the devil. There's so little documentation on it, that you basically need to work for LNRS to actually get any experience with the damn thing. If it had an SQL layer, I'd spend time evangelising it, but it's not worth learning another query language for. There exists a world where it got open-sourced before Hadoop was built, and in that world it's probably everywhere.
Spooky23 5y ago
I get a kick out of stuff like this - I’m mostly an exec these days, but I recently prototyped a small database system to feed a business process in SQLite on my laptop. To my amusement, my little SQLite prototype smoked the “enterprise” database. Turns out that a MacBook Pro SSD performs better than the SAN, and the query planner needs more tlc. We ended up running the queries off my laptop for a few days while the DBAs did their thing.
- PhilipTai 5y ago
  Right. Local storage is much more performant and cost effective than network storage. I tried to run some iops sensitive workload on cloud. It turns out I need to pay several thousands dollar per month for the performance I can get on a $100 nvme ssd.
  - noduerme 5y ago
    I'm working on a web app right now that does a lot of heavy/live/realtime processing with workers. The original thought was to run those workers on the node servers and stream the results over a socket, charging by the CPU/minute. But it surprised me that the performance looks a lot better up to about 16 workers if the user just turns over a few cores to local web workers and runs it on their own box. As long as they don't need 64 cores or something, it's faster to run locally, even on an older machine. Thread transport is slow but sockets are slower at a distance; the bottlenecks are in the main thread anyway. So I've been making sure parts are interchangeable between web workers and "remote socket web workers" assigned to each instance of the app remotely.
- KptMarchewa 5y ago
  Who the fuck thinks it's a good idea to run database on networked storage?
  - Spooky23 5y ago
    You might have heard of this little company in Seattle called “Amazon”.
SNosTrAnDbLe 5y ago
I work in the Analytics space and been mostly on Java and I am so glad that other people feel the same. At this point, people have become afraid of suggesting something other than Spark. I see something written in Rust to be much better at problems like this. I love the JVM but it works well with transactional workloads and starts showing its age when its dealing with analytical loads. The worst thing is then people start doing weak references and weird off the heap processing usually by a senior engineer but really defeats the purpose of the JVM
- robertlagrant 5y ago
  > I love the JVM but it works well with transactional workloads and starts showing its age when its dealing with analytical loads. This is interesting. Can you elaborate a bit?
  - SNosTrAnDbLe 5y ago
    Analytical loads deal with very large datasets in the order of terabytes even after you compress them. These workloads dont change much so keeping them in the heap eventually results in long processing pauses because JVM tries to recover this memory. However, in most cases, this data is not meant to be garbage collected. For transactions, once you have persisted the data, it should be garbage collected so the pattern works. There are a lot of other aspects that I can probably think of but the above one is the most important in my mind.
- wiz21c 5y ago
  I guess your company is running on Java and running something else would cost a lot in training, recruiting, understanding, etc. But down the line, defeating the JVM will be understood only by the guy who did it... Then that guy will leave... Then the newcomers will rewrite the thing in Spark 'cos it will feel safer. Rinse-repeat... (I'm totally speculating, but your story seems so true that it inspired me :-)
  - SNosTrAnDbLe 5y ago
    Some of it is what you mentioned about training and hiring costs that but mostly its this creation of the narrative that it will scale someday in the future. This is usually done by that engineer(s) and they are good at selling so a different opinion is ignored or frowned upon by the leadership. I have now seen this anti pattern in multiple places now
willvarfar 5y ago
Yes, the whole idea of sending “agents” to do processing is poor performing and things like snowflake and Trino, where queries go to already deployed code, run rings around it. Furthermore, pyspark is by far the most popular and used spark, and it’s also got the absolute world-worst atrocious mechanical sympathy. Why? Developer velocity trumps compute velocity any day? (I want the niceness of python and the performance of eg firebolt. Why must I pick?) (There is a general thing to get spark “off heap” and use generic query compute on the spark sql space, but it is miles behind those who start off there)

b9a2cab5 5y ago

Abstractions almost always end up leaky. Spark SQL, for example, does whole-stage codegen which collapses multiple project/filter stages into a single compiled stage, but your underlying data format still needs to be memory friendly (i.e. linear accesses, low branching, etc.). The codegen is very naive and the JVM JIT can only do so much. What I've seen is that you need people who deeply understand the system (e.g. Spark) to be able to tune for these edge cases (e.g. see [1] for examples of some of the tradeoffs between different processing schemes). Those people are expensive (think $500k+ annual salaries) and are really only cost effective when your compute spend is in the tens of millions or higher annually. Everyone else is using open source and throwing more compute at the problem or relying on their data scientists/data engineers to figure out what magic knob to turn. [1]: https://www.vldb.org/pvldb/vol11/p2209-kersten.pdf https://www.vldb.org/pvldb/vol11/p2209-kersten.pdf

paulluuk 5y ago
What's wrong with relying on data engineers for data engineering?
- disgruntledphd2 5y ago
  Spark is very very odd to tune. Like, it seems (from my limited experience) to have the problems common to distributed data processing (skew, it's almost always skew) but because it's lazy, people end up really confused as to what actually drives the performance problems. That being said, Spark is literally the only (relatively) easy way to run distributed ML that's open source. The competitors are GPU's (if you have a GPU friendly problem) and running multiple Python processes across the network. (I'm really hoping that people will now school me, and I'll discover a much better way in the comments).
- b9a2cab5 5y ago
  Data engineers should be building pipelines and delivering business value, not fidgeting with some JVM or Spark parameter that saves them runtime on a join (or for that matter, from what I've seen at a certain bigco, building their own custom join algorithms). That's why I said it's only economical for big companies to run efficient abstractions and everyone else just throws more compute at the issue.

Waterluvian 5y ago

What reminded me of this the other day is how MacOS will grow your cursor if you “shake” it to help you find it on a big screen. I was thinking about how they must have a routine that’s constantly taking mouse input, buffering history, and running some algorithm to determine when user input is a mouse “shake”. And how many features like this add up to eat up a nontrivial amount of resources.

nostrademons 5y ago
That particular example seems like something that's probably a lot cheaper than you'd initially think. The OS has to constantly take mouse input anyway to move the pointer and dispatch events to userspace. It also needs to record the current and new position of the mouse pointer to dispatch the events. Detecting whether the mouse is being "shaken" can be done with a ring buffer of mouse velocities over the last second or two of ticks. At 60 fps, that's about 120 ints = 480 bytes. Since you don't need to be precise, you can take Manhattan distance (x + y) rather than Euclidean distance (sqrt(x^2 + y^2)), which is a basically negligible computation. Add up the running total of the ring buffer - and you don't even need to visit each element, just keep a running total in a variable, add the new velocity, subtract the velocity that's about to be overwritten - and if this passes a threshold that's say 1-2 screen widths, the mouse is being "shaken" and the pointer should enlarge. In total you're looking at < 500 bytes and a few dozen CPU cycles per tick for this feature.
- Waterluvian 5y ago
  Thanks for the thoughtful analysis and napkin math. You may very well be right. I wonder if this is true in practice or if they suffer from any interface abstractions and whatnot.
  - rollcat 5y ago
    On every modern (past few decades) platform, the mouse cursor is a hardware sprite with a dedicated, optimized, *fast* path thru every layer of the stack, just to shave off a few ms of user-perceived latency. Grab a window and shake it violently, you'll notice it lags a few pixels behind the cursor - that's the magic in action. In some places there's no room left for unnecessary abstractions, I can imagine most of the code touching mouse / cursor handling is in that category.

iaw 5y ago

Could you elaborate on other systems besides Ballista? (which looks great btw, thank you for sharing)

mynameisash 5y ago
U-SQL on Azure Data Lake Analytics[0] and ECL from HPCC Systems[1] are two I have fairly extensive experience with. There may be other such platforms, but those and Ballista are the three on my radar. [0] https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/data-lake-analytics/data-lake-analytics-u-sql-sdk.md#working-directory https://github.com/MicrosoftDocs/azure-docs/blob/master/arti... [1] https://wiki.hpccsystems.com/pages/viewpage.action?pageId=28803509 https://wiki.hpccsystems.com/pages/viewpage.action?pageId=28...
- disgruntledphd2 5y ago
  ECL is the devil. There's so little documentation on it, that you basically need to work for LNRS to actually get any experience with the damn thing. If it had an SQL layer, I'd spend time evangelising it, but it's not worth learning another query language for. There exists a world where it got open-sourced before Hadoop was built, and in that world it's probably everywhere.

Spooky23 5y ago

I get a kick out of stuff like this - I’m mostly an exec these days, but I recently prototyped a small database system to feed a business process in SQLite on my laptop. To my amusement, my little SQLite prototype smoked the “enterprise” database. Turns out that a MacBook Pro SSD performs better than the SAN, and the query planner needs more tlc. We ended up running the queries off my laptop for a few days while the DBAs did their thing.

PhilipTai 5y ago
Right. Local storage is much more performant and cost effective than network storage. I tried to run some iops sensitive workload on cloud. It turns out I need to pay several thousands dollar per month for the performance I can get on a $100 nvme ssd.
- noduerme 5y ago
  I'm working on a web app right now that does a lot of heavy/live/realtime processing with workers. The original thought was to run those workers on the node servers and stream the results over a socket, charging by the CPU/minute. But it surprised me that the performance looks a lot better up to about 16 workers if the user just turns over a few cores to local web workers and runs it on their own box. As long as they don't need 64 cores or something, it's faster to run locally, even on an older machine. Thread transport is slow but sockets are slower at a distance; the bottlenecks are in the main thread anyway. So I've been making sure parts are interchangeable between web workers and "remote socket web workers" assigned to each instance of the app remotely.
KptMarchewa 5y ago
Who the fuck thinks it's a good idea to run database on networked storage?
- Spooky23 5y ago
  You might have heard of this little company in Seattle called “Amazon”.

SNosTrAnDbLe 5y ago

I work in the Analytics space and been mostly on Java and I am so glad that other people feel the same. At this point, people have become afraid of suggesting something other than Spark. I see something written in Rust to be much better at problems like this. I love the JVM but it works well with transactional workloads and starts showing its age when its dealing with analytical loads. The worst thing is then people start doing weak references and weird off the heap processing usually by a senior engineer but really defeats the purpose of the JVM

robertlagrant 5y ago
> I love the JVM but it works well with transactional workloads and starts showing its age when its dealing with analytical loads. This is interesting. Can you elaborate a bit?
- SNosTrAnDbLe 5y ago
  Analytical loads deal with very large datasets in the order of terabytes even after you compress them. These workloads dont change much so keeping them in the heap eventually results in long processing pauses because JVM tries to recover this memory. However, in most cases, this data is not meant to be garbage collected. For transactions, once you have persisted the data, it should be garbage collected so the pattern works. There are a lot of other aspects that I can probably think of but the above one is the most important in my mind.
wiz21c 5y ago
I guess your company is running on Java and running something else would cost a lot in training, recruiting, understanding, etc. But down the line, defeating the JVM will be understood only by the guy who did it... Then that guy will leave... Then the newcomers will rewrite the thing in Spark 'cos it will feel safer. Rinse-repeat... (I'm totally speculating, but your story seems so true that it inspired me :-)
- SNosTrAnDbLe 5y ago
  Some of it is what you mentioned about training and hiring costs that but mostly its this creation of the narrative that it will scale someday in the future. This is usually done by that engineer(s) and they are good at selling so a different opinion is ignored or frowned upon by the leadership. I have now seen this anti pattern in multiple places now

willvarfar 5y ago

Yes, the whole idea of sending “agents” to do processing is poor performing and things like snowflake and Trino, where queries go to already deployed code, run rings around it. Furthermore, pyspark is by far the most popular and used spark, and it’s also got the absolute world-worst atrocious mechanical sympathy. Why? Developer velocity trumps compute velocity any day? (I want the niceness of python and the performance of eg firebolt. Why must I pick?) (There is a general thing to get spark “off heap” and use generic query compute on the spark sql space, but it is miles behind those who start off there)