8 ms·

COST in the land of databases

203 points by jchanimal 9y ago

aub3bhat 9y ago
The author should spend some time delivering solutions to "real problems". It's easy to be snarky while writing a blogpost with toy datasets but it's difficult to actually deliver production solution in an organization. The real COST is often not the time consumed but costs that arise from maintaining heterogeneous architecture and remaining legally compliant. A database or hadoop cluster is not just an execution engine, but rather a managable, reproducible execution engine. Sure you can code up a smarter algorithm on your personal i9 workstation. However having something available as part of DB or Hadoop ecosystem enables reproducible execution while taking care of compliance (e.g. privacy), security, access control etc. Eventually we will have a containerized extensions to DB query execution flow that will allow us to have best of both worlds but for 99% practicing data scientists downloading a Gigabyte sized subset to their own workstation is not a viable option. The real opportunity lies in ability to add arbitrary compute/memory capacity to DB query execution flows.
- zie 9y ago
  I agree, but at what cost? 10% performance hit, sure, no problem, VM's take about that much from running on real hardware. but a 100% performance hit or worse... you must be crazy. Also, downloading 1GB is no big deal, downloading 1TB is a big deal, so I think your example has more merit in the > 1TB range. i.e. things that don't fit on a laptop easily.
  - aub3bhat 9y ago
    1Mb vs 1Gb vs 1Tb distinction is a function of information contained in data. If its PII even 1Mb is an orginizational headache from security/risk-management perspective. Also most commodity compute nodes have allocation around 16 Gb. The big difference comes in 16~256 Gb range where having a single powerful node can make a huge difference.
    - zie 9y ago
      Sure, but 'laptop' can be changed very easily into any container or VM box somewhere that is tied to security/etc but still on a single box. Again sizing here is sort of beside the point, the point is, if its 500% worse to use than just doing it on a single instance/compute node, you better be offering A LOT of magic awesomeness that makes the performance drain acceptable. 10% cost for the benefits that this stuff can give you, that is pretty easy to swallow. 100%-500% perf. cost is just a very, very, very hard to swallow pill.
- JoachimSchipper 9y ago
  Auditing is a real argument; but is decently-written specialized code really that much less reproducible than Hadoop/SQL? (I'll accept "yes", but note that that also rules out e.g. the fairly common practice of using R or pandas for some ad-hoc processing.)
  - aub3bhat 9y ago
    also rules out e.g. the fairly common practice of using R or pandas for some ad-hoc processing. This is essentially the reason why all organizations are adopting Spark, since it allows you to write imperative code on dataframes and build ML models.
    - rspeer 9y ago
      No, "all organizations" are not adopting Spark. You do not need Spark to use dataframes. And distributed computing is terrible for machine learning. Maybe you've worked at a job or two where nobody can comprehend not using distributed computing, as you describe, but it's nonsense to claim that "all organizations" work that way.
      - aub3bhat 9y ago
        No, "all organizations" are not adopting Spark. All organizations which already have a Hadoop cluster. You do not need Spark to use dataframes. Never claimed this. To clarify Spark allows you to directly port Pandas code while leveraging existing Hadoop cluster infrastructure. And distributed computing is terrible for machine learning. Distributed computing (Both traditional hadoop/spark and latest TF/PyTorch with parameter server) are essential for scaling ML beyond a certain point. Maybe you've worked at a job or two where nobody can comprehend not using distributed computing, as you describe, but it's nonsense to claim that "all organizations" work that way. If you have experience routinely training models on Terabytes of data intended for production deployment. I am happy to hear. There is a vast difference between training a model on your machine for research and building a reliable ML system that scales across large datasets and teams while taking infrastructure costs into account.
- jhayward 9y ago
  I will note that his local computation could just as easily have been done on a single, fully managed/secured "cloud" node with much the same performance. I think it misses the point to say that not downloading the data is where the cost lies.
  - aub3bhat 9y ago
    I agree but sadly we don't have any plug-in solutions that operate in similar manner. How do you allocate capacity? What if your node has hard upper limit? How do you ensure security e.g. secomp, jails etc. The moment data crosses the DB boundry you need significant investment in ensuring compliance. The reason Spark ecosystem has been so popular is because it enables these types of computations without breaking the model.
    - Terretta 9y ago
      This line of discussion worries me, in case others think your security assertions are true and could be lead astray. Today’s OSS big data query execution environments are BnL neither secure nor compliant. This is not a knock. They were simply not designed for those constraints. They were built for academic or single tenant use cases without separation of duties / control. This also applies to commercial products such as Splunk. There is tremendous investment going on the last couple years in raising the security and compliance bar. We’ve worked on this with the usual suspects. But barring a handful of proprietary stacks (the big three CSPs, and a couple enterprise on prem bare metal distros) getting close, we are not there yet. Trying to land with your security or risk teams or regulators that the CSP pulled it off will likely take you longer than provisioning a compliant laptop you’d keep locked up with two keys. Unless you have several tens of man years invested in in-depth security wrapping these environments, or can choose a big three CSP w/o answering to anyone, today I’d still recommend the trusted laptop build approach for truly sensitive algorithms and computations.
      - aub3bhat 9y ago
        today I’d still recommend the trusted laptop build approach for truly sensitive algorithms and computations. You are utterly wrong. Algorithms and computations (especially ML kind) are never sensitive, its the data which is always sensitive. And that ALREADY exists on the cluster. If an Organization already has a Hadoop cluster containing data. You are suggesting that somehow having it downloaded to a secured laptop is better? Than say Spark running on top of the cluster? I think you are deeply mistaken. The cluster instances are already protected (if not you have a bigger problems). Also while an organization might not have Hadoop, they surely have an RDBMS, in which case the algorithms are even more useful.
- geofft 9y ago
  That's absolutely true, but I think the author's argument is likely to be that the popularity of Spark and Hadoop in the first place is misguided, and in a world where people were paying attention to performance instead of novelty, we'd have something like Spark and something like Hadoop that was optimized for a single computer, not for scaling for the fun of scaling.
  - aub3bhat 9y ago
    scaling for the fun of scaling. I think your arguments are misguided, for every compute bound task where Hadoop/Spark undeperform, 1000 other ETL type tasks where hadoop is indispensable. As a result any organization running a large hadoop cluster will already have underused compute capacity for free and a maintainance staff which already taking care of the cluster. Thus from an organizations perspective the time difference is not material especially for batch jobs, this is the reason why Presto and Spark have been so successful. They enabled underutilized hadoop clusters to be used for ML and data science while delivering reasonable performance for Zero cost.
    - Terretta 9y ago
      > ”... Staff already taking care of the cluster.” “The” cluster, singular ... This is the catch. If you need security and compliance, you can’t today derive the benefits of re-use by other teams. Given today’s distros, and assuming your threat model needs to account for insider threat, you need a different cluster for each data ownership grouping and data sensitivity level. For four teams with three levels of data, you’d need twelve completely independent clusters. Unless, as noted above, you’ve done a ton of in-house multi-tenancy work to provide full stack security and compliance assurances and audits.
      - aub3bhat 9y ago
        However you may split clusters by role/access. You will have I/O bound ETL/Interactive tasks as primary use case. Thus the investment in security will be constant whether you reuse the same infrastructure for compute/memory bound ML tasks or not.
- mcguire 9y ago
  "A factor of 10x isn't a horrible tax to pay for re-using re-usable infrastructure, like your database. A factor of 60x-130x is more debatable, but I think the CIDR paper is totally worth talking about. I'm a bit disappointed that there aren't reasonable baselines, but not as much as that the SIGMOD paper doesn't have any baselines at all." His "smarter algorithms" aren't.
- lambda 9y ago
  > However having something available as part of DB or Hadoop ecosystem enables reproducible execution while taking care of compliance (e.g. privacy), security, access control etc. You are conflating a couple of things here. Of course for real analyses, you want to run them on standardized environments with appropriate access controls. But nothing says that "standardized environment" has to use a distributed database that, from all accounts, seems to harm more than help. You could just have a RDBMS or even filesystem, which is appropriately backed up or replicated, on a single beefy VM or physical machine. > The real opportunity lies in ability to add arbitrary compute/memory capacity to DB query execution flows. Only if the tools used to provide that ability actually allow you to take advantage of it. If scaling up to 160 nodes doesn't allow you to be any faster than 1 single laptop, you probably could have spent a lot less money on just adding more RAM to a single beefy server than a 160 node cluster.
dswalter 9y ago
For anyone who is just passing by, the article attempts to verify some claims from recent VLDB and SIGMOD papers that claim decent* performance on graph algorithms in RDBMs. It's worth reading not only for the technical merits of the discussion, but also because the style of the article is ...feisty. *for a given definition of decent.
- aaronblohowiak 9y ago
  money quote: >None of them present any evidence that they are any better than a single-threaded implementation on any problem at any scale. So, if nothing else the published papers are not yet right. But I suspect they are mostly wrong.
- mcguire 9y ago
  There is one fundamental step you should always take when reading systems papers with benchmarks: Always look for what the authors are trying to hide. It's (a) always there, and (b) always hilarious. From operating systems comparisons that need "we didn't actually slow down the processor physically" comparisons to look good to these "we look great, compared to that guy over there licking the floor."
fredliu 9y ago
Not sure if it's related, and AWS Athena is not a Database in a strict sense (or in any sense at all since it's not a transactional DB), but it seems to offer both scale and lower cost if it fits your requirements (read only, DW like traffic)? Anybody has experience using it in prod as a DW replacement?
- dswalter 9y ago
  I would argue this question is not really related, but we use PrestoDB (the technology AWS Athena is built using), and it is particularly great for when data becomes too large to fit in a "regular" database. PrestoDB and Apache Impala are the two major open-source surprisingly-fast-distributed-SQL-query-planning-engines. If you pair either of them with a columnar data storage format like Parquet or ORC, you can make analytical workloads work on inconveniently-sized data. AWS Athena abstracts the cluster management for you and runs PrestoDB on spare EC2 instances under the hood.
  - fredliu 9y ago
    As mentioned in the other comment, I was actually only thinking about use cases like Athenta + csv_on_s3. and that's the really "low cost" approach I was referring to for analytical data.
- electrum 9y ago
  The term database or RDBMS here seems to be mean runs SQL or processes [something close to] relational algebra. Based on that definition, Athena is definitely a database. Athena is built on Presto, which has full support for transactions. Each connector provides varying support (isolation level, multi-statement writes, etc.) for transactions, limited by the underlying data source.
  - fredliu 9y ago
    I do know Athena is built on top of Presto and it can pair with lots of different data sources. But somehow when talking about Athena I always end up thinking about using big CSVs on S3 as data source, so what I really meant was Athena+S3 I guess. Edit: regarding transaction though, I wonder if Athena could even be used as a "meta sharding" layer, when all of the underlying data sources support transaction, but data is too large to fit in a single instance. The advantage would be to not implement sharding logic in code. Not sure about the performance though. Just a thought. Edit2: not sure if I read it correctly, but looks like right now Athena does not support transactions yet (http://docs.aws.amazon.com/athena/latest/ug/creating-tables.html http://docs.aws.amazon.com/athena/latest/ug/creating-tables....) although using Presto there could be a future that they do support.
yazr 9y ago
Is this guy for real? He is limited by not having an "expensive macbook with 4 real cores" ?? He gave up on a test because it "paged out the 15GB of data". Of course at these scales a cluster is x5 slower. Can someone please explain - am i missing something here ?
- tlb 9y ago
  It'd obviously be a nice property of systems if they were efficient at any scale. But currently we have two kinds of systems, designed for small and large scale.
  - philipkglass 9y ago
    If a system is designed for large scale problems, it seems like it really should be validated against problems that are impractical to run on a single commodity server. The problem with the results the blog author pokes fun at isn't (necessarily) that they trade high overhead for high scalability. The problem is more that these papers are mostly dealing with such small problems that simple one-laptop processing offers competitive or even superior performance. There's no useful evidence provided either way when a proposed "big data" approach is tested on inappropriately small inputs.
    - shub 9y ago
      Good luck finding anything public about graph processing on a dataset too large to fit on a single machine. I can launch an AWS instance with 128 cores and 4 TB RAM--how many triples is too many for that monster? Tens of billions? Hundreds of billions?
      - fnord123 9y ago
        The common crawl has 3.28 billion pages. That should result in plenty of edges for 2017. (Shouldn't it?)
        shub 9y ago
mjb 9y ago
That was a good read, but I think the author's conclusion is a little bit too punchy (although it did make it a fun read). Figuring out how to scale-out non-trivial problems is difficult. Many of these scale-out solutions are going to be worse than local processing, and many won't add meaningful value at any scale. However, there's real value in approaching difficult problems, and there's real value in "the fastest way to do X in a cluster" even if it's not "the fastest way to do X". That's because scale-out is a real requirement to do a lot of interesting things, and because if we can find better ways to scale out we can start to compete with single-machine solutions (and "custom cluster" and "supercomputer" solutions) then we have made real progress towards something that's faster at all scales. It's a valuable research goal. Now, it would be great if the database community (and the systems research community at large) was more explicit about that. They should set the standard that authors need to clearly differentiate between "improved on result X under constraints Y" and "moved the state-of-the-art forward under all constraints". It's also good to read systems research skeptically, and think about whether you really need to pay the cost of scale-out. For companies who's output isn't primarily "research", it makes a lot of sense to constantly look for "the cheapest way to do X", not "the cheapest way to do X in a cluster".
- lambda 9y ago
  Note that the author himself is working on a framework for distributed differential dataflow computations (dataflow computations in which it is very cheap to make small updates to the inputs and see the differences in the outputs), and he has plans to apply the same metrics (comparing to a single-threaded implementation) to his own work once he gets a little further: https://github.com/frankmcsherry/blog/blob/master/posts/2017-03-28.md https://github.com/frankmcsherry/blog/blob/master/posts/2017... I think that answering the "fastest way to do X in a cluster" question is only particularly interesting if X is something that you need to do in a cluster. But clearly they are testing X that is capable of being done on a mid-range laptop. There's a real question of whether the X that they are testing will actually scale to larger datasets than the laptop (or a decently beefy single node) could handle; it's entirely possible that the cluster solution will blow up at that point. And of course, if the single-threaded implementation is faster, then you could presumably just do "X in a cluster" by doing "X on a single node" and using the rest of the nodes to mine your CPU bound cryptocurrency of choice.
jandrewrogers 9y ago
Tangentially, graph platforms have long had the issue that performance claims are astonishingly unreliable and come with so many caveats and qualifications (documented or not) that the claims don't generalize usefully in practice. One of the reasons I got out of the graph database business was the pervasive lack of rigor and repeatability when it comes to measuring scalability and performance across platforms; there is a cost asymmetry to properly addressing the volume of questionable performance measurements made in the graph database world. In fairness, graph workload performance, compared to other types of workloads, is unusually sensitive to a diverse set of details that are rarely captured adequately by researchers. As often as not, you are actually measuring things unrelated to the graph algorithm itself. Measuring graph databases properly would require a level of specification for how to measure such things that hasn't happened yet. And many people like it that way, it makes it relatively easy to contrive details that make your particular algorithm or implementation look good despite the reality that serious weaknesses are pervasive across graph database platforms. A generally, objectively strong graph database platform doesn't exist AFAIK -- they all kind of suck in unique ways.
- eternalban 9y ago
  A graph is a mathematical object. It has well defined properties, such as 'degree', 'radius', etc. Just as in e.g. relational domain you can propose well defined operations (such as join two relations based on a common foreign key) and devise benchmarks for these operations, it should be possible to do the same precise thing for graph databases. For a graph database, you would want (a) ingress, (b) comps, and (c) recovery benchmarks. The (b) comps per above can be based on a set of common patterns of queries applied to a parametric graph spec. > In fairness, graph workload performance, compared to other types of workloads, is unusually sensitive to a diverse set of details that are rarely captured adequately by researchers. That reads: we don't understand graphs.
mcguire 9y ago
Haven't read the article yet, but these slides (from HOTOS 2015) are beautiful: [PDF] https://www.usenix.org/sites/default/files/conference/protected-files/hotos15_slides_mcsherry.pdf https://www.usenix.org/sites/default/files/conference/protec... That article is https://www.usenix.org/conference/hotos15/workshop-program/presentation/mcsherry https://www.usenix.org/conference/hotos15/workshop-program/p...
harveywi 9y ago
I love this quote by Paul Barham: "You can have a second computer once you've shown you know how to use the first one."
- elvinyung 9y ago
  Is there actually a source for this? The COST paper uses it, but I still haven't found what it's quoting.
microcolonel 9y ago
I am overloaded with acronyms. For crying out loud can we please just say what we mean instead of sending everyone down an acronym wormhole three whitepapers deep?
- mcguire 9y ago
  From this post: "Our first paper is Scalable Distributed Subgraph Enumeration or "SEED". We will have a future post about rules for picking acronyms." I suspect the author would agree.
incan1275 9y ago
He has some great points about SIGMOD/VLDB reviewing - they emphasize far too much on "algorithmic novelty", instead of impactful simple systems. However, in terms of graph processing, I too think in many instances, single machine processing is too limiting. In general, I think people would be more convinced if there were real cases in industry where engineers switched from distributing graph algos to single-machine algos because they found performance benefits were better. In all my years, I haven't heard of a case like this (it's always been the other way around).

203 points by jchanimal 9y ago

The author should spend some time delivering solutions to "real problems". It's easy to be snarky while writing a blogpost with toy datasets but it's difficult to actually deliver production solution in an organization. The real COST is often not the time consumed but costs that arise from maintaining heterogeneous architecture and remaining legally compliant. A database or hadoop cluster is not just an execution engine, but rather a managable, reproducible execution engine. Sure you can code up a smarter algorithm on your personal i9 workstation. However having something available as part of DB or Hadoop ecosystem enables reproducible execution while taking care of compliance (e.g. privacy), security, access control etc. Eventually we will have a containerized extensions to DB query execution flow that will allow us to have best of both worlds but for 99% practicing data scientists downloading a Gigabyte sized subset to their own workstation is not a viable option. The real opportunity lies in ability to add arbitrary compute/memory capacity to DB query execution flows.

zie 9y ago
I agree, but at what cost? 10% performance hit, sure, no problem, VM's take about that much from running on real hardware. but a 100% performance hit or worse... you must be crazy. Also, downloading 1GB is no big deal, downloading 1TB is a big deal, so I think your example has more merit in the > 1TB range. i.e. things that don't fit on a laptop easily.
- aub3bhat 9y ago
  1Mb vs 1Gb vs 1Tb distinction is a function of information contained in data. If its PII even 1Mb is an orginizational headache from security/risk-management perspective. Also most commodity compute nodes have allocation around 16 Gb. The big difference comes in 16~256 Gb range where having a single powerful node can make a huge difference.
  - zie 9y ago
    Sure, but 'laptop' can be changed very easily into any container or VM box somewhere that is tied to security/etc but still on a single box. Again sizing here is sort of beside the point, the point is, if its 500% worse to use than just doing it on a single instance/compute node, you better be offering A LOT of magic awesomeness that makes the performance drain acceptable. 10% cost for the benefits that this stuff can give you, that is pretty easy to swallow. 100%-500% perf. cost is just a very, very, very hard to swallow pill.
JoachimSchipper 9y ago
Auditing is a real argument; but is decently-written specialized code really that much less reproducible than Hadoop/SQL? (I'll accept "yes", but note that that also rules out e.g. the fairly common practice of using R or pandas for some ad-hoc processing.)
- aub3bhat 9y ago
  also rules out e.g. the fairly common practice of using R or pandas for some ad-hoc processing. This is essentially the reason why all organizations are adopting Spark, since it allows you to write imperative code on dataframes and build ML models.
  - rspeer 9y ago
    No, "all organizations" are not adopting Spark. You do not need Spark to use dataframes. And distributed computing is terrible for machine learning. Maybe you've worked at a job or two where nobody can comprehend not using distributed computing, as you describe, but it's nonsense to claim that "all organizations" work that way.
    - aub3bhat 9y ago
      No, "all organizations" are not adopting Spark. All organizations which already have a Hadoop cluster. You do not need Spark to use dataframes. Never claimed this. To clarify Spark allows you to directly port Pandas code while leveraging existing Hadoop cluster infrastructure. And distributed computing is terrible for machine learning. Distributed computing (Both traditional hadoop/spark and latest TF/PyTorch with parameter server) are essential for scaling ML beyond a certain point. Maybe you've worked at a job or two where nobody can comprehend not using distributed computing, as you describe, but it's nonsense to claim that "all organizations" work that way. If you have experience routinely training models on Terabytes of data intended for production deployment. I am happy to hear. There is a vast difference between training a model on your machine for research and building a reliable ML system that scales across large datasets and teams while taking infrastructure costs into account.
jhayward 9y ago
I will note that his local computation could just as easily have been done on a single, fully managed/secured "cloud" node with much the same performance. I think it misses the point to say that not downloading the data is where the cost lies.
- aub3bhat 9y ago
  I agree but sadly we don't have any plug-in solutions that operate in similar manner. How do you allocate capacity? What if your node has hard upper limit? How do you ensure security e.g. secomp, jails etc. The moment data crosses the DB boundry you need significant investment in ensuring compliance. The reason Spark ecosystem has been so popular is because it enables these types of computations without breaking the model.
  - Terretta 9y ago
    This line of discussion worries me, in case others think your security assertions are true and could be lead astray. Today’s OSS big data query execution environments are BnL neither secure nor compliant. This is not a knock. They were simply not designed for those constraints. They were built for academic or single tenant use cases without separation of duties / control. This also applies to commercial products such as Splunk. There is tremendous investment going on the last couple years in raising the security and compliance bar. We’ve worked on this with the usual suspects. But barring a handful of proprietary stacks (the big three CSPs, and a couple enterprise on prem bare metal distros) getting close, we are not there yet. Trying to land with your security or risk teams or regulators that the CSP pulled it off will likely take you longer than provisioning a compliant laptop you’d keep locked up with two keys. Unless you have several tens of man years invested in in-depth security wrapping these environments, or can choose a big three CSP w/o answering to anyone, today I’d still recommend the trusted laptop build approach for truly sensitive algorithms and computations.
    - aub3bhat 9y ago
      today I’d still recommend the trusted laptop build approach for truly sensitive algorithms and computations. You are utterly wrong. Algorithms and computations (especially ML kind) are never sensitive, its the data which is always sensitive. And that ALREADY exists on the cluster. If an Organization already has a Hadoop cluster containing data. You are suggesting that somehow having it downloaded to a secured laptop is better? Than say Spark running on top of the cluster? I think you are deeply mistaken. The cluster instances are already protected (if not you have a bigger problems). Also while an organization might not have Hadoop, they surely have an RDBMS, in which case the algorithms are even more useful.
geofft 9y ago
That's absolutely true, but I think the author's argument is likely to be that the popularity of Spark and Hadoop in the first place is misguided, and in a world where people were paying attention to performance instead of novelty, we'd have something like Spark and something like Hadoop that was optimized for a single computer, not for scaling for the fun of scaling.
- aub3bhat 9y ago
  scaling for the fun of scaling. I think your arguments are misguided, for every compute bound task where Hadoop/Spark undeperform, 1000 other ETL type tasks where hadoop is indispensable. As a result any organization running a large hadoop cluster will already have underused compute capacity for free and a maintainance staff which already taking care of the cluster. Thus from an organizations perspective the time difference is not material especially for batch jobs, this is the reason why Presto and Spark have been so successful. They enabled underutilized hadoop clusters to be used for ML and data science while delivering reasonable performance for Zero cost.
  - Terretta 9y ago
    > ”... Staff already taking care of the cluster.” “The” cluster, singular ... This is the catch. If you need security and compliance, you can’t today derive the benefits of re-use by other teams. Given today’s distros, and assuming your threat model needs to account for insider threat, you need a different cluster for each data ownership grouping and data sensitivity level. For four teams with three levels of data, you’d need twelve completely independent clusters. Unless, as noted above, you’ve done a ton of in-house multi-tenancy work to provide full stack security and compliance assurances and audits.
    - aub3bhat 9y ago
      However you may split clusters by role/access. You will have I/O bound ETL/Interactive tasks as primary use case. Thus the investment in security will be constant whether you reuse the same infrastructure for compute/memory bound ML tasks or not.
mcguire 9y ago
"A factor of 10x isn't a horrible tax to pay for re-using re-usable infrastructure, like your database. A factor of 60x-130x is more debatable, but I think the CIDR paper is totally worth talking about. I'm a bit disappointed that there aren't reasonable baselines, but not as much as that the SIGMOD paper doesn't have any baselines at all." His "smarter algorithms" aren't.
lambda 9y ago
> However having something available as part of DB or Hadoop ecosystem enables reproducible execution while taking care of compliance (e.g. privacy), security, access control etc. You are conflating a couple of things here. Of course for real analyses, you want to run them on standardized environments with appropriate access controls. But nothing says that "standardized environment" has to use a distributed database that, from all accounts, seems to harm more than help. You could just have a RDBMS or even filesystem, which is appropriately backed up or replicated, on a single beefy VM or physical machine. > The real opportunity lies in ability to add arbitrary compute/memory capacity to DB query execution flows. Only if the tools used to provide that ability actually allow you to take advantage of it. If scaling up to 160 nodes doesn't allow you to be any faster than 1 single laptop, you probably could have spent a lot less money on just adding more RAM to a single beefy server than a 160 node cluster.

dswalter 9y ago

For anyone who is just passing by, the article attempts to verify some claims from recent VLDB and SIGMOD papers that claim decent* performance on graph algorithms in RDBMs. It's worth reading not only for the technical merits of the discussion, but also because the style of the article is ...feisty. *for a given definition of decent.

aaronblohowiak 9y ago
money quote: >None of them present any evidence that they are any better than a single-threaded implementation on any problem at any scale. So, if nothing else the published papers are not yet right. But I suspect they are mostly wrong.
mcguire 9y ago
There is one fundamental step you should always take when reading systems papers with benchmarks: Always look for what the authors are trying to hide. It's (a) always there, and (b) always hilarious. From operating systems comparisons that need "we didn't actually slow down the processor physically" comparisons to look good to these "we look great, compared to that guy over there licking the floor."

fredliu 9y ago

Not sure if it's related, and AWS Athena is not a Database in a strict sense (or in any sense at all since it's not a transactional DB), but it seems to offer both scale and lower cost if it fits your requirements (read only, DW like traffic)? Anybody has experience using it in prod as a DW replacement?

dswalter 9y ago
I would argue this question is not really related, but we use PrestoDB (the technology AWS Athena is built using), and it is particularly great for when data becomes too large to fit in a "regular" database. PrestoDB and Apache Impala are the two major open-source surprisingly-fast-distributed-SQL-query-planning-engines. If you pair either of them with a columnar data storage format like Parquet or ORC, you can make analytical workloads work on inconveniently-sized data. AWS Athena abstracts the cluster management for you and runs PrestoDB on spare EC2 instances under the hood.
- fredliu 9y ago
  As mentioned in the other comment, I was actually only thinking about use cases like Athenta + csv_on_s3. and that's the really "low cost" approach I was referring to for analytical data.
electrum 9y ago
The term database or RDBMS here seems to be mean runs SQL or processes [something close to] relational algebra. Based on that definition, Athena is definitely a database. Athena is built on Presto, which has full support for transactions. Each connector provides varying support (isolation level, multi-statement writes, etc.) for transactions, limited by the underlying data source.
- fredliu 9y ago
  I do know Athena is built on top of Presto and it can pair with lots of different data sources. But somehow when talking about Athena I always end up thinking about using big CSVs on S3 as data source, so what I really meant was Athena+S3 I guess. Edit: regarding transaction though, I wonder if Athena could even be used as a "meta sharding" layer, when all of the underlying data sources support transaction, but data is too large to fit in a single instance. The advantage would be to not implement sharding logic in code. Not sure about the performance though. Just a thought. Edit2: not sure if I read it correctly, but looks like right now Athena does not support transactions yet (http://docs.aws.amazon.com/athena/latest/ug/creating-tables.html http://docs.aws.amazon.com/athena/latest/ug/creating-tables....) although using Presto there could be a future that they do support.

yazr 9y ago

Is this guy for real? He is limited by not having an "expensive macbook with 4 real cores" ?? He gave up on a test because it "paged out the 15GB of data". Of course at these scales a cluster is x5 slower. Can someone please explain - am i missing something here ?

tlb 9y ago
It'd obviously be a nice property of systems if they were efficient at any scale. But currently we have two kinds of systems, designed for small and large scale.
- philipkglass 9y ago
  If a system is designed for large scale problems, it seems like it really should be validated against problems that are impractical to run on a single commodity server. The problem with the results the blog author pokes fun at isn't (necessarily) that they trade high overhead for high scalability. The problem is more that these papers are mostly dealing with such small problems that simple one-laptop processing offers competitive or even superior performance. There's no useful evidence provided either way when a proposed "big data" approach is tested on inappropriately small inputs.
  - shub 9y ago
    Good luck finding anything public about graph processing on a dataset too large to fit on a single machine. I can launch an AWS instance with 128 cores and 4 TB RAM--how many triples is too many for that monster? Tens of billions? Hundreds of billions?
    - fnord123 9y ago
      The common crawl has 3.28 billion pages. That should result in plenty of edges for 2017. (Shouldn't it?)
      - shub 9y ago

mjb 9y ago

That was a good read, but I think the author's conclusion is a little bit too punchy (although it did make it a fun read). Figuring out how to scale-out non-trivial problems is difficult. Many of these scale-out solutions are going to be worse than local processing, and many won't add meaningful value at any scale. However, there's real value in approaching difficult problems, and there's real value in "the fastest way to do X in a cluster" even if it's not "the fastest way to do X". That's because scale-out is a real requirement to do a lot of interesting things, and because if we can find better ways to scale out we can start to compete with single-machine solutions (and "custom cluster" and "supercomputer" solutions) then we have made real progress towards something that's faster at all scales. It's a valuable research goal. Now, it would be great if the database community (and the systems research community at large) was more explicit about that. They should set the standard that authors need to clearly differentiate between "improved on result X under constraints Y" and "moved the state-of-the-art forward under all constraints". It's also good to read systems research skeptically, and think about whether you really need to pay the cost of scale-out. For companies who's output isn't primarily "research", it makes a lot of sense to constantly look for "the cheapest way to do X", not "the cheapest way to do X in a cluster".

lambda 9y ago
Note that the author himself is working on a framework for distributed differential dataflow computations (dataflow computations in which it is very cheap to make small updates to the inputs and see the differences in the outputs), and he has plans to apply the same metrics (comparing to a single-threaded implementation) to his own work once he gets a little further: https://github.com/frankmcsherry/blog/blob/master/posts/2017-03-28.md https://github.com/frankmcsherry/blog/blob/master/posts/2017... I think that answering the "fastest way to do X in a cluster" question is only particularly interesting if X is something that you need to do in a cluster. But clearly they are testing X that is capable of being done on a mid-range laptop. There's a real question of whether the X that they are testing will actually scale to larger datasets than the laptop (or a decently beefy single node) could handle; it's entirely possible that the cluster solution will blow up at that point. And of course, if the single-threaded implementation is faster, then you could presumably just do "X in a cluster" by doing "X on a single node" and using the rest of the nodes to mine your CPU bound cryptocurrency of choice.

jandrewrogers 9y ago

Tangentially, graph platforms have long had the issue that performance claims are astonishingly unreliable and come with so many caveats and qualifications (documented or not) that the claims don't generalize usefully in practice. One of the reasons I got out of the graph database business was the pervasive lack of rigor and repeatability when it comes to measuring scalability and performance across platforms; there is a cost asymmetry to properly addressing the volume of questionable performance measurements made in the graph database world. In fairness, graph workload performance, compared to other types of workloads, is unusually sensitive to a diverse set of details that are rarely captured adequately by researchers. As often as not, you are actually measuring things unrelated to the graph algorithm itself. Measuring graph databases properly would require a level of specification for how to measure such things that hasn't happened yet. And many people like it that way, it makes it relatively easy to contrive details that make your particular algorithm or implementation look good despite the reality that serious weaknesses are pervasive across graph database platforms. A generally, objectively strong graph database platform doesn't exist AFAIK -- they all kind of suck in unique ways.

eternalban 9y ago
A graph is a mathematical object. It has well defined properties, such as 'degree', 'radius', etc. Just as in e.g. relational domain you can propose well defined operations (such as join two relations based on a common foreign key) and devise benchmarks for these operations, it should be possible to do the same precise thing for graph databases. For a graph database, you would want (a) ingress, (b) comps, and (c) recovery benchmarks. The (b) comps per above can be based on a set of common patterns of queries applied to a parametric graph spec. > In fairness, graph workload performance, compared to other types of workloads, is unusually sensitive to a diverse set of details that are rarely captured adequately by researchers. That reads: we don't understand graphs.

mcguire 9y ago

Haven't read the article yet, but these slides (from HOTOS 2015) are beautiful: [PDF] https://www.usenix.org/sites/default/files/conference/protected-files/hotos15_slides_mcsherry.pdf https://www.usenix.org/sites/default/files/conference/protec... That article is https://www.usenix.org/conference/hotos15/workshop-program/presentation/mcsherry https://www.usenix.org/conference/hotos15/workshop-program/p...

harveywi 9y ago

I love this quote by Paul Barham: "You can have a second computer once you've shown you know how to use the first one."

elvinyung 9y ago
Is there actually a source for this? The COST paper uses it, but I still haven't found what it's quoting.

microcolonel 9y ago

I am overloaded with acronyms. For crying out loud can we please just say what we mean instead of sending everyone down an acronym wormhole three whitepapers deep?

mcguire 9y ago
From this post: "Our first paper is Scalable Distributed Subgraph Enumeration or "SEED". We will have a future post about rules for picking acronyms." I suspect the author would agree.

incan1275 9y ago

He has some great points about SIGMOD/VLDB reviewing - they emphasize far too much on "algorithmic novelty", instead of impactful simple systems. However, in terms of graph processing, I too think in many instances, single machine processing is too limiting. In general, I think people would be more convinced if there were real cases in industry where engineers switched from distributing graph algos to single-machine algos because they found performance benefits were better. In all my years, I haven't heard of a case like this (it's always been the other way around).