11 ms·
Database Performance at Scale – A free book
- gajus 3y agoI appreciate that "Just the PDF" actually gives me the PDF and not 10 more popups to subscribe to someones newsletter.
- deleted 3y ago[deleted]
- Nezteb 3y agoDirect URL: https://link.springer.com/content/pdf/10.1007/978-1-4842-9711-7.pdf https://link.springer.com/content/pdf/10.1007/978-1-4842-971...
- mindondrugs 3y agoIm not sure how this qualifies as open source when the repo for the book[0] is essentially empty? [0] https://github.com/Apress/db-performance-at-scale https://github.com/Apress/db-performance-at-scale
- deleted 3y ago[deleted]
- pipo234 3y agoYes, seems like the GitHhub repo was a bit of an after thought: Top right: > About > SOURCE CODE for "Database Performance at Scale: A Practical Guide (Apress, 2023)," by Felipe Cardeneti Mendes, Piotr Sarna, Pavel Emelyanov & Cynthia Dunlop But readme.md: > This repository ACCOMPANIES Database Performance at Scale:A Practical Guide by Felipe Cardeneti Mendes, Piotr Sarna, Pavel Emelyanov & Cynthia Dunlop (Apress, 2023). [ my emphasis ]
- iudqnolq 3y agoIn the context of technical books "source code for" often means "source code for [readers of the book to reference while reading]" rather than "source code for [building the book]"
- maximinus_thrax 3y agoI disagree. Words have meaning. 'Open source' means 'open source' in all contexts. For comparison, https://doc.rust-lang.org/book/ https://doc.rust-lang.org/book/ is an open source book. A PDF with a CC license without a repo of the publishing artifacts is not an open source book. It's just a free book.
- iudqnolq 3y agoThe question is not whether it's open source, the question is what it is.
- mdaniel 3y agoI wondered about that, too, but in the body of the article they claim the license is a Creative Commons flavor so I interpreted it as a marketingdroid faux pas that the source is not available but that the content is open I guess the distinction would be if someone wanted to upstream a change, not fork it, through what mechanism would that take place, but I'm going with "highly unlikely" on that one
- colesantiago 3y agoWhy don’t more companies / startups choose ScyllaDB rather than Postgres or MySQL? ScyllaDB is a C++ version of Cassandra so it looks like speed and scalability is a complete advantage over the Java based Cassandra and Discord is using ScyllaDB at scale too.
- olivermuty 3y agoBecause I (we) want rigid schemas with transaction wrapped DDL migrations more than I want blazing speed at Discords scale.
- michaelmior 3y agoYou can have a rigid schema with Cassandra/ScyllaDB. Transactions are a whole other thing though for sure.
- riku_iki 3y agoPlus complex queries with rich capabilities
- joeatwork 3y agoThere are a bunch of common workloads (including i’m guessing HN itself) where the trade offs for distributed databases make them a lot harder to use well, or where distributed databases don’t make sense.
- whynotmaybe 3y agoGovernment agencies I've worked usually build an app and put it in production without very few evolutions for 15+ years. Data is the most important stuff for them and being able to read it is therefore very important. A nosql DB structure can usually only be read by the code that goes with it and if nobody understands the code you're doomed. In 2017, I had to extract data from an app from 1994. The app was built with a language that wasn't widely used at the time and had disappeared today. It disappeared before all the software companies put their doc on the internet. You can't run it on anything older than windows2k. There was only one guy in Montréal that knew how to code and run that thing. As the DB engine was relational with a tool like sqlplus, I was able to extract all the data and we rebuilt the app based on the structure we found in the database. If you want to do that with Nosql, you must maintain a separate documentation that explains your structure... And we all now how dev like documentation.
- nilslindemann 3y agoThe first code example is on page 89.
- cogman10 3y ago> Noticing that a certain NoSQL database was recently trending on the front page of Hacker News, Patrick picked it for his backend stack I feel attacked! :D
- maximinus_thrax 3y agoHaven't have time to look over it. For people who did, is this a generic 'Database performance' book or a longform pamphlet for ScyllaDB ?
- biohazard__ 3y agoThis is already addressed: "It’s not “about” ScyllaDB – but it’s highly relevant for anyone using or considering it" It does contain a plethora of references to other databases, and it does exemplify concepts using ScyllaDB. IIRC they also made it clear in the front matter.
- maximinus_thrax 3y agoI'm referring to the content itself, not what is advertised about it. The article also states that the book is open source, when it's actually not, so I guess I have trust issues. But I'll take the fact that you created an account just to write this comment as a positive signal that the book is generic and not only about ScyllaDB.
- pc86 3y agoOther than making the PDF freely available what in your mind would need to happen to make this book open source? I'm not even sure what an "open source" book would be in the FOSS meaning of the phrase. From the edition notice page of the PDF: > Open Access This book is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/ http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. And the comment you're replying to says "It does contain a plethora of references to other databases" so I'm not sure the snarky "I'm referring to the content itself, not what is advertised about it." is necessary or particularly helpful.
- 3y ago
- renegade-otter 3y agoIs it just me, or is the basic knowledge of database normalization and indexing kind of a lost art these days? How many teams out there just "add cache and more hardware" when no one ran an EXPLAIN on that one core query in a giant table? Perhaps we should do that before talking about anything "at scale".
- sb8244 3y agoI wrote a system that used the database as essentially a rules engine for "what work to execute next?" There were constraints about max throughput per key, waiting periods, etc. That system broke me down and built me back up over time. Really deep explaining, understanding the internals of how the data is actually being processed, then finding ways to improve it. I had 1-2 other people who were really interested in postgres that would help work on debugging. We would continuously get 100x query improvements every time we dove in. Including after we thought we maxed it out. I hope everyone gets the chance to work on something like that. It was really fun.
- PTOB 3y agoI would slaver at mouth over that opportunity.
- renegade-otter 3y agoIf anyone starts a new side project, do yourself a favor - write your data access layer in straight SQL. With minimal wiring and safe variable substitution that most libraries provide, you will learn a lot.
- derefr 3y ago> a rules engine for "what work to execute next?" I'm afraid to inform you, that you have built a cluster workload manager. (Where a "cluster workload manager" is a type of job scheduler designed for — usually HPC — clusters, that prioritizes among ongoing abstract batch "jobs" that can each broken down into more granular atomic scheduleable work-units; and which accounts the often-unpredictable resource-consumption outcomes of those work-units, to the account of the job and its owning user+group+org, to then weight further executions of that job or other jobs owned by that user+group+org, so that ultimately, jobs can proceed at a QoS/rate that's 1. "fair", and 2. in proportion to the amount that a given customer is willing to pay for prioritization. "Business rules" in such a system, are expressed in terms of conditional coefficients or formulae to apply to the "costs" used to re-weight jobs.) Examples of cluster workload management software: • Slurm (https://en.wikipedia.org/wiki/Slurm_Workload_Manager https://en.wikipedia.org/wiki/Slurm_Workload_Manager) • IBM Spectrum LSF (https://www.ibm.com/products/hpc-workload-management https://www.ibm.com/products/hpc-workload-management) • technically, Kubernetes' kube-scheduler — though it's the most trivial possible implementation of one, without any of the ongoing-fairness-of-QoS logic. The key thing to know about these, is that it's nearly impossible to build one that lives "quiesced" inside a database (as you've probably found.) Imagine taking an OS process scheduler and trying to keep the state for that canonically inside a database. Then imagine multiplying the write load by 1000x, while keeping the realtime read requirements the same. It's just a bad idea. You can certainly persist the state of these systems into a DB — but what you really want is one or more active scheduler nodes, that keep a big in-memory ledger matrix, that can be mutated moment-to-moment, without needing to persist every single change to disk in some WAL/journal/etc. Like a regular OS scheduler. Or like, say, a game physics engine — very similar in how it's all about tiny little "integration" steps, many times per second.
- Kudotap 3y agoBeautiful. Been aiming to learn how to scale MySQL databases so I can run my apps on VMs without having to used managed dbs like Aurora or Azure Managed Database
- huijzer 3y agoHow big are your apps? Wouldn’t the most simple and reliable way be to get one beefy MySQL server and let all the app instances connect to that?
- zeroCalories 3y agoOne large VM won't give you replication/sharding, failover, or backups.
- Aeolun 3y agoWith one large VM, for most use cases, you do not need anything but the backups.
- zeroCalories 3y agoSure, if your workload is small and you have a SLA of "it will be fine."
- Aeolun 3y agoMost businesses do not need an SLA on ther webservice beyond ‘call us if it doesn’t work’.
- jamesblonde 3y agoCheck out RonDB (https://www.rondb.com/blog https://www.rondb.com/blog) which is a fork of MySQL Cluster but has support for elasticity with no downtime, backups, etc. Disclaimer: i am involved with RonDB.
- hintymad 3y agoI wish there are authoritative books or papers on how to build object stores like S3 or software-defined storage in general. It looks object stores in public clouds have been so successful that few companies or research groups have been working on their own. Yes, I'm aware that we have systems like MinIO and Ceph, but so many questions are left unanswered, like how to achieve practically unlimited throughput like S3 does, like what kind of hardware would be optimal for S3's workload, like how to optimize for large-scan incurred by analytics workload, which S3 is really good at; like how to support strong consistency like S3 does without impacting system performance visibly even though S3 internally must have metadata layer, storage layer, and an index layer, like how to shrink or expand clusters without impacting user experience, like how to write an OSD that squeezes out every bit of hardware performance (vitastor claims so, but there's not many details), and the list goes on.
- emmanueloga_ 3y ago> how to achieve practically unlimited throughput like S3 does > like what kind of hardware would be optimal for S3's workload > how to support strong consistency like S3 does without impacting system performance I think most of these questions are first and most importantly _hardware_, _money_ and _physics_ questions. I have no expertise in this matter, but I think this would be a good proxy answer: you make a system _seem_ to have unlimited throughput by interconnecting _a lot_ of the fastest available storage solutions with the fastest available network cards, putting it as close as possible to where your users are going to be. All of this is extremely expensive so you would need to have deep pockets, as amazon has, to make it possible (or a lot of investment). I suspect with the right hardware and physical deployment locations you could fine tune Ceph or MinIO or whatnot to similar performance as S3. S3 is an object storage so the distributed system aspects of its implementation should definitely be a lot easier than, say, distributed SQL (not saying either is an "easy" thing to accomplish). If you are interested in which hardware to use for a SAN, I found these benchmarks that may be exactly what you are looking for :-) [1] -- 1: https://www.spec.org/storage2020/results/ https://www.spec.org/storage2020/results/
- hintymad 3y ago
- gigatexal 3y ago“A free book” aka a book by a database vendor that wants to skew the premise of this-is-just-a-discussion-on-performance-at-scale-vendor-agnostic to that very vendor. Nothing is free. It’s all marketing.
- benjaminwootton 3y agoYes it’s marketing, but if you have a complex product or service offering, a short book can be a good way to get the thinking across.
- dacryn 3y agoI do like the premise though of this one - here is how to optimize your database - here is why that works from a theorethical perspective - by the way, our product makes this process easier that's fine by me
- carpintech 3y agoAlex Debrie: " Damn, this is an excellent resource. I read the first few chapters closely and plan to return to the rest. Lots of interesting stuff + frank discussion of how to truly evaluate different database options." https://x.com/alexbdebrie/status/1709960305122418898?s=20 https://x.com/alexbdebrie/status/1709960305122418898?s=20