6 ms·
Something I learned about writing robust code is that scalability needs to be tested up-front. Test with 0, 1, and many where the latter is tens of millions, no
by jiggawatts 3y ago
Something I learned about writing robust code is that scalability needs to be tested up-front. Test with 0, 1, and many where the latter is tens of millions, not just ten.
I've seen production databases that had 40,000 tables for valid reasons.
I've personally deployed an app that needed 80,000 security groups in a single LDAP domain, just for it. I can't remember what the total group number of groups across everything was, but it was a decent chunk of a million.
Making something like Git, or a file system, or a package manager? Test what happens with millions of objects! Try billions and see where your app breaks. Fix the issues even if you never think anyone will trigger them.
It's not about scaling to some arbitrary number, it's about scaling, period.
- vincnetas 3y agocould you elaborate on 40,000 tables DB? I want to learn what could be valid reasons for that?
- phyrex 3y agoRepresent every entity (like “person” or “post”) in a large graph using a relational database. You can get to 40k rather quickly
- vincnetas 3y agoIs this valid reason to use RDBM like this?
- phyrex 3y agoWhy not? I don't think there are many graph databases that are set up to handle multiple petabytes of data, so RDBMs make a good storage layer at that scale
- Timon3 3y agoDo you have any example for a project with 40k entities? I'd love to see how they handle the complexity.
- horse_dung 3y agoDatabase audit tool and they needed test what happens when an excessive number of tables is hit??? :)
- jasonjayr 3y agoNot the OP, but in our case, 300 tables x 300 customers (different 'schemas/dbs') == single mysql instance with 90000+ tables.
- vincnetas 3y agoSure multi tenant DB's, but lets limit this to per project DB's. 300 tables is quite reasonable, but multi thousands?
- wiredfool 3y agoI've got a db that hosts postgresql versions of CSVs/XLSs that are uploaded/harvested to an open data portal (as part of the portal). There are ~10k of them in there (+-), and could certainly see more (O(5k)) if some of the CSVs were parsed better.
- stef25 3y agoNever understood why each customer would have their own DB. Must be a nightmare to maintain.
- indeed30 3y agoIt makes enterprise sales easier, since it removes a common objection from security, privacy and compliance teams.
- Cthulhu_ 3y agoWouldn't having separate databases (with separate users (per organization)) make more sense from a security point of view? I have no knowledge of these things, I've never actually worked with more than one database in a mysql instance. edit: I tell a lie, I separated the forums and wordpress databases on a website I run.
- vincnetas 3y agoOne DB server can have multiple DB's. In this case we are talking about single DB (not server) containing multi thousands of tables. And im curious what is the use case for such designs.
- jiggawatts 3y agoSAP with a bunch of plugins plus custom tables added for various purposes. This is for managing the finances of 200K staff across 2,500 locations.
- stef25 3y agoIsn't that just bad design on the part of SAP ?
- miroljub 3y agoNon necessarily. Would it be "better" if they had one table with json/xml/whatever and handled schema in code? They made a trade-off they found right. When they hit the limit with their approach, they even implemented their own DB (S4/Hana) to support their system.
- tambourine_man 3y agoI was surprised as well, I feel one may need a DB to sort that DB. A meta DB :)
- pravus 3y agoI worked at an educational institution where we ran an academic-focused Enterprise Resource Planning (ERP) system that was fairly large. Not quite 40k tables, but it had over 4k. To give you an idea of how this was organized: * Most simple things like a "Person" were multiple tables because you had to include audits and historical changes for each field * A "Person" wasn't even all that useful because it included guests or other fairly transient entities like vendor contacts so you had an explosion of more tables as you classified roles into "Student", "Faculty", "Employee", etc... (many with histories as above). * Addresses and other non-core demographic information were usually sharded into all sorts of categories like "primary", "parent's", "last known good", "good for mailing", etc... (more histories, etc...) * All coded information like label types such as "STUDENT", or "MAILING" were always handled as separate validation tables with strict FK constraints and usually included extra meta information like descriptions and usage notes within parts of the system. * Each functional sub-system (HR, Payroll, AR, AP, etc.) had its own dedicated schema. * All external jobs, processes, and external integrations were configured separately. * All enterprise integrations usually had a whole a dedicated schema for configuration. * Most parts of the interactive web UI were database driven (Oracle's Apache mod PL/SQL) with many templates and other components stored in large collections of tables. I'll stop there, but basically just imagine a very large application that tries to be 100% database-driven. That's how you get a lot of tables.
- Spivak 3y agoAnd honestly, I kinda get it. Until you run into a case where your volume is such that you physically can't run on the db then run it on the db. I run all my job processing off the DB and couldn't be happier. I have to hit "can't run along side the real data" and "can't run in its own db" before I'll need to consider something else. It probably feels weird for devs to drive the UI off the db but it's just Wordpress by another name.
- Cthulhu_ 3y agoI've worked with / on an application like that, it had all form fields awkwardly configured in a database, plus a complicated database migration script to add, remove and update those fields. When I rewrote the application I just hardcoded the form fields, nobody should need to do a database migration to change an otherwise mostly static form.
- marcosdumay 3y agoAny ERP will bring you near that.
- lenkite 3y agoSaw ~60k tables in one famous ecommerce company backend primarily due to sharding - spread across multiple DB's of-course.
- horse_dung 3y agoI would agree with scale orders of magnitude higher than you can possibly imagine. But once you know what your scaling limits are (and there are always are limits) and what the (pre)failure behaviour looks like… we’ll you don’t have to fix them…
- miroljub 3y agoWhile this is true in some cases, more frequently I saw apps designed and able to handle millions of users and billions of transactions that ended being used by tens of users and hundreds of transactions. All the effort spent on testing and optimizations for scaling purpose was a waste of time and resources, that could be better spent elsewhere. I'm not telling one should not care, or code sloppy, but there is a balance where code is just good enough for the purpose. There's a lot of truth in this "don't do premature optimization".
- heavenlyblue 3y agoYeah and these apps probably would never work with that many of users in practice because they missed a few things here and there and the only way to fix them would have been to have that amount of traffic in the first place.
- jasfi 3y agoI think the GP was saying that it should scale without breaking. It can get slow, fine, that's a different challenge. But it shouldn't segfault (as an example).
- yebyen 3y agoThe gist of it is this: many load tests don't even consider the actual potential volume of traffic. But that's fine, if you're using a load tester, you don't have to estimate the traffic well - even to within an order of magnitude. You can just add a couple extra zeroes, and see where it breaks. Failing to do this simple thing will usually lead to objectively worse software, and there's a chance that some day you'll need to handle that much traffic. But that chance isn't the sole reason why you're doing that load test. The reason is to improve the software. You're identifying defects by stressing the limits. When you're doing a load test (or any test really) the possible outcomes are basically three: (1) it works! (2) it broke. (3) huh, that's interesting. If your tests are always coming up (1) then you're not obtaining any benefit from them. Don't you want to know where the limiting factors are in your app? If you're able to remove those limits, but not for production (at least not right now), wouldn't it be great to know what will break next month (or next year) when you do? Think of the person who writes unit tests for every piece of code, but not as TDD. There's a school of thought that you should write the test first, then write the simplest code that passes, and that's fine but not what I'm talking about. Imagine a person who writes perfect code and perfect tests. Every code works, every test passes confirming that it worked. What is even the value of writing the test? That's what load testing under only the expected conditions is like. We already know the software works under those conditions (likely) because it's already in production, handling that amount of load. So while there is value in a load test that runs prior to deployment, in order to check that nothing of the change is likely to induce a break under the expected/existing load, it's a different kind of testing and produces different value than a stress test that is designed to hopefully induce a failure and show you where there is a defect in code. Where it segfaults, for example. And just because you've identified a limiting factor outside the bounds of what expected activity is likely to go through the system soon, doesn't mean you need to fix it now. Having one less "known unknown" on the table is a thing of value. Now that stress won't be able to surprise you later, when that parameter has drifted into the danger zone because of organic development, and now it's becoming a thing in the way.
- switch007 3y agoAt what cost?
- 5e92cb50239222b 3y agoAmong mainline Linux filesystems, xfs started doing this first. The test suite is still named xfstests, although many more filesystems rely on it now. They regularly test xfs on enormous filesystems which 99.9% of us will never see, both with hundreds of billions of tiny files, and relatively small numbers of very large ones, plus various mixes of the two. Pushing it into edge cases like billions of files in one directory without any nesting. I really like that strong engineering culture and that's why I prefer xfs for most stuff.
- silvestrov 3y ago> edge cases like billions of files Sometimes edge cases can quickly detect bugs that only happend rarely under normal circumstances and therefore is difficult to reproduce/debug. E.g. when programming in C for little endian computers it can be a good idea to test code on big endian CPUs as the difference in endianess can reveal "out of bounds" writes for pointers.
- crabbone 3y agoThis is really not unique to XFS... anyone who worked on a filesystem in at least the last decade would tell you that a test like the one similar to what OP inadvertently created are commonplace. Unlike with many user-space applications, filesystems have very well-defined range of conditions they have to work in. Eg. every filesystem worth its salt will come with a limit on number of everything in it, i.e. number of files, groups, links and so on. And these limits are tested, they aren't conjectures. Ask any filesystem developer how many metadata operations per second can their program do, and they will likely be able to answer you in their sleep. This might be surprising on the consumer end of the deal, but to the developers there's nothing new here.
- hinkley 3y agoThese sorts of exercises help with performance tuning in the small. One thing you should learn, and many don't, about perf analysis is that you start getting serious artifacting in the data for tiny functions that get called an awful lot. I've found a lot of tangible improvements from removing 50% of the calls to a function that the profiler claims takes barely any time. Profilers lie. You have to know what they lie about. When I'm trying to optimize leaf- or near-leaf-node functions I've been known to wrap the call with a for loop that runs the same operation 10, 100, 1000 times in a row, just so I can see if some change has a barely-double-digit effect on performance. These predictions usually hold up in production. Just be very, very sure not to commit that for loop. Or use representative data that is ridiculously large compared to the average case.
- notTooFarGone 3y agoI mean that's how you get k8s for projects that in reality will never need it. Now you have a developer that is only doing k8s. Managing overhead and minimizing it is really something to keep in mind. So your App can't handle 100000 concurrent Users? As long as there is a plan how you could enable that in case of emergency there is really no incentive to have all that premature optimization for 90% of companies imo.
- Karellen 3y ago> Making something like Git, or a file system, or a package manager? Test what happens with millions of objects! Test with, say, one of largest open-source projects in existence at the time? Like, for instance, the Linux kernel?
- compiler-guy 3y agoThe kernel is big, but it isn't _that_ big in the grand scheme of things. The project from the original article here is bigger, and many companies have projects bigger than that.
- layer8 3y agoWhen Git was first released, the Linux kernel sources had less than 20,000 files. It currently has around 70,000 files. It’s not nothing, but it also isn’t millions.
- zoomablemind 3y ago> ...scalability needs to be tested up-front. I'd rephrase it: if expecting massive or longterm use - know how that thing is built/designed. Picking a technology based on general popularity or vendor's marketing is not a way to solve _your_ problem.
- dahfizz 3y agoScaling code isn't always as simple as rewriting your search function to be faster. What if scaling to millions of objects forces real tradeoffs for the hundreds of objects case? It feels like you're asking people to only create Postgres, but SQLite has a perfectly valid use case as well. In this case, git checking the access time of 500k files is fundamentally slow. The only way around this is to change how git tracks files, which all come with other usability tradeoffs. Git itself supports a fsmonitor that makes handling more files faster, but very few people use it because the tradeoffs aren't worth it.
- adamckay 3y ago> It feels like you're asking people to only create Postgres, but SQLite has a perfectly valid use case as well. I don't mean this to be a "well actually" comment, but because I found it interesting when I learnt this a few weeks ago - some limits for SQLite [1] are actually higher than the limits for Postgres [2] (specifically the number of columns in a table and the maximum size of a single field). 1 - https://www.sqlite.org/limits.html https://www.sqlite.org/limits.html 2 - https://www.postgresql.org/docs/current/limits.html https://www.postgresql.org/docs/current/limits.html
- lordgrenville 3y agoI've been working with some SQLite databases that are >100GB lately, and wondering if this is a bad idea. The theoretical max size is 140TB, but there's a big gap between can and should.
- crabbone 3y agoPeople who make filesystems test this stuff and will be able to tell you the ballpark figure for performance of this kind of operation even w/o testing. Testing here isn't the problem... The problem here is that we need a reasonably small interface for filesystem to enable competing implementations, so, for example, we don't have a filesystem interface for bulk metadata operations, because this is an unusual request (most user-space applications which consume filesystem services don't need it). So, we can only query individual files for metadata changes through "legal" means (i.e. through the documented interface). And now you end up in a situation where instead of fetching all the necessary information in a single query, the performance impact of your query scales linearly with the number of items queried. Even if Git developers anticipated this performance bottleneck, there's not much they can do w/o doing some other undesirable stuff. Any solution created outside of the filesystem would risk de-synchronization with the filesystem (i.e. something that watches the state of the filesystem dies and needs to be restarted, either loosing old changes or changes done between the restarts). Another solution could try going behind the documented filesystem interface, and try to salvage this information directly from the known filesystems... which would be a lot of work compounded with the potential to screw up your filesystem. Maybe if we'd have Git integrated with the kernel and be able to thus integrate better with at least the in-kernel filesystems. But this would still put people on anything but Linux at a disadvantage, and even on Linux, if you wanted some filesystem that's not in the kernel, you'd also have the same problem...
- SAI_Peregrinus 3y agoRather, test at the extreme points. If you're using a size_t counter, test 0, 1, and SIZE_MAX items. Etc for other sizes & types.