11 ms·

> Rule 1. You can't tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hac

• maxk42 6y ago
  This is particularly exasperating for me. I can't tell you how many times in my professional career I've ended up speeding up systems by removing two or three layers of improperly-implemented "caching" and using good ol' MySQL and a basic understanding of algorithmic time complexity to simplify things.
  • flukus 6y ago
    Me too. I've seen a few systems that replaced their simple request requiring 10,000 queries "optimized" by requiring 10,000 cache lookups when they should have just added some joins. The bottleneck is the network latency, not the database. The worst I've seen is an nHibernate cache stored in a session variable, half the database was being serialized/deserialized on every http request. Fortunately that was a small database. Even with in memory caches I've seen systems grind to a halt by death of a thousand cuts, dictionary based entity attribute systems where each attribute is looked up individually. There seems to be a mentality that constant lookup == free lookup and devs don't seem to realize that constant * $bignumber == $biggerNumber. Caching shouldn't be granular. Obligatory latency numbers every programmer should know: https://gist.github.com/jboner/2841832 https://gist.github.com/jboner/2841832
    • YarickR2 6y ago
      It very much depends. I've seen too many cases when business logic was moved to the database layer, with absolutely disastrous effects. Scaling application backends doing map-reduce on a pre-filtered datasets is way easier than scaling database to handle five-way joins. Yet,I've seen opposing cases too, when rdbms was used as a huge key-value storage , and was pushing hundreds of thousands of rows to tens of backends, sucking it's bandwidth dry.
    • noisy_boy 6y ago
      Having worked with Hibernate recently on a smallish database (that isn't expected to grow too big), my take away is to minimize the network trips. If you have to check something for a list/set of things and all of that can be done from, say, two not so big tables, fetch the superset of records via a join with as much criteria pushed to predicates as efficiently possible and handle rest of the logic in application code. It is so much more quicker. Of course, like most things in software, conditions apply like network latency/total no. of records being fetched by the query/potential for data to suddenly grow etc.
      • bernawil 6y ago
        Not to cache simple trips to the database is pretty uncontroversial, the are more controversial cases elsewhere. Say you consume a Restful API to hidrate order data in another system. Do you fetch orders as: /orders?ids=id-1,id-2,id-3 or separate calls to /orders/id-x which can be cached and retrieved by id as a memoized function? Well if you had to pick only one the second is probably better, but the best would be abstracting away order fetching in application code to always fetch single orders and behind the scenes looking up the cache for singles and pooling all the misses into a single request to the plural endpoint.
        • flukus 6y ago
          If this is a common occurrence the former would almost always be better, the second would be absolutely glacial. > but the best would be abstracting away order fetching in application code to always fetch single orders and behind the scenes looking up the cache for singles and pooling all the misses into a single request to the plural endpoint. Unless a significant amount of requests have 100% cache hits then I doubt a local cache will make much of a difference at all, all it's saving is a bit of bandwidth.
  • earthboundkid 6y ago
    My tech lead once speed up our system by just turning off Memcached. :-)
  • cvrjk 6y ago
    I had worked on an application that saw less than 100 writes per minute and about 1k reads per min and used a caching layer in front of the DB. Not only was the cache actually slowing us down, it was also inconsistent with the DB. Can't even begin to express the amount of lost dev time and productivity. We couldn't get rid of it, because someone above was convinced that we would need it to scale in the future.
    • Cthulhu_ 6y ago
      There are only two hard things in computer science: naming things, cache invalidation, and off-by-one errors. I'm trying to never implement any caching if I can help it. The database itself does caching already as well. And if you DO need caching, keep your hands off of the application; add a cache layer in front, or between the application and the database. But don't invent it yourself.
      • bernawil 6y ago
        And if you DO need caching, keep your hands off of the application; add a cache layer in front, or between the application and the database. But don't invent it yourself. so, redis?
  • pitay 6y ago
    My problem was people using ORM frameworks and having very little knowledge of the database they were using. Slow as molasses, difficult to figure out what is going on because of many layers of extra "stuff".
  • oh-4-fucks-sake 6y ago
    Same boat. Basically every day.
• farhaven 6y ago
  Do you work at my company? Because I have a coworker that is always proposing _exactly_ that solution. No matter what issues the code has, for him its usage of MySQL is always "the worst moment". Asking for benchmark just gets a repeat of "our worst moment is MySQL and we can solve that with some NoSQL cache".
  • fizwhiz 6y ago
    My dude's just trying to get a promo, I guarantee it.
• paulryanrogers 6y ago
  Never read the release notes for MySQL. Unless phrases like "server exit" or 'regression' don't bother you.
• enahs-sf 6y ago
  Wholeheartedly agree. What you're describing is like the ideal mysql use case. Also, even the most basic next steps performance-wise will probably tithe you over well into the TBs of data. The headroom for mysql is quite a bit higher than most people think unless you're operating at facebook scale.
• TeMPOraL 6y ago
  That case you've seen speaks of the guy's inexperience and lack of understanding. I have a problem with this rule because what I see happening is people taking it to heart and no longer thinking about what they're doing performance-wise. And then the program is working 1000x slower than it should, at no extra gain (and often a loss) of readability or safety, just because someone decided to use O(n) data structure where O(1) would do, or keeps repeating the same computation thousand times instead of ensuring it's done once. So to your "let stuff work", I want to also add: "understand the work you're putting the computer through", and "don't do stupid things".
  • keithnz 6y ago
    I think maybe you are misreading the rule, it doesn't say don't optimize, it says when optimizing, don't guess from the code where the bottleneck is, go measure it.
    • vvanders 6y ago
      Yeah but some things like how you structure your data(which then drives CPU cache misses) aren't something you can easily adjust/tune. Usually when you encounter one of those it's a rewrite/rearchitecture of a whole module/subsystem before you see any gains. Been there done that, not excited to repeat it again.
      • tonyhb 6y ago
        The other rule is choose your data structures wisely.
        • tikiman163 6y ago
          The number of people who don't understand when to use a list/array VS dictionary/hash table vs lookup object is too damn high. A huge amount of basic optimization is constantly at their fingertips and they nearly always choose to make a list/array and use linq to join/merge multiple relational data sets instead of optimized standard objects.
• zimpenfish 6y ago
  > a Redis caching layer and a complex custom lookup scheme Bane of my life. A few gigs ago, they had a global Redis cache, a Redis cache per server, an in-app cache, and then MySQL. Needless to say, there were many, MANY bugs that came down to cache coherency and race conditions between them. [edit: it was a global Redis, not clustered.]
  • hckr1292 6y ago
    Did we both work at a certain edtech company? Seems oddly specific and exactly how I remember our monolith being structured
    • zimpenfish 6y ago
      Alas, that wasn't edtech, it was photos related.
• aprdm 6y ago
  hehehhe I have actually put some data that do not change more often than once a week hardcoded in the code base and commited in source control... whenever a change needs to happen in the data the CI/CD runs and deploys a new version of the app. You don't wanna know how that was done before. It is like < 1 GB of JSON as well.
  • indiv0 6y ago
    Please tell me you have git LFS configured to handle this.
    • aprdm 6y ago
      Yup !
  • dkarl 6y ago
    I wish we had a deployment process trivial enough to support that. On the other hand, eventually you want to farm that job out to customer success or an implementation engineer, and then it's not so fun having the data in source control.
• throwaway894345 6y ago
  I ran into this early in my career when a senior engineer at a startup reviewed my Python PR for operating on a CSV dataset and insisted I rewrite it in Pandas for performance. It was a very simple program with naive Python (a handful of lines), but the Pandas version was far longer and more complex (I had to have the senior engineer help because it took some advanced Pandas-fu, and he spent nearly a full day on it), and it ultimately ended up being an order of magnitude slower because we ultimately had to call back into Python for each cell. I actually don’t think “premature optimization” is all that bad in the general case, but you have to be educated about it (and I’ve met a lot of people in Python shops who think that Pandas or multiprocessing will cure all performance ails), and you should specifically think about how costly is a given optimization going to be to maintain or back out of if you’re wrong about the performance benefits. In general, I’ve never worked on a Python project where we didn’t have to do weird, inordinately expensive things to work around performance issues (though I’ve never worked on a rudimentary CRUD app either) nor has “just throw Pandas/C/multiprocessing at it” ever adequately addressed our most significant performance bottlenecks (usually the solution looks something like Spark, all to do something that would have been sufficiently performant with naive Java or Go). This might just be my experience working on nontrivial SaaS apps; maybe if you’re just doing straight data science or CRUD apps or workloads that aren’t latency-sensitive (mind you, we struggled to keep per-request performance in the tens of seconds, so I use “latency-sensitive” very loosely), Python/Pandas will be just fine. We also ran into other problems with Python, such as packaging and distribution; notably our lambda functions were routinely too large because the pandas branch of the dependency tree was itself more than half of the permitted artifact size (to work around, we switched to Fargate tasks, which have a much larger size limit but take 30s or minutes to boot up).
  • bpyne 6y ago
    "I ran into this early in my career when a senior engineer at a startup reviewed my Python PR for operating on a CSV dataset and insisted I rewrite it in Pandas for performance." Did a performance problem exist? If not, then I would never promote that person to senior engineer. A working program should only be re-engineered for performance if it wasn't meeting the performance contract agreed upon when the program was written.
• deckarep 6y ago
  I see this behavior a lot too. Inexperienced engineers will nitpick about what is often minor “performance optimization”, clearly not seeing the bigger picture. Example, why should we spend precious developer time to rewrite some code using something that is often less readable when it’s called once every 30 seconds? To the folks who do this: you are better off spending the time making data-driven decisions and optimizing for the big picture. In other words measure first than come up with an optimization that has a large impact on the system. Not this micro-optimization, I-love-to-tickle-myself stuff. Learn to see the bigger picture.
• userbinator 6y ago
  It seems that a subset of developers are somehow convinced that adding complexity is a good thing for performance and should be done first, when it should really be a last resort. Their usual rebuttal is "it's scalable" and other buzzword-laden phrases. I wonder if it's a form of "big data envy" or just regular "architecture astronautism".
  • Cthulhu_ 6y ago
    Honestly, it's the other way around usually; simplicity is scalable. A stateless service talking to a database is fine. You can scale out the service, and scale up the database. It's even easier when you use a cloud provider, their relational database offerings can scale from a wordpress blog's database to enterprise scale, tens of thousands of transactions / second. The problem is that it's boring, and there's a lot of developers that create work and complexity to make their own jobs interesting.
    • Aeolun 6y ago
      No, no. We need all these layers of proxies, caches and various databases to make things more reliable!
• avasthe 6y ago
  This is taken to other extreme many times. Example: Google Chrome codebase was allocating lot of std::string and also someone used a Set to check membership of single item. [1] I mean, if you say like this, many people don't even care about algorithm complexity. Doesn't help that people want to write Python in the monster that is C++. https://groups.google.com/a/chromium.org/forum/m/#!msg/chromium-dev/EUqoIz2iFU4/kPZ5ZK0K3gEJ https://groups.google.com/a/chromium.org/forum/m/#!msg/chrom...
  • account42 6y ago
    I very much agree that many people use rules like these as excuses to write shitty code. From the post you linked though: > Not reserving space in a vector when the size is known std::vector::reserve() is actually not something you should always use when you are adding a number of elements as it will (typically) grow the vector to exactly what you ask. If your function then gets called in a loop to append to the same vector multiple times you end up with quadradtic run time that is normally avoided by the geometric growth done when you just append without reserving.
  • coldtea 6y ago
    >and also someone used a Set to check membership of single item. [1] Depending on where it was done (fast path or not), this could be just fine.
    • avasthe 6y ago
      You don't know when something comes to hot path. if (std::set(itr.begin(), itr.end()).count(element)) { _____ } is it tempting for someone than something like std::find(itr.begin(), itr.end(), element) != itr.end()) ?? I don't know. That said, C++ STL quite undiscoverable.
      • coldtea 6y ago
        >You don't know when something comes to hot path. That's the point of the first advice though...
• combatentropy 6y ago
  We all start out this way, don't we? I think what drove it out of me was having to maintain. Okay, you wrote this thing, now you need to change the way a few features work and add some others. Especially effective is if you take turns being on call, getting paged in the middle of the night when your multilayer architecture has a hiccup and now you have to troubleshoot it. I wish there was a quick remedy for such people. But usually they are new, based on what I said. So they should be Junior Developer, not Architect. Sure, you can suggest things, but the answer is nope. A lot of people, though, may never get much experience supporting their own architectures, because they swoop in, then swoop out, never staying at a job long enough. Or support gets assigned to another group. One thing I like about Agile is that the team that creates the code is the one that supports it.
• yashap 6y ago
  It’s insane how often people stick Memchaced/Redis in front of MySQL/Postgres, completely unnecessarily. So many otherwise decent developers assume that MySQL/Postgres is too slow/doesn’t scale, without even trying. Or similarly, how frequently people shard databases unnecessarily. A single MySQL/Postgres server, on a beefy machine, can handle an absolutely massive amount of work, with great performance, assuming you’ve indexed well and don’t have to run super crazy queries. I’ve worked on some pretty high traffic systems, with pretty large data volumes, and very, very rarely have I actually needed to cache simple DB reads. And it’s hard to properly cache complex ones anyways, because cache invalidation is so hard for complex queries. Likewise, it’s very rare that I’ve needed sharding. SOMETIMES you need these things, but mostly people are adding a lot of extra complexity and cost for no good reason.
  • nojvek 6y ago
    I don’t think people realize that databases themselves have a caching layer internally. Redis isn’t magical, you still have to send network packets to the other server. Even when reading from disk via mmap, hot pages are in memory. Sometimes postgresql is faster than redis because all it needs to do is read something from memory and spit it out in the right format.
    • yashap 6y ago
      Totally. If the hot pages of your indexes and hot pages of your records mostly fit in memory, DBs mostly read from memory anyways. If they don’t mostly fit in memory, things can slow down a bit, but you’re likely better off vertically scaling your DB (more RAM) than adding a cache. Looking up a normal sized record by id, with good networking in your data centre, takes ~1-2 ms round trip whether you’re reading from Redis/Memcached or MySQL/Postgres, and either can handle massive read load if sized properly. The cache just ~doubles your costs, is one more thing to patch/maintain, and introduces new sorts of bugs/outages.
• hansvm 6y ago
  No comment on your particular example, but the opposite approach is more common than I'd personally like and also not ideal -- if you're doing anything nontrivial it's probably worth doing a back of the envelope calculation to ensure you have enough disk, memory, and network throughput for your favorite solution to handle the expected load.
• cel1ne 6y ago
  > Can't we just let stuff work? I'm no fan of MySQL, but can't we admit that a ubiquitous and battle-tested piece of technology, applied to a canonical use case, on tiny data under near-ideal circumstances, is probably going to work just fine? I call that "separation of error-domains", meaning trying to find interfaces where you can separate off parts of your application, so debugging gets easier: Is the bug in the database, or in our code?
• cs02rm0 6y ago
  Yeah. I got told to put in a caching for a system that made very slow calls through a handful of proxies to the other side of the world and back. We went with redis for the "PoC". I'd tried explaining that if a query is only made once every 24 hours and the data can't be cached longer than that because it's considered too out of date then a cache is pointless extra complexity. He wasn't having it though, so he made us build it and demo it to a room full of people. Fortunately the room understood the simple explanation and he listened to them where he wouldn't listen to the dev team, so a few weeks of work was scrapped there and then.
• thdrdt 6y ago
  It is all about lack of knowledge because checking where the bottlenecks are is one thing. Knowing why they are there is another. A retailer app was very slow. The developers proposed a newer faster server with a newer Oracle version. Then I took a look at one of the slowest queries. Changed it so it could use the indexes in a better way and the query went from +20 to 0.7 seconds. The developers measured the query was slow, they checked the query used indexes and they were right that a server with more RAM and a newer Oracle version could improve the speed. But they missed that the query had to fetch a lot of data (using indexes) before it could start filtering the data. The only thing I did was to change the query so it could filter the data first.
  • Annatar 6y ago
    Bolt - 95 cents knowing where to put the bolt - 95 bucks per hour.
  • Cthulhu_ 6y ago
    Sounds like ops vs dev (or DBA) right there; it does make sense to a point that ops would grow into that kind of behaviour, given that they can't really allocate resources into performance enhancements for software (especially if it's from an external party or off-the-shelf). To them it's a black box. But if it's an in-house thing then there should be open communication lines. The SRE paradigm makes sense there, where I see a SRE as part dev, part ops. They can identify perf issues as a software problem and either fix it or send it back to the authors.
    • thdrdt 6y ago
      It was more like dev & dev. I am not criticizing anyone because I was just lucky to notice the query could be rewritten. But it was also the fact that I had a little more knowledge about how the db engine handles queries. So we all looked in the right direction, we all found the bottleneck but the solution was different based on different knowledge.
      • strgcmc 6y ago
• dvfjsdhgfv 6y ago
  > <1GB, moderate read volume, super low write volume MySQL database It looks like an excellent candidate to put it entirely in RAM and trigger sync on writes only, why on Earth would you need anything else.
  • dkarl 6y ago
    That's effectively what MySQL is going to do.
    • dvfjsdhgfv 6y ago
      For the most part, yes, but I was thinking about explicitly proloading all tables and indexes into RAM or even using the MEMORY engine, if it makes sense in that particular case.
• specialist 6y ago
  I feels ya. My most recent team had our hottest dataset in dynamodb. Because cloud. So much effort to get our web service's P99 <50ms. The whole dataset fit easily into RAM. Thru (way too much) effort, I was able to introduce Redis. First shared. Then eventually one mini-instance per EC2, running alongside nginx & nodejs.