10 ms·

In my experience most people creating CRUD websites with Django will hit a CPU bottleneck.

• onepointsixC 7y ago
  Would Django going async ease that a bit or not really?
  • sojmq 7y ago
    async doesn't decrease the amount of CPU cycles needed to do work, in any case it only increases it.
• traverseda 7y ago
  I've got more than a decade of on-and-off web-dev experience, a lot of integrating data-science projects with a web-interface, and I've never seen anything like that. You have to work pretty hard to hit a CPU bottleneck, and I can't imagine how you'd do that building a simple CRUD website. Can you explain a bit more about how the people you know are hitting that bottleneck? I mean I've literally built CRUD apps on an esp32 microcontroller using python, and they work performantly. If real python with a real web-framework can't do the same then something is going horrible wrong.
  • gdfasfklshg4 7y ago
    If you do a bunch of filtering/joining etc that should be done in SQL by grabbing whole tables you can hit a CPU bottleneck... But then maybe your problems are deeper that choice of language.
    • takeda 7y ago
      That's trend that I see often. The other day I saw someone providing a response to argument that when using NoSQL database you need to plan in advance how the data is accessed (so you can pick the right key). The response was "don't you need to do that in every database?". So many people immediately dismiss relational databases, then re-implement all that functionality in their apps with various bugs and performance bottlenecks. ORM is another bad technique. It supposedly promises you that you don't need to know SQL to use, and that is true for the simplest examples, but you absolutely have to know it for anything less trivial, then you have to figure out how to write query in ORM to get a desired SQL statement (it makes it very difficult to use advanced SQL functionality), and that's not the end. ORM constantly will make unnecessary SQL queries and by default request all fields, even if you don't use them, adding additional performance bottleneck. In my current workplace thanks to ORM we make on average 10 queries per request and at peak generate 1Gbps throughput from/to database because of those inefficiencies. I think the way to go is SQL support in IDE, I recently saw PyCharm with DataGrip where if you configure it to connect to a database it will start recognizing SQL in string statements and start treating it like code (so you now have autocomplete, refactoring etc). I think this is probably the proper way to do it and I wish that other IDEs would have similar features.
      • zzzeek 7y ago
        > It supposedly promises you that you don't need to know SQL to use please support this assertion with an ORM whose documentation promises this. > ORM constantly will make unnecessary SQL queries and by default request all fields, as does "SELECT * FROM table" if you don't write out the fields and use a buffering database adapter (which is the case for nearly all Python database adapters), so, when using an ORM, you need to give it instructions over what columns you need to fetch. This is not unusual nor even anything a library could possibly guess for you if you do not give it this intent.
        • int_19h 7y ago
          The problem is that once you start fetching specific columns in your queries, the resulting objects aren't entities, just records - i.e. it's not really object-oriented, which is the main allure of ORM. ORMs that are more honest about this, such as SQLAlchemy, are generally better than those that try to pretend that you really are dealing with entities.
          • collinmanderson 7y ago
            I use .values() and .values_list() to fetch raw records instead of objects. It still allows me to construct the query using the ORM which is nice.
            • int_19h 7y ago
              I think that "ORM" is a bit of a misnomer for that. I'd call it "language-integrated query", if not for the potential confusion with LINQ (which stands for exactly that, not coincidentally).
              • takeda 7y ago
                There's a clear distinction between query builder and ORM though you can especially see this with SQLAlchemy. But anyway even query builder is not that great. Recently used PyCharm's integration with DataGrip. Basically the way it works is that if you configured a database in your project and let PyCharm fetch its schema, suddenly the IDE started recognizing the SQL statements providing autocomplete not only the statements but also table names etc, it also offered refactoring which created migration scripts. After using that I think that's the proper way of solving the impedance mismatch and at that point you no longer need ORM or query builders. I hope other IDEs will start doing the same thing.
• pytester 7y ago
  IME database always bottlenecks first.
• acdha 7y ago
  Your experience is very much unlike my own. Every project I’ve seen hit issues with the database (e.g. unindexed queries, changing data access patterns) first, followed by I/O (especially other services like S3, search, etc.), and RAM usage before CPU became a major factor. It’s possible, of course, but I’ve usually seen it as a symptom of not having a good culture around monitoring and troubleshooting — e.g. I remember someone porting an entire site to Jinja2 alleging performance wins, and it’s true that template rendering got (IIRC) 10-15% faster but they’d missed that 99.999% of the total runtime was being taken up by unoptimized database access generating many thousands of queries.
  • qxnqd 7y ago
    Of course, if you are incompetent enough to write queries that use no indexes and you don't detect that before you push to prod, you will have problems there. What I mean is: when you start optimising, no matter how well you do, you will eventually hit a CPU performance wall. Then you will realise that only a rewrite will get you out of that hole, and by then it will be late.
    • acdha 7y ago
      In my experience, many sites never hit that point of having CPU-bound Python code which they can’t afford; the ones which do hit that wall are usually large enough that they can carve out room for a C/Rust library, microservice, etc. rather than rewriting the entire thing.
      • frankwiles 7y ago
        I concur, I’ve worked on at least a dozen “web scale” Django sites and I can’t think of a single one where we were CPU bound.
• roddds 7y ago
  In my experience it doesn't sound like you have a lot of experience anywhere near CRUD websites with Django or otherwise.
• djstein 7y ago
  just going to post my usual answer to these types of comments: if it's good enough for Instagram, it's good enough for us.