16 ms·

> Python 3.11 is up to 10-60% faster than Python 3.10. Horrayy!! Does anyone has any experience with working with python in a very large scale? Most of the b

• ZeKK14 4y ago
  The typing issues are being solved through mypy: http://mypy-lang.org/ http://mypy-lang.org/ A large majority of packages are still missing typing hints, but the ecosystem is moving in the right direction on that issue.
  • robertwt7 4y ago
    What’s the difference between mypy and python built in type hint?
    • ehutch79 4y ago
      mypy, as well as things like pylance, enforce the type hints, the interpreter ignores them during execution.
• jsmeaton 4y ago
  I work on a reasonably large Django project (1M+ LOC) with approximately 100 developers and mostly have a reasonable time debugging. We use mypy, lots of linters (some custom flake8 plugins), and a fairly strict layered architecture[0] that's enforced with tooling[1]. Without the tooling it's a nightmare. My previous Django project with approx 500k LOC had linting and some typing and that was a mess. [0] https://github.com/octoenergy/conventions/blob/master/patterns.md https://github.com/octoenergy/conventions/blob/master/patter... [1] https://github.com/seddonym/import-linter https://github.com/seddonym/import-linter
  • jonatron 4y ago
    Is there any chance of making Octopus's vendor.bundle.js less than 15MB? It takes time to load on any device, and the size makes the site unusable on mobile.
    • jsmeaton 4y ago
      There are some efforts going into the frontend to reduce these sizes. There's a bit of a transition between frameworks happening.
  • robertwt7 4y ago
    What’s the difference between mypy and python built in type hint?
    • sk0g 4y ago
      MyPy provides static analysis that uses the type hints to actually, well, do things with it. Type hints are the specification, but MyPy|PyLance|Pyright|Pyre use those hints for and type checking.
  • jbkkd 4y ago
    200+ developers at this point
• smallerfish 4y ago
  My advice - go with Kotlin. You get the JVM, all of the Java ecosystem, type checking (of course), and a sane, modern language.
  • rattray 4y ago
    Kotlin may have many advantages over Python as a language (subjective), but the Java ecosystem is not an attraction over the Python ecosystem to me (both have flaws, but the Java world has a lot of bloated stuff)
  • mardifoufs 4y ago
    Why not just java? I thought it caught up to kotlin in the past few years
    • cutler 4y ago
      Hardly. Despite the introduction of raw string literals Java still can't handle a regex without having to double-escape metacharacters.
      • germandiago 4y ago
        How can you even say this? Regex is very specific. It has added records, streams, virtual threads (preview I think) working with native better, pattern matching... Java is like C++. Everyone talks bad about it but at the end of the day it is one of the most Getting things done language. I am not sure why I should use Kotlin. Llokd nice! But for sure docs and support are not even close to Java.
        • alex_suzuki 4y ago
          Are you seriously comparing Java to C++? One excels at helping you shoot yourself in the foot. The other gets things done…
          • sk0g 4y ago
• HPsquared 4y ago
  JPMorgan's Athena platform is 35M lines of Python. Probably not too relevant here though as it's all Python2 code...
  • jonatron 4y ago
    Do you know if BAML Quartz is similar (size and Python 2)?
    • wirthjason 4y ago
      Any number of resumes on LinkedIn with “Python 2 to 3 migration” will say yes.
  • Hamuko 4y ago
    Do they maintain all parts of their software stack? Third-party libraries are leaving Python 2 behind (or rather, have largely done so).
• groestl 4y ago
  Yeah, been on multiple, and for me it felt like a nightmare (maybe I'm just not smart enough) I'll use python for small tools, depending on the planned deployment. But as a general rule, everything north of 2 months developer time (being it just me, or multiple devs) gets a language with static types, at least for the non-glue main components. Glad that JS got TS types for that. My approach has served me well so far.
  • groestl 4y ago
    Never tried mypy though, I must say. Maybe that saves the day.
    • guilherme-puida 4y ago
      I find mypy to be underwhelming in comparison to Typescript, but maybe it's just the ecosystem. Most JS libraries nowadays are either written in Typescript or have type definitions, but some big Python libraries don't (or the types are not complete).
      • cinntaile 4y ago
        I recently started using mypy and my experience so far is that it's a bit of a pain to get it to play nicely with numpy.
• LtWorf 4y ago
  For typed loading data, my benchmarks show 3.11 is actually slower than 3.10. https://ltworf.github.io/typedload/performance.html https://ltworf.github.io/typedload/performance.html
  • ehutch79 4y ago
    So typed python runs slower on 3.11?
    • LtWorf 4y ago
      Not just typed, but when accessing types at runtime. But I didn't investigate on which change exactly is making it slower.
• matsemann 4y ago
  ~500k loc python project here. Detest it. For being such an old and established language, the ecosystem/community is very immature. Most stuff is unstable, there are thousands of ways of doing anything, no one has settled on which tooling to use so all projects are a miss-match of what was in vogue the month it got started. Typing is almost useless, like Typescript in the early days where you really cannot trust it not blowing up runtime even though you think you have typed everything. Especially since most libraries do stuff with (*args, *kwargs) and the type checker has to give up, or decorations not propagating types correctly, or Django ORM lying about types. You think you're safe, but secretly everything is just typed as Any. No safety enforced between modules, so every piece of code you write can be considered part of a modules public API as people will import and use it. Really lacking stdlib making fugly list comprehensions with mutations the standard way of solving things that no one will understand in a years time. Dependencies are a hell, everything break all the time, and "works on my computer" could lead to days of trying to figure out the setup on a coworkers computer where it doesn't work.
  • tpict 4y ago
    I don’t disagree with the “early TypeScript” comparison, but what’s the issue with args, *kwargs?
    • matsemann 4y ago
      The problem is that you lose all help from tooling/IDEs. Like in Celery, the definition is "shared_task(*args, *kwargs)". This gives you no indication of what parameters you actually can use. Opening up the code doesn't help, as it's many layers down. The decorated function ends up untyped, but with some new methods on it that again are untyped. But like originalfunction.delay(...) should have the params of the original decorated function. But no, all that is lost. Just pray that the docs are correct.*
      • Too 4y ago
        ParamSpec from 3.10 is adding some improvements to type hinting decorators and wrapper functions that just forward args and kwargs. https://peps.python.org/pep-0612/ https://peps.python.org/pep-0612/
• StevePerkins 4y ago
  I currently work at a startup that uses Python for everything. Since we're data science oriented (and Python is the only programming language the data scientists kinda-sorta know). And the CTO fears having to hire for more than one backend language (even though we have more people with prior Java experience than Python, and all of the backend API developers hate working with Python). We get by, but it's pretty awful. People will give you all sorts of arguments for why loose typing and an NPM-like sloppy ecosystem are advantages for Python. And for the domains where Python really shines (i.e. devops, and manual data-wrangling), maybe these are advantageous. But what they DON'T tell you is that best practices for large-scale backend development call for bolting-on so many extra things, and accepting so many constraints, that it wipes out the purported advantages. You'll need to use at least one linter, if not two, and a style checker to enforce the use of type hints. This gets you to something that is maybe 50-75% as effective as a Java compiler, but will never be any better than that, and will never have anywhere near the same level of IDE integration. If you have a lot of dependencies (and you will), then you'll need a proper build system like Poetry. Which puts you right back in that Maven world, that escaping from was supposed to be one of the advantages. And the dependency ecosystem is SO... FREAKING... BAD. Unlike Java, where most of the major libraries are professional in nature with corporate financial backing, most Python libraries are "pure" community and written by unpaid volunteers or hobbyists. Many of whom have NO understanding of semantic versioning, so you find yourself having to pin ALL of your dependencies to specific fixed versions so the spider web doesn't tangle up. Upgrading anything is a nightmare. You'll be told that it's easier to hire for. But then you'll find that the candidate pool is largely ops or QA/QE people looking to transition into engineering, boot camp grads, and other people with no professional experience in backend API development or even Python itself. Most capable candidates that you hire will take the job for other reasons (e.g. wanting some exposure to data science or ML), and will complain constantly about having to work with Python for general backend services. You can get by, but as the project or company grows, you'll probably find yourself either re-writing Python portions, or at least deprecating them as legacy and eventually migrating to some other new greenfield thing. I can't imagine any sane reason to migrate a Java or .NET project in the opposite direction.
  • BuckRogers 4y ago
• deleted 4y ago
  [deleted]
• zrlname 4y ago
• carapace 4y ago
  I've been on large projects written in Python, it's no good. (I say this as a Python fan!) For scripts and small projects or rapid prototyping Python is pretty much king. (Although Red/Rebol is a contender. Check out their GUI examples!) For medium-sized projects (100K LoC, 1-10 devs) you might use Go or you could experiment with the usual suspects: Lisp or Nim or Haskell or OCaml or whatever you want. For a large project (millions of LoC, 100's of devs) I would use Ada or Java unless the problem was very Erlang-shaped (in which case you would use Erlang (or Elixir.)) For front-end work I'll personally never use anything but Elm for the foreseeable future. Based on economics. Compared to Elm the entire JS ecosystem is a incredibly massive boondoggle, a total waste. (I'm cranky this morning and I'm practically trolling here. Apologies to those who are not entertained. I mean it though: in my considered opinion Elm mocks JS, brutally.)
  • hbrn 4y ago
    Could you elaborate on this more? What exactly happens between 20k and 100k LOC (or 2 devs and 10 devs) that makes Go more suitable than Python? What exactly happens between 100k and 1kk LOC (or 100 devs and 500 devs) that makes Java more appropriate than Go? How do you even find this boundary when same apps in Java and Python might have 2-3x LOC count difference? I've seen this mantra repeated for decades (technologies X/Y/Z for small/medium/large projects) and could never make sense of it. For example Angular was always sold as a technology for "large" projects (as opposed to React). Where is Angular now? All those "large" projects are now legacy that everyone despises. And if you have 100+ developers on a "large" Angular project I bet you're are feeling fucked now. Since everything breaks at scale, I'm actually into the opposite mantra: only the tech stacks that are great at small scale are capable of being great at large scale. Note: capable, not guaranteed. I always prefer layering complexity on top of a simple technology, instead of praying that complexity inside of a complex technology will perfectly match my needs.
    • carapace 4y ago
      > Could you elaborate on this more? > What exactly happens between 20k and 100k LOC (or 2 devs and 10 devs) that makes Go more suitable than Python? > What exactly happens between 100k and 1kk LOC (or 100 devs and 500 devs) that makes Java more appropriate than Go? > How do you even find this boundary when same apps in Java and Python might have 2-3x LOC count difference? I've been thinking about those questions for roughly thirty years, and I still don't have exact answers. (If I did I'd be famous and hopefully rich, but that's another story, eh?) Speaking in broad generalities, the main thing seems to be something like the number of separate concepts you have to keep in mind to make (non-breaking) changes to the system. The more that the system you're using can do for you, the more suitable it is for larger-scale projects. > I've seen this mantra repeated for decades (technologies X/Y/Z for small/medium/large projects) and could never make sense of it. I gotta say, it's not a "mantra", it's experience. Have you not worked with various languages on various sized projects? (In re: Angular, to me that was an obvious shitshow from day one. I would never hire anyone who admitted to ever thinking Angular was a good idea. Same thing in re: Heroku, while I'm at it. Lots of people do lots of foolish things when computers get in the mix.) > I'm actually into the opposite mantra: only the tech stacks that are great at small scale are capable of being great at large scale. Note: capable, not guaranteed. I don't see how that follows? (I also don't see how that's "opposite" to the other "mantra".)