14 ms·

Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a

• SanderNL 3y ago
  It sounds I/O heavy, but you mention it being CPU-heavy in which case I’d say Python is just not the right tool for the job although you may be able to cope with multiprocessing.
• iknownothow 3y ago
  I would consider the following optimizations first before attempting to rewrite an HTTP API since you already did the hard part: 1. For multiples processes use `gunicorn` [1]. Runs your app across multiple processes without you having to touch your code much. It's the same as having the n instances of the same backend app where n being the number of CPU cores you're willing to throw at it. One backend process per core, full isolation. 2. For multiple threads use `gunicorn` + `gevent` workers [2]. Provides multiprocessing + multithreaded functionality out of the box if you have IO intensive. It's not perfect but works very well in some situations. 3. Lastly, if CPU is where you have a bottleneck, that means you have some memory to spare (even if it's not much). Throw some LRU cache or cachetools [3] over functions that return the same result or functions that do expensive I/O. [1]: https://www.joelsleppy.com/blog/gunicorn-sync-workers/ https://www.joelsleppy.com/blog/gunicorn-sync-workers/ [2]: https://www.joelsleppy.com/blog/gunicorn-async-workers-with-gevent/ https://www.joelsleppy.com/blog/gunicorn-async-workers-with-... [3]: https://pypi.org/project/cachetools/ https://pypi.org/project/cachetools/
  • xmaayy 3y ago
    > 1. For multiples processes use `gunicorn` This will load up multiple processes like you say. OP loads a large dataset and gUnicorn would copy that dataset in each process. I have never figured out shared memory with gUnicorn.
    • sanderjd 3y ago
      One way to achieve similar performance is redis or memcached running on the same node. It really depends on the workload too. If it is lookups by key without much post-processing, that architecture will probably work well. If it's a lot of scanning, or a lot of post-processing, in-process caching might be the way to go, maybe with some kind of request affinity so that the cache isn't duplicated across each process.
    • zbentley 3y ago
      > gUnicorn would copy that dataset in each process Assuming you're on Linux/BSD/MacOS, sharing read-only memory is easy with Gunicorn (as opposed to actual POSIX shared memory, for which there are multiprocessing wrappers, but they're much harder to use). To share memory in copy-on-write mode, add a call to load your dataset into something global (i.e. a global or class variable or an lru_cache of a free/class/static method) in gunicorn's "when_ready" config function[1]. This will load your dataset once on server start, before any processes are forked. After processes are forked, they'll gain access to that dataset in copy-on-write mode (this behavior is not specific to python/gunicorn; rather, it's a core behavior of fork(2)). If those processes do need to mutate the dataset, they'll only mutate their copy-on-write copies of it, so their mutations won't be visible to other parallel Gunicorn workers. In other words, if one request in a parallel=2 gunicorn mutates the dataset, a subsequent request has only a 50% likelihood of observing that mutation. If you do need mutable shared memory, you could either check out databases/caches as other commenters have mentioned (Redislite[2] is a good way to embed Redis as a per-application cache into Python without having to run or configure a separate server at all; you can launch it in gunicorn's "when_ready" as well), or try true shared memory[3][4] 1. https://docs.gunicorn.org/en/stable/settings.html#when-ready https://docs.gunicorn.org/en/stable/settings.html#when-ready 2. https://pypi.org/project/redislite/ https://pypi.org/project/redislite/ 3. https://docs.python.org/3/library/multiprocessing.html#shared-ctypes-objects https://docs.python.org/3/library/multiprocessing.html#share... 4. https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem...
  • danpalmer 3y ago
    These don't really apply to the parent commenter's scenario. 1) gunicorn or any solution with multiple processes is going to just multiply the RAM usage. Using 10-100GB of RAM per effective thread makes this sort of problem very RAM bound, to the point that it can be hard to find hardware or VM support. 2) This isn't I/O bound. 3) If your service is fundamentally just looking up data in a huge in-memory data store, adding LRU caching around that is unlikely to make much of a difference because you're a) still doing a lookup in memory, just for the cache rather than the real data, and b) you're still subject to the GIL for those cache lookups. I've also written services like this, we only loaded ~5GB of data, but it was sufficient to be difficult to manage in a few ways like this. The GIL-ectomy will probably have a significant impact on these sorts of use cases.
    • kayodelycaon 3y ago
      For #1, would copy on write help? Or does python store the counters on the objects?
      • danpalmer 3y ago
        Ha! Yes! Unfortunately I know this because of terrible reasons. Python is reference counted so copy-on-write doesn't work for this with Python objects (note: if your Python object is actually just a reference to a native object in a library all bets are off, may work or may not). We had an issue with the service I mentioned above where VMs with ~6GB RAM weren't working, because at the point that gunicorn forked there was instantaneously >10GB RAM usage because everything got copied. We had to make sure that the data file was only loaded after the daemon fork, which unfortunately limits the benefits of that fork, part of the idea is that you do all your setup before forking so that you know you've started cleanly.
• lfkdev 3y ago
  You could have just use gunicorn and spawn multiple workers maybe
• nesarkvechnep 3y ago
  If your data doesn't change, you can leverage HTTP caching and lift a huge burden off of your service.
• severino 3y ago
  May I ask why you didn't consider writing that quick implementation in Java in the first place?
• vorticalbox 3y ago
  Why not load the data into sqlite dB and let the clients query that? Is there a reason you're loading 10s/100s gb into memory?
• datadeft 3y ago
  I don't think that Python was designed for this. I found it largely unsuited for such work. It is much easier to saturate IO with (random order) F#, Rust or Java (that I have used for in scenarios you mentioned).
• xcv123 3y ago
  > I may have missed something You did not miss anything. The GIL prevents parallel multi threading.
• jeremycarter 3y ago
  Similar experience. Even with multi process and threads python is slow, very slow. Java, Go and .NET all provide a very performant out of box experience.
  • __d 3y ago
    Python is both an interpreter, and quite dynamic. Both of these lead to lower performance when compared to less dynamic, compiled solutions. All of Java, Go, and .NET are compiled and (much) less dynamic. This is absolutely an expected outcome.
    • cypress66 3y ago
      Node is pretty performant for anything IO related, not compiled and reasonably dynamic.
      • __d 3y ago
        I think it's worth the clarification that Javascript is usually JITed; (C)Python isn't. And that CPython's I/O isn't really the problem: some of its async event loop implementations are fairly competitive with Node. But still ... yes. Javascript has benefited from two decades of intensive, well-funded work by the best people in the business, with clear focus on performance as a high priority goal. Not to take away from those who work on Python, but I think it's fair to say the effort has had orders of magnitude difference. I don't have a deep enough understanding to say whether the nature of Python or Javascript makes one better suited for performance optimization than the other. Python is perhaps able to benefit from seeing what's been done with Javascript, although of course Javascript has stood on the shoulders of its own giants.
    • mike_ivanov 3y ago
      "absolutely an expected outcome." Good day. Is it the right time to talk to you about Common Lisp?
      • tsimionescu 3y ago
• Waterluvian 3y ago
  No, that’s about right. The response, which isn’t technically wrong, is “unless you’re CPU bound, your application should be parallized with a WSGI. You shouldn’t be loading all that up in memory so it shouldn’t matter that you run 5 Python processes that each handle many many concurrent I/O bound requests.” And this is kinda true… I’ve done it a lot. But it’s very inflexible. I hate programming architectures/patterns/whatnot where the answer is “no you’re doing it wrong. You shouldn’t be needing gigs of memory for your web server. Go learn task queues or whatever.” They’re not always wrong, but very regularly it’s the wrong time to worry about such “anti patterns.”
  • dotnet00 3y ago
    Yes, this is even more the case in languages that are popular with more "applied" programming audiences, like scientific computing. Telling them "no you should be using this complicated DBMS" (or whatever other acronym) is not productive. It tends to get them exceptionally mad because their concern isn't the ideal way to write the code and architect the system, they simply want to write just enough code to continue their research, and even if they did care about proper architecture, they don't have the time or interest in learning/testing a new library for every little thing. They'd rather be putting that time reading up on their field of research.
    • 9dev 3y ago
      This stance always rubbed me the wrong way a bit. Effectively, code is one of the tools a researcher uses to do their work. As soon as their work interacts with other people, for example when publishing a purportedly reproducible study or supplying novel algorithms to developers, they have a responsibility to deliver proper work that can be used and understood by other people. This is something we expect of every other profession, yet scientists appear to somehow have no concern for such lowly ambitions. To be clear, I’m not advocating for data scientists to write production-grade webapps. But I absolutely think they should be bothered to write code that fulfills minimal requirements, is reproducible, documented, and mostly bug-free.
      • 3y ago
• threatripper 3y ago
  You have a single big data structure that can't be shared easily between multiple processes. Can't you use multiprocessing with that? Maybe mapping the data structure to a file and mmapping that in multiple processes? Maybe wrapping the whole thing in database instead of just using one huge nested dictionary? To me multi-threading sounds so much less painful than all the alternatives that I could imagine. Just adding multi-threading could give you >10x improvement on current hardware without much extra work if your data structure plays nice.
  • TylerE 3y ago
    I'd go for a db, yeah, or if that's a really painful mapping, this, erm, is actually the sort of thing Go is pretty good at it, and it's not too hard to write a fairly simple program that will traverse your data structure and communicate via a JSON api or something. That's a useful technique in general - separate the big heavy awkward thing from your main web processes. While I hate how verbose and inexpressive it is, Go does hit a sweet spot of fairly good performance, even multi-core, while still being GCed so it's not nearly as foreign for a native python user.
  • dathinab 3y ago
    > You have a single big data structure that can't be shared easily between multiple processes. Can't you use multiprocessing with that? Maybe mapping the data structure to a file and mmapping that in multiple processes? Maybe wrapping the whole thing in database instead of just using one huge nested dictionary? ton of additional complexity, not worth it for many use-cases and anything on the line of "using multiple processes or threads to increase python performance" does have (or at least did have) quite a bunch of additional foot guns in python In that context porting a very trivial ad-hoc application to Java (or C# or Rust, depending on what knowhow exist in the Team) would faster or at least not much slower to do. But it would be reliable estimable by reducing the chance for any unexpected issues, like less perf then expected. Basically the moment "use mmap" or "use multi-processing" is a reasonable recommendation for something ad-hocish there is something rally wrong with the tools you use IMHO.
    • nine_k 3y ago
• brightball 3y ago
  This is actually one of the reasons I was drawn to Ruby over Python. Ruby also has the GIL but jRuby is an excellent option when needed.
  • antod 3y ago
    I wonder what lead to JRuby attracting support while Jython not? I know the Jython creator went on to other things (was it eg IronPython for dotnet?). I suppose it was the inverse with dotnet - eg IronPython surviving while IronRuby seems dead. Is it just down to corporate sponsorship?
    • empthought 3y ago
      Twitter used JRuby and invested heavily for a time.
    • brightball 3y ago
      JRuby has been pretty actively maintained for about 15 years and had a big release this year. It’s an impressive project.
      • krylon 3y ago
        I looked into it a long time ago (~10-12 years?), and was disappointed JRuby could not use extensions written in C. It's not surprising in retrospect, for obvious reasons, but has there been some progress in this area?
• wood_spirit 3y ago
  That’s right. In the past, for read-only data, I’ve used a disk file and relied on the the OS page cache to keep it performant. For read-write, using a raw file safely gets risky quickly. And alternative languages with parallelism runs rings around python. So getting rid of the GIL and allowing parallelism will be a big boon.
• rrishi 3y ago
  I am not too deeply experienced with Python so forgive my ignorance. But I am curious to understand why you were not able to utilize the concurrency tools provided in Python. A quick google search gave me these relevant resources 1. An intro to threading in Python (https://realpython.com/intro-to-python-threading/#conclusion-threading-in-python https://realpython.com/intro-to-python-threading/#conclusion...) 2. Speed Up Your Python Program With Concurrency (https://realpython.com/python-concurrency/ https://realpython.com/python-concurrency/) 3. Async IO in Python: A Complete Walkthrough (https://realpython.com/async-io-python/ https://realpython.com/async-io-python/) Forgive me for my naivety. This topic has been bothering me for quite a while. Several people complain about the lack of threading in Python but I run into plenty of blogs and books on concurrency in Python. Clearly there is a lack in my understanding of things.
  • aardvark179 3y ago
    Threading in Python is fine if your threads are io bound or spend their time in a C extension which releases the GIL, if you are bound then the GIL means effectively one thread can run at a time and you gain no advantage from multiple threads.
  • teraflop 3y ago
    The whole point of the GIL is that even if you use Python's threading or asyncio, you don't get any benefits from scaling beyond a single CPU core, because all of your threads (or coroutines) are competing for a single lock. They run "concurrently", but not actually in parallel. The pages you linked explain this in more detail. In theory, multiprocessing could allow you to distribute the workload, but in a situation like OP describes -- just serving API requests based on a data structure -- the overhead of dispatching requests would likely be bigger than the cost of just handling the request in the first place. And your main server process is still a bottleneck for actually parsing the incoming requests and sending responses. So you're unlikely to see a significant benefit.
  • deleted 3y ago
    [deleted]
  • wmwmwm 3y ago
• nwallin 3y ago
  > I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module. If you've already written it with the multithreading module, it is a drop in replacement. Your data structure will live in shared memory and can be accessed by all processes concurrently without incurring the wrath of the GIL. Obviously this does not fix the issue of Python just being super slow in general. It just lets you max out all your CPU cores instead of having just one core at 100% all the time.
  • AlphaSite 3y ago
    Python is also going to get a JIT eventually, so they’re fixing that too! One of the concerns with no gil was that it would make certain optimisations harder for the JIT, but it’s very cool to see both being worked on.
  • nine_k 3y ago
    Multiprocessing is great. But then every process keeps its own copy of hundreds of gigabytes of stuff. May be okay, depending on how many processes you spawn. If the bulk of the data is immutable (or at least never mutated), it can be safely shared though, via shared memory.
    • zbentley 3y ago
      > every process keeps its own copy of hundreds of gigabytes of stuff. May be okay, depending on how many processes you spawn That depends on how you're using multiprocessing. If you're using the "spawn" multiprocessing-start method (which was set to the default on MacOS a few years ago[1], unfortunately), then every process re-starts python from the beginning of your program and does indeed have its own copy of anything not explicitly shared. However, the "fork" and "forkserver" start methods make everything available in python before your multiprocessing.Pool/Process/concurrent.futures.ProcessPoolExecutor was created accessible for "free" (really: via fork(2)'s copy-on-write semantics) in the child processes without any added memory overhead. "fork" is the default startup mode on everything other than MacOS/Windows[2]. I find that those differing defaults are responsible for a lot of FUD around memory management regarding multiprocessing (some of which can be found in these comments!); folks who are watching memory while using multiprocessing on MacOS or Windows observe massively different memory consumption behavior than folks on Linux/BSD (which includes folks validating in Docker on MacOS/Windows). There's an additional source of FUD among folks who used Python on MacOS before the default was changed from "fork" to "spawn" and who assume the prior behavior still exists when it does not. This sometimes results in the humorously counterintuitive situation of someone testing some Python code in Docker on MacOS/Windows observing far better performance inside Docker (and its accompanying virtual machine) than they observe when running that same code natively directly on the host operating system. If you're on MacOS (not Windows) and wish to use the "fork" or "forkserver" behaviors of multiprocessing for memory sharing, do "export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES" in your shell before starting Python (modifying os.environ or calling os.setenv() in Python will not work), and then call "multiprocessing.set_start_method("fork", force=True)" in your entry point. Per the linked GitHub issue below, this can occasionally cause issues, but in my experience it does so rarely if ever. 1. https://github.com/python/cpython/issues/77906 https://github.com/python/cpython/issues/77906 2. https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods https://docs.python.org/3/library/multiprocessing.html#conte...
• strictfp 3y ago
  My tip for this is Node.js and some stream processing lib like Highland. You can get ridiculous IO parallelism with a very little code and a nice API. Python just scales terribly, no matter if you use multi-process or not. Java can get pretty good perf, but you'll need some libs or quite a bit of code to get nonblocking IO sending working well, or you're going to eat huge amounts of resources for moderate returns. Node really excels at this use case. You can saturate the lines pretty easily.
  • pid-1 3y ago
    Isn't Node single threaded, just like Python?
    • krylon 3y ago
      Python is technically multithreaded, but the GIL means only one thread can execute interpreter code at a time. If you use libraries written in C/C++, the library code can run in multiple threads simultaneously if they release the GIL. I vaguely recall Node used to run multiple threads under the hood for disk I/O, but it might use kqueue/epoll these days.
    • strictfp 3y ago
      Node is essentially a single-threaded API to a very capable multithreaded engine. https://youtu.be/ztspvPYybIY https://youtu.be/ztspvPYybIY
  • goatlover 3y ago
    Wouldn't Elixir or Go be better for this use case? Node still blocks on compute heavy tasks.
  • hughesjj 3y ago
    0_o Did I miss something? Does nodes/highland have good shared memory semantics these days? I've always felt the best analogy to python concurrency was (node)js, but I admittedly haven't kept up all that well.
  • porridgeraisin 3y ago
• qbasic_forever 3y ago
  Are you just reading from this data structure? If so I wouldn't do any locking or threading, I'd just use asyncio to serve up read requests to the data and it should scale quite well. Multithreading/processing is best for CPU limited workloads but this sounds like you're really just IO-bound (limited by the very high IO of reading from that data structure in memory). If you're allowing writes to the shared data structure... I'd ask myself am I using the right tool for the job. A proper database server like postgres will handle concurrent writers much, much better than you could code up hastily. And it will handle failures, backups, storage, security, configuration, etc. far better than an ad hoc solution.
  • tsimionescu 3y ago
    Reading from memory is really not IO. Perhaps you're suggesting doing something like mmapping a file to memory, putting the data structure in that memory, and then using asyncio on the file to serve things, but this would only work if you can compute byte ranges inside the file to serve ahead of time, in which case there are much simpler solutions anyway. Most likely, when receiving a query they need to actually search through the datastructure based on the query, and it's very likely that this is the bottleneck, not just reading some memory.
  • Jtsummers 3y ago
    > I'd just use asyncio to serve up read requests to the data and it should scale quite well. Quoting GP: >> often CPU heavy We have to take their word for it that it's actually CPU heavy work, but if they're not lying and not mistaken then asyncio would do nothing for them.
• deleted 3y ago
  [deleted]
• TylerE 3y ago
  Spin up as many processes as you need, map connections 1:1 to processes if possible.