4 ms·

I'd wish people stop post all crappy blogposts about shallow language-comparisons. Saying language X does Y in 4 lines instead of language Zs 5 lines just does

I'd wish people stop post all crappy blogposts about shallow language-comparisons. Saying language X does Y in 4 lines instead of language Zs 5 lines just doesn't say much unless you put it in a bigger perspective. (And I like Python very much and hate Java).

atl2008 18y ago
So write something that gives that bigger perspective.
fauigerzigerk 18y ago
That's true, and I also like Python more than Java. What's missing in these comparisons are some important architectural consequences of using Python (or ruby or PHP) for web apps. The one reason why I haven't been using Python as much as I like is the GIL (global interpreter lock). The GIL requires that you use multiple processes to make use of multiple CPUs or cores. And that means you cannot keep much data in memory as it would be duplicated in each process. Some applications benefit a lot from keeping much of the data in memory most of the time, and if you have complex data structures (like graphs) memcached (or similar solutions) is no replacement as you have to rebuild those data structures on each access. Any solution that invalidates process relative adresses incurs a huge overhead in order to access that data. I know the debate around threads and how problematic they are and how much better a multi process model scales. That's all very well, but threads are the only way to use multiple cores and access in memory data structures fast. So using Python, for me, means having to write much of my application in C++ (or Java or C#) and access that backend process from the Python web frontend. This approach comes with a lot of complexity and it's a lot more work than using Java or C# in the first place. On the other hand, keeping a lot of data in memory isn't necessary for most applications as the data structures and the parallelism lives in the DBMS, which is written in C/C++ anyway. It's also questionable whether Java is a good solution for my scenario. Java and C# need about twice as much memory as C++ for the exact same data structures, and the garbage collector gets slower the more memory a process allocates. I know there are more options but this post is getting lengthy :-)
- anewaccountname 18y ago
  If the data structures are static, you can build them, then fork off a bunch of python processes. They will all have the data, but it will only be in memory once. Just make sure to never invoke GC after that point... (unless python's GC isn't compacting). Also make sure to never reference anything, or you will increment it's reference count and trigger copy-on-write...
  - fauigerzigerk 18y ago
    The data structures are not static. It's effectively an in memory database, so the data keeps changing.
- anthonyrubin 18y ago
  PostgreSQL is not multi-threaded and uses System V IPC. Threads are certainly not the only option.
  - fauigerzigerk 18y ago
    So how would you do it? I have a large graph on which a number of clients run queries, complex analyses and updates at the same time. I've been pondering various solutions like shared memory, memory mapped files, relying on the disk cache, memcached, etc. But they're all either not performing well because they cause a lot of serialisation in and out of process memory, or they cause overly complex designs or both. I do appreciate any advice I can get... [edit] Traditional DBMS architectures like that of postgres ,to my knowledge, rely on shared memory or memory mapped files (I'm sure this varies). They swap on disk pages in and out of memory and those pages retain their on-disk structure. There is little if any translation into in-memory data structures. Yes I could do that, but it's incredibly complex. Every single data structure basically has to be implemented on top of a byte array. I wouldn't be able to use any existing libraries for trees, lists, sets, hashtables, etc.
    - ii 18y ago
      Try ZODB/ZEO, you will be able to access native python data structures (possibly cached in memory) from multiple python processes. http://www.zope.org/Products/ZEO/ZEOFactSheet http://www.zope.org/Products/ZEO/ZEOFactSheet

atl2008 18y ago

So write something that gives that bigger perspective.

fauigerzigerk 18y ago

That's true, and I also like Python more than Java. What's missing in these comparisons are some important architectural consequences of using Python (or ruby or PHP) for web apps. The one reason why I haven't been using Python as much as I like is the GIL (global interpreter lock). The GIL requires that you use multiple processes to make use of multiple CPUs or cores. And that means you cannot keep much data in memory as it would be duplicated in each process. Some applications benefit a lot from keeping much of the data in memory most of the time, and if you have complex data structures (like graphs) memcached (or similar solutions) is no replacement as you have to rebuild those data structures on each access. Any solution that invalidates process relative adresses incurs a huge overhead in order to access that data. I know the debate around threads and how problematic they are and how much better a multi process model scales. That's all very well, but threads are the only way to use multiple cores and access in memory data structures fast. So using Python, for me, means having to write much of my application in C++ (or Java or C#) and access that backend process from the Python web frontend. This approach comes with a lot of complexity and it's a lot more work than using Java or C# in the first place. On the other hand, keeping a lot of data in memory isn't necessary for most applications as the data structures and the parallelism lives in the DBMS, which is written in C/C++ anyway. It's also questionable whether Java is a good solution for my scenario. Java and C# need about twice as much memory as C++ for the exact same data structures, and the garbage collector gets slower the more memory a process allocates. I know there are more options but this post is getting lengthy :-)

anewaccountname 18y ago
If the data structures are static, you can build them, then fork off a bunch of python processes. They will all have the data, but it will only be in memory once. Just make sure to never invoke GC after that point... (unless python's GC isn't compacting). Also make sure to never reference anything, or you will increment it's reference count and trigger copy-on-write...
- fauigerzigerk 18y ago
  The data structures are not static. It's effectively an in memory database, so the data keeps changing.
anthonyrubin 18y ago
PostgreSQL is not multi-threaded and uses System V IPC. Threads are certainly not the only option.
- fauigerzigerk 18y ago
  So how would you do it? I have a large graph on which a number of clients run queries, complex analyses and updates at the same time. I've been pondering various solutions like shared memory, memory mapped files, relying on the disk cache, memcached, etc. But they're all either not performing well because they cause a lot of serialisation in and out of process memory, or they cause overly complex designs or both. I do appreciate any advice I can get... [edit] Traditional DBMS architectures like that of postgres ,to my knowledge, rely on shared memory or memory mapped files (I'm sure this varies). They swap on disk pages in and out of memory and those pages retain their on-disk structure. There is little if any translation into in-memory data structures. Yes I could do that, but it's incredibly complex. Every single data structure basically has to be implemented on top of a byte array. I wouldn't be able to use any existing libraries for trees, lists, sets, hashtables, etc.
  - ii 18y ago
    Try ZODB/ZEO, you will be able to access native python data structures (possibly cached in memory) from multiple python processes. http://www.zope.org/Products/ZEO/ZEOFactSheet http://www.zope.org/Products/ZEO/ZEOFactSheet