6 ms·

The article doesn't mention a hidden gem in Python's standard library - typed arrays! Those are in package `array` and they are a very barebones version of num

• vbezhenar 7y ago
  That's how you deal with that problem with Java as well: arrays of primitives. If you need to operate on million of points, you can either use Point[] which will incur something like 16 MB additional memory or use two int[] arrays which won't incur any extra overhead short of few bytes. Your code won't be pretty, but it'll be fast and you can always hide this weirdness behind pretty API.
  • 0815test 7y ago
    > That's how you deal with that problem with Java as well: arrays of primitives. It just goes to prove that you can indeed write FORTRAN in any language!
    • llukas 7y ago
      This is actually a feature - python array slicing syntax is very similar to fortran.
  • cesarb 7y ago
    Also known as the "Struct of Arrays" approach.
    • wtallis 7y ago
      "Struct of Arrays", to be contrasted with "Array of Structs", because in Java it's actually "Array of pointers to Struct". In languages where "Array of Structs" is actually possible, the decision of which to use is less clear-cut and depends on how big the Struct is, what the access patterns are like, and whether you're trying to perform SIMD operations on multiple Structs at once.
      • vbezhenar 7y ago
        That's not only about pointers. Every Java object incurs an overhead of around 16 bytes (architecture and JVM dependent, of course, but it's there).
    • corysama 7y ago
      Several times in Python I've used the built-in CTypes library to cut down the memory footprint by using giant Arrays of (literally C) Structs.
      • danudey 7y ago
        I was bored and had time on my hands, so I played around a bit with some of the suggestions in this thread, but it seems as though a naive ctypes approach is the worst possible approach in the two cases I tested (arrays of integers, arrays of x,y struct). Here are my code and results: https://gist.github.com/danudey/94b5442b617a734a2366e824be159beb https://gist.github.com/danudey/94b5442b617a734a2366e824be15... Is there an approach that you use that produces better results than this? Because my naive approach is unmanageably worse for both performance (not recorded) and impact on Python's memory usage.
  • nullwasamistake 7y ago
    Or you can just use a more memory efficient language like Java. Python and JS use tons of memory compared to strongly typed VM languages like C#, Go, Java. People lament Java memory usage but Python/Node use several times more
    • whatshisface 7y ago
      >People lament Java memory usage but Python/Node use several times more The people who lament Java's memory usage never complain about Python or Ruby because... if you're worried about the JVM you would never consider Python!
      • nullwasamistake 7y ago
        I'm mostly referring to those that say the JVM is too "resource heavy" then go on to use JS/Python/Ruby instead, which have much higher runtime overhead. In my experience many dev think the scripting language heritage means these languages have smaller/lighter runtimes
        • drej 7y ago
          Not that I want to go down the rabbit hole, but I probably fall into the category of “JVM is heavy, but I use Python” Sure, memory consumption and CPU perf are both pretty bad in Python, but latency and memory footprint of the runtime itself are pretty good, so it’s ideal for tooling, crons, lambdas etc. JVM is comparatively efficient once you have the JVM ready, but that sure takes tons of resources. I’m hoping Graal changes this. I don’t like JVM-based languages, but I do like technological progress.
          • alexhutcheson 7y ago
            Python startup time can be significant for non-trivial programs as well, and it's been a big problem for Mercurial and other projects. For example, see these threads: - https://mail.python.org/pipermail/python-dev/2014-May/134528.html https://mail.python.org/pipermail/python-dev/2014-May/134528... - https://mail.python.org/pipermail/python-dev/2018-May/153296.html https://mail.python.org/pipermail/python-dev/2018-May/153296... If your script is expected to be run interactively on a frequent basis, then Go, Rust, C++, or even Bash (for simple stuff) will give you much lower user-perceived latency than Python.
  • rcfox 7y ago
    It will depend on your application, but you might also consider just one array of [x, y, x, y, x, y, ...] to improve cache locality.
  • adrianratnapala 7y ago
    > Your code won't be pretty, Depends on the algorithms that use that code. Two separate arrays is not a bad way to represent a vector-of-tuples. And so if you think of everything as a vectorised operation (and if that makes sense for your use case) then the code can be clean enough. Efficiency-wise it depends on how the locality of reference falls out. There are cases where it's more cache-efficient to store the pairs next to each other, but again for big vectorised operations you might lose nothing by using two cache-ways instead of one.
• nudpiedo 7y ago
  Until you end up rediscovering the information storage skills used by databases and lower level programs: buffers with fixed sizes and implicit type associations. I would say that Python is the right tool for some things, not for others. Why not use Go, or Nim or C to store/access/return these data in a data structure that transforms the values from/to python?
  • Seirdy 7y ago
    That's what C-extensions are for :P Python is a great high-level wrapper around low-level programs (c.f. SciPy)