10 ms·

How big are PHP arrays (and values) really?

150 points by there 15y ago

maratd 15y ago
As the author mentions, if you run into a use-case where you need to store 100000 integers in memory, then you should use one of the many alternative structures available. Some of them were explicitly designed to store integers in an efficient manner. Arrays weren't designed to store integers efficiently or anything else for that matter. They were designed to be fast and easy to use.
- mbell 15y ago
  I wouldn't call using 56 bytes to store 8 bytes of actual data efficient.
  - dangrossman 15y ago
    That'd be why he said "arrays weren't designed to store integers efficiently".
    - ktr 15y ago
      From the comment: > then you should use one of the many alternative structures available. Some of them were explicitly designed to store integers in an efficient manner. From the article: > But if you do want to save memory you could consider using an SplFixedArray for large, static arrays. ... It basically does the same thing, but if you run it, you’ll notice that it uses “only” 5600640 bytes. That’s 56 bytes per element ... EDIT: formatting.
      - maratd 15y ago
        That is only one of the alternatives and in my opinion, not a very good one. I forget the exact details, but there is an extension by the guy who wrote igbinary that is specifically designed for this use-case.
- skeptical 15y ago
  Considering PHP's scope and its limitations (lacks things like threads for example) I would say that is an extremely rare scenario. If you're storing 100.000 integers on a PHP data structure you're most likely doing it wrong.
  - maratd 15y ago
    > Considering PHP's scope and its limitations (lacks things like threads for example) Please look up PHP-FPM. I use it in production, with great success. > If you're storing 100.000 integers on a PHP data structure you're most likely doing it wrong. What if I had a ton of price points and I needed to do statistical analysis? Well, those would be floats, but you get my point. PHP is capable of doing the work efficiently, you just need to put a little thought in it first or you'll very quickly trash your box.
    - epochwolf 15y ago
      Isn't PHP-FPM a fastcgi process manager? I don't see any references to threading in it's documentation. I also recall the php documentation saying php is unsafe with multiple threads due to a large number of libraries that are not coded to be thread-safe.
      - maratd 15y ago
        > Isn't PHP-FPM a fastcgi process manager It is. > I don't see any references to threading Wow. I always thought it was threaded. My mistake. It seems it launches multiple child processes. There is some form of memory sharing going on, though. Looks like it's some sort of a hybrid. Now I'm just thoroughly confused ...
        Gigablah 15y ago
        You can take a look at php-pcntl. http://www.php.net/manual/en/book.pcntl.php http://www.php.net/manual/en/book.pcntl.php http://www.php.net/manual/en/function.pcntl-fork.php http://www.php.net/manual/en/function.pcntl-fork.php
rorrr 15y ago
PHP arrays are not really arrays, they are sort of hash-maps. You can do things like $arr = array(1 => 10, "1" => 11); Or even $arr = array('他妈的我的生活' => 5); But at the same time you can treat them as regular zero-based arrays. $arr = array(); $arr[] = 1; $arr[] = 2; $arr[] = 3; $arr[] = 'dog';
- fhars 15y ago
  And once you relalize that they are really hashes you also realize that a fast and memory efficient representation will use at least nine machine words or 72 bytes per value: 1 + x for the pointer in the bucket array (you don't want 100% occupancy to avoid hash collisions), 2 for the cons cell of the list stored in the bucket (the second element of which will usually be NULL if you have a good hash function and low occupancy), 2 for the key-value pair pointed to by the first element of the cons cell representing the index and the value, and two each for the integer key and value plus their tag words). And you may also want to store some metadata (like the length of the list stored in each bucket) with the hash, so the PHP array in the example is within a factor of two from the naive optimum for a dynamically typed language without specialized arrays. Its just the price you have to pay for not caring about the type of your "array" keys.
  - obtu 15y ago
    That or specialize. Lua, Javascript, and PyPy all have more efficient storage of integer-keyed hashes. Lua's solution is particularly simple and could easily be transferred to PHP.
    - lysium 15y ago
      In PHP you can mix the types of the key, so that won't work.
      - lysium 15y ago
        In another thread, dchest points out http://news.ycombinator.com/item?id=3360369 http://news.ycombinator.com/item?id=3360369 that Lua 5.0 allows mixed-typed keys, too, so that may work.
nestlequ1k 15y ago
Interesting. Wonder how this compares to python and ruby memory handling.
- InfernalH 15y ago
  I tried with a "pure" Array - for Ruby it's 1Mb for 100 000 elements EDIT: and for Hash with h[i] = i, it's ~6Mb
  - masklinn 15y ago
    Does that account for the size taken by the integers themselves?
    - jeltz 15y ago
      I assume so. If the integers fit in 31 or 63 bits, depending on your architecture, they are inlined in the pointer.
      - masklinn 15y ago
        OK (because that's not the case in CPython, I think)
        jeltz 15y ago
        Yeah, I too am pretty sure that CPython does not inline integers in the pointers. And from a quick glance at the source code I saw nothing such. Inlining integers in pointers by shifting up and adding 1 is a quite common trick though and I have seen it in more programming language implementations than MRI. I think at least some Prolog implementation and older versions of Spidermonkey (newer versions use a similar trick with doubles).
        masklinn 15y ago
        > Inlining integers in pointers by shifting up and adding 1 is a quite common trick though and I have seen it in more programming language implementations than MRI. Yeah, I know about it, I just did not think MRI had bothered with it anymore than CPython.
baby 15y ago
I'm kinda hijacking this thread because I always wonder how to handle vars in PHP. Should we use short named vars like $a, $b? Should we avoid always using the same var and changing its type? $a = 30; $a = "thing";
- ars 15y ago
  Someone downmodded you for being offtopic, but I'll answer you anyway. The name of the var doesn't matter at all. And you can change the type of a var at will. There is nothing at all in PHP that will be better if you avoid changing the type, so just do what is clearest for your program.
- conradfr 15y ago
  I don't think someone picking up your code will be pleased with all variables named $a :) Use "long" names and use unset() if worried about memory consumption. And more on topic, you can use unset on a array item !
pacmon 15y ago
Did anyone else even try running his suggested code? I think maybe he has a problem with his setup. I don't get those numbers. My numbers for PHP 5.3.8: Windows 7 - 8524568 bytes (using range) 3600584 bytes (using SplFixedArray) Fedora 14 - 7724600 bytes (using range) 3200568 bytes (using SplFixedArray) *edit - Added numbers for SplFixedArray
- Robin_Message 15y ago
  Are you on 32-bit, because all his numbers are for 64-bit, except the large table near the beginning, which matches roughly with the numbers you give.
  - pacmon 15y ago
    Yes and No. I run a 64-bit Windows 7 machine, but oddly enough the Fedora 14 machine is 32 bit. Yet it yields similar numbers to the 64-bit Windows.
    - dangrossman 15y ago
      Sounds like you're running a 32-bit interpreter on your 64-bit machine.
    - Klinky 15y ago
      Are you running a 64-bit build of PHP though?
- nikic 15y ago
  Yep, I already got some comments on that. You are either using a 32 bit system or a 32 bit binary (at least I think that the binaries PHP distributes for Windows are compiled for 32 bit, so even if you are on a 64 bit Windows you'll still get 32 bit numbers). The Windows number still is 8 bytes per element larger than the number I wrote (76 per element). This might have various reasons, one could be that it was compiled with head protection :)
  - 15y ago
wvenable 15y ago
Has anyone run similar tests on the soon-to-be-released PHP 5.4? From what I understand, one of the changes in that release is reduced memory consumption. Andi Gutmans has said that PHP 5.4 could lower PHP's memory footprint by as much as 35%.
- nikic 15y ago
  I actually did most of my tests with PHP 5.4 and trunk binaries, but also tested PHP 5.3 and the numbers didn't change. PHP 5.2 used 8 bytes less, because the circular GC was introduced only in PHP 5.3. By the way, you can test that yourself too. The codepad I posted the same on has a switch for PHP 5.2, 5.3 and 5.4, so you can easily see for yourself :)
lang 15y ago
php is not unique in having big memory footprint. AFAIC Python and Ruby are also memory hogs. An interesting question is how memory efficient a dynamic PL can be. Given that in modern computers memory access (cache misses) is fairly expensive it probably makes sense to trade instructions for memory.
- masklinn 15y ago
  > AFAIC Python and Ruby are also memory hogs. Even more so in some areas, for instance a Python `int` is not a machine integer but a full-blown object.
  - lvh 15y ago
    It's only always a full-blown object in CPython. Smarter implementations, notably PyPy, will do escape analysis, and never actually end up allocating those objects. (My point is that not having unboxed types does not imply being a memory hog. You just need a smarter implementation.)
    - masklinn 15y ago
      > Smarter implementations, notably PyPy, will do escape analysis, and never actually end up allocating those objects. Objects in a collection (which is what we're talking about here) escape kind-of by default. Until type-specialized collections are merged in PyPy (if they are not yet), it'll have the same issue as CPython.
legooolas 15y ago
> A union is a means to make some value accessible as various types. For example if you do a zvalue_value->lval you’ll get the value interpreted as an integer. If you use zvalue_value->ht on the other hand the value will be interpreted as a pointer to a hashtable (aka array). This is not valid C usage of unions. They are _only_ for use as a method to save space, not for conversion between types, despite it being a very common usage of unions. This can cause all manner of problems when compiler optimizations such as type-based alias analysis are used. EDIT: Turns out I'm completely wrong on this and it's fine from C99 onwards.
- jronkone 15y ago
  I don't see how unions could save space, care to explain?
  - legooolas 15y ago
    If you have multiple items of which you're only going to store one at a time but want them in the same "structure" then you use a union. (i.e. temporally disjoint)
- chadaustin 15y ago
  Actually, the upcoming C1X standard makes type punning via union legal. In C99, alias analysis works behaves exactly as you said, but using unions to convert between floats and ints is so common that it's legal in C1X, VC++, and I believe gcc.
  - legooolas 15y ago
    gcc does in some cases but not all (see http://gcc.gnu.org/onlinedocs/gcc-4.1.1/gcc/Optimize-Options.html#index-fstrict_002daliasing-542 http://gcc.gnu.org/onlinedocs/gcc-4.1.1/gcc/Optimize-Options... but not all). Still, the current standard says it's not legal so it seems like a bad idea to rely on undefined behaviour...
  - cygx 15y ago
    Type-punning through unions is already legal in C99, but there's a known error in Annex J, listing it incorrectly as unspecified behaviour. See http://stackoverflow.com/a/8513748/48015 http://stackoverflow.com/a/8513748/48015
noselasd 15y ago
What does memory_get_usage() actually do ? Does it report the "heap" size assigned to the process, or does it use PHP internal counters for the allocated user data/variables ? A C malloc subsystem will assign a whole lot of virtual memory, in steps of pages, or more if it decides to attach a piece using mmap(). In order to make this test case relevant, I'd say one have to know what memory_get_usage() does - it's at least meaningless to determine the overhead of an array based on it, if for whatever reason creating the 1. PHP array in a program also initializes "big" memory pools that count towards the memory usage.
- ezyang 15y ago
  This is easy to check in the source. memory_get_usage() calls zend_memory_usage(), which accesses the size field on a global structure mm_heap, which is updated by PHP's memory allocation system (e.g. if you call *_zend_mm_alloc_int)
  - hardtke 15y ago
    I was debugging a php memory issue yesterday, and noticed that at some point my get_memeory_usage() value became much, much smaller than the memory footprint recorded by top (20 MB in get_memory_usage(), 500 MB in top RES). It was a sudden jump during a loop execution according to top. Does top give accurate memory estimates for php scripts?
iampims 15y ago
A coworker just pointed me to: http://us.php.net/SplFixedArray http://us.php.net/SplFixedArray

150 points by there 15y ago

As the author mentions, if you run into a use-case where you need to store 100000 integers in memory, then you should use one of the many alternative structures available. Some of them were explicitly designed to store integers in an efficient manner. Arrays weren't designed to store integers efficiently or anything else for that matter. They were designed to be fast and easy to use.

mbell 15y ago
I wouldn't call using 56 bytes to store 8 bytes of actual data efficient.
- dangrossman 15y ago
  That'd be why he said "arrays weren't designed to store integers efficiently".
  - ktr 15y ago
    From the comment: > then you should use one of the many alternative structures available. Some of them were explicitly designed to store integers in an efficient manner. From the article: > But if you do want to save memory you could consider using an SplFixedArray for large, static arrays. ... It basically does the same thing, but if you run it, you’ll notice that it uses “only” 5600640 bytes. That’s 56 bytes per element ... EDIT: formatting.
    - maratd 15y ago
      That is only one of the alternatives and in my opinion, not a very good one. I forget the exact details, but there is an extension by the guy who wrote igbinary that is specifically designed for this use-case.
skeptical 15y ago
Considering PHP's scope and its limitations (lacks things like threads for example) I would say that is an extremely rare scenario. If you're storing 100.000 integers on a PHP data structure you're most likely doing it wrong.
- maratd 15y ago
  > Considering PHP's scope and its limitations (lacks things like threads for example) Please look up PHP-FPM. I use it in production, with great success. > If you're storing 100.000 integers on a PHP data structure you're most likely doing it wrong. What if I had a ton of price points and I needed to do statistical analysis? Well, those would be floats, but you get my point. PHP is capable of doing the work efficiently, you just need to put a little thought in it first or you'll very quickly trash your box.
  - epochwolf 15y ago
    Isn't PHP-FPM a fastcgi process manager? I don't see any references to threading in it's documentation. I also recall the php documentation saying php is unsafe with multiple threads due to a large number of libraries that are not coded to be thread-safe.
    - maratd 15y ago
      > Isn't PHP-FPM a fastcgi process manager It is. > I don't see any references to threading Wow. I always thought it was threaded. My mistake. It seems it launches multiple child processes. There is some form of memory sharing going on, though. Looks like it's some sort of a hybrid. Now I'm just thoroughly confused ...
      - Gigablah 15y ago
        You can take a look at php-pcntl. http://www.php.net/manual/en/book.pcntl.php http://www.php.net/manual/en/book.pcntl.php http://www.php.net/manual/en/function.pcntl-fork.php http://www.php.net/manual/en/function.pcntl-fork.php

rorrr 15y ago

PHP arrays are not really arrays, they are sort of hash-maps. You can do things like $arr = array(1 => 10, "1" => 11); Or even $arr = array('他妈的我的生活' => 5); But at the same time you can treat them as regular zero-based arrays. $arr = array(); $arr[] = 1; $arr[] = 2; $arr[] = 3; $arr[] = 'dog';

fhars 15y ago
And once you relalize that they are really hashes you also realize that a fast and memory efficient representation will use at least nine machine words or 72 bytes per value: 1 + x for the pointer in the bucket array (you don't want 100% occupancy to avoid hash collisions), 2 for the cons cell of the list stored in the bucket (the second element of which will usually be NULL if you have a good hash function and low occupancy), 2 for the key-value pair pointed to by the first element of the cons cell representing the index and the value, and two each for the integer key and value plus their tag words). And you may also want to store some metadata (like the length of the list stored in each bucket) with the hash, so the PHP array in the example is within a factor of two from the naive optimum for a dynamically typed language without specialized arrays. Its just the price you have to pay for not caring about the type of your "array" keys.
- obtu 15y ago
  That or specialize. Lua, Javascript, and PyPy all have more efficient storage of integer-keyed hashes. Lua's solution is particularly simple and could easily be transferred to PHP.
  - lysium 15y ago
    In PHP you can mix the types of the key, so that won't work.
    - lysium 15y ago
      In another thread, dchest points out http://news.ycombinator.com/item?id=3360369 http://news.ycombinator.com/item?id=3360369 that Lua 5.0 allows mixed-typed keys, too, so that may work.

nestlequ1k 15y ago

Interesting. Wonder how this compares to python and ruby memory handling.

InfernalH 15y ago
I tried with a "pure" Array - for Ruby it's 1Mb for 100 000 elements EDIT: and for Hash with h[i] = i, it's ~6Mb
- masklinn 15y ago
  Does that account for the size taken by the integers themselves?
  - jeltz 15y ago
    I assume so. If the integers fit in 31 or 63 bits, depending on your architecture, they are inlined in the pointer.
    - masklinn 15y ago
      OK (because that's not the case in CPython, I think)
      - jeltz 15y ago
        Yeah, I too am pretty sure that CPython does not inline integers in the pointers. And from a quick glance at the source code I saw nothing such. Inlining integers in pointers by shifting up and adding 1 is a quite common trick though and I have seen it in more programming language implementations than MRI. I think at least some Prolog implementation and older versions of Spidermonkey (newer versions use a similar trick with doubles).
        masklinn 15y ago
        > Inlining integers in pointers by shifting up and adding 1 is a quite common trick though and I have seen it in more programming language implementations than MRI. Yeah, I know about it, I just did not think MRI had bothered with it anymore than CPython.

baby 15y ago

I'm kinda hijacking this thread because I always wonder how to handle vars in PHP. Should we use short named vars like $a, $b? Should we avoid always using the same var and changing its type? $a = 30; $a = "thing";

ars 15y ago
Someone downmodded you for being offtopic, but I'll answer you anyway. The name of the var doesn't matter at all. And you can change the type of a var at will. There is nothing at all in PHP that will be better if you avoid changing the type, so just do what is clearest for your program.
conradfr 15y ago
I don't think someone picking up your code will be pleased with all variables named $a :) Use "long" names and use unset() if worried about memory consumption. And more on topic, you can use unset on a array item !

pacmon 15y ago

Did anyone else even try running his suggested code? I think maybe he has a problem with his setup. I don't get those numbers. My numbers for PHP 5.3.8: Windows 7 - 8524568 bytes (using range) 3600584 bytes (using SplFixedArray) Fedora 14 - 7724600 bytes (using range) 3200568 bytes (using SplFixedArray) *edit - Added numbers for SplFixedArray

Robin_Message 15y ago
Are you on 32-bit, because all his numbers are for 64-bit, except the large table near the beginning, which matches roughly with the numbers you give.
- pacmon 15y ago
  Yes and No. I run a 64-bit Windows 7 machine, but oddly enough the Fedora 14 machine is 32 bit. Yet it yields similar numbers to the 64-bit Windows.
  - dangrossman 15y ago
    Sounds like you're running a 32-bit interpreter on your 64-bit machine.
  - Klinky 15y ago
    Are you running a 64-bit build of PHP though?
nikic 15y ago
Yep, I already got some comments on that. You are either using a 32 bit system or a 32 bit binary (at least I think that the binaries PHP distributes for Windows are compiled for 32 bit, so even if you are on a 64 bit Windows you'll still get 32 bit numbers). The Windows number still is 8 bytes per element larger than the number I wrote (76 per element). This might have various reasons, one could be that it was compiled with head protection :)
- 15y ago

wvenable 15y ago

Has anyone run similar tests on the soon-to-be-released PHP 5.4? From what I understand, one of the changes in that release is reduced memory consumption. Andi Gutmans has said that PHP 5.4 could lower PHP's memory footprint by as much as 35%.

nikic 15y ago
I actually did most of my tests with PHP 5.4 and trunk binaries, but also tested PHP 5.3 and the numbers didn't change. PHP 5.2 used 8 bytes less, because the circular GC was introduced only in PHP 5.3. By the way, you can test that yourself too. The codepad I posted the same on has a switch for PHP 5.2, 5.3 and 5.4, so you can easily see for yourself :)

lang 15y ago

php is not unique in having big memory footprint. AFAIC Python and Ruby are also memory hogs. An interesting question is how memory efficient a dynamic PL can be. Given that in modern computers memory access (cache misses) is fairly expensive it probably makes sense to trade instructions for memory.

masklinn 15y ago
> AFAIC Python and Ruby are also memory hogs. Even more so in some areas, for instance a Python `int` is not a machine integer but a full-blown object.
- lvh 15y ago
  It's only always a full-blown object in CPython. Smarter implementations, notably PyPy, will do escape analysis, and never actually end up allocating those objects. (My point is that not having unboxed types does not imply being a memory hog. You just need a smarter implementation.)
  - masklinn 15y ago
    > Smarter implementations, notably PyPy, will do escape analysis, and never actually end up allocating those objects. Objects in a collection (which is what we're talking about here) escape kind-of by default. Until type-specialized collections are merged in PyPy (if they are not yet), it'll have the same issue as CPython.

legooolas 15y ago

> A union is a means to make some value accessible as various types. For example if you do a zvalue_value->lval you’ll get the value interpreted as an integer. If you use zvalue_value->ht on the other hand the value will be interpreted as a pointer to a hashtable (aka array). This is not valid C usage of unions. They are _only_ for use as a method to save space, not for conversion between types, despite it being a very common usage of unions. This can cause all manner of problems when compiler optimizations such as type-based alias analysis are used. EDIT: Turns out I'm completely wrong on this and it's fine from C99 onwards.

jronkone 15y ago
I don't see how unions could save space, care to explain?
- legooolas 15y ago
  If you have multiple items of which you're only going to store one at a time but want them in the same "structure" then you use a union. (i.e. temporally disjoint)
chadaustin 15y ago
Actually, the upcoming C1X standard makes type punning via union legal. In C99, alias analysis works behaves exactly as you said, but using unions to convert between floats and ints is so common that it's legal in C1X, VC++, and I believe gcc.
- legooolas 15y ago
  gcc does in some cases but not all (see http://gcc.gnu.org/onlinedocs/gcc-4.1.1/gcc/Optimize-Options.html#index-fstrict_002daliasing-542 http://gcc.gnu.org/onlinedocs/gcc-4.1.1/gcc/Optimize-Options... but not all). Still, the current standard says it's not legal so it seems like a bad idea to rely on undefined behaviour...
- cygx 15y ago
  Type-punning through unions is already legal in C99, but there's a known error in Annex J, listing it incorrectly as unspecified behaviour. See http://stackoverflow.com/a/8513748/48015 http://stackoverflow.com/a/8513748/48015

noselasd 15y ago

What does memory_get_usage() actually do ? Does it report the "heap" size assigned to the process, or does it use PHP internal counters for the allocated user data/variables ? A C malloc subsystem will assign a whole lot of virtual memory, in steps of pages, or more if it decides to attach a piece using mmap(). In order to make this test case relevant, I'd say one have to know what memory_get_usage() does - it's at least meaningless to determine the overhead of an array based on it, if for whatever reason creating the 1. PHP array in a program also initializes "big" memory pools that count towards the memory usage.

ezyang 15y ago
This is easy to check in the source. memory_get_usage() calls zend_memory_usage(), which accesses the size field on a global structure mm_heap, which is updated by PHP's memory allocation system (e.g. if you call *_zend_mm_alloc_int)
- hardtke 15y ago
  I was debugging a php memory issue yesterday, and noticed that at some point my get_memeory_usage() value became much, much smaller than the memory footprint recorded by top (20 MB in get_memory_usage(), 500 MB in top RES). It was a sudden jump during a loop execution according to top. Does top give accurate memory estimates for php scripts?

iampims 15y ago

A coworker just pointed me to: http://us.php.net/SplFixedArray http://us.php.net/SplFixedArray