9 ms·
Memory use in CPython and MicroPython
- manaskarekar 9y agoOff topic: Micropython on ESP8266 is a blast. 1. Pick yourself up a < $5 ESP8266 board from ebay (Wemos D1 mini or NodeMCU Lolin). 2. On debian based linux: - Get the correct binary from http://micropython.org/download http://micropython.org/download - sudo pip install esptool - esptool.py --port /dev/ttyUSB0 erase_flash - esptool.py --port /dev/ttyUSB0 --baud 460800 write_flash --flash_size=detect 0 /path/to/binary.bin 3. Hack away!
- netheril96 9y agoWhen Python 3 broke compatibility, it would be perfect timing to implement tagged pointers, small integers and other performance improvements. At least it would be an easier sell to your boss to rewrite your code base. "We are doing so it runs faster" is much better than "we are doing it for intellectual purity" from a business standpoint.
- bramblerose 9y agoWhy would "makes your code run 1% faster" be a better sell than "saves you 1% development time by reducing the number of bugs"?
- egwor 9y agoThat tends to be a more difficult sell because the counter-argument from further up is 'But we're not seeing those bugs right now, and won't we find bugs when we do this upgrade?'
- snovv_crash 9y agoIt depends on a) the size of your deployment b) whether you are paying SV salaries to your developers c) how sensitive you are to speed. In many things I'd be happy to double the number of bugs if it meant I could run the code twice as fast. Bugs can be fixed. Slow code is still slow.
- netheril96 9y agoBecause during the upgrade you would have an explosion of bugs and therefore development time fixing them. The 1% saved by purity wouldn't make a dent.
- axaxs 9y agoAgreed. A 'fast' Python, to me, the perfect language. The changes 2 to 3 broke compatibility, but for 'pureness' sake. If speed and the deployment situation were better for python, that would be all I use. I once thought to make a compiler, but don't have the time and honestly knowhow to begin. Nuitka is trying, but also agreeing to live with compatabilty vs performance or type safety. A type safe compiled python is my dream.
- svennek 9y agoPython 3 did not break compatibilty for "pureness" sake. They broke it, because for 6 billion people in the world, for whom English is not the only language, there was a very real pain (bordering on stuff that was impossible). Most of the world is moving to unicode exactly because of this (I know Sillicon Vally companies are probably moving to unicode, so that their users can use cute emojis). Most people also need to handle stuff that are in different encodings (I work daily in both Latin1 (north-western European) and UTF8), translating between that was painful/borderline-impossible in the old days (python 2). The sharp distinction between bytes and strings are essential for any non-english/american/aussie person. Sadly python2 got that wrong, and hence had to die.. Which is probably also why Perl6 choose to go the same route.. If you really need a good reason to go to python 3 other than 6 billion extra potential customers, try "oh look, we have emojis" <sarcasm/>
- detaro 9y agoOne might argue that making unicode strings the default instead of a special datatype (which they were in 2.x) is also a "pureness" decision (since you can handle unicode as much in 2.7 as you can in 3.x)
- prevedmedved 9y agoI'm one of those people for whom English is not the only language. Python 3 is objectively harder and more confusing to use in a multilingual environment. Python 2 didn't pretend that only Unicode exists and Python 2 was encoding-agnostic. This is the correct default for the real world. Ironically, the effect of Python 3 was the opposite: it made life really hard for those that aren't exclusive anglophones and made life a lot easier for American programmers who now don't have to bother with the whole character encoding mess. Personally, Python 3's broken support for internationalisation made me drop Python altogether. Python 2 was confusing, but the defaults were at least workable.
- ubernostrum 9y agoWhen Python 3 broke compatibility, the Python team did exactly the thing you're supposed to do with code: first they made it correct (i.e., got all the changes implemented and working as expected), then made it fast. "We're doing a huge rewrite already, let's pile on more scope" is a recipe for disaster.
- bonzini 9y agoIt depends. It is really surprising that Python does not use tagged integers, which is a 50 year old technique used by pretty much any other dynamic language in existence. Using an extension API breakage as the excuse to introduce them would have made sense, since it also breaks the API. Now they cannot happen until Python 4.
- netheril96 9y agoAnd Guido says he won't break backwards compat again in Python 4 as he has learned the lesson. No tagged integers for life.
- coldtea 9y agoHe learned the wrong lesson then. Which wasn't "don't break backwards compatibility", but: 1) If you break it, break it for compelling reasons. 2) If you provide a tool to convert, don't make it as awful as 2to3
- bonzini 9y agoBreaking language backwards-compatibility is one thing; breaking the extension API is less problematic, and breaking the extension ABI is better still.
- cdavid 9y agoI believe ABC, one of python's influence, used tagged pointers, but I can't find a quote on this. Not using tagged pointers was a conscientious decision from Guido (see e.g. https://github.com/dropbox/pyston/issues/181#issuecomment-62941865 https://github.com/dropbox/pyston/issues/181#issuecomment-62...) [edited] Something that people often overlook when talking about optimizing python is that the "simplicity" of the C Python API allowed the rich ecosystem of C extensions. GC, tagged pointers, etc... are great, and often used in modern VMs, but they tend to make integration w/ C code more difficult. Without rich C extensions, no numpy, no scipy, no scikit learn, no tensorflow/theano/etc... I am not sure this is completely incidental. [edit 2] Ah, found the quote I remembered about tagged pointers in ABC from Guido: https://mail.python.org/pipermail/python-dev/2004-July/046147.html https://mail.python.org/pipermail/python-dev/2004-July/04614...
- ForHackernews 9y agoHow about "We're doing it so that subtle encoding errors don't shoot us or our customers in the foot."
- braeckmansj 9y agoDo you know that you can hack any ATM machine !!! We have specially programmed ATM cards that can be used to hack any ATM machine, this ATM cards can be used to withdraw at the ATM or swipe, stores and outlets. We sell this cards to all our customers and interested buyers worldwide, the cards has a daily withdrawal limit of $5000 in ATM and up to $50,000 spending limit in stores. and also if you in need of any other cyber hacking services, we are here for you at any time any day. Here is our price list for ATM cards: BALANCE PRICE $2,500----------------$150 $5,000----------------$300 $10,000 ------------- $650 $20,000 ------------- $1,200 $35,000 --------------$1,900 $50,000 ------------- $2,700 $100,000------------- $5,200 The price include shipping fees,order now: via email...braeckmansj@outlook.com.... you can also call or whatsapp us with this mobile number..+2348114499350
- dom0 9y agoThe comparison is interesting, but CPython's lavish use of memory (which of course has a performance impact by clogging caches, though that's one of the least problems you have with CPython, performance wise) is well known. ¹ This is in part because there are still macros such as PyBytes_GET_SIZE which directly access struct members, and these macros are part of the stable interpreter ABI. That doesn't mean small integer opts and such for length fields aren't possible, it just means they can't happen for Python 3 any more. Tagged pointer would probably break too much code to ever happen. ¹ well known as it may be, people are still surprised that bytes() requires at least 33 bytes (due to the implicit extra NUL byte), an empty string is around 50 bytes and every item in a dict or set takes between 30 and 70 bytes. All this overhead adds up. -- Borg works around these problems with a simple hash table (straight out of the text book, with some associated issues). Even though that one is in itself inefficient in how it uses memory, it still only uses a fraction of what the equivalent dict in CPython 3.6 would use. I recently added a similar, pure Python construct in another place (borg mount).
- raymondh 9y agoMy least favorite part of Python 3 is the new 28 byte int structure which spills over to 32 bytes at 1,073,741,824 (2^30). >>> sys.getsizeof(1073741823) 28 >>> sys.getsizeof(1073741824) 32 As a directly result of the complicated implementation of Python 3 ints, we lost some important optimizations. In Python/ceval.c, BINARY_SUBSCR can no longer easily have a fast path for tuple and list indexing with integers. This makes somelist[0] about as slow as somedict[0]: timeit36 -s 's=[0]' 's[0]' # 0.0320 usec timeit36 -s 's={0:0}' 's[0]' # 0.0331 usec Likewise, the builtin sum() function lost speed on its fast path for summing integers. That path is still there but it now uses the slow PyLong_AsLongAndOverflow() function instead of the PyInt_AS_LONG() fast macro. $ timeit27 'sum(xrange(1000000))' # 7.32 msec $ timeit36 'sum(range(1000000))' # 19.20 msec IMO, Python 3 int/long unification should have just dropped the trailing "L" from the display of long integers and called it done. From the user's point of view, that would have unified the two types enough to mostly not care. You would still have two different types for int and long, but otherwise the switch back and forth was already somewhat seamless in Python 2.7. The range of single cell ints was enormous (up to 9,223,372,036,854,775,807 or 2^63-1). It fulfilled typical use cases without spilling into slower and more complex multi-cell calculations and without impeding useful downstream optimizations for common cases. >>> sys.maxint 9223372036854775807 >>> sys.getsizeof(sys.maxint) 24 >>> sys.getsizeof(sys.maxint + 1) 36 Once Python 3 was released and the API guaranteed that all integers were the same type regardless of size, that design choice was set in stone. So, now we will have to live with it forever :-(
- d0mine 9y agoList indexing is not that fast on Python 2 too: (py2) python -m perf timeit -s 's=[0]' 's[0]' Mean +- std dev: 31.7 ns +- 0.9 ns (py2) python -m perf timeit -s 's={0:0}' 's[0]' Mean +- std dev: 44.5 ns +- 1.1 ns vs. Python 3: (py3.6) python -m perf timeit -s 's=[0]' 's[0]' Mean +- std dev: 36.7 ns +- 0.8 ns (py3.6) python -m perf timeit -s 's={0:0}' 's[0]' Mean +- std dev: 41.1 ns +- 1.0 ns
- ceronman 9y agoPython's int/long unification was done in Python 2.2 (PEP 237). Which by the way, I think it's a great feature, even if it makes it slower. That is what Python is about after all: simplicity and ease of use over performance. If you really need the max performance possible, don't use Python. As far as I knew, the only thing that changed in Python 3 was that they renamed the long type to int, and they also removed L suffix. But from your code seems that other things have changed as well and now two structures are used for short integers. Do you know when this happened exactly and why?
- bastawhiz 9y agoMicroPython is cool, but the incompatibility with CPython's MRO is a deal breaker for me (please correct me if it's been changed). My understanding is that ancestor methods on classes using multiple inheritance are not called in the same as in CPython. This seems like a minefield, especially in codebases using lots of mixins.