5 ms·

PyPy v7.1 released; now uses UTF-8 internally for Unicode strings

153 points by mattip 8y ago

ChrisSD 8y ago
So does this mean that RPython is now UTF-8 only?
- mattip 8y ago
  No. RPython still supports unicode strings. It now also supports utf8 string classes for things like regular expressions and ffi calls. PyPy just uses the non-unicode rpython apis.
  - akvadrako 8y ago
    UTF-8 is Unicode - your post doesn’t make much sense.
    - mattip 8y ago
      In rpython (a restricted python2 language) unicode refers to a specific python object. Utf-8 classes refers to rpython classes that hold a bytes, not unicode, but are aware of the encoding of the bytes.
    - simcop2387 8y ago
      UTF-8 is one way to encode Unicode code points. There's also UTF-16LE, UTF16-BE, UTF-32LE, UTF-32BE, and others like UTF-7. This makes pypy use UTF-8 internally for the additional memory and speed savings, at the cost of some time later on when you use certain features that require the string index to be built.
spenrose 8y ago
Great work, thank you so much! I was looking for PyPy vs C-Python 3.x speed comparisons on the site and not finding any. Do you publish them?
- mattip 8y ago
  Yeah, about that. Speed.pypy.org is having problems updating, the seven year old fork of codespeed needs refreshing. The benchmarks there are python2, so we would need to rework them for python3. In the meantime, the python perf and performance packages are driving speed.python.org, and I started trying to get pypy3 runs reported there, but we need to provide a warmup parameter for each benchmark so that too will take some time. The best benchmark of course is your use case, try it out and let us know where we are slow
hsivonen 8y ago
Is there a write-up of the details of the PyPy UTF-8 Unicode string internals?
- mattip 8y ago
  Not really. The UnicodeObject has utf8 bytes, a length in codepoints, and an optional index map that is only calculated if needed to traverse the string. Most strings won’t need the map. String matching is utf8 aware, and indexes by codepoints. The main speedup is in converting unicode to internal representation once, then passing around the object with no further conversions. Not sure that answered the question though.
Animats 8y ago
Oh, nice. From PyPy's Twitter feed: "unicode-utf8 just got merged! Unicode strings are now internally represented as utf-8 in PyPy, with an optional extra index data structure to make indexing O(1). We'll write a blog post about it eventually." Python strings are random access indexable. UTF-8 has variable length characters, so it is not directly randomly accessable. So the Python 3 representation inflates them to 1, 2 or 4 bytes per character, depending on the longest character in the string. In practice, most strings are not random accessed. Most subscripts are +-1 from a previous subscript. So most actions can be optimized into uses of "advance one UTF-8 char" or "back up one UTF-8 char". Hard cases require generating an index array of the entire string. That takes a full string scan. Apparently that's what PyPy is doing. This raises the cost of functions like "find()", which search the string for something and return an integer which is a position in the string suitable for random access use. I'd once suggested that "find", etc. return an opaque object which was really a byte index into the string. If you used that in a subscript, the right thing would happen. If you converted it to an integer, the index array would have to be computed. Is PyPy doing that, or does the use of "find" force index generation?
- marmaduke 8y ago
  > most strings are not random accessed But they do expose the buffer interface which requires uniform size
  - mattip 8y ago
    That conversion should not happen only when needed, i.e. when interfacing via ffi to system calls.
- mjevans 8y ago
  In various languages, there are generally two cases where I'll use the result of 'find' (or similar). * Is there ANY occurrence of the byte sequence within the 'string'? (I only care if it's 'invalid', right at the start or end, etc) * I'm about to use it to create a splice / as part of parsing. I could be convinced to use strings.Split() if I knew the resulting objects were more like go's slices. Thin array bounds wrappers indirectly pointing to the same backing array.
therealmarv 8y ago
why is PyPy not just skipping current Python 3.6 effort and goes directly to 3.7 or at least doing that once in a while (skipping in between versions) ?
- pritambaral 8y ago
  I'd guess because that would require implementing the 3.6 features anyway.
hermitdev 8y ago
Curious, any one know why 64-bit Windows isn't supported? Is it because a long is only 32-bits? I'd potentially willing to help with the development to get that working. Current day job is a 64-bit Windows shop in finance and we have several teams expanding their use of Python with very large datasets. A 64-bit PyPy on Windows could be big for us.
- mattip 8y ago
  Here is an explanation and a suggestion on how to implement it. http://doc.pypy.org/en/release-2.6.x/windows.html#what-is-missing-for-a-full-64-bit-translation http://doc.pypy.org/en/release-2.6.x/windows.html#what-is-mi... Please help us, we are a small team. The best way to reach us is through IRC on the #pypy channel
truth_seeker 8y ago
Is anybody using pypy for big data and machine learning tasks ?

153 points by mattip 8y ago

ChrisSD 8y ago

So does this mean that RPython is now UTF-8 only?

mattip 8y ago
No. RPython still supports unicode strings. It now also supports utf8 string classes for things like regular expressions and ffi calls. PyPy just uses the non-unicode rpython apis.
- akvadrako 8y ago
  UTF-8 is Unicode - your post doesn’t make much sense.
  - mattip 8y ago
    In rpython (a restricted python2 language) unicode refers to a specific python object. Utf-8 classes refers to rpython classes that hold a bytes, not unicode, but are aware of the encoding of the bytes.
  - simcop2387 8y ago
    UTF-8 is one way to encode Unicode code points. There's also UTF-16LE, UTF16-BE, UTF-32LE, UTF-32BE, and others like UTF-7. This makes pypy use UTF-8 internally for the additional memory and speed savings, at the cost of some time later on when you use certain features that require the string index to be built.

spenrose 8y ago

Great work, thank you so much! I was looking for PyPy vs C-Python 3.x speed comparisons on the site and not finding any. Do you publish them?

mattip 8y ago
Yeah, about that. Speed.pypy.org is having problems updating, the seven year old fork of codespeed needs refreshing. The benchmarks there are python2, so we would need to rework them for python3. In the meantime, the python perf and performance packages are driving speed.python.org, and I started trying to get pypy3 runs reported there, but we need to provide a warmup parameter for each benchmark so that too will take some time. The best benchmark of course is your use case, try it out and let us know where we are slow

hsivonen 8y ago

Is there a write-up of the details of the PyPy UTF-8 Unicode string internals?

mattip 8y ago
Not really. The UnicodeObject has utf8 bytes, a length in codepoints, and an optional index map that is only calculated if needed to traverse the string. Most strings won’t need the map. String matching is utf8 aware, and indexes by codepoints. The main speedup is in converting unicode to internal representation once, then passing around the object with no further conversions. Not sure that answered the question though.

Animats 8y ago

Oh, nice. From PyPy's Twitter feed: "unicode-utf8 just got merged! Unicode strings are now internally represented as utf-8 in PyPy, with an optional extra index data structure to make indexing O(1). We'll write a blog post about it eventually." Python strings are random access indexable. UTF-8 has variable length characters, so it is not directly randomly accessable. So the Python 3 representation inflates them to 1, 2 or 4 bytes per character, depending on the longest character in the string. In practice, most strings are not random accessed. Most subscripts are +-1 from a previous subscript. So most actions can be optimized into uses of "advance one UTF-8 char" or "back up one UTF-8 char". Hard cases require generating an index array of the entire string. That takes a full string scan. Apparently that's what PyPy is doing. This raises the cost of functions like "find()", which search the string for something and return an integer which is a position in the string suitable for random access use. I'd once suggested that "find", etc. return an opaque object which was really a byte index into the string. If you used that in a subscript, the right thing would happen. If you converted it to an integer, the index array would have to be computed. Is PyPy doing that, or does the use of "find" force index generation?

marmaduke 8y ago
> most strings are not random accessed But they do expose the buffer interface which requires uniform size
- mattip 8y ago
  That conversion should not happen only when needed, i.e. when interfacing via ffi to system calls.
mjevans 8y ago
In various languages, there are generally two cases where I'll use the result of 'find' (or similar). * Is there ANY occurrence of the byte sequence within the 'string'? (I only care if it's 'invalid', right at the start or end, etc) * I'm about to use it to create a splice / as part of parsing. I could be convinced to use strings.Split() if I knew the resulting objects were more like go's slices. Thin array bounds wrappers indirectly pointing to the same backing array.

therealmarv 8y ago

why is PyPy not just skipping current Python 3.6 effort and goes directly to 3.7 or at least doing that once in a while (skipping in between versions) ?

pritambaral 8y ago
I'd guess because that would require implementing the 3.6 features anyway.

hermitdev 8y ago

Curious, any one know why 64-bit Windows isn't supported? Is it because a long is only 32-bits? I'd potentially willing to help with the development to get that working. Current day job is a 64-bit Windows shop in finance and we have several teams expanding their use of Python with very large datasets. A 64-bit PyPy on Windows could be big for us.

mattip 8y ago
Here is an explanation and a suggestion on how to implement it. http://doc.pypy.org/en/release-2.6.x/windows.html#what-is-missing-for-a-full-64-bit-translation http://doc.pypy.org/en/release-2.6.x/windows.html#what-is-mi... Please help us, we are a small team. The best way to reach us is through IRC on the #pypy channel

truth_seeker 8y ago

Is anybody using pypy for big data and machine learning tasks ?