4 ms·

These are really nice libraries, but they indicate there is a problem with input data having "gigabytes of ASCII floats", where you store floats as strings inst

These are really nice libraries, but they indicate there is a problem with input data having "gigabytes of ASCII floats", where you store floats as strings instead of binary formats(which doesn't require parsing anything). for scripts and user-input, i'd rather use the standard stuff that has been tried and tested(strtod/strtold).

hacker_9 6y ago
Sometimes third parties give you json and that's it, in these cases these libraries are useful if being fastest to react is a constraint.
asicsp 6y ago
>On my Apple M1 MacBook, using a realistic data file (canada), we get that fast_float can far exceeds a gigabyte per second, and get close to 2 GB/s. The conventional C function (strtod) provided by the default Apple standard library does quite poorly on this benchmark.
MaxBarraclough 6y ago
If it's so cheap to decode that it's barely slower than copying the bytes, the problem largely goes away, no? No doubt you're right that it would be more efficient to store the binary representation natively, especially assuming portability isn't an issue, and especially if a zero-copy solution were used, but many real-world systems have to cope with data formats that aren't the most efficient. Huge amounts of data are shipped as XML and JSON.
- FrozenVoid 6y ago
  If performance matters, storage space should matter too.
  - MaxBarraclough 6y ago
    Representing numbers in decimal ASCII/Unicode shouldn't be too bad in that regard, should it? If we're representing a sparse matrix, the '0' character occupies one byte, whereas a 32-bit representation occupies 4 (whether floating point or fixed point). Both representations should lend themselves to compression.
    - jstimpfle 6y ago
      Text requires at least two bytes (it needs a separator), and negative numbers require yet another byte. So it's hard to make a point that they occupy less storage. But there is another advantage: it can represent infinitely large numbers.
      - MaxBarraclough 6y ago
        > Text requires at least two bytes (it needs a separator), and negative numbers require yet another byte. True. > it's hard to make a point that they occupy less storage No, like I said, it would occupy less storage for sparse data. That's true even with separator characters (2 bytes as against 4). Doubtless there are much better solutions for representing sparse data that aren't human readable. A simple (index, value) dictionary, say. > it can represent infinitely large numbers Right, there's no upper bound beyond the limitations of the systems, although again a non-human-readable bignum format could do this more efficiently.
    - retrac 6y ago
      The algorithm give above is an excellent, extremely fast compression algorithm for ASCII-encoded numbers :)
  - maccard 6y ago
    You don't always have control over your input sources, and you may not be responsible for the output format. As I posted elsewhere in this thread, my consumer desktop pc has a 10Gb network card, and has an SSD with a read speed of over 5x that, and 2TB capacity. If my data source is ascii formatted floats and I have to work with that, then I have to work with that.

hacker_9 6y ago

Sometimes third parties give you json and that's it, in these cases these libraries are useful if being fastest to react is a constraint.

asicsp 6y ago

>On my Apple M1 MacBook, using a realistic data file (canada), we get that fast_float can far exceeds a gigabyte per second, and get close to 2 GB/s. The conventional C function (strtod) provided by the default Apple standard library does quite poorly on this benchmark.

MaxBarraclough 6y ago

If it's so cheap to decode that it's barely slower than copying the bytes, the problem largely goes away, no? No doubt you're right that it would be more efficient to store the binary representation natively, especially assuming portability isn't an issue, and especially if a zero-copy solution were used, but many real-world systems have to cope with data formats that aren't the most efficient. Huge amounts of data are shipped as XML and JSON.

FrozenVoid 6y ago
If performance matters, storage space should matter too.
- MaxBarraclough 6y ago
  Representing numbers in decimal ASCII/Unicode shouldn't be too bad in that regard, should it? If we're representing a sparse matrix, the '0' character occupies one byte, whereas a 32-bit representation occupies 4 (whether floating point or fixed point). Both representations should lend themselves to compression.
  - jstimpfle 6y ago
    Text requires at least two bytes (it needs a separator), and negative numbers require yet another byte. So it's hard to make a point that they occupy less storage. But there is another advantage: it can represent infinitely large numbers.
    - MaxBarraclough 6y ago
      > Text requires at least two bytes (it needs a separator), and negative numbers require yet another byte. True. > it's hard to make a point that they occupy less storage No, like I said, it would occupy less storage for sparse data. That's true even with separator characters (2 bytes as against 4). Doubtless there are much better solutions for representing sparse data that aren't human readable. A simple (index, value) dictionary, say. > it can represent infinitely large numbers Right, there's no upper bound beyond the limitations of the systems, although again a non-human-readable bignum format could do this more efficiently.
  - retrac 6y ago
    The algorithm give above is an excellent, extremely fast compression algorithm for ASCII-encoded numbers :)
- maccard 6y ago
  You don't always have control over your input sources, and you may not be responsible for the output format. As I posted elsewhere in this thread, my consumer desktop pc has a 10Gb network card, and has an SSD with a read speed of over 5x that, and 2TB capacity. If my data source is ascii formatted floats and I have to work with that, then I have to work with that.