5 ms·

> If your code is successfully making that your bottleneck your code has problems. I disagree. My use cases involve loading, processing, rendering, and saving

> If your code is successfully making that your bottleneck your code has problems. I disagree. My use cases involve loading, processing, rendering, and saving point clouds with millions to billions of points. For precision and file size reasons, coordinates are stored as 3 x int32 instead of 3 x float or 3 x double. Every time I load the points from disk I have to convert them to floats for rendering, or doubles for processing. Vice-versa for storing the processed results to disk. Just because you don't need it, doesn't mean nobody needs it.

titzer 8y ago
You should consider using WebAssembly if you process so many numbers.
- mschuetz 8y ago
  Javascript is surprisingly fast at processing that many numbers, using TypedArrays. You just had to avoid DataView up until now but the V8 team recently fixed the performance of DataView so it should have similar performance as TypedArray access and therefore be around 10x faster in future versions of chrome.
  - kbumsik 8y ago
    Although they are fast, TypedArrays and DataView only deal with setting/getting data from a buffer. They are totally unrelated to numerical calculations mentioned in OP.
xxs 8y ago
Unless you care about the Z flag the instruction doesn't offer that much (perhaps longer pipeline due to reduced size)
maccard 8y ago
> For precision and file size reasons, coordinates are stored as 3 x int32 instead of 3 x float or 3 x double I get storing them as float instead of double but into instead of float for size/precision doesn't really make sense to me. A float is 32 bits, so will be the same size as an int(32). If it's precision you're worried about,you're going to lose the precision as soon as you convert it from an int to a float to use.
- mschuetz 8y ago
  Single precision floats loose precision as coordinates get larger, hence "floating" point precision. Point cloud data often consists of outdoor/aerial scans covering multiple kilometers and floats cannot accurately represent large coordinate values like that. Translating to origin and/or rescaling only works to a limited extent and isn't a very robust solution. Integers on the other hand can be used to represent coordinates in a fixed precision. E.g. if you want to store coordinates in milimeter precision, you multiply your double precision meter coordinates by 1000, and then cast the resulting coordinate to 32 bit integers. > If it's precision you're worried about,you're going to lose the precision as soon as you convert it from an int to a float to use. Yes, that is why you convert ints to doubles for processing and only use floats for tasks where limited accuracy is okay, e.g. rendering.
  - maccard 8y ago
    > Single precision floats loose precision as coordinates get larger, hence "floating" point precision. I've plenty of experience with floating point numbers. > Translating to origin and/or rescaling only works to a limited extent and isn't a very robust solution. Curious as to why you think this isn't the solution? double precision just kicks the can farther down the line, whereas a fixed point offset origin gives you the "best" of both (albeit with slightly more code) > Yes, that is why you convert ints to doubles for processing and only use floats for tasks where limited accuracy is okay, e.g. rendering. Right, so you store as 32 bit values, and do the processing in double precision, but convert to single precision for rendering (from double precision)? So you were never going to store them as float on disk.
    - olliej 8y ago
      Presumably they don’t think that’s the problem because doubles already exceed the precision of the integer types they’re using. I do get this decision - point cloud data for scanned geography tends to be uniformly separated over a large area, a 32 bit float very rapidly starts to accrue significant error if the geometry is large. Presumably he olde fixed point arithmetic would be more precise (for this specific purpose) than a 32bit float (which has thrown away significant amounts of precision for the exponent). But again, 64bit float has 53bits of precision, which is larger than int32 that is in their data format. It’s also probably enough precision for more or less anything outside of extreme ends of science :)
olliej 8y ago
Storing to memory is orders of magnitude slower than an floating point conversion. This isn’t “conversion doesn’t need to be fast”, this is “conversion is so fast already that that won’t be your bottleneck”. Even if it was slow, given what you are describing I would assume that you performance would be bottlenecked on IO even with a pure software floating point conversion.
- deleted 8y ago
  [deleted]

titzer 8y ago

You should consider using WebAssembly if you process so many numbers.

mschuetz 8y ago
Javascript is surprisingly fast at processing that many numbers, using TypedArrays. You just had to avoid DataView up until now but the V8 team recently fixed the performance of DataView so it should have similar performance as TypedArray access and therefore be around 10x faster in future versions of chrome.
- kbumsik 8y ago
  Although they are fast, TypedArrays and DataView only deal with setting/getting data from a buffer. They are totally unrelated to numerical calculations mentioned in OP.

xxs 8y ago

Unless you care about the Z flag the instruction doesn't offer that much (perhaps longer pipeline due to reduced size)

maccard 8y ago

> For precision and file size reasons, coordinates are stored as 3 x int32 instead of 3 x float or 3 x double I get storing them as float instead of double but into instead of float for size/precision doesn't really make sense to me. A float is 32 bits, so will be the same size as an int(32). If it's precision you're worried about,you're going to lose the precision as soon as you convert it from an int to a float to use.

mschuetz 8y ago
Single precision floats loose precision as coordinates get larger, hence "floating" point precision. Point cloud data often consists of outdoor/aerial scans covering multiple kilometers and floats cannot accurately represent large coordinate values like that. Translating to origin and/or rescaling only works to a limited extent and isn't a very robust solution. Integers on the other hand can be used to represent coordinates in a fixed precision. E.g. if you want to store coordinates in milimeter precision, you multiply your double precision meter coordinates by 1000, and then cast the resulting coordinate to 32 bit integers. > If it's precision you're worried about,you're going to lose the precision as soon as you convert it from an int to a float to use. Yes, that is why you convert ints to doubles for processing and only use floats for tasks where limited accuracy is okay, e.g. rendering.
- maccard 8y ago
  > Single precision floats loose precision as coordinates get larger, hence "floating" point precision. I've plenty of experience with floating point numbers. > Translating to origin and/or rescaling only works to a limited extent and isn't a very robust solution. Curious as to why you think this isn't the solution? double precision just kicks the can farther down the line, whereas a fixed point offset origin gives you the "best" of both (albeit with slightly more code) > Yes, that is why you convert ints to doubles for processing and only use floats for tasks where limited accuracy is okay, e.g. rendering. Right, so you store as 32 bit values, and do the processing in double precision, but convert to single precision for rendering (from double precision)? So you were never going to store them as float on disk.
  - olliej 8y ago
    Presumably they don’t think that’s the problem because doubles already exceed the precision of the integer types they’re using. I do get this decision - point cloud data for scanned geography tends to be uniformly separated over a large area, a 32 bit float very rapidly starts to accrue significant error if the geometry is large. Presumably he olde fixed point arithmetic would be more precise (for this specific purpose) than a 32bit float (which has thrown away significant amounts of precision for the exponent). But again, 64bit float has 53bits of precision, which is larger than int32 that is in their data format. It’s also probably enough precision for more or less anything outside of extreme ends of science :)

olliej 8y ago

Storing to memory is orders of magnitude slower than an floating point conversion. This isn’t “conversion doesn’t need to be fast”, this is “conversion is so fast already that that won’t be your bottleneck”. Even if it was slow, given what you are describing I would assume that you performance would be bottlenecked on IO even with a pure software floating point conversion.

deleted 8y ago
[deleted]