3 ms·
If you want a thorough understanding: you'll want to look up "numerical computation", "numerical methods" or "computational methods"; in particular computing er
by cbhl 4y ago
If you want a thorough understanding: you'll want to look up "numerical computation", "numerical methods" or "computational methods"; in particular computing error bounds and error propagation. Typically covered in bachelor's level university-level Math, CS, or Engineering departments.
If you just want to fix the odd behavior: adjust the schema so that you only work with whole numbers. For example, in the database schema, you can use DECIMAL instead of REAL/DOUBLE columns, or use two columns to specify the ratio of two integers (for example num/denom in frame rates in various video containers/codecs). In the application code: work only in whole numbers (e.g. cents, satoshis) instead of fractionals, using bigint or string types as applicable instead of e.g. double.
- gowld 4y agoOr Fixed Point numbers .
- flopriore 4y agoDon't Fixed Point numbers add more issues? Let's assume you have a machine that accepts 8-bits fixed-point numbers, with 6 bits for integers and 2 for decimals. For simplicity, let's use decimal digits. If you must represent the number 2.013, the resulting number would be +00002.01 So you cut the third decimal digit and wasted 4 bits with useless zeros. At the same time, if you represented the same number in 8 bits floating point, you would have all the digits.
- brianwawok 4y agoDepends what you track. Fixed point is great for money. Floating point is not. Using an integer cents is easier than both.