4 ms·

Wow. Could you share the equation/code you worked on? I can't imagine a mathematical object being 84KB long, that's insanely huge

okaleniuk 2y ago
Sure! https://github.com/akalenuk/mesh-experiments/blob/master/eq/isotropic_plane.py https://github.com/akalenuk/mesh-experiments/blob/master/eq/... It's not even a particularly large system. Only 6 linear equations.
SPACECADET3D 2y ago
Author of Symbolica here: Symbolica is used in physics calculations to do arithmetic on rational polynomials that are hundreds of megabytes long. For my physics research I have worked with expressions that was just shy of a terabyte long and had > 100M terms. The way that works is that you stream terms from disk, perform manipulations on them and write them to disk again. Using a mergesort, terms that add up can be identified by sorting them to be adjacent.
- actionfromafar 2y ago
  I love how merge-sort which was once used on punch cards because RAM was tight is still relevant. :)
  - lanstin 2y ago
    Knuth's section on sorting data on tape drives is surprisingly relevant to big data running over S3 buckets and streaming into compute.
    - ThomasBHickey 2y ago
      I can remember the days! I suppose as problems grow in size, it's not too surprising that older methods of coping with what seemed like lots of data are still applicable.
- glimshe 2y ago
  This is fascinating. What is the real-world application of these polynomials? I mean, what technical-scientific problems can only be solved with such large objects? I love the thought of "if I don't solve this enormous polynomial, this solar panel won't have very good efficiency" or something like that.
  - SPACECADET3D 2y ago
    These polynomials appear when computing Feynman diagrams, which are used to make predictions for the Large Hadron Collider. The collider can measure collisions with such astonishing precision (<1% error) that predictions of the same order are also needed. The more precise you want to be, the more terms in a series approximation of the mathematical description of the collision you need to compute. For example, I computed the fifth-order approximation of the QCD beta function, which governs how intense the strong force affects matter. This takes 5 days of symbolic manipulations (pattern matching, substitutions, rational polynomial arithmetic, etc) on 32 cores. The large polynomials appear in the middle of the computation, often referred to as intermediate expression swell. This also happens when you do a Gaussian elimination or compute greatest common divisors: the final result will be small, but intermediately, the expressions can get large.
    - eh_why_not 2y ago
      Is there a document/book you can recommend that includes a simplified example/tutorial of this computation process from beginning to end? (Or, what are the right keywords to search for such a thing?) I'm looking for something like: here's the particle interaction we will work on, this is a very simple Feynman diagram, and here's the simplified data the LHC gave us about it, here's the resulting equation from which we'll derive a series, etc. Not looking for how to program it, but actually for seeing the problem structure, and the solution design from beginning to end. (Familiar with high level physics concepts, and comfortable with any math).
      - Y_Y 2y ago
        I like Peskin's Introduction to QFT, but Zee's QFT in a Nutshell is also good.
jgalt212 2y ago
Indeed, it's hard to me to imagine such an expression without any of the following qualities: - immediately converges to zero - immediately heads to infinity - is dominated by only a few terms (thus obviating the needs for the other X million terms)
- 6gvONxR4sf7o 2y ago
  It sounds like the problem is that it’s dominated by a few terms, but finding them is tricky. If you have 1.0 x + 2.0 x - 3.0 x - 5/4 x + x - x + x - x … (etc for a few GB) and then quadratic terms, it’ll take work to figure out whether the dominant term is linear or quadratic. Cancelling out the potentially important intermediate terms (especially in higher dimensions) sounds like a mess.
cha42 2y ago
The largest fancy math object I know is probably the proof of the following theorem: https://link.springer.com/chapter/10.1007/978-3-030-51074-9_4 https://link.springer.com/chapter/10.1007/978-3-030-51074-9_... The proof is 200Gb large. I am quiet sure now even larger proof exists, in particular thet exhaust some combinatorial property on graphs.

okaleniuk 2y ago

Sure! https://github.com/akalenuk/mesh-experiments/blob/master/eq/isotropic_plane.py https://github.com/akalenuk/mesh-experiments/blob/master/eq/... It's not even a particularly large system. Only 6 linear equations.

SPACECADET3D 2y ago

Author of Symbolica here: Symbolica is used in physics calculations to do arithmetic on rational polynomials that are hundreds of megabytes long. For my physics research I have worked with expressions that was just shy of a terabyte long and had > 100M terms. The way that works is that you stream terms from disk, perform manipulations on them and write them to disk again. Using a mergesort, terms that add up can be identified by sorting them to be adjacent.

actionfromafar 2y ago
I love how merge-sort which was once used on punch cards because RAM was tight is still relevant. :)
- lanstin 2y ago
  Knuth's section on sorting data on tape drives is surprisingly relevant to big data running over S3 buckets and streaming into compute.
  - ThomasBHickey 2y ago
    I can remember the days! I suppose as problems grow in size, it's not too surprising that older methods of coping with what seemed like lots of data are still applicable.
glimshe 2y ago
This is fascinating. What is the real-world application of these polynomials? I mean, what technical-scientific problems can only be solved with such large objects? I love the thought of "if I don't solve this enormous polynomial, this solar panel won't have very good efficiency" or something like that.
- SPACECADET3D 2y ago
  These polynomials appear when computing Feynman diagrams, which are used to make predictions for the Large Hadron Collider. The collider can measure collisions with such astonishing precision (<1% error) that predictions of the same order are also needed. The more precise you want to be, the more terms in a series approximation of the mathematical description of the collision you need to compute. For example, I computed the fifth-order approximation of the QCD beta function, which governs how intense the strong force affects matter. This takes 5 days of symbolic manipulations (pattern matching, substitutions, rational polynomial arithmetic, etc) on 32 cores. The large polynomials appear in the middle of the computation, often referred to as intermediate expression swell. This also happens when you do a Gaussian elimination or compute greatest common divisors: the final result will be small, but intermediately, the expressions can get large.
  - eh_why_not 2y ago
    Is there a document/book you can recommend that includes a simplified example/tutorial of this computation process from beginning to end? (Or, what are the right keywords to search for such a thing?) I'm looking for something like: here's the particle interaction we will work on, this is a very simple Feynman diagram, and here's the simplified data the LHC gave us about it, here's the resulting equation from which we'll derive a series, etc. Not looking for how to program it, but actually for seeing the problem structure, and the solution design from beginning to end. (Familiar with high level physics concepts, and comfortable with any math).
    - Y_Y 2y ago
      I like Peskin's Introduction to QFT, but Zee's QFT in a Nutshell is also good.

jgalt212 2y ago

Indeed, it's hard to me to imagine such an expression without any of the following qualities: - immediately converges to zero - immediately heads to infinity - is dominated by only a few terms (thus obviating the needs for the other X million terms)

6gvONxR4sf7o 2y ago
It sounds like the problem is that it’s dominated by a few terms, but finding them is tricky. If you have 1.0 x + 2.0 x - 3.0 x - 5/4 x + x - x + x - x … (etc for a few GB) and then quadratic terms, it’ll take work to figure out whether the dominant term is linear or quadratic. Cancelling out the potentially important intermediate terms (especially in higher dimensions) sounds like a mess.

cha42 2y ago

The largest fancy math object I know is probably the proof of the following theorem: https://link.springer.com/chapter/10.1007/978-3-030-51074-9_4 https://link.springer.com/chapter/10.1007/978-3-030-51074-9_... The proof is 200Gb large. I am quiet sure now even larger proof exists, in particular thet exhaust some combinatorial property on graphs.