4 ms·

What could possibly be so memory-intensive about gron? I suppose it could make sense for ungronning, but not in the forward direction.

• nojs 3y ago
  Deeply nested structures would get expanded a lot
• haileys 3y ago
  It buffers all of its output statements in memory before writing to stdout: https://github.com/tomnomnom/gron/blob/master/main.go#L204 https://github.com/tomnomnom/gron/blob/master/main.go#L204
  • codetrotter 3y ago
    Why?
    • mongol 3y ago
      It appears to be so it can sort the lines. Not sure how useful that is however.
      • loeg 3y ago
        It shouldn't need to buffer the output to do that, right?
        • swsieber 3y ago
          Correct.
      • CGamesPlay 3y ago
        An ironic near-miss on the UNIX philosophy. There's a great UNIX tool that will handle sorting arbitrarily large files!
        • tgv 3y ago
          It will mess up array indices, though.
          • codetrotter 3y ago
            Wouldn’t “sort -n” work with indices?
            • tgv 3y ago
              It's tricky to specify the sorting criterium: you have to indicate the column. Gron's output looks like this: a.b[0].c.d[0]: ... a.b[0].e[0].f: ...
  • xiphias2 3y ago
    Here's the fastgron printer if you want to compare: https://github.com/adamritter/fastgron/blob/main/src/print_gron.cpp https://github.com/adamritter/fastgron/blob/main/src/print_g...
• salted-fry 3y ago
  I remembered where some of my old files are and re-tested; forward-gron was "only" about 7GB for the 15MB file. gron -u was the real killer, clocking in around 53GB.
  • sph 3y ago
    Yeah that's a bug, no amount of buffering can justify that amount of memory. And gron -u in theory should use less memory than gron-ifying a JSON, as you just have to fill a data structure in memory as you go.
    • zimpenfish 3y ago
      > as you just have to fill a data structure in memory as you go You don't know the size, shape, or type of any of the levels in the data structure until you get to a line specifying one part of it. If you did, yep, it would be trivial!
      • sph 3y ago
        But you do: if the first line is `users[15].name.family_name = "Foo"`, all you need to know is there: there is an array of users, each is a map containing a field called name, which is a map with a field called family_name. If users[14] is a string, or there are 1500 users, the amount of memory usage to ungrok that line is exactly the same. Prove me wrong, I can't think of any way it would not be trivial, provided one uses the correct datastructures.
        • zimpenfish 3y ago
          > there is an array of users How big is it? All you know at this point is that it's at least 16 entries long. If the next line starts with `users[150]`, now it's 151 entries. Next line might make it 2000 entries long. You have no idea until you see the line. > each is a map But a map of what? `string -> object`? Ok, the next line is `users[15].flange[15]` which means your map is now `string -> (object|array)`. Then the next line is `users[15].age = 15` and you've got `string -> (object|array|int)`. Each line can change what you've got and in Go this isn't a trivial thing to handle without resorting to `interface{}` (or `any`) all over the show and reflection to handle the management of the data structures. > Prove me wrong, I can't think of any way it would not be trivial, provided one uses the correct datastructures. All I can suggest is that you try to build `ungron` in Go and have it correctly handle disordered input. If you find a better way of doing it, I'd be happy to hear about it because I spent several months fighting Go in 2021-22 trying to optimise this without success.
• 1letterunixname 3y ago
  What happens when you ignore software engineering.