4 ms·

Go and a not tiny amount of memory

19 points by relyio 9y ago

peterwwillis 9y ago
Every time I read one of these it's like watching someone in college learn about databases, but it's actually a company.
- voidlogic 9y ago
  Thank you for making the comment I wanted to make, but was too afraid of being down-voted for being an elitist UNIX beard. Also funny that they admit the CPU use might not really be a problem... but try to solve it anyway and add a lot of complexity to their system. I have learned to KISS and am I am surprised they are solving this problem and don't have more pressing product development. Doing something like this is pretty standard, but doing it before its needed smells of misused engineering time. Also they didn't even really need to do most of the work they did here, they could have re-used existing out-of-heap Go memory mgmt solutions like: https://github.com/teh-cmc/mmm https://github.com/teh-cmc/mmm . But I get it, building things is fun :) But like the parent said, I'd expect it to be a personal rather than a company project.
- jaytaylor 9y ago
  This. Why not use one of the many full-featured RDBMS or KV stores available for free? This is an already solved problem. Go isn't the best tool for all cases.
- dang 9y ago
  Regardless of how wrong or ignorant someone is, please don't be a jerk in comments here. That's far worse in its systemic effect on the site. It looks like the entire thread needs to learn this. If someone is wrong or ignorant, teach them something. Then we all learn. Alternatively, it's fine not to comment.
  - peterwwillis 9y ago
    Uh. I never said anyone or anything was wrong or ignorant. My comment was a meta comment; there's nothing to teach.
    - dang 9y ago
      Maybe I misunderstood you, but to me it read like a putdown. Snarky putdowns are the kind of thing we're trying to avoid here, especially when the putdown is apparently for lack of knowledge.
jzwinck 9y ago
So you use a GC and have occasional CPU usage spikes. You correctly identify the cause. You then use "unsafe" code to go behind the GC's back and build some stuff which probably has crash bugs or worse, and by your analysis solves a problem which was not significant. I shouldn't be surprised. The red flag was flying as soon as IOP was used in singular form. Input/Output Operations Per.
- deleted 9y ago
  [deleted]
- zackelan 9y ago
  > I shouldn't be surprised. The red flag was flying as soon as IOP was used in singular form. > Input/Output Per. IOPS can also be "I/O operations per second" and in that context "1 IOP" is a meaningful quantity. Particularly when AWS and other cloud providers sell capacity in units of IOPS: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-io-characteristics.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-io-c...
henpa 9y ago
I wonder if libraries such as freecache[1] and bigcache[2] wouldn't offer a better solution for this use case. [1] https://github.com/coocood/freecache https://github.com/coocood/freecache [2] https://github.com/allegro/bigcache https://github.com/allegro/bigcache

19 points by relyio 9y ago

Every time I read one of these it's like watching someone in college learn about databases, but it's actually a company.

voidlogic 9y ago
Thank you for making the comment I wanted to make, but was too afraid of being down-voted for being an elitist UNIX beard. Also funny that they admit the CPU use might not really be a problem... but try to solve it anyway and add a lot of complexity to their system. I have learned to KISS and am I am surprised they are solving this problem and don't have more pressing product development. Doing something like this is pretty standard, but doing it before its needed smells of misused engineering time. Also they didn't even really need to do most of the work they did here, they could have re-used existing out-of-heap Go memory mgmt solutions like: https://github.com/teh-cmc/mmm https://github.com/teh-cmc/mmm . But I get it, building things is fun :) But like the parent said, I'd expect it to be a personal rather than a company project.
jaytaylor 9y ago
This. Why not use one of the many full-featured RDBMS or KV stores available for free? This is an already solved problem. Go isn't the best tool for all cases.
dang 9y ago
Regardless of how wrong or ignorant someone is, please don't be a jerk in comments here. That's far worse in its systemic effect on the site. It looks like the entire thread needs to learn this. If someone is wrong or ignorant, teach them something. Then we all learn. Alternatively, it's fine not to comment.
- peterwwillis 9y ago
  Uh. I never said anyone or anything was wrong or ignorant. My comment was a meta comment; there's nothing to teach.
  - dang 9y ago
    Maybe I misunderstood you, but to me it read like a putdown. Snarky putdowns are the kind of thing we're trying to avoid here, especially when the putdown is apparently for lack of knowledge.

jzwinck 9y ago

So you use a GC and have occasional CPU usage spikes. You correctly identify the cause. You then use "unsafe" code to go behind the GC's back and build some stuff which probably has crash bugs or worse, and by your analysis solves a problem which was not significant. I shouldn't be surprised. The red flag was flying as soon as IOP was used in singular form. Input/Output Operations Per.

deleted 9y ago
[deleted]
zackelan 9y ago
> I shouldn't be surprised. The red flag was flying as soon as IOP was used in singular form. > Input/Output Per. IOPS can also be "I/O operations per second" and in that context "1 IOP" is a meaningful quantity. Particularly when AWS and other cloud providers sell capacity in units of IOPS: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-io-characteristics.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-io-c...

henpa 9y ago

I wonder if libraries such as freecache[1] and bigcache[2] wouldn't offer a better solution for this use case. [1] https://github.com/coocood/freecache https://github.com/coocood/freecache [2] https://github.com/allegro/bigcache https://github.com/allegro/bigcache