5 ms·

I have a generational art program that I wrote under Go 1.4 which was absolutely crushed by 1.5's GC. The program heavily abuses goroutines, so I'm cautiously o

by Kapura 10y ago

I have a generational art program that I wrote under Go 1.4 which was absolutely crushed by 1.5's GC. The program heavily abuses goroutines, so I'm cautiously optimistic that this optimisation would allow me to keep my version of Go current once again.

_ak 10y ago
Have you tested it with the GC improvements in 1.6 and the 1.7 beta? If so, did they have any impact on your program's performance?
- Kapura 10y ago
  I did test in 1.6, and the performance and output were still significantly worse than in 1.4. The part I'm saddest about is that because some of the goroutines abuse involved literally building in a race condition, the aggressive garbage collection made the resulting output less artistically interesting (on top of taking an hour to complete instead of a few minutes).
  - iainmerrick 10y ago
    That doesn't sound like something that's likely to get fixed. More like a loophole that's now gone.
    - Kapura 10y ago
      I think you're probably right, but this whitepaper gave me a sliver of hope that my goroutines might be free to work some nondeterministic magic once again
  - h8liu 10y ago
    interesting work. if I am understanding correctly, you are essentially using the randomness generated by the scheduler to produce art pieces. do you have a public link to your work?
    - deleted 10y ago
      [deleted]
    - Kapura 10y ago
      Alas, I've been hording almost all of the most interesting images I've created using the process, but I wrote up a short piece when I got the initial program running [1], and I do have a few examples of pictures generated using the derivations of the code that still make use of the core of the algorithm (these are all huge images!) [2][3][4][5][6]. [1] https://medium.com/@kapuramax/procedural-image-generation-in-go-7a57ff2e2e90#.7dxx5hzci https://medium.com/@kapuramax/procedural-image-generation-in... [2] http://www.nobledust.com/full_size/bullet.r0g0b0.x1000y1000.blur5.ch100.cpu2.png http://www.nobledust.com/full_size/bullet.r0g0b0.x1000y1000.... [3] http://www.nobledust.com/full_size/b.brg.rr0.90.blur1.ch8.cpu8.png http://www.nobledust.com/full_size/b.brg.rr0.90.blur1.ch8.cp... [4] http://www.nobledust.com/full_size/a.jpg http://www.nobledust.com/full_size/a.jpg [5] http://www.nobledust.com/reds.png http://www.nobledust.com/reds.png [6] http://www.nobledust.com/full_size/mosey.png http://www.nobledust.com/full_size/mosey.png
      - jerf 10y ago
        Looking over your code, did you ever run it single-threaded to compare the performance? I see you spawning what appears to be a crapton of goroutines, each of which performs a very small number of operations, then terminates. As cheap as goroutines are, they still have some cost to startup and shutdown, and it's not a good idea to spawn a goroutine if it's going to do roughly the same order of work inside the goroutine as the spawn & shutdown costs. This is a pessimal situation. I'm not sure if you're seeing the GC fail or if you're seeing the scheduler choke under the harsh load. If you just want nondeterminacy, I'd suggest using math/rand directly, not accidents of scheduling... which, as was just proved, can disappear as quickly as they appear. In fact it doesn't even take something as serious as a language upgrade for that to happen, that kind of indeterminancy can disappear if you so much as add a member to a struct or have another process on your system eating 100% of one CPU or something. It's not reliable, not even a little.
        Kapura 10y ago
gtirloni 10y ago
What is a generational art program? If you don't mind me asking.
- overcast 10y ago
  Senior finger painting classes.
  - 0xdeadbeefbabe 10y ago
    In Florida.
- Kapura 10y ago
  Well, in my case specifically, it was a program that generated coloured images based on a set of simple rules (namely, don't reuse the same RGB colour for any pixel & try to be as close to the colour of neighbouring pixels as possible). I wrote a brief overview around the time I finished the first version[1]. More recently, I wrote a new routine which generates several of those and then averages the RGB channel values on a per-pixel basis, and then repeatedly re-averages the original images with those average images to create more "generations." I haven't written anything up about that because the code required to sum and average pixels in a PNG is significantly less interesting. [1] https://medium.com/@kapuramax/procedural-image-generation-in-go-7a57ff2e2e90#.7dxx5hzci https://medium.com/@kapuramax/procedural-image-generation-in...
- spriggan3 10y ago
  s/generational/generative ?
vessenes 10y ago
That's interesting -- my use case had up to 2 minute GC pauses in 1.4, and now runs without a hiccup. For a variety of reasons, some code-side and some on Go's side. I didn't know there were use cases that were negatively affected by GC work done in the last year or two.
giovannibajo1 10y ago
Did you open a bug report with some datasets (profile dumps, etc.)? Go developers treat these kinds of regressions very very seriously. They absolutely want to avoid situations were users are pinned to an old version because of a performance regression.

by Kapura 10y ago

_ak 10y ago

Have you tested it with the GC improvements in 1.6 and the 1.7 beta? If so, did they have any impact on your program's performance?

Kapura 10y ago
I did test in 1.6, and the performance and output were still significantly worse than in 1.4. The part I'm saddest about is that because some of the goroutines abuse involved literally building in a race condition, the aggressive garbage collection made the resulting output less artistically interesting (on top of taking an hour to complete instead of a few minutes).
- iainmerrick 10y ago
  That doesn't sound like something that's likely to get fixed. More like a loophole that's now gone.
  - Kapura 10y ago
    I think you're probably right, but this whitepaper gave me a sliver of hope that my goroutines might be free to work some nondeterministic magic once again
- h8liu 10y ago
  interesting work. if I am understanding correctly, you are essentially using the randomness generated by the scheduler to produce art pieces. do you have a public link to your work?
  - deleted 10y ago
    [deleted]
  - Kapura 10y ago
    Alas, I've been hording almost all of the most interesting images I've created using the process, but I wrote up a short piece when I got the initial program running [1], and I do have a few examples of pictures generated using the derivations of the code that still make use of the core of the algorithm (these are all huge images!) [2][3][4][5][6]. [1] https://medium.com/@kapuramax/procedural-image-generation-in-go-7a57ff2e2e90#.7dxx5hzci https://medium.com/@kapuramax/procedural-image-generation-in... [2] http://www.nobledust.com/full_size/bullet.r0g0b0.x1000y1000.blur5.ch100.cpu2.png http://www.nobledust.com/full_size/bullet.r0g0b0.x1000y1000.... [3] http://www.nobledust.com/full_size/b.brg.rr0.90.blur1.ch8.cpu8.png http://www.nobledust.com/full_size/b.brg.rr0.90.blur1.ch8.cp... [4] http://www.nobledust.com/full_size/a.jpg http://www.nobledust.com/full_size/a.jpg [5] http://www.nobledust.com/reds.png http://www.nobledust.com/reds.png [6] http://www.nobledust.com/full_size/mosey.png http://www.nobledust.com/full_size/mosey.png
    - jerf 10y ago
      Looking over your code, did you ever run it single-threaded to compare the performance? I see you spawning what appears to be a crapton of goroutines, each of which performs a very small number of operations, then terminates. As cheap as goroutines are, they still have some cost to startup and shutdown, and it's not a good idea to spawn a goroutine if it's going to do roughly the same order of work inside the goroutine as the spawn & shutdown costs. This is a pessimal situation. I'm not sure if you're seeing the GC fail or if you're seeing the scheduler choke under the harsh load. If you just want nondeterminacy, I'd suggest using math/rand directly, not accidents of scheduling... which, as was just proved, can disappear as quickly as they appear. In fact it doesn't even take something as serious as a language upgrade for that to happen, that kind of indeterminancy can disappear if you so much as add a member to a struct or have another process on your system eating 100% of one CPU or something. It's not reliable, not even a little.
      - Kapura 10y ago

gtirloni 10y ago

What is a generational art program? If you don't mind me asking.

overcast 10y ago
Senior finger painting classes.
- 0xdeadbeefbabe 10y ago
  In Florida.
Kapura 10y ago
Well, in my case specifically, it was a program that generated coloured images based on a set of simple rules (namely, don't reuse the same RGB colour for any pixel & try to be as close to the colour of neighbouring pixels as possible). I wrote a brief overview around the time I finished the first version[1]. More recently, I wrote a new routine which generates several of those and then averages the RGB channel values on a per-pixel basis, and then repeatedly re-averages the original images with those average images to create more "generations." I haven't written anything up about that because the code required to sum and average pixels in a PNG is significantly less interesting. [1] https://medium.com/@kapuramax/procedural-image-generation-in-go-7a57ff2e2e90#.7dxx5hzci https://medium.com/@kapuramax/procedural-image-generation-in...
spriggan3 10y ago
s/generational/generative ?

vessenes 10y ago

That's interesting -- my use case had up to 2 minute GC pauses in 1.4, and now runs without a hiccup. For a variety of reasons, some code-side and some on Go's side. I didn't know there were use cases that were negatively affected by GC work done in the last year or two.

giovannibajo1 10y ago

Did you open a bug report with some datasets (profile dumps, etc.)? Go developers treat these kinds of regressions very very seriously. They absolutely want to avoid situations were users are pinned to an old version because of a performance regression.