7 ms·

Poop: Performance Optimizer Observation Platform

130 points by davikr 3y ago

AndyKelley 3y ago
heh, I didn't expect to see poop on the front page of hacker news. If you're curious about the poop-making process, I created it and released it all on one 6 hour twitch stream [1]. [1]: https://www.twitch.tv/videos/1845437356 https://www.twitch.tv/videos/1845437356
- devsda 3y ago
  Even with the context, this comment sounded so intriguing and wrong at the same time that I hesitated while opening the link. Watched a few mins and bookmarked it for another day. There were many other name suggestions that are good but I'm glad you kept the initial name.
  - notThrowingAway 3y ago
    Note that twitch VODs don't stay up forever, Andrew did mention that he would upload to peertube, but I don't know if he did.
- _yb2s 3y ago
  Does this need a [NSFW] tag?
  - DANmode 3y ago
    Twitch?!
- pkphilip 3y ago
  I know how to make poop, bro. Been doing it for a long time now.
- dundarious 3y ago
  Kindof off-topic, but has anyone had any success in using perf_event_open from within WSL, or is that unsupported? I tried `sysctl kernel.perf_event_paranoid=0` and the following, which works under "real" linux, but fails under WSL: struct perf_event_attr attr = { .type = PERF_TYPE_HARDWARE, .size = sizeof(struct perf_event_attr), .config = PERF_COUNT_HW_INSTRUCTIONS, .disabled = 1, /* only need to read once, not updating values, but fails with 0 also */ .exclude_kernel = 1, .exclude_hv = 1 }; int fd = perf_event_open(&attr, 0, -1, -1, 0); /* fails, returns -1 */ Microsoft's kernel appears to have the right CONFIG_* as well, so I feel like it must be supported in some capacity, even if limited.
birdiesanders 3y ago
I'll be making poop part of the CI development process starting Monday.
cmehdy 3y ago
Makes sense, it's for stuff nobody likes dealing with but everybody has to do.
warent 3y ago
Not to be rude but this source code... makes me wonder if I ever want to use Zig. It looks barely readable https://github.com/andrewrk/poop/blob/main/src/main.zig https://github.com/andrewrk/poop/blob/main/src/main.zig
- brosciencecode 3y ago
  Out of curiosity, what makes it unreadable to you?
  - slekker 3y ago
    Not OP but to me it is a bit difficult to read due to the syntax highlighting colours (mainly red and brown)
- dinglejungle 3y ago
  This might be more of an indictment on 'readable' being a meaningful concept (beyond simple personal preference) than it is of Zig's syntax.
- proto_lambda 3y ago
  I've never touched zig and know next to nothing about it, but that all looks reasonable to me at a glance. What exaxtly do you find so bad?
- LanternLight83 3y ago
  I've never used Zig or C++, mostly Bash, Python, and Lisps, but have recently began learning C. I see C++ in search results, and it looks like C with higher level features in a register that reminds me of Java. I see this, and it looks similer: C with higher level features that remind me more of Python. I'm sure it's got other important features and priorities that set it apart (oft' discussed here), but that's what I see, and I dig it. To be clear there are definitely a few ideoms I'd need to learn before considering this readable (it ain't psudocode), but I generally see all the building blocks and recognize what they're achiving. Maybe I'd have put them together slightly differently, but that could be equally true in Python. I'd rather learn this next than C++. The density of how it's been written reminds me of some lisp code I've read, some implementations of state machines can be hard to follow too (in the classic "but where does it do the thing?" sense). I attribute a lot of the stuff that I don't follow to the lower-level C influence; I'm working for the first time with memory managment, structs, static types, pointers, bitwise opperations, etc. Think I made my point in there somewhere >z> Would love to know where you're coming at it from, I'd imagine that the C or 'kinda functional control flow' aspects might explain what looks most forign to you too (unless you have broader experience with low level languages)?
j-pb 3y ago
Nobody should do benchmarking like this anymore. Memory layout of your code has such a big impact on performance on modern computers that measuring performance without removing that variable leads to wild goose chases, where you think you improved something, but in reality you incidentally got the compiler to move the code around a bit. Emery Berger has an excelent talk on this [1], and a causal profiler that they developed called Coz[2]. Branch miss-predicts might be somewhat invariant to that, but still, one of the main points of the talk is that people do too much eyeball statistics, mistaking the variance of the underlying stochastic process with actual signal. Coz is pretty trivial to set up with Zig too. 1. https://m.youtube.com/watch?v=r-TLSBdHe1A https://m.youtube.com/watch?v=r-TLSBdHe1A 2. https://github.com/plasma-umass/coz https://github.com/plasma-umass/coz
- ororroro 3y ago
  Sometimes you just need to check Poop to make sure your apps still healthy.
  - deleted 3y ago
    [deleted]
- krona 3y ago
  Causal profiling still uses hypothesis testing, so I don't get your point. Ultimately, one needs to know if software A is 'better' or 'worse' than software A' for any given definition of those terms.
- adrian_b 3y ago
  Coz appears to use a very good method for profiling multi-threaded applications, but which seems to be not applicable to single-threaded applications and I do not see how it would help to avoid the variability caused by different memory layouts, which, as you rightly point out, may confound profiling results.
  - dundarious 3y ago
    Yes, just because it's from a team connected with the old Stabilizer tool, it doesn't mean it shares the insights and benefits of that kind of testing. At least, that's my understanding based on my limited knowledge of coz, and a little bit more knowledge of https://github.com/ccurtsinger/stabilizer https://github.com/ccurtsinger/stabilizer
CrampusDestrus 3y ago
Poop: it's the shit!
- MeteorMarc 3y ago
  The name poop does not hold in the intercultural sense. Indeed, in northern dutch poep means shit. In southern dutch poepen means having sexual intercourse.
  - sverhagen 3y ago
    Southern Dutch as in the North of Belgium, right?
- thawab 3y ago
  Fun fact: the platform is built using zig, which means poop in Arabic.
- yashBonde 3y ago
  It's just not working, I `poop`ed all day long!
ftxbro 3y ago
> "The MIT License (Expat) Copyright (c) Poop Contributors Permission is hereby granted..."
aprasadh 3y ago
should I use fork or spoon the poop? how I do merge my changes to poop? I don't want to make a pull request.
nine_k 3y ago
A perfect way to prevent adoption in larger and more serious companies, but become poopular among younger audience.
- 6510 3y ago
  how young? I'm having a hard time containing my inner 5 year old. Apparently he has a poopulence of jokes ready for the making.
- mcdonje 3y ago
  A company refusing to use the most appropriate tool because of the name is much more offensive to me than a tool with a childish name.
- devsda 3y ago
  Slim chance that it really gains wider adoption and becomes a standard tool but it will be an interesting conversation starter for sure. Imagine you are struggling with performance of some tool and a more knowledgeable senior comes in and says "you need to use poop" or "just poop it!".
  - 6510 3y ago
    Who knows, peristaltics pushed it to the front page already.
mpweiher 3y ago
Big POOP by Sam Adams http://www.youtube.com/watch?v=GBtqQwcJoN0 http://www.youtube.com/watch?v=GBtqQwcJoN0
pacoverdi 3y ago
(phonetically) related: weboob https://pypi.org/project/weboob/ https://pypi.org/project/weboob/
- matheusmoreira 3y ago
  Such a cool project. Essentially a collection of custom HTTP clients for every supported web site.
- quesera 3y ago
  They've apparently renamed to Woob: https://woob.tech/ https://woob.tech/ > Woob (Web Outside of Browsers) is a library which provides a Python standardized API and data models to access websites. Interesting idea -- basically an API on top of a passel of site-specific web scraper modules.
sirwitti 3y ago
Upvoted just for the name and acronym!

130 points by davikr 3y ago

heh, I didn't expect to see poop on the front page of hacker news. If you're curious about the poop-making process, I created it and released it all on one 6 hour twitch stream [1]. [1]: https://www.twitch.tv/videos/1845437356 https://www.twitch.tv/videos/1845437356

devsda 3y ago
Even with the context, this comment sounded so intriguing and wrong at the same time that I hesitated while opening the link. Watched a few mins and bookmarked it for another day. There were many other name suggestions that are good but I'm glad you kept the initial name.
- notThrowingAway 3y ago
  Note that twitch VODs don't stay up forever, Andrew did mention that he would upload to peertube, but I don't know if he did.
_yb2s 3y ago
Does this need a [NSFW] tag?
- DANmode 3y ago
  Twitch?!
pkphilip 3y ago
I know how to make poop, bro. Been doing it for a long time now.
dundarious 3y ago
Kindof off-topic, but has anyone had any success in using perf_event_open from within WSL, or is that unsupported? I tried `sysctl kernel.perf_event_paranoid=0` and the following, which works under "real" linux, but fails under WSL: struct perf_event_attr attr = { .type = PERF_TYPE_HARDWARE, .size = sizeof(struct perf_event_attr), .config = PERF_COUNT_HW_INSTRUCTIONS, .disabled = 1, /* only need to read once, not updating values, but fails with 0 also */ .exclude_kernel = 1, .exclude_hv = 1 }; int fd = perf_event_open(&attr, 0, -1, -1, 0); /* fails, returns -1 */ Microsoft's kernel appears to have the right CONFIG_* as well, so I feel like it must be supported in some capacity, even if limited.

birdiesanders 3y ago

I'll be making poop part of the CI development process starting Monday.

cmehdy 3y ago

Makes sense, it's for stuff nobody likes dealing with but everybody has to do.

warent 3y ago

Not to be rude but this source code... makes me wonder if I ever want to use Zig. It looks barely readable https://github.com/andrewrk/poop/blob/main/src/main.zig https://github.com/andrewrk/poop/blob/main/src/main.zig

brosciencecode 3y ago
Out of curiosity, what makes it unreadable to you?
- slekker 3y ago
  Not OP but to me it is a bit difficult to read due to the syntax highlighting colours (mainly red and brown)
dinglejungle 3y ago
This might be more of an indictment on 'readable' being a meaningful concept (beyond simple personal preference) than it is of Zig's syntax.
proto_lambda 3y ago
I've never touched zig and know next to nothing about it, but that all looks reasonable to me at a glance. What exaxtly do you find so bad?
LanternLight83 3y ago
I've never used Zig or C++, mostly Bash, Python, and Lisps, but have recently began learning C. I see C++ in search results, and it looks like C with higher level features in a register that reminds me of Java. I see this, and it looks similer: C with higher level features that remind me more of Python. I'm sure it's got other important features and priorities that set it apart (oft' discussed here), but that's what I see, and I dig it. To be clear there are definitely a few ideoms I'd need to learn before considering this readable (it ain't psudocode), but I generally see all the building blocks and recognize what they're achiving. Maybe I'd have put them together slightly differently, but that could be equally true in Python. I'd rather learn this next than C++. The density of how it's been written reminds me of some lisp code I've read, some implementations of state machines can be hard to follow too (in the classic "but where does it do the thing?" sense). I attribute a lot of the stuff that I don't follow to the lower-level C influence; I'm working for the first time with memory managment, structs, static types, pointers, bitwise opperations, etc. Think I made my point in there somewhere >z> Would love to know where you're coming at it from, I'd imagine that the C or 'kinda functional control flow' aspects might explain what looks most forign to you too (unless you have broader experience with low level languages)?

j-pb 3y ago

Nobody should do benchmarking like this anymore. Memory layout of your code has such a big impact on performance on modern computers that measuring performance without removing that variable leads to wild goose chases, where you think you improved something, but in reality you incidentally got the compiler to move the code around a bit. Emery Berger has an excelent talk on this [1], and a causal profiler that they developed called Coz[2]. Branch miss-predicts might be somewhat invariant to that, but still, one of the main points of the talk is that people do too much eyeball statistics, mistaking the variance of the underlying stochastic process with actual signal. Coz is pretty trivial to set up with Zig too. 1. https://m.youtube.com/watch?v=r-TLSBdHe1A https://m.youtube.com/watch?v=r-TLSBdHe1A 2. https://github.com/plasma-umass/coz https://github.com/plasma-umass/coz

ororroro 3y ago
Sometimes you just need to check Poop to make sure your apps still healthy.
- deleted 3y ago
  [deleted]
krona 3y ago
Causal profiling still uses hypothesis testing, so I don't get your point. Ultimately, one needs to know if software A is 'better' or 'worse' than software A' for any given definition of those terms.
adrian_b 3y ago
Coz appears to use a very good method for profiling multi-threaded applications, but which seems to be not applicable to single-threaded applications and I do not see how it would help to avoid the variability caused by different memory layouts, which, as you rightly point out, may confound profiling results.
- dundarious 3y ago
  Yes, just because it's from a team connected with the old Stabilizer tool, it doesn't mean it shares the insights and benefits of that kind of testing. At least, that's my understanding based on my limited knowledge of coz, and a little bit more knowledge of https://github.com/ccurtsinger/stabilizer https://github.com/ccurtsinger/stabilizer

CrampusDestrus 3y ago

Poop: it's the shit!

MeteorMarc 3y ago
The name poop does not hold in the intercultural sense. Indeed, in northern dutch poep means shit. In southern dutch poepen means having sexual intercourse.
- sverhagen 3y ago
  Southern Dutch as in the North of Belgium, right?
thawab 3y ago
Fun fact: the platform is built using zig, which means poop in Arabic.
yashBonde 3y ago
It's just not working, I `poop`ed all day long!

ftxbro 3y ago

> "The MIT License (Expat) Copyright (c) Poop Contributors Permission is hereby granted..."

aprasadh 3y ago

should I use fork or spoon the poop? how I do merge my changes to poop? I don't want to make a pull request.

nine_k 3y ago

A perfect way to prevent adoption in larger and more serious companies, but become poopular among younger audience.

6510 3y ago
how young? I'm having a hard time containing my inner 5 year old. Apparently he has a poopulence of jokes ready for the making.
mcdonje 3y ago
A company refusing to use the most appropriate tool because of the name is much more offensive to me than a tool with a childish name.
devsda 3y ago
Slim chance that it really gains wider adoption and becomes a standard tool but it will be an interesting conversation starter for sure. Imagine you are struggling with performance of some tool and a more knowledgeable senior comes in and says "you need to use poop" or "just poop it!".
- 6510 3y ago
  Who knows, peristaltics pushed it to the front page already.

mpweiher 3y ago

Big POOP by Sam Adams http://www.youtube.com/watch?v=GBtqQwcJoN0 http://www.youtube.com/watch?v=GBtqQwcJoN0

pacoverdi 3y ago

(phonetically) related: weboob https://pypi.org/project/weboob/ https://pypi.org/project/weboob/

matheusmoreira 3y ago
Such a cool project. Essentially a collection of custom HTTP clients for every supported web site.
quesera 3y ago
They've apparently renamed to Woob: https://woob.tech/ https://woob.tech/ > Woob (Web Outside of Browsers) is a library which provides a Python standardized API and data models to access websites. Interesting idea -- basically an API on top of a passel of site-specific web scraper modules.

sirwitti 3y ago

Upvoted just for the name and acronym!