4 ms·

Interesting, but I'd think that just running your profiling in a server environment would have the same effect as doing everything in one, no? And for much less

by _bxg1 6y ago

Interesting, but I'd think that just running your profiling in a server environment would have the same effect as doing everything in one, no? And for much less hassle

Analemma_ 6y ago
You'd be amazed how few developers even know about profiling, or think it's a method of last resort when other attempts to track down performance issues have failed. One of my standard interview questions is, "A customer comes to you and says that [app] 'is slow', with no additional information". You look at the code and there are no glaring inefficiencies, no O(n!) algorithms, etc. What's your next step?", and less than a quarter of candidates ever mention profiling. "Sympathy for the hardware" is, IMO, sort of a hack to be an imperfect solution to this problem.
- Ididntdothis 6y ago
  I see the same . People will do anything to avoid using a profiler. It's really weird.
  - deleted 6y ago
    [deleted]
  - thaumasiotes 6y ago
    People will try to avoid using tools they aren't already using. You see the same thing with debuggers.
- javajosh 6y ago
  I also see this, and I think its worse than that because so many devs concentrate only on the static expression of the code and have little understanding of what things look like at runtime. Ironically, client-side coders are much better at this, thanks to excellent tools. Speaking of tools, server-side profiling is almost always weird and it pays to have someone show you the ropes. (Java has some of the best profiling tools, interestingly).
- scott00 6y ago
  Is profiling actually a good answer to that question? Profiling is very useful for improving the performance of software under a particular workload, but unless you have very good non-anonymized telemetry anything other than trying to get more information about the customer's situation is going to be a wild goose chase. I would definitely start with talking to the customer to get more details, and ideally observe the customer using the app before I got anywhere near a profiler.
  - paranoidrobot 6y ago
    Profiling is really useful for any moderately complex application, particularly where you have dependencies upon a bunch of third party code. While your code may look all nice and sane, not have any glaring performance issues - what you may not realise is that you're using a framework that does things in unexpected ways. The devs I work with are running into the reality of this at the moment - they blame the servers/network/database/whatever as being slow/shit/etc. I go take a look and see that network traffic is at capacity. They've got some 'magic' caching framework that promises to speed things up by storing data in Redis. What's happening in the background though is that this caching framework is doing dumb things like asking for the same key over and over again - so we've got a multi-Gbit stream of the same handful of keys being asked for thousands of times a second. These are the sorts of things profiling can tell you just by looking at the amount of calls to Redis.
    - deleted 6y ago
      [deleted]
- jeffbee 6y ago
  The upside is if you know how to use the profiler, especially if you know a lot about PMUs and whatnot, you'll look like a damned wizard to many people.
  - tehjoker 6y ago
    PMU = Performance Monitor Unit. Never heard of the term before. Thanks!
- bluedino 6y ago
  We had a customer who complained about a part of our site being very slow, it drew a bunch of Canvas stuff back in like 2011. Eventually we got them online to do a screen share, and sure enough it took over a minute to render the page (it was a map). They were using an 8 year old CPU. Switching them from IE6 to Firefox fixed the issue.
  - SilasX 6y ago
    I'm glad you didn't have to upgrade their hardware!
  - bdcravens 6y ago
    Had a similar issue, but switching browsers wasn't an option. This was in a hospital environment in 2002: slow computers, running IE as the standard. A large form with a ton of HTML select dropdowns totally killed performance. Had to rewrite the thing to use just a few (dynamically generating options, before heavy Javascript apps were common), which fixed things.
- devonkim 6y ago
  The reality is that a lot of profilers cause a sort of Heisenberg effect when running in production where it can slow down your code so much that it's not as meaningful anymore. This is now the time your ops engineers like me will be wagging their finger saying the application should have had instrumentation or APM support built-in like via OpenTracing or that ebpf and friends could be useful if your production machines were reasonably up to date. In most occasions I've seen, the majority of engineering teams outside massive scale companies with gobs of resources even today wind up debugging performance problems like it's 1999 with application-external hypotheses and checking for smoking guns like high page fault rates, dropped packets, etc. (eg. USE methodology).
  - jeffbee 6y ago
    Sampling cycle count or LBR with linux perf events is almost invisible to performance, maybe a 1% hit to throughput as a general rule. The problems come from languages where the PC is irrelevant, like Python, but nobody uses Python because it's fast, so using a python profiler like pyflame should be fine, even in production.
    - devonkim 6y ago
      Profiling in production with the JVM using something like YourKit or JProfiler is the typical case for myself. Ironically, I've found profiling with Python in production easier for the reasons you've mentioned. If something is down or running slowly already, adding another 3%+ latency is hardly going to be an issue. Architecturally, with big monolithic programs that do too many things attaching a profiler to try to analyze 1% of the program's responsibilities or surface area becomes a risk to other production operations unfortunately. In most cases slowdowns happen because of resource saturation, things timing out, blocking on shared resources. In the first scenario, trying to run a profiler can exacerbate the problem or even fail to start, so the only way forensics can be done there is by emitting observability data prior to the failure point. Other approaches taken have been the more Erlang style "let it fail" methodology which is fine for newer projects but represents a rewrite for most systems in practice and is thus far, far beyond profiling discussions.
  - 6y ago
kasey_junk 6y ago
For performance centric uses you tend to run benchmarks and performance tests a lot. Having production quality hardware for the dev cycle is extremely valuable for these uses.
- cowsandmilk 6y ago
  Sure, but my IDE doesn’t have be running there. Code compiles and is benchmarked on prod hardware (and OS environment), I write code on machine designed for my productivity.
- gav 6y ago
  Performance testing is hugely valuable but nothing beats having performance metrics captured and made available from production. You never really know how things are going to perform until you see them under real load against real data. There’s been many times I’ve been bitten by this, including the one time we found that a small portion of users would set the number of results returned to the max and scroll through hundreds of pages of results, blowing the cache for everyone. I tend to hear “we run performance tests so we don’t get regressions” a lot more than “we monitor the performance of releases to canaries and rollback”. It shouldn’t be one or the other, both are invaluable.
  - ColanR 6y ago
    > small portion of users would set the number of results returned to the max and scroll through hundreds of pages of results Yup, that sounds like me. I do exactly that whenever the pages are slow to load. Rather wait longer, once.
- _bxg1 6y ago
  I'm not saying remove it from the dev cycle; you can stand up a remote server for whatever testing happens during your normal workflow without a) buying a physical server box to use as your workstation, or b) constraining yourself to working via a terminal.
bluedino 6y ago
Servers (especially VPS) can be incredibly inconsistent with performance

by _bxg1 6y ago

Interesting, but I'd think that just running your profiling in a server environment would have the same effect as doing everything in one, no? And for much less hassle

Analemma_ 6y ago

You'd be amazed how few developers even know about profiling, or think it's a method of last resort when other attempts to track down performance issues have failed. One of my standard interview questions is, "A customer comes to you and says that [app] 'is slow', with no additional information". You look at the code and there are no glaring inefficiencies, no O(n!) algorithms, etc. What's your next step?", and less than a quarter of candidates ever mention profiling. "Sympathy for the hardware" is, IMO, sort of a hack to be an imperfect solution to this problem.

Ididntdothis 6y ago
I see the same . People will do anything to avoid using a profiler. It's really weird.
- deleted 6y ago
  [deleted]
- thaumasiotes 6y ago
  People will try to avoid using tools they aren't already using. You see the same thing with debuggers.
javajosh 6y ago
I also see this, and I think its worse than that because so many devs concentrate only on the static expression of the code and have little understanding of what things look like at runtime. Ironically, client-side coders are much better at this, thanks to excellent tools. Speaking of tools, server-side profiling is almost always weird and it pays to have someone show you the ropes. (Java has some of the best profiling tools, interestingly).
scott00 6y ago
Is profiling actually a good answer to that question? Profiling is very useful for improving the performance of software under a particular workload, but unless you have very good non-anonymized telemetry anything other than trying to get more information about the customer's situation is going to be a wild goose chase. I would definitely start with talking to the customer to get more details, and ideally observe the customer using the app before I got anywhere near a profiler.
- paranoidrobot 6y ago
  Profiling is really useful for any moderately complex application, particularly where you have dependencies upon a bunch of third party code. While your code may look all nice and sane, not have any glaring performance issues - what you may not realise is that you're using a framework that does things in unexpected ways. The devs I work with are running into the reality of this at the moment - they blame the servers/network/database/whatever as being slow/shit/etc. I go take a look and see that network traffic is at capacity. They've got some 'magic' caching framework that promises to speed things up by storing data in Redis. What's happening in the background though is that this caching framework is doing dumb things like asking for the same key over and over again - so we've got a multi-Gbit stream of the same handful of keys being asked for thousands of times a second. These are the sorts of things profiling can tell you just by looking at the amount of calls to Redis.
  - deleted 6y ago
    [deleted]
jeffbee 6y ago
The upside is if you know how to use the profiler, especially if you know a lot about PMUs and whatnot, you'll look like a damned wizard to many people.
- tehjoker 6y ago
  PMU = Performance Monitor Unit. Never heard of the term before. Thanks!
bluedino 6y ago
We had a customer who complained about a part of our site being very slow, it drew a bunch of Canvas stuff back in like 2011. Eventually we got them online to do a screen share, and sure enough it took over a minute to render the page (it was a map). They were using an 8 year old CPU. Switching them from IE6 to Firefox fixed the issue.
- SilasX 6y ago
  I'm glad you didn't have to upgrade their hardware!
- bdcravens 6y ago
  Had a similar issue, but switching browsers wasn't an option. This was in a hospital environment in 2002: slow computers, running IE as the standard. A large form with a ton of HTML select dropdowns totally killed performance. Had to rewrite the thing to use just a few (dynamically generating options, before heavy Javascript apps were common), which fixed things.
devonkim 6y ago
The reality is that a lot of profilers cause a sort of Heisenberg effect when running in production where it can slow down your code so much that it's not as meaningful anymore. This is now the time your ops engineers like me will be wagging their finger saying the application should have had instrumentation or APM support built-in like via OpenTracing or that ebpf and friends could be useful if your production machines were reasonably up to date. In most occasions I've seen, the majority of engineering teams outside massive scale companies with gobs of resources even today wind up debugging performance problems like it's 1999 with application-external hypotheses and checking for smoking guns like high page fault rates, dropped packets, etc. (eg. USE methodology).
- jeffbee 6y ago
  Sampling cycle count or LBR with linux perf events is almost invisible to performance, maybe a 1% hit to throughput as a general rule. The problems come from languages where the PC is irrelevant, like Python, but nobody uses Python because it's fast, so using a python profiler like pyflame should be fine, even in production.
  - devonkim 6y ago
    Profiling in production with the JVM using something like YourKit or JProfiler is the typical case for myself. Ironically, I've found profiling with Python in production easier for the reasons you've mentioned. If something is down or running slowly already, adding another 3%+ latency is hardly going to be an issue. Architecturally, with big monolithic programs that do too many things attaching a profiler to try to analyze 1% of the program's responsibilities or surface area becomes a risk to other production operations unfortunately. In most cases slowdowns happen because of resource saturation, things timing out, blocking on shared resources. In the first scenario, trying to run a profiler can exacerbate the problem or even fail to start, so the only way forensics can be done there is by emitting observability data prior to the failure point. Other approaches taken have been the more Erlang style "let it fail" methodology which is fine for newer projects but represents a rewrite for most systems in practice and is thus far, far beyond profiling discussions.
- 6y ago

kasey_junk 6y ago

For performance centric uses you tend to run benchmarks and performance tests a lot. Having production quality hardware for the dev cycle is extremely valuable for these uses.

cowsandmilk 6y ago
Sure, but my IDE doesn’t have be running there. Code compiles and is benchmarked on prod hardware (and OS environment), I write code on machine designed for my productivity.
gav 6y ago
Performance testing is hugely valuable but nothing beats having performance metrics captured and made available from production. You never really know how things are going to perform until you see them under real load against real data. There’s been many times I’ve been bitten by this, including the one time we found that a small portion of users would set the number of results returned to the max and scroll through hundreds of pages of results, blowing the cache for everyone. I tend to hear “we run performance tests so we don’t get regressions” a lot more than “we monitor the performance of releases to canaries and rollback”. It shouldn’t be one or the other, both are invaluable.
- ColanR 6y ago
  > small portion of users would set the number of results returned to the max and scroll through hundreds of pages of results Yup, that sounds like me. I do exactly that whenever the pages are slow to load. Rather wait longer, once.
_bxg1 6y ago
I'm not saying remove it from the dev cycle; you can stand up a remote server for whatever testing happens during your normal workflow without a) buying a physical server box to use as your workstation, or b) constraining yourself to working via a terminal.

bluedino 6y ago

Servers (especially VPS) can be incredibly inconsistent with performance