4 ms·

I'd be happy if the developers were given an "average" device for testing, and not the newest-shiniest-flagship one. I get it why for development it's benefici

I'd be happy if the developers were given an "average" device for testing, and not the newest-shiniest-flagship one. I get it why for development it's beneficial for a test device to be as fast as possible (because there less time is wasted for (eg.) an app to start after a change in code), but a lot of modern applications don't do more than they did 10 years ago, but eat up a lot more resources for the same output.

muskox2 6y ago
I find this is true for phones also (or even moreso). Google maps is a bad offender, getting slower every year despite doing essentially the same thing.
iudqnolq 6y ago
Google (allegedly, in public) does this, although for some NIH-smelling reason they've invented their own representative low-end phone instead of just buying some.
pitay 6y ago
I'd like it if they tested their distributable product on a hard drive sometimes, not just SSD all the time. I've come across a game that had major stutter problems, very noticeable on a HDD but still noticeable on SSD, just noticeable on a ramdisk, problem was frequent saving level data to disk and having some things block while they did that. The developer got lots of reports about performance, they said that people should upgrade their computer, game was 2d and graphically simple, the developer did not understand why people were having performance problems and didn't really care, it is essentially abandoned at this point and still for sale. Another game also had major stutter problems which would last a few seconds each, turns out it was reading a file (which should be read once at startup) over 100 times a second almost all the time it is open. Submitted bug reports in both places they support with log output from the file monitoring I did. No response to either of them. Firefox generally runs fine, if the HDD where your profile is located is saturated with IO then pages will not load until Firefox is able to write to the disk, this is with disk cache turned off. This isn't unexpected behaviour and not a bug like I would consider the previous two examples, but it is noticeable. I really would like developers to test on a HDD even if only every few releases, and would also like them to use a monitoring tool on their software sometimes. This is just so their products don't have really bad behaviour on the system it is running on which the developers are completely oblivious to.
klelatti 6y ago
I've been intentionally developing a data science type app - one where run times are critical - on a 12' MacBook. I know run times will feel great on a 6 Core i7 machine for my fairly simple test cases. With the MacBook I get immediate feedback if something I've done creates a slowdown.
arnaudsm 6y ago
I kept my 10 yo laptop only for testing my apps on regular consumer hardware, not my typical $3k overkill dev machine.
whatshisface 6y ago
Another reason why you want a blazing fast test device is that running with certain debugging and profiling tools attached can degrade performance substantially.
penagwin 6y ago
I definitely agree things should be tested on realistic hardware - but one of the big reasons for me at least is using a decked out machine is because of all the dev tools + browser tabs, etc. IntelliJ might be using 6GB, chrome another 6GB, 6GB for teams/discord/figma/gitkraken/vscode/spotify. That's not including any image editing software or vm's but I'm already needing over 16GB of ram.
mamcx 6y ago
Exist an incredible good reason: Because MOST of the tools we must/need to use have atrocious performance. Not bad. Terrible. I use Rust, and I like as language. But is terrible experience for compile times. I know what a good one is (Delphi, Pascal) and after it no single compile time language get it close (with exception of GO). The terrible thing is that the base of all is C/C++/JS and their toolchain is the SLOW incarnate. Then even if I dodge the use of them, I still get impacted, eventually. For example, I use TailwindCSS... that is great. But it use node. Node is terrible. I must use the speedier machine I can pay because soon or later I will deal with node, c++/llvm/c, Android Studio (damm!) Android emulator (mega damm!) and other tools like that. And the tricky part ("why not just not use them?") is that they are terrible at performance but maybe good for something else (or: I must use them... I'm contractor!). So only IF I have total control and can supply my needs with better tools I could use a old machine of the past. But I can't.
- guitarbill 6y ago
  It would be awesome if Rust were faster, of course. At least with Rust, you get the borrow checker, not just compiling. It feels like you're getting something more in return (even though I don't know which step of compilation actually takes the most time). But that would still suck. Luckily, there is a solution that works in many cases, rust-analyser. Obviously having basic syntax checking in an editor is preferable, because it shortens the feedback loop. I think this is why a lot of IDEs do it, not just for Rust. It's still a work-around, but makes it more bearable. I'm not hugely experienced in Rust development, but it would be awesome if cargo could compile files when they are saved, so incremental build times and running tests might feel shorter. You could still get into trouble with complex dependencies between code units, but better > perfect. Visual Studio's "Live Unit Testing" takes this to an extreme (yeah, VS itself has poor performance, but doesn't mean the idea is bad). Basically, if tools must have bad performance, because they are doing a lot of work for us (?), then at least make the development experience not suffer from that bad performance. That's what I care about more than just perf, it's how it impacts me, right? In general, I still think compiling provides a decent trade-off. Yes, I need a beefier machine, and yes, my dev cycle is just a bit longer. But people using the actual product, they get the benefits. You could go with e.g. Python, where maybe one or two tests take less time to run, but ultimately you'd just be shifting the burden somewhere else. It isn't much consolation in the moment, but worth remembering.
Rule35 6y ago
One thing I saw was someone logging all system calls their software generated and comparing that between runs. They'd been bitten by some minor library upgrade that had changed directory scanning which caused huge multiples of normal latency which on an SDD wasn't noticeable but on an HDD went from trivial to insane. That was handy because it didn't actually involve testing on a slow device but mostly showed what we needed to know. There are tools like https://github.com/schoentoon/slowpokefs https://github.com/schoentoon/slowpokefs which try to make your fast storage act like a HDD which could be better for black-box testing.
habosa 6y ago
It would be nice if it was easier to have a setup which is fast for development but realistically slow for testing. I develop my personal website on a 6-year-old Windows laptop which was only $750 at the time. It definitely keeps me from making a slow website! But it also makes my devserver much slower than I'd like. I wish I could have speedy tools but a slow browser (and no, Chrome's CPU throttling isn't really the same as a truly crappy computer)

muskox2 6y ago

I find this is true for phones also (or even moreso). Google maps is a bad offender, getting slower every year despite doing essentially the same thing.

iudqnolq 6y ago

Google (allegedly, in public) does this, although for some NIH-smelling reason they've invented their own representative low-end phone instead of just buying some.

pitay 6y ago

I'd like it if they tested their distributable product on a hard drive sometimes, not just SSD all the time. I've come across a game that had major stutter problems, very noticeable on a HDD but still noticeable on SSD, just noticeable on a ramdisk, problem was frequent saving level data to disk and having some things block while they did that. The developer got lots of reports about performance, they said that people should upgrade their computer, game was 2d and graphically simple, the developer did not understand why people were having performance problems and didn't really care, it is essentially abandoned at this point and still for sale. Another game also had major stutter problems which would last a few seconds each, turns out it was reading a file (which should be read once at startup) over 100 times a second almost all the time it is open. Submitted bug reports in both places they support with log output from the file monitoring I did. No response to either of them. Firefox generally runs fine, if the HDD where your profile is located is saturated with IO then pages will not load until Firefox is able to write to the disk, this is with disk cache turned off. This isn't unexpected behaviour and not a bug like I would consider the previous two examples, but it is noticeable. I really would like developers to test on a HDD even if only every few releases, and would also like them to use a monitoring tool on their software sometimes. This is just so their products don't have really bad behaviour on the system it is running on which the developers are completely oblivious to.

klelatti 6y ago

I've been intentionally developing a data science type app - one where run times are critical - on a 12' MacBook. I know run times will feel great on a 6 Core i7 machine for my fairly simple test cases. With the MacBook I get immediate feedback if something I've done creates a slowdown.

arnaudsm 6y ago

I kept my 10 yo laptop only for testing my apps on regular consumer hardware, not my typical $3k overkill dev machine.

whatshisface 6y ago

Another reason why you want a blazing fast test device is that running with certain debugging and profiling tools attached can degrade performance substantially.

penagwin 6y ago

I definitely agree things should be tested on realistic hardware - but one of the big reasons for me at least is using a decked out machine is because of all the dev tools + browser tabs, etc. IntelliJ might be using 6GB, chrome another 6GB, 6GB for teams/discord/figma/gitkraken/vscode/spotify. That's not including any image editing software or vm's but I'm already needing over 16GB of ram.

mamcx 6y ago

Exist an incredible good reason: Because MOST of the tools we must/need to use have atrocious performance. Not bad. Terrible. I use Rust, and I like as language. But is terrible experience for compile times. I know what a good one is (Delphi, Pascal) and after it no single compile time language get it close (with exception of GO). The terrible thing is that the base of all is C/C++/JS and their toolchain is the SLOW incarnate. Then even if I dodge the use of them, I still get impacted, eventually. For example, I use TailwindCSS... that is great. But it use node. Node is terrible. I must use the speedier machine I can pay because soon or later I will deal with node, c++/llvm/c, Android Studio (damm!) Android emulator (mega damm!) and other tools like that. And the tricky part ("why not just not use them?") is that they are terrible at performance but maybe good for something else (or: I must use them... I'm contractor!). So only IF I have total control and can supply my needs with better tools I could use a old machine of the past. But I can't.

guitarbill 6y ago
It would be awesome if Rust were faster, of course. At least with Rust, you get the borrow checker, not just compiling. It feels like you're getting something more in return (even though I don't know which step of compilation actually takes the most time). But that would still suck. Luckily, there is a solution that works in many cases, rust-analyser. Obviously having basic syntax checking in an editor is preferable, because it shortens the feedback loop. I think this is why a lot of IDEs do it, not just for Rust. It's still a work-around, but makes it more bearable. I'm not hugely experienced in Rust development, but it would be awesome if cargo could compile files when they are saved, so incremental build times and running tests might feel shorter. You could still get into trouble with complex dependencies between code units, but better > perfect. Visual Studio's "Live Unit Testing" takes this to an extreme (yeah, VS itself has poor performance, but doesn't mean the idea is bad). Basically, if tools must have bad performance, because they are doing a lot of work for us (?), then at least make the development experience not suffer from that bad performance. That's what I care about more than just perf, it's how it impacts me, right? In general, I still think compiling provides a decent trade-off. Yes, I need a beefier machine, and yes, my dev cycle is just a bit longer. But people using the actual product, they get the benefits. You could go with e.g. Python, where maybe one or two tests take less time to run, but ultimately you'd just be shifting the burden somewhere else. It isn't much consolation in the moment, but worth remembering.

Rule35 6y ago

One thing I saw was someone logging all system calls their software generated and comparing that between runs. They'd been bitten by some minor library upgrade that had changed directory scanning which caused huge multiples of normal latency which on an SDD wasn't noticeable but on an HDD went from trivial to insane. That was handy because it didn't actually involve testing on a slow device but mostly showed what we needed to know. There are tools like https://github.com/schoentoon/slowpokefs https://github.com/schoentoon/slowpokefs which try to make your fast storage act like a HDD which could be better for black-box testing.

habosa 6y ago

It would be nice if it was easier to have a setup which is fast for development but realistically slow for testing. I develop my personal website on a 6-year-old Windows laptop which was only $750 at the time. It definitely keeps me from making a slow website! But it also makes my devserver much slower than I'd like. I wish I could have speedy tools but a slow browser (and no, Chrome's CPU throttling isn't really the same as a truly crappy computer)