12 ms·

> There’s benchmarks and code example both on this post and on the rest of the site showing this (...) I feel this sort of approach lacks honesty and fails to

> There’s benchmarks and code example both on this post and on the rest of the site showing this (...) I feel this sort of approach lacks honesty and fails to be objective. Anyone can cherry-pick a customized test where their stuff comes out as the best of class, no matter how underperforming it is. To have a proper apples-to-apples comparison, standard benchmarks are the way to go. Everything else sounds like snake oil and hand-waving.

tekknolagi 4y ago
You, above: >> Basically it’s hard to be fully precise, (...) > Not really. Just put together a benchmark and let it speak for itself. You seem a little bit split on this one :) Yes, creating a fair benchmark is hard. Yes, it's hard to get other people to evaluate their things using your benchmark. Yes, it's hard to do an apples-to-apples comparison.
- simplotek 4y ago
  > You seem a little bit split on this one :) Not really. By "putting up a benchmark" I mean an objective and verifiable standard set of performance tests that everyone interested can contribute their best effort. I means nothing if you roll out a cherry-picked ad-hoc test that compares a corner case of your best effort to a half-assed underperforming implementation of a competitor you chose because it suits you best. There are standard benchmarks out there, which were already mentioned in this discussion. It takes virtually no effort to roll out Tamagui's best effort. Not using those actually requires more effort than using them, which casts doubt over the validity of self-serving cherry-picked ad-hoc tests.
  - nwienert 4y ago
    The benchmarks are open source and adopted from competitor libraries, standard tests that style libraries have used for a long time. Nothing deceptive. The other benchmark posted tests more framework level stuff, but could be repurposed to show a more “typical app screen” with some effort.
  - redbar0n 4y ago
    The standard benchmark mentioned (krausest) is for JS frameworks. But Tamagui is not a framework. (It uses React as the framework on web, so would have similar perfomance to that in krausest.) Tamagui is a style and component library (plus optimizing compiler). Do you know of any standard benchmarks for any of those?
nwienert 4y ago
There's no such thing as a standard benchmark, but I benchmarked 8 different setups using benchmark source from two competitor libraries. If you have a better idea let me know. In the ideal you'd have to benchmark a heavy screen that you'd have to write for every competitor, and then provide timings across many different facets - initial load, runtime, total time, window resize, etc. As far as I know no one does that for anything, even backend stuff as it's just too much effort (even the TechEmpower benchmarks are micro and not like this). If anything though Tamagui will look even better there. I've open sourced the results, re-used benchmarks from rival libraries, shown my work, and even published the 3 benchmarks that don't do as well for Tamagui. It's as Apples-to-Apples as it gets and it's not measuring anything weird this is straightforward stuff.
- nicoburns 4y ago
  > There's no such thing as a standard benchmark There actually is for web frameworks https://github.com/krausest/js-framework-benchmark https://github.com/krausest/js-framework-benchmark
  - nwienert 4y ago
    Almost but not quite true same. Rendering a typical app screen is probably the “right” ACID test, this is testing something very different - insert/remove row etc. I’d love to make a benchmark someday despite my bias, maybe just a profile page or feed or something generic, and could have side tests for animations, theme changes, responsive styles, and logical styles. Those are the key areas for a style library. Tamagui is actually 10x or more at a few of them.
  - redbar0n 4y ago
    I think he meant there’s no standardized benchmark for styling and component libraries, or optimizing compilers, all of which Tamagui are. (Tamagui isn’t a framework, so wouldn’t fit into krausest benchmark, as you know).

tekknolagi 4y ago

You, above: >> Basically it’s hard to be fully precise, (...) > Not really. Just put together a benchmark and let it speak for itself. You seem a little bit split on this one :) Yes, creating a fair benchmark is hard. Yes, it's hard to get other people to evaluate their things using your benchmark. Yes, it's hard to do an apples-to-apples comparison.

simplotek 4y ago
> You seem a little bit split on this one :) Not really. By "putting up a benchmark" I mean an objective and verifiable standard set of performance tests that everyone interested can contribute their best effort. I means nothing if you roll out a cherry-picked ad-hoc test that compares a corner case of your best effort to a half-assed underperforming implementation of a competitor you chose because it suits you best. There are standard benchmarks out there, which were already mentioned in this discussion. It takes virtually no effort to roll out Tamagui's best effort. Not using those actually requires more effort than using them, which casts doubt over the validity of self-serving cherry-picked ad-hoc tests.
- nwienert 4y ago
  The benchmarks are open source and adopted from competitor libraries, standard tests that style libraries have used for a long time. Nothing deceptive. The other benchmark posted tests more framework level stuff, but could be repurposed to show a more “typical app screen” with some effort.
- redbar0n 4y ago
  The standard benchmark mentioned (krausest) is for JS frameworks. But Tamagui is not a framework. (It uses React as the framework on web, so would have similar perfomance to that in krausest.) Tamagui is a style and component library (plus optimizing compiler). Do you know of any standard benchmarks for any of those?

nwienert 4y ago

There's no such thing as a standard benchmark, but I benchmarked 8 different setups using benchmark source from two competitor libraries. If you have a better idea let me know. In the ideal you'd have to benchmark a heavy screen that you'd have to write for every competitor, and then provide timings across many different facets - initial load, runtime, total time, window resize, etc. As far as I know no one does that for anything, even backend stuff as it's just too much effort (even the TechEmpower benchmarks are micro and not like this). If anything though Tamagui will look even better there. I've open sourced the results, re-used benchmarks from rival libraries, shown my work, and even published the 3 benchmarks that don't do as well for Tamagui. It's as Apples-to-Apples as it gets and it's not measuring anything weird this is straightforward stuff.

nicoburns 4y ago
> There's no such thing as a standard benchmark There actually is for web frameworks https://github.com/krausest/js-framework-benchmark https://github.com/krausest/js-framework-benchmark
- nwienert 4y ago
  Almost but not quite true same. Rendering a typical app screen is probably the “right” ACID test, this is testing something very different - insert/remove row etc. I’d love to make a benchmark someday despite my bias, maybe just a profile page or feed or something generic, and could have side tests for animations, theme changes, responsive styles, and logical styles. Those are the key areas for a style library. Tamagui is actually 10x or more at a few of them.
- redbar0n 4y ago
  I think he meant there’s no standardized benchmark for styling and component libraries, or optimizing compilers, all of which Tamagui are. (Tamagui isn’t a framework, so wouldn’t fit into krausest benchmark, as you know).