5 ms·

"What differentiates FASTER are its cache-optimized index that achieves very high performance — up to 160 million operations per second when data fits in memory

"What differentiates FASTER are its cache-optimized index that achieves very high performance — up to 160 million operations per second when data fits in memory;" I really dislike when papers make performance claims like this in the introduction. That "160 million" number is so meaningless at face value because everything from the runtime environment to the hardware is going to play a huge role in ops. I rather see how it compares against other key-value stores across a number of configurations and use cases.

nolok 8y ago
Agreed. Hardware and software has had a fair amount of upgrades since then, but reading that I could pseudo-remember that quote by the memcache guy that goes something like "you can't tell if that or that tweak is giving you a few % more requests because you're being i/o limited by your network link anyway".
- dormando 8y ago
  close enough. heavily batched on a 48 core I can pull 50 million keys/sec over localhost. if you remove syscalls and use it as a library it should double at least. writes are another story, but they're slower because nobody asks for them to be faster.
haney 8y ago
It'd be awesome if they provided a comparison of other tools on the same hardware like, "on an AWS M4.xlarge instance we were able to achieve 160m/ops/sec when the dataset fit into memory where as redis only did X" Lacking that I agree it's a pretty meaningless stat.
- mgraczyk 8y ago
  Check out the linked paper for a detailed performance comparison.
- ddorian43 8y ago
  Don't do benchmarks on shared-host.
  - ebikelaw 8y ago
    If that’s your production environment then that’s where you should run them.
    - dymk 8y ago
      That's not why you don't run them on a shared host. It's because every other tenant on the machine is going to make run of the same benchmark unpredictable, and it will likely vary greatly through the day. Even taking multiple runs of each benchmark isn't sufficient, because you don't know the usage patterns of other tenants.
      - pvg 8y ago
        You're making it sound like performance on such hosts in unknowable which isn't really accurate. 'Multiple runs of each benchmark' is vague enough to be potentially insufficient in just about any environment to boot.
        eklavya 8y ago
        Variance matters though. Test where you can reasonably be sure about low variance.
        pvg 8y ago
        Sure. But it can be measured and accounted for. The idea that you can't or shouldn't benchmark such environments seems weird, given that they're pretty popular.
        deleted 8y ago
        [deleted]
- andrewstuart 8y ago
  No benchmark means anything on an EC2 shared instance (or probably any other cloud instance) because you don't know what else is running on the machine.
  - haney 8y ago
    What about running the benchmark multiple times on instances of the same type? I get that it would be noisy but lots of workloads run on shared instances so it’s a useful measuring stick in that way.
    - deleted 8y ago
      [deleted]
feyman_r 8y ago
Links to 'our website' on the github page follows through to this paper [1]. Haven't read through it, but I did see a graph comparing Intel TBB, RocksDB and Masstree. [1] https://www.microsoft.com/en-us/research/uploads/prod/2018/08/p809-chandramouli.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/0...
kodablah 8y ago
And: "FASTER achieves higher throughput than current systems, by more than two orders of magnitude, and scales better than current pure in-memory data structures, for in-memory working sets." Looking at 7.2 of the paper, they probably mean "more than 2x", definitely not exponentially faster in most cases. Still nice work though.
- jamesblonde 8y ago
  Two orders of magnitude is patently not true. MySQL Cluster has been benched at 200m ops/s (where operations are part of 2PC transactions!). And that in 2015! https://www.mysql.com/why-mysql/benchmarks/mysql-cluster/ https://www.mysql.com/why-mysql/benchmarks/mysql-cluster/ The echo chamber of silicon valley is an important reason why MySQL Cluster is not more popular.
  - CyberDildonics 8y ago
    As well as being owned by Oracle.
  - cliffordj 8y ago
    From your link: "This was achieved with 32 (out of a maximum 48) data nodes." The project being discussed is benchmarked on a single node, not a cluster. This is why they compare performance to RocksDB.
xjia 8y ago
So 160 million packets per second?
hedora 8y ago
The comment about being two orders of magnitude faster than other stores is also pretty annoying. Many systems perform at most 1 disk access for data sets with working sets 100’s of times larger than DRAM, and that’s essentially optimal for those workloads. It would be nice if they said what workload patterns they’ve optimized for, and which ones they perform poorly on.
imhoguy 8y ago
This. Somebody already filled an issue https://github.com/Microsoft/FASTER/issues/2 https://github.com/Microsoft/FASTER/issues/2

nolok 8y ago

Agreed. Hardware and software has had a fair amount of upgrades since then, but reading that I could pseudo-remember that quote by the memcache guy that goes something like "you can't tell if that or that tweak is giving you a few % more requests because you're being i/o limited by your network link anyway".

dormando 8y ago
close enough. heavily batched on a 48 core I can pull 50 million keys/sec over localhost. if you remove syscalls and use it as a library it should double at least. writes are another story, but they're slower because nobody asks for them to be faster.

haney 8y ago

It'd be awesome if they provided a comparison of other tools on the same hardware like, "on an AWS M4.xlarge instance we were able to achieve 160m/ops/sec when the dataset fit into memory where as redis only did X" Lacking that I agree it's a pretty meaningless stat.

mgraczyk 8y ago
Check out the linked paper for a detailed performance comparison.
ddorian43 8y ago
Don't do benchmarks on shared-host.
- ebikelaw 8y ago
  If that’s your production environment then that’s where you should run them.
  - dymk 8y ago
    That's not why you don't run them on a shared host. It's because every other tenant on the machine is going to make run of the same benchmark unpredictable, and it will likely vary greatly through the day. Even taking multiple runs of each benchmark isn't sufficient, because you don't know the usage patterns of other tenants.
    - pvg 8y ago
      You're making it sound like performance on such hosts in unknowable which isn't really accurate. 'Multiple runs of each benchmark' is vague enough to be potentially insufficient in just about any environment to boot.
      - eklavya 8y ago
        Variance matters though. Test where you can reasonably be sure about low variance.
        pvg 8y ago
        Sure. But it can be measured and accounted for. The idea that you can't or shouldn't benchmark such environments seems weird, given that they're pretty popular.
        deleted 8y ago
        [deleted]
andrewstuart 8y ago
No benchmark means anything on an EC2 shared instance (or probably any other cloud instance) because you don't know what else is running on the machine.
- haney 8y ago
  What about running the benchmark multiple times on instances of the same type? I get that it would be noisy but lots of workloads run on shared instances so it’s a useful measuring stick in that way.
  - deleted 8y ago
    [deleted]

feyman_r 8y ago

Links to 'our website' on the github page follows through to this paper [1]. Haven't read through it, but I did see a graph comparing Intel TBB, RocksDB and Masstree. [1] https://www.microsoft.com/en-us/research/uploads/prod/2018/08/p809-chandramouli.pdf https://www.microsoft.com/en-us/research/uploads/prod/2018/0...

kodablah 8y ago

And: "FASTER achieves higher throughput than current systems, by more than two orders of magnitude, and scales better than current pure in-memory data structures, for in-memory working sets." Looking at 7.2 of the paper, they probably mean "more than 2x", definitely not exponentially faster in most cases. Still nice work though.

jamesblonde 8y ago
Two orders of magnitude is patently not true. MySQL Cluster has been benched at 200m ops/s (where operations are part of 2PC transactions!). And that in 2015! https://www.mysql.com/why-mysql/benchmarks/mysql-cluster/ https://www.mysql.com/why-mysql/benchmarks/mysql-cluster/ The echo chamber of silicon valley is an important reason why MySQL Cluster is not more popular.
- CyberDildonics 8y ago
  As well as being owned by Oracle.
- cliffordj 8y ago
  From your link: "This was achieved with 32 (out of a maximum 48) data nodes." The project being discussed is benchmarked on a single node, not a cluster. This is why they compare performance to RocksDB.

xjia 8y ago

So 160 million packets per second?

hedora 8y ago

The comment about being two orders of magnitude faster than other stores is also pretty annoying. Many systems perform at most 1 disk access for data sets with working sets 100’s of times larger than DRAM, and that’s essentially optimal for those workloads. It would be nice if they said what workload patterns they’ve optimized for, and which ones they perform poorly on.

imhoguy 8y ago

This. Somebody already filled an issue https://github.com/Microsoft/FASTER/issues/2 https://github.com/Microsoft/FASTER/issues/2