3 ms·

I have a hard time understanding the reasoning for this. What argument do they have as to why people shouldn't know about the performance?

by Drakim 11y ago

KaiserPro 11y ago
Because VMware is generally quite slow. However it is very easy to manage large estates of machines using the out of the box tools. Unlike Xen or KVM
- rodgerd 11y ago
  oVirt would like a word with you.
  - tomaac 11y ago
    and proxmox
wila 11y ago
The reasoning is that it is so easy to make mistakes in performance testing and then draw the wrong conclusion. As somebody else mentions, this is testing NFSv3 in the VMware setup against local disk in bhyve. While still interesting, it certainly is not a 1 on 1 comparison. edit: to expand a bit on that. - the VMware storage appliance gets 24GB, the bhyve config isn't mentioned, but the host OS can access 32 GB. - as he does mention his test load fits in the ARC, again interesting, but normally you would test real-world loads where your workload does not completely fit in the cache.
- Drakim 11y ago
  Sure, I mean, but, anybody can say that. "We'd rather not you publish your review of our product, it's so easy for the reviewer to make a mistake and say something negative about our product. So, you are not allowed to publish any examination of our product."
  - wila 11y ago
    You ARE allowed to post test results, but they like to review your tests to make sure you didn't make any obvious mistakes. Quote from the .pdf file higher up in this thread: <quote> You may use the Software to conduct internal performance testing and benchmarking studies, the results of which you (and not unauthorized third parties) may publish or publicly disseminate; provided that VMware has reviewed and approved of the methodology, assumptions and other parameters of the study. Please contact VMware at benchmark@vmware.com to request such review. </ quote> For example here's a review from storagereview.com on VSAN http://www.storagereview.com/vmware_virtual_san_review_overview_and_configuration http://www.storagereview.com/vmware_virtual_san_review_overv...
    - rodgerd 11y ago
      "You may publish results which make us look good."
      - wila 11y ago
        Don't think so if your testing has obvious flaws.
        Drakim 11y ago
        There is clearly a conflict of interest if the producer is the one who gets to decide if your testing has flaws or not. There would be nothing but their own integrity preventing them from simply labeling any test with poor results as being "flawed".
- tmd83 11y ago
  Its true that good benchmark is extremely hard to get right specially for complex tools like these or databases. But I honestly think thats the secondary reason, the primary being stopping someone from truly evaluating their products. Essentially you will then have to rely on sales reps and some hearsay from other users and you are less likely to have someone who have actually worked with multiple comparable products in similar use cases.
devonkim 11y ago
Very similar reasons for why you're not allowed to publish benchmarks for Oracle products as well. It's just plain bad PR at a point if someone does a terrible job. On the other hand, it also means that if you're an unscrupulous vendor, you can just squash benchmarks that do not make you look better than your competition. The lack of information available in general is what is frustrating for many folks though even if it's completely anecdotal evidence.
acdha 11y ago
As wila noted, benchmarks are easy to get wrong. It also helps to remember that VMware has been in the enterprise space for a long time and a lot of people in that field have been burned by benchmarketing where a competitor would run an ad campaign hoping nobody would notice that their big performance lead was due to misconfiguration[1]. Even now, where it's easier to check the details, it's fairly common to see things like the HTTP/2 demo from yesterday (https://news.ycombinator.com/item?id=10091689 https://news.ycombinator.com/item?id=10091689) which overstated gains by misconfiguring the HTTP/1.1 server used for comparison. That wasn't too bad because both were their own servers but if you imagine that they were comparing their shiny new server to e.g. Apache or IIS that way and you'd have the average late-90s web server sales pitch. I don't like these terms but it's easy to understand why they became so common. 1. Stuff like different defaults for fsync, “accidentally” using different block sizes on a RAID array / filesystem / NFS mount, forgetting to use the same write-back/through settings on both, etc.

I have a hard time understanding the reasoning for this. What argument do they have as to why people shouldn't know about the performance?

by Drakim 11y ago

I have a hard time understanding the reasoning for this. What argument do they have as to why people shouldn't know about the performance?

KaiserPro 11y ago

Because VMware is generally quite slow. However it is very easy to manage large estates of machines using the out of the box tools. Unlike Xen or KVM

rodgerd 11y ago
oVirt would like a word with you.
- tomaac 11y ago
  and proxmox

wila 11y ago

The reasoning is that it is so easy to make mistakes in performance testing and then draw the wrong conclusion. As somebody else mentions, this is testing NFSv3 in the VMware setup against local disk in bhyve. While still interesting, it certainly is not a 1 on 1 comparison. edit: to expand a bit on that. - the VMware storage appliance gets 24GB, the bhyve config isn't mentioned, but the host OS can access 32 GB. - as he does mention his test load fits in the ARC, again interesting, but normally you would test real-world loads where your workload does not completely fit in the cache.

Drakim 11y ago
Sure, I mean, but, anybody can say that. "We'd rather not you publish your review of our product, it's so easy for the reviewer to make a mistake and say something negative about our product. So, you are not allowed to publish any examination of our product."
- wila 11y ago
  You ARE allowed to post test results, but they like to review your tests to make sure you didn't make any obvious mistakes. Quote from the .pdf file higher up in this thread: <quote> You may use the Software to conduct internal performance testing and benchmarking studies, the results of which you (and not unauthorized third parties) may publish or publicly disseminate; provided that VMware has reviewed and approved of the methodology, assumptions and other parameters of the study. Please contact VMware at benchmark@vmware.com to request such review. </ quote> For example here's a review from storagereview.com on VSAN http://www.storagereview.com/vmware_virtual_san_review_overview_and_configuration http://www.storagereview.com/vmware_virtual_san_review_overv...
  - rodgerd 11y ago
    "You may publish results which make us look good."
    - wila 11y ago
      Don't think so if your testing has obvious flaws.
      - Drakim 11y ago
        There is clearly a conflict of interest if the producer is the one who gets to decide if your testing has flaws or not. There would be nothing but their own integrity preventing them from simply labeling any test with poor results as being "flawed".
tmd83 11y ago
Its true that good benchmark is extremely hard to get right specially for complex tools like these or databases. But I honestly think thats the secondary reason, the primary being stopping someone from truly evaluating their products. Essentially you will then have to rely on sales reps and some hearsay from other users and you are less likely to have someone who have actually worked with multiple comparable products in similar use cases.

devonkim 11y ago

Very similar reasons for why you're not allowed to publish benchmarks for Oracle products as well. It's just plain bad PR at a point if someone does a terrible job. On the other hand, it also means that if you're an unscrupulous vendor, you can just squash benchmarks that do not make you look better than your competition. The lack of information available in general is what is frustrating for many folks though even if it's completely anecdotal evidence.

acdha 11y ago

As wila noted, benchmarks are easy to get wrong. It also helps to remember that VMware has been in the enterprise space for a long time and a lot of people in that field have been burned by benchmarketing where a competitor would run an ad campaign hoping nobody would notice that their big performance lead was due to misconfiguration[1]. Even now, where it's easier to check the details, it's fairly common to see things like the HTTP/2 demo from yesterday (https://news.ycombinator.com/item?id=10091689 https://news.ycombinator.com/item?id=10091689) which overstated gains by misconfiguring the HTTP/1.1 server used for comparison. That wasn't too bad because both were their own servers but if you imagine that they were comparing their shiny new server to e.g. Apache or IIS that way and you'd have the average late-90s web server sales pitch. I don't like these terms but it's easy to understand why they became so common. 1. Stuff like different defaults for fsync, “accidentally” using different block sizes on a RAID array / filesystem / NFS mount, forgetting to use the same write-back/through settings on both, etc.