6 ms·

"All our servers are built on powerful Octo-Core machines" pretty much guarantees they're using some cheaper than E5 Xeons to save money, I'm wondering if it's

by stu2010 10y ago

"All our servers are built on powerful Octo-Core machines" pretty much guarantees they're using some cheaper than E5 Xeons to save money, I'm wondering if it's something in the Xeon-D line. Has anyone specifically characterized what they're using? Could be Xeon D-1540s or similar, or they could also be selling 4-core hyperthreaded E3 Xeons as "8 core".

joecot 10y ago
Edit: I went ahead and signed up for an account and made a container. The floating IP for the container appears to be an LA IP address. Their host appears to be ZenLayer, a Chinese hosting company that can apparently do co-location in LA, so while the IPs geolocate to China, it's possible they are indeed hosting in LA. The CPU is a E5-2630 v4. Original: I'm pretty sure they're entirely hosted on AWS. Given that they say they're hosted in Los Angeles, I think they mean us-west-1. Their API uses an AWS address as their endpoint, their authentication is just a veneer over AWS's authentication (including basically find and replacing header variables). They previously had docs that showed how to add floating IPs to the containers, and all the IPs were AWS Elastic IPs. I'm pretty sure the docs specifically stated they were on AWS last time Hyper came up [1] (Hyper.sh had linked off the Hyper article), but now when I look it's not there. So either in a few days they've moved their infrastructure off AWS and just left their API up there (and are doing some crazy stuff to redirect elastic IPs), or they moved everything but their API off Amazon a while ago and hadn't updated their docs, or they've decided to make the fact they're on AWS less visible, while they're competing with Amazon's own container service. I have the feeling it's option #3. To answer your question though, I think they're using M4 AWS instances [2], so Xeon E5-2686 Broadwell or Xeon E5-2676 Haswell. Probably the m4.10xlarge, since they talk about the 10 GB networking the containers use. 1. https://news.ycombinator.com/item?id=12873089 https://news.ycombinator.com/item?id=12873089 2. https://aws.amazon.com/ec2/instance-types/#m4 https://aws.amazon.com/ec2/instance-types/#m4
- djsumdog 10y ago
  I have a feeling a huge chunk of the projects we see here are run on AWS. Duckduckgo runs on AWS. For getting things started quickly at a low initial cost, it's usually more viable to use a hosted solution (AWS, RackSpace, Digital Ocean, etc.) However once you get big enough, the cost savings usually start falling the other way. Several companies I've been at have moved from using hosting to running their own boxes, either co-located or in their own data center (the CTO like to call this, "moving to our own private cloud" or some other marketing bullshit). Even then, careful decisions are made on to what to host locally and what to keep on a managed service due to cost.
  - joecot 10y ago
    Having this on AWS is perfectly fine. I even think it's viable to look at Hyper.sh as an alternative to Elastic Container Service, even though both are being hosted on AWS. But if it's hosted on AWS, it's important that people actually know that. If someone wants to build a highly reliable system, and they pick 2 resources -- let's say Hyper.sh and AWS ECS as the most likely candidates for that -- it's pretty important for the customer to know that both resources they're relying on are on the same service, and even possibly in the exact same data center, as that affects how effective their redundancy actually is.
    - IanCal 10y ago
      As another side of the same point, hyper on AWS shifts the balance of costs for me as if I want to store my data in S3 it changes whether or not I've got to pay for network egress.
  - justinhj 10y ago
    There are plenty of big companies that stay on AWS. Netflix is the obvious example. Companies like Zynga and Activision are moving gradually from self hosted to AWS (from what I hear).
    - vidarh 10y ago
      Once you are big enough to be able to negotiate prices that are nowhere near the published prices for AWS, it clearly can start becoming cost-competitive...
- gnawux 10y ago
  Not on aws at all. The hyper.sh API address is us-west-1.hyper.sh, which looks like the AWS style, however, it is not an AWS address and it is located in an independent IDC around Los Angels.
  - joecot 10y ago
    Edit: Tried it and confirmed it's indeed not on AWS, updated my previous comment. Original: Not on AWS at all doesn't seem possible The docs for Floating IPs [1] list 52.68.129.19 as an example, which is an AWS Elastic IP. The docs for the API [2] says "Hyper.sh API signature algorithm is based on AWS Signature Version 4", and then proceeds to explain the differences, which is variable names. The API Domain is us-west-1.hyper.sh, which is the same URL schema as AWS (us-west-1 is also AWS's North California region). Maybe the containers themselves are somehow not on AWS? Sure. But not on AWS at all doesn't seem to be the answer. 1. https://docs.hyper.sh/Feature/network/fip.html https://docs.hyper.sh/Feature/network/fip.html 2. https://docs.hyper.sh/Reference/API/2016-04-04%20[Ver.%201.23]/index.html https://docs.hyper.sh/Reference/API/2016-04-04%20[Ver.%201.2...
    - gnawux 10y ago
      You can try the api address us-west-1.hyper.sh
      - joecot 10y ago
        Resolves to 65.255.36.153 and 65.255.36.154, which don't have reverse DNS, but Maxmind Geolocates to China.
        gnawux 10y ago
        In Los Angels
    - gnawux 10y ago
      Based on AWS v4 does not means aws v4 is employed, on the contrary, the docs explains the signature of hyper, which uses different literatures. At last, you can try it. Then you will found it is totally different from AWS.
- ones_and_zeros 10y ago
  I don't think AWS allows you to do nested virtualization. Perhaps they have a hack?
- gnawux 10y ago
  Another thing to remind you is, E5-2630v4 was released in Q1 2016, which is not an old CPU http://ark.intel.com/products/92981/Intel-Xeon-Processor-E5-2630-v4-25M-Cache-2_20-GHz http://ark.intel.com/products/92981/Intel-Xeon-Processor-E5-...
gnawux 10y ago
Does the CPU model matter?
- yberreby 10y ago
  Yes, it does. Not all CPU cores are created equal. Performance depends on microarchitecture, cache size, memory bandwidth and latency, clock speed, hardware acceleration of specific features, and hyper-threading capabilities, to name a few. Take the Xeon E5-2403[1] and the Xeon E5-2637 v4[2]. Both are quad-core Xeons, but they differ by pretty much everything except core count. Here's a comparison of their performance: http://cpubenchmark.net/compare.php?cmp%5B%5D=1827&cmp%5B%5D=2790 http://cpubenchmark.net/compare.php?cmp%5B%5D=1827&cmp%5B%5D.... Granted, this is an artificial benchmarks, but the results speak for themselves. In this case, the Xeon E5-2637 v4 is almost three times faster than its little brother, the Xeon E5-2403. Quantifying CPU performance by number of cores is disingenuous at best, and dishonest at worst. [1]: http://ark.intel.com/products/64615/Intel-Xeon-Processor-E5-2403-10M-Cache-1_80-GHz-6_40-GTs-Intel-QPI http://ark.intel.com/products/64615/Intel-Xeon-Processor-E5-... [2]: http://ark.intel.com/products/92983/Intel-Xeon-Processor-E5-2637-v4-15M-Cache-3_50-GHz http://ark.intel.com/products/92983/Intel-Xeon-Processor-E5-...
- module0000 10y ago
  Yes! Running applications that can't cope with NUMA means you need to know what model of CPU, and especially know whether your N-cores are on the same socket or not.

by stu2010 10y ago

joecot 10y ago

Edit: I went ahead and signed up for an account and made a container. The floating IP for the container appears to be an LA IP address. Their host appears to be ZenLayer, a Chinese hosting company that can apparently do co-location in LA, so while the IPs geolocate to China, it's possible they are indeed hosting in LA. The CPU is a E5-2630 v4. Original: I'm pretty sure they're entirely hosted on AWS. Given that they say they're hosted in Los Angeles, I think they mean us-west-1. Their API uses an AWS address as their endpoint, their authentication is just a veneer over AWS's authentication (including basically find and replacing header variables). They previously had docs that showed how to add floating IPs to the containers, and all the IPs were AWS Elastic IPs. I'm pretty sure the docs specifically stated they were on AWS last time Hyper came up [1] (Hyper.sh had linked off the Hyper article), but now when I look it's not there. So either in a few days they've moved their infrastructure off AWS and just left their API up there (and are doing some crazy stuff to redirect elastic IPs), or they moved everything but their API off Amazon a while ago and hadn't updated their docs, or they've decided to make the fact they're on AWS less visible, while they're competing with Amazon's own container service. I have the feeling it's option #3. To answer your question though, I think they're using M4 AWS instances [2], so Xeon E5-2686 Broadwell or Xeon E5-2676 Haswell. Probably the m4.10xlarge, since they talk about the 10 GB networking the containers use. 1. https://news.ycombinator.com/item?id=12873089 https://news.ycombinator.com/item?id=12873089 2. https://aws.amazon.com/ec2/instance-types/#m4 https://aws.amazon.com/ec2/instance-types/#m4

djsumdog 10y ago
I have a feeling a huge chunk of the projects we see here are run on AWS. Duckduckgo runs on AWS. For getting things started quickly at a low initial cost, it's usually more viable to use a hosted solution (AWS, RackSpace, Digital Ocean, etc.) However once you get big enough, the cost savings usually start falling the other way. Several companies I've been at have moved from using hosting to running their own boxes, either co-located or in their own data center (the CTO like to call this, "moving to our own private cloud" or some other marketing bullshit). Even then, careful decisions are made on to what to host locally and what to keep on a managed service due to cost.
- joecot 10y ago
  Having this on AWS is perfectly fine. I even think it's viable to look at Hyper.sh as an alternative to Elastic Container Service, even though both are being hosted on AWS. But if it's hosted on AWS, it's important that people actually know that. If someone wants to build a highly reliable system, and they pick 2 resources -- let's say Hyper.sh and AWS ECS as the most likely candidates for that -- it's pretty important for the customer to know that both resources they're relying on are on the same service, and even possibly in the exact same data center, as that affects how effective their redundancy actually is.
  - IanCal 10y ago
    As another side of the same point, hyper on AWS shifts the balance of costs for me as if I want to store my data in S3 it changes whether or not I've got to pay for network egress.
- justinhj 10y ago
  There are plenty of big companies that stay on AWS. Netflix is the obvious example. Companies like Zynga and Activision are moving gradually from self hosted to AWS (from what I hear).
  - vidarh 10y ago
    Once you are big enough to be able to negotiate prices that are nowhere near the published prices for AWS, it clearly can start becoming cost-competitive...
gnawux 10y ago
Not on aws at all. The hyper.sh API address is us-west-1.hyper.sh, which looks like the AWS style, however, it is not an AWS address and it is located in an independent IDC around Los Angels.
- joecot 10y ago
  Edit: Tried it and confirmed it's indeed not on AWS, updated my previous comment. Original: Not on AWS at all doesn't seem possible The docs for Floating IPs [1] list 52.68.129.19 as an example, which is an AWS Elastic IP. The docs for the API [2] says "Hyper.sh API signature algorithm is based on AWS Signature Version 4", and then proceeds to explain the differences, which is variable names. The API Domain is us-west-1.hyper.sh, which is the same URL schema as AWS (us-west-1 is also AWS's North California region). Maybe the containers themselves are somehow not on AWS? Sure. But not on AWS at all doesn't seem to be the answer. 1. https://docs.hyper.sh/Feature/network/fip.html https://docs.hyper.sh/Feature/network/fip.html 2. https://docs.hyper.sh/Reference/API/2016-04-04%20[Ver.%201.23]/index.html https://docs.hyper.sh/Reference/API/2016-04-04%20[Ver.%201.2...
  - gnawux 10y ago
    You can try the api address us-west-1.hyper.sh
    - joecot 10y ago
      Resolves to 65.255.36.153 and 65.255.36.154, which don't have reverse DNS, but Maxmind Geolocates to China.
      - gnawux 10y ago
        In Los Angels
  - gnawux 10y ago
    Based on AWS v4 does not means aws v4 is employed, on the contrary, the docs explains the signature of hyper, which uses different literatures. At last, you can try it. Then you will found it is totally different from AWS.
ones_and_zeros 10y ago
I don't think AWS allows you to do nested virtualization. Perhaps they have a hack?
gnawux 10y ago
Another thing to remind you is, E5-2630v4 was released in Q1 2016, which is not an old CPU http://ark.intel.com/products/92981/Intel-Xeon-Processor-E5-2630-v4-25M-Cache-2_20-GHz http://ark.intel.com/products/92981/Intel-Xeon-Processor-E5-...

gnawux 10y ago

Does the CPU model matter?

yberreby 10y ago
Yes, it does. Not all CPU cores are created equal. Performance depends on microarchitecture, cache size, memory bandwidth and latency, clock speed, hardware acceleration of specific features, and hyper-threading capabilities, to name a few. Take the Xeon E5-2403[1] and the Xeon E5-2637 v4[2]. Both are quad-core Xeons, but they differ by pretty much everything except core count. Here's a comparison of their performance: http://cpubenchmark.net/compare.php?cmp%5B%5D=1827&cmp%5B%5D=2790 http://cpubenchmark.net/compare.php?cmp%5B%5D=1827&cmp%5B%5D.... Granted, this is an artificial benchmarks, but the results speak for themselves. In this case, the Xeon E5-2637 v4 is almost three times faster than its little brother, the Xeon E5-2403. Quantifying CPU performance by number of cores is disingenuous at best, and dishonest at worst. [1]: http://ark.intel.com/products/64615/Intel-Xeon-Processor-E5-2403-10M-Cache-1_80-GHz-6_40-GTs-Intel-QPI http://ark.intel.com/products/64615/Intel-Xeon-Processor-E5-... [2]: http://ark.intel.com/products/92983/Intel-Xeon-Processor-E5-2637-v4-15M-Cache-3_50-GHz http://ark.intel.com/products/92983/Intel-Xeon-Processor-E5-...
module0000 10y ago
Yes! Running applications that can't cope with NUMA means you need to know what model of CPU, and especially know whether your N-cores are on the same socket or not.