14 ms·
Show HN: VPS Comparison – Automated tests to compare VPS by yourself
- slitaz 9y agoNice and thorough work!
- joedicastro 9y agoThanks! Very nice of you.
- neom 9y agoI am not easily impressed when it comes to vps and cloud stuff, but this is very impressive. Kudos. Curious why you didn't include ec2?
- merb 9y agoec2 is different however lightsail would be comparable. sadly there is still no instance outside US.
- neom 9y agoWhy is ec2 different?
- merb 9y agoyou can combine different network adapters/storage options and instance types and also create your own virtual network, with various other options. while these VPS providers only allow a specific amount of storage for a specific type of instance and also have a included traffic option while ec2 does not have that. I mean more and more VPS providers have an API, but they still don't have the customizability and services like a cloud has, yet they are preferable in many use cases.
- joedicastro 9y agoThanks! Very kind of you! Well, I would like to include more, in fact I would include Ligthsail for sure if they offered a datacenter in Europe. But to start something like this you have to start small, or the costs will bury you pretty son. As a pet and personal project, the way to limit the cost was setting a clear seat of boundaries, like plans under $5 and available in Europe. And being one-man band to make this affordable in terms of time and effort I had to keep it small at the beginning. I worked on this in my spare time for more than two months. If I find a way to sustain the costs, or the providers itself offer me a way to test their machines, I have no problem to expand the comparison. But in the meantime, I share the tools because I knew from the beginning that I only have limited resources. And after all, reproducibility, independence and transparency are the keys IMHO to have a trustful comparison.
- tedmiston 9y agoCurious to know if anyone uses Vultr whether they actually hit 100% uptime, or if that's more marketing and they just issue credits for the SLA regularly.
- yunyu 9y agoI've had one instance of network downtime lasting 40 minutes in December which was SLA refunded. No incidents since then.
- dijit 9y agoAnecdatum inbound: Sometimes they'll reboot my instance due to host errors they claim, seems to be an automated email. Last one was January 17, before that was Dec 16, before that was Oct 29, then Oct 22 Each time the instance in question was down for less than a minute, no credit was issued. I have instances in Sydney, Los Angeles and New Jersey
- lima 9y ago100% uptime is an unrealistical claim. There are so many things that can go wrong with a VPS (that is, by design, only ever running on one node at the time). The probability of that node failing or a messed up route taking offline all routers at once is low, but it's not 0%. In order to even come close to 100%, you need a multi-region distributed system and even then, so many things can go wrong. Look at their outage credits (https://www.vultr.com/sla/ https://www.vultr.com/sla/) if you want to get realistical percentages. Even Google Cloud Platform - which has multiple layers of redundancy at every level - does not claim 100% uptime: https://cloud.google.com/compute/sla https://cloud.google.com/compute/sla Disclaimer: I work for a competitor who would never advertise 100% uptime - it's a known fact that shit happens
- joedicastro 9y agoI totally agree with this, after all is common sense. Even if all the hardware were perfect, and all the software were bugless, you would still have to deal with physics and with those imperfect and error-prone machines that set up all the things called humans.
- throwit2mewillU 9y agoStrasboug (DE) - not for quite some time.
- joedicastro 9y agoYou're right! another people pointed to same error as well and was fixed in the repository with a PR. I'm sorry for that, was a huge mistake, I hope that nobody have been offended by that.
- deleted 9y ago[deleted]
- nodomain 9y agoVery impressive reference. Would be interested in ec2 as well. And I think "datacenter in the EU" does not necessarily mean "bound to EU law" since the parent company is in the US, right?
- efdee 9y agoIf the data is stored in the EU, it is bound to EU law.
- joedicastro 9y agoThanks! Well, I would like to include more, in fact I would include Ligthsail for sure if they offered a datacenter in Europe. But to start something like this you have to start small, or the costs will bury you pretty son. As a pet and personal project, the way to limit the cost was setting a clear seat of boundaries, like plans under $5 and available in Europe. And being one-man band to make this affordable in terms of time and effort I had to keep it small at the beginning. Maybe, If providers offer me free credits to make the tests or if I found a way to assume the costs, them I could think to expand the project.
- gketuma 9y agoNicely done.
- joedicastro 9y agoThanks! Very kind of you.
- id122015 9y agoI thought linode and digitalocean were clouds not VPS
- vxxzy 9y agocloud is just a fancy word for Virtualization.
- zephod 9y agoI think virtualization is a fancy word for cloud.
- codefined 9y agoVirtualisaton is the basic task you're doing, you're making a "virtual" computer. The idea of the "cloud" is more that these virtual machines are hosted in a room somewhere far away with other virtual computers there as well. For instance, I can install VMWare and get virtualisation going on my laptop, but I wouldn't consider that the cloud. Similar, you can get bare metal computers on the cloud which aren't virtualised.
- neom 9y agoThat's not strictly true, most folks agree that cloud services typically have an "on the fly" type elastic nature.
- vxxzy 9y agoAt the end of the day there exists a hypervisor... Virtualization.
- Danihan 9y agoYep, afaik Amazon ec2 is still Xen.
- dijit 9y agoSo do VPS providers. Ask for a VPS and you'll get it in the same amount of time as EC2. The difference is only surrounding automated deployment (cloudformation) or services that are offered (RDS)
- vldx 9y agoI was asked recently (2017) by Linode to provide scanned my credit card (after paying, then eventually refunded). The reason - fraud prevention, whatever that means. Certainly leaves bad flavor and decided to go w/ DO.
- scrps 9y agoI encountered the same thing (also in 2017) with the addition that they wanted front and back scans of a government-issued ID as well. Having to pay with a credit card before you can use paypal didn't earn them any points either. Suffice to say I did not go with Linode.
- JohnTHaller 9y agoPayPal's fraud detection sucks, so I can definitely understand companies wanting to verify/charge a credit card directly before accepting PayPal.
- joedicastro 9y agoYes, probably is one of the reasons why they do those checks. I suppose also that depends a lot on the kind of plan that are you going to use, probably they do more checks for more expensive plans.
- EduardoBautista 9y agoThat just happened to me on Saturday. I decided to go with DO. You get what you pay for.
- joedicastro 9y agoI did not have to make that with them, but I had to pass a weird process with OVH in order to confirm my identity through a phone call, and in the process have to wait several hours until I have my server available.
- wdb 9y ago
- dijit 9y agoThere's a VPS provider whom I am infatuated with[0] and I'd love an easy objective way to compare them with Amazon/Linode. The only concern is that they only have 2 datacenters and they're both in the Netherlands. [0]: https://tilaa.com https://tilaa.com
- iagovar 9y agoTheir pricing model seems similar to http://clouding.io http://clouding.io, but it forces you to get more CPU if you want, say, 8GB RAM. It's a bit cheaper with HDD instead of SSD though.
- NKCSS 9y agoJust wondering; why do you prefer them over TransIP for instance?
- dijit 9y agoEase of access to engineers, transparency when it matters. And I've never had a node go down, that helps.
- joedicastro 9y agoWell, the only thing that you need is to execute the tests on the three of the providers, and compare them yourself if you want. Those are the keys of the project so far, reproducibility, independency and transparency. I'm inclined to expand the review, but only if I find a way to get free credits from providers or another way to finance the tests. But right now, the best thing I can offer you is the same tools that I used here in the repository itself.
- proyb2 9y agoI saw Lynis is use for audit and harden VPS, have anyone use it? I haven't harden my Linode.
- joedicastro 9y agoWell, it's very useful, and in continue development. In the end you have to know what you are doing, but it will point you to very useful tips and show some warnings that you can look to improve the security of you server. But in this, I still don't know two SysAdmins that concur on the best way to harden a server. There is a lot of things to consider and there are very strong opinions on how to do certain things. But without doubt, it's a very useful tool, even if you know what you are doing you can use it like a checklist to know if you are forgetting something. It's not perfect (nothing is, and less in this field) but I can only recommend it.
- mxuribe 9y agoReally helpful details; great job on this!
- joedicastro 9y agoThanks!
- sjs382 9y agoHalf-baked startup idea: because VPS performance is so dependent on the VPS's neighbors, create a service that spins up a bunch and then compares them on demand. Keep the highest performing one. Charge your users to perform these tests.
- jtokoph 9y agoInteresting. Any ideas on how to monitor/respond to a new noisy neighbor? Ongoing monitoring to see if performance degrades? Can you filter out your own usage from the performance rating?
- anc84 9y agoFeature request #1: Don't test on machine we currently test on in a neighbor.
- NKCSS 9y agoYou mean the 'Netflix Model'? This is what they do all the time on AWS; when needed, spin up a bunch of VM's; test performance, drop the ones that perform worse and use the better ones.
- tuananh 9y agodo you have any source for this?
- joedicastro 9y agoI don't know, but as NKCSS says, you could do probably that in AWS and being a client like Netflix. But stressing instances regularly to test them to find a better one is something that you could probably do on your own, but as a service I have serious doubts if the providers would like that from a given scale. I'm not sure, but I don't think that would be something rare to have you account suspended. Perhaps, if you arrive a some type of agreement with them and their "business model" is compatible wit that, maybe. But, don't give me too much credit in this, I'm only speculating.
- vorpalhex 9y agoExtremely useful! I'm currently on DO (and I love them) but I've been eyeing vultr and this definitely pushed me to give them a go.
- joedicastro 9y agoThanks! I'm glad that is helpful to you.
- smcleod 9y agoI have servers on Vultr, AWS, Azure, physical hardware in data centres and a few others. The physical servers are by far the fastest obviously, but when looking at the VPS / 'cloud' services Vultr is by _far_ the fastest and most reliable servers out of the lot, a close second would be Sitehost (AU/NZ Only). On the other end of the scale, the least reliable services we've experienced is easily those hosted on Azure where constant network issues, unreliable performance and random unexplained outages are frequent. Rackspace used to be good, going back say 2 or so years ago, then it felt like they were struggling although their technology far exceeded that of AWS - Amazon's marketing and targeting of management level sales seemed to outweigh and caused them to buckle.
- joedicastro 9y agoThanks for share this! You know, this is the kind of of knowledge that we all like to have, but even then, all we have is your word (I don't have any reason to doubt it), what I tried to share here is a way to back all of that (at least performance, reliability is a different kind of monster) with numbers.
- jazoom 9y agoLet me add my word. Vultr is powerful and reliable. Recently they cut their prices roughly in HALF. They were the best (IMHO) before that. Now it's not even a competition.
- h1d 9y agoLinode seems to provide pretty much exact same plans, but their control panel is a bit too geeky (you need a few extra steps to launch a OS because launching an instance means powering on an empty machine without a disk), touching Vultr control panel feels pretty easy. I just hope they both get block storage deployed on all their regions.
- joedicastro 9y agoLinode seems to be working in a new version of their control panel, I hope that they do a great work, because for too much people the current one is a entry barrier, even when they do a very good work.
- SmellTheGlove 9y agoEverything I've wanted to know but been too lazy to do on my own. This is excellent, thank you for sharing.
- joedicastro 9y agoYou're welcome! Thanks!
- Elect2 9y agoWhat I cared the most about is CPU limit. On all these vps if you keep 100% CPU usage they will treat you as a noisy neighbor. But none of them clearly tell you the baseline of CPU resource that "safe" to use.
- joedicastro 9y agoI hope that there was a easy way to know that clearly, but happens the same with support, is very hard to know what kind of support they are going to offer you until you experience it firsthand. That's one of the reasons why I started this, all the reviews and comparisons along the web have some kind of bias -even when is positive and truthful by grateful users- and is very hard to know what is right from wrong. I wanted to offer some way to have transparency and reproducibility all the way.
- throwasehasdwi 9y agoThis is why I hate using AWS and similar virtual servers. I use them for quick dev stuff then get a non shared server. Massive datacenter scaling doesn't save that much money. The money in cloud is made from huge over-provisioning. Most providers will artificially limit you below the published limits if you use "too much" CPU or internal network bandwidth. You think that 1Gbps link between your machines isn't shared? See what happens when you try to use it full tilt for a couple days.
- Elect2 9y agoAs far as I know, AWS and Google cloud instances do not limit your vCPU usage.(Exclude aws t1&t2 instance and Google micro&small instance) . And also they do not limit your bandwidth usage, they bill you instead.
- extra88 9y ago> The money in cloud is made from huge over-provisioning. Did you mean "overselling," selling more than your infrastructure could support if every customer used all you sold them?
- jsmthrowaway 9y ago
- Sami_Lehtinen 9y agoI've been very keen with https://UpCloud.com https://UpCloud.com at first their servers don't seem to be cheap. But when you factor in the actual performance, those do provide a good on bang ratio. So far I don't have any complaints about UpCloud. But it's not the same story with OVH. UpCloud didn't throttle the disk I/O, even it was truly insane, due to a software bug. That's what I recommend, if you're looking for good instead of just cheap. UpCloud doesn't oversell resources, so full CPU / RAM utilization is totally acceptable. Also VPS resources are guaranteed and performance is pretty much constant. Scaleway is actually quite good when you factor in the performance / price ratio with it. (VC1S) But that's not something I would recommend for 'production use'. Yet, it's very nice for hobby projects. All systems are under constant monitoring. CPU, RAM, Disk I/O & Networking. Unfortunately the data collected isn't something I can publish. Btw. OVH SSD VPS 1 does provide IPv6, yet it's not being tested. Why? - Yes, you'll need to manually add the routing once to enable it.
- joedicastro 9y agoThanks for the information, it's very helpful. Yes, I know about the OVH plan and IPv6, but in the hurry and in the middle of all the tasks to accomplish I left behind a few things. I'll keep improving it. Thanks!
- mysterydip 9y agoThis is just what I needed! I'm downsizing from a colo and looking for just this sort of data. Vultr seems really appealing especially with the ability to upload my own iso (My preferred linux distro isn't one of the common host offerings). I would love to hear people's experiences good and bad with them as a provider.
- joedicastro 9y agoThanks!
- h1d 9y agoOut of curiosity, which distro do you need to use?
- mysterydip 9y agoTiny core. I use debian or ubuntu where required, but prefer the features and minimal footprint of tiny.
- halfeatenpie 9y agoGreat start to a comparison review. I've used all the service providers available and these opinions include my historical experience with them (I've used all providers for many years, my account with DO is since 2013). Most of my service uses were focused in those service provider's North America locations, Asia locations, and Oceania locations. Note I did do a short comparison between DigitalOcean and Vultr back in 2014 here: https://vpsboard.com/threads/digitalocean-vs-vultr-the-asian-face-off.3688/ https://vpsboard.com/threads/digitalocean-vs-vultr-the-asian... Most service providers here are fairly decent. From my experience, OVH's KVM VPSes weren't worth the time, but their VMWare VMs were absolutely top notch. Linode has had two major security events over the last few years, one of which was in early 2016 which included release of credit card information. However, they also offer paypal payments now and are fairly top notch in terms of performance. (They've taken approaches to fix their problems from 2016 with this blog post here: https://blog.linode.com/2016/02/19/security-investigation-retrospective/ https://blog.linode.com/2016/02/19/security-investigation-re... ). DigitalOcean is fairly reliable. Their Singapore location early on had regular packet loss events, but they've been shaping up pretty nicely over time. Vultr has been a major problem for me. Vultr for a while (on their storage instances in Japan) had terrible I/O which took a solid two hours just to install Ubuntu 14.04. When contacted support, they stated they "stopped a noisy neighbor and to retry" but no real improvements were seen. We had a service running on a VC2 that irregularly required a large amount of CPU power (higher than 0.5 load on a 1 CPU instance for a period of 1 hour a night). The same task was completed faster with VMs from DigitalOcean and Linode. But the biggest problem I've experienced is their network. For the longest time their Japan location was single-homed and routing was funky. We'd regularly see any traffic going outside of Japan being routed back through their Los Angeles location, so adding 300+ ms latency for any of our users in Korea, China, and anywhere in South-East Asia. Today it's much better with 40 ms latency from their Japan location to our Korea client, and a solid 80 ms latency to our Singapore client. A colleague of ours had very important clients who required 100% uptime (which Vultr agreed to) but over a period of a month had regular packet loss events or network loss (regularly 40 minutes per event and around one event every few days monitored via external monitoring systems such as uptimerobot, statuscake, and a GCP server running Icinga2). Didn't receive much help from Vultr's engineers nor could they find the issue. They ended up moving to Google Cloud Platform. They did end up getting SLA credit from them but no reason to have SLA credit when you're not on their platform anymore. I regularly return to Vultr every few months to every year or so to see how it's going and they've come a long way with the services they're offering. I'd recommend them for a dev environment and testing over a period of an hour a day or so. However, I would caution with production systems. Your mileage may vary, but I'd suggest reviewing the quality of the product in addition to the price tag per resource.
- lvh 9y agoThis is a great set of tests. Once you move out of the side-project phase and start turning stuff into a business, you probably care about security features too. Unfortunately, most VPS providers don't have a great track record there. With the big Cloud providers, you have tools like AWS' VPCs and security groups to help fence off parts of your app from the rest of the world. In a perfect world, you wouldn't care so much about network reachability and care a lot more about, say, mTLS or at-rest crypto. But as a young company, managing that probably isn't the best thing to spend your resources on right now. Much like Google's BeyondCorp: just because you can expose some apps to the internet with years of work and millions in resources doesn't mean that a VPN or a SG isn't a pretty good idea for now -- if nothing else, it'll do a pretty decent job protecting you from the background radiation of the Internet :)
- joedicastro 9y agoThanks! Great comment, I agree.
- thinkMOAR 9y agoOh that is a nice overview, perhaps regarding IP reputation, include an overview which of those providers has the most ip addresses on RBL lists too? Performance is nice, but if you are already on (a) blacklist (s) from your start.. :) since most of those providers, have a lovely reputation in regards of compromised machines being abused in botnet and other malicious activities... i think it might be an interesting index.
- joedicastro 9y agoInteresting idea... it didn't though of that for sure. I thought of talking about how permissive the providers are with certain type of contents (like legal adult content), but it is an information not easy to collect and so I went with the official legal terms/ToS instead. I'll have to think about that, and how collect that information. Thanks!
- corford 9y agoVery nice. One other datapoint that could be useful for some: Terraform provider support (and how capable the provider is). Terraform has providers for Digital Ocean, Scaleway and Lightsail but nothing for Linode or Vultr.
- joedicastro 9y agoThat's a good idea, I suppose that would have to do the same with another tools like Apache libcloud or Ansible itself. I'll add this suggestion as an issue to the project. Thanks!
- tpetry 9y agoA terraform provider for Vultr is the really missing part for seitching from DO to Vultr.
- joedicastro 9y agoI suppose that is the same with another tools and providers. The community size and the work that the providers put on this makes a big difference IMHO. Another reason to share this in the comparison. Thanks!
- hathym 9y agoNice, I recommand that you include the 1€ arubacloud vps.
- dylz 9y agoGone now..
- joedicastro 9y agoThanks! I would take into account.
- mtw 9y agoThanks - I found the framework performance towards the bottom very interesting. What about including other popular frameworks? Ruby on Rails, Laravel, ASP.NET, symfony, express.js
- joedicastro 9y agoThanks to you! Yes, there is always that possibility, is a question only of time, more tests implies more time to complete the full batch. And of course, more time to implement it. I have the intention to keep improving it, and thanks to the Ansible tags, the tests can be run individually and independently (I tried from the beginning to make them as atomic as possible for this same purpose). Thus is very easy to extend.
- HugoDaniel 9y agoIt would be nice to include hetzner :)
- NKCSS 9y agoAnd maybe TransIP and LeaseWeb
- joedicastro 9y agoI put you here the same reply that I gave to another user, because the answer is essentially the same. Anyway, thanks for you feedback. I understand you, but as I said in another comments, I would like to expand the comparison, but in the end is a question of time and costs. If I find a way to sustain the costs, or the providers itself offer me a way to test their machines, I have no problem to expand the comparison. But in the meantime, I share the tools because I knew from the beginning that I only have limited resources. And after all, reproducibility, independence and transparency are the keys IMHO to have a trustful comparison.
- fapjacks 9y agoI have two dedicated machines through Hetzner and they're great! Great value.
- porker 9y agoWould love to see Joyent included. Silly pricing, hopefully performance that justifies it. https://www.joyent.com/pricing/cloud/compute https://www.joyent.com/pricing/cloud/compute
- joedicastro 9y agoI understand you, but as I said in another comments, I would like to expand the comparison, but in the end is a question of time and costs. If I find a way to sustain the costs, or the providers itself offer me a way to test their machines, I have no problem to expand the comparison. But in the meantime, I share the tools because I knew from the beginning that I only have limited resources. And after all, reproducibility, independence and transparency are the keys IMHO to have a trustful comparison.
- notamy 9y ago> I have serious doubts about the OVH’s and Scaleway’s unlimited traffic, seems more marketing strategy than real to me (joe di castro). Anecdotal, of course, but I have an application on OVH (on the USD $3.50/month plan) that pushes/pulls >10TB/month - coming close to saturating the 100/100 connection at times - and I've not heard anything from OVH about it being an issue.
- joedicastro 9y agoOh! That's nice to know, I'm glad to hear that and that they keep that promise. I knew that they have their own infrastructure, but even then, I really have doubts because in my experience almost all of the providers have "little marketing white lies".
- RX14 9y agoScaleway are owned by one of the largest ISPs in France. Bandwidth is typically brought symmetric, but home traffic is nearly entirely download. It's my assumption that they use their server buisness to offset their bandwidth costs for their underutilised outbound links.
- joedicastro 9y agoYes, Free is very well known here in my country, it has a great reputation. Your assumption makes sense to me, and probably is close to the truth.
- zzzcpan 9y agoYeah, they have no problem with that. But I do recall years ago when a few customers were running hundreds of seedboxes OVH changed ToS and asked them to leave.
- dawnerd 9y agoIt seems to depend on if you generate complaints. If you're not causing problems for them they won't likely care. Goes for most seedboxes, actually.
- RX14 9y agoI'm a little confused as to why comparing single-threaded performance (for scaleway) is more "fair". Surely you should compare both, to allow people to compare for single-threaded and multi-threaded workloads.
- joedicastro 9y agoI understand you, but is more fair in the sense of comparing apples to apples if you are going to do one test of a kind. Currently all of this tests as a batch job take several hours, doing more tests, means more time and more job for me. Specially that when I took that decision I didn't gather all the data automatically and I have to collect all of that by hand and put it on the tables. There is a reason for all. Now that I have (almost, I have to add 1 more) all the tests automated and I collect all the data with a python script, I can start to think to do all kind of "alchemy" and improve the comparison. Thanks for you feedback. I'll add this as an issue into the repository to improve the comparison.
- solotronics 9y agohey if you would like a Softlayer / IBM Bluemix instance to test I would be happy to provide gene.gaddy@ibm.com
- joedicastro 9y agoThanks for the offer. I'll take into account.
- rkv 9y agoWould be nice if the "Notes" section referenced the matching table entry. Example: >Send root password by email Yes No[1] No >Notes: >1. DigitalOcean sends you the passwords only if you don’t use SSH keys, in plain text. Otherwise, great comparisons.
- joedicastro 9y agoThanks! Yeah, I agree, that would be nice, but as I automate the creation of those tables and cut/paste them every time that I run a set of new tests, to do that would require a lot of manual job to me. That's the reason why I don't do that, to keep it manageable.
- raybb 9y agoHow long did it take you to write this up?
- joedicastro 9y agoMore than two months in my spare time. To collect the "static" information, and create the Ansible roles and python scripts to collect the "dynamic" info (the results from the benchmarks) and finally compose the final document. I lost a lot of time waiting for the benchmark tests to finish ;-)