6 ms·
Enjoy your free service, but I would avoid paying for Digital Ocean for any serious project. No custom kernel support 1. Digital Ocean does not allow you to r
by tshtf 11y ago
Enjoy your free service, but I would avoid paying for Digital Ocean for any serious project.
No custom kernel support
1. Digital Ocean does not allow you to run your own kernel natively.
2. Digital Ocean droplet kernels are infrequently updated.
3. These kernels often contain relevant security vulnerabilities.
4. This has been a known issue since 2013.
5. You can kexec a kernel, but this is an annoying workaround.
https://www.digitalocean.com/community/questions/how-can-i-boot-my-compiled-kernel-image-on-gentoo https://www.digitalocean.com/community/questions/how-can-i-b...
Private networking is a joke
1. Your private IP addresses are accessible by everyone in the same datacenter.
https://www.digitalocean.com/community/tutorials/how-to-isolate-servers-within-a-private-network-using-iptables https://www.digitalocean.com/community/tutorials/how-to-isol...
IPv6 support
1. Took forever to implement, and the timetable broke promises to customers.
2. Inferior. Digital Ocean still won't give you a /64 per standards.
- pbreit 11y agoI bet this deal drives improvements on all of those points (or at least the ones that matter).
- rdtsc 11y ago> No custom kernel support How many people run custom kernels? I have used a kernel with realtime extensions enabled but that was a very special case and wouldn't run that in a VM anyway.
- tshtf 11y ago> How many people run custom kernels? How many people want to run an updated kernel without known security vulnerabilities? Or with fixes to relevant issues?
- rdtsc 11y ago> How many people want to run an updated kernel without known security vulnerabilities? How many people call that "a custom kernel"? Haven't heard anybody call an upstream distro kernel update a "custom kernel". Is this a typical conversation people have? "Hey Jim, did you update the servers to get the latest OpenSSL security fixes? - Yap, I compiled and installed a custom kernel". Maybe they need to run a better distro with faster security update response?
- jsmthrowaway 11y agoIt's specific to VM images. "Custom" here means "not baked into Xen," since your filesystem is not considered when spawning a domU kernel except in limited circumstances. In the Xen world, your kernel is provided by your hosting provider. You can apt-get all day and nothing will happen. That is what custom means in this context. "Not yours." Read accordingly; you've made the same flawed point at several spots in this thread.
- rdtsc 11y agoAh, I understand now, thanks for explaining. I thought they used KVM for some reason... But I guess if they use Xen they yeah, they are stuck with whatever kernel they get.
- jsmthrowaway 11y agoIt's a slightly similar story under KVM in these scenarios. Customer kernels are trickier.
- rdtsc 11y agoDepending on what kind of level of virtualization they opted for, I've run Windows on Linux under KVM. They are doing the "boot a kernel" mode instead of fully virtufalized hardware mode probably to save on resources.
- danellis 11y agoDO uses KVM.
- andrewsomething 11y agoJust to confirm, we use KVM across our fleet at DO.
- DanielDent 11y agoI've got both Xen & KVM systems under my care. It's no longer true that Xen needs to mean managing the kernel outside the VM. PVGRUB can be specified as the 'kernel' to boot, which will chainload a grub which can be managed inside the VM, which lets you run any kernel you wish and manage the boot process as you would on a non-virtualized system. Amazon uses Xen for their EC2 product, and as I understand they too now set people up with pvgrub.
- makomk 11y agoVery few people run actual custom kernels, but most people want to run the distro-supplied kernel for their distro of choice (including security updates to it) and most decent providers configure their VMs to allow custom kernels so that their users can do this.
- rdtsc 11y ago> people want to run the distro-supplied kernel I read "custom kernel" as in "replace distro-supplied kernel and compile your own with some custom flags and patches".
- makomk 11y agoWell, you can do that too, but the point is that you have control over what kernel your VMs are running and not the hosting provider. From a technical perspective it makes little difference where that kernel comes from; either you control what kernel you're running or you don't.
- rdtsc 11y agoI was at least thinking that once you pick a distro, they'd be first in line updating the security patches for it (or at least as fast as say your own ops team would). But I guess that is not the case.
- jsmthrowaway 11y agoThis is especially big for RHEL, OEL, and friends. Yeah, CentOS is cool for your startup but a big portion of the valley wants support contracts so they can stop doing OS grunt work, and if your provider doesn't roll RHEL you get to deploy your own, and AFAIK that is not possible on DO (and requires quite a bit of work on Linode, its closest competitor in the space; DO is not AMZN). Deploying RHEL in a supported way requires using their kernels. It's your virtual machine. You should be able to pick a kernel. This isn't for running Andrew Morton patches, as some of the comments imply.
- guan 11y ago
- jorgecurio 11y agoMost people don't consider custom kernel as a requirement for serious projects. I don't see why this is a joke.
- deleted 11y ago[deleted]
- timdorr 11y agoIf you want to maintain your server well and keep the kernel updated against security vulnerabilities, that's a pretty big deal. It would suck to be the victim of a root escalation and have no way of preventing it.
- gruez 11y agohttps://news.ycombinator.com/item?id=11031421 https://news.ycombinator.com/item?id=11031421
- viraptor 11y agoIt's less about customised kernel and more about an up-to-date one. You may want to patch at your own schedule (faster, or slower than DO). Any serious project will take updates (especially security updates) seriously.
- mwpmaybe 11y agoIf you want to run your own LXC or Docker you might need a custom kernel to enable aufs.
- mayank 11y ago> 1. Your private IP addresses are accessible by everyone in the same datacenter. This is pretty bad compared to an AWS VPC -- you basically have to manage and sync your own iptables between all your nodes.
- alrs 11y agoI would suggest that instead of trying to maintain shared state (which IP addresses are blessed) across all of your nodes, you look into using ipsec. Those internal interfaces aren't for security, they're to segment cheap/fast network traffic internal to the dc from expensive/slow/metered traffic that hits the Internet. https://en.wikipedia.org/wiki/Opportunistic_encryption https://en.wikipedia.org/wiki/Opportunistic_encryption
- JoshTriplett 11y agoPerhaps, but at the same time, you shouldn't be using IP addresses as a security mechanism. Assume the connection between your hosts is compromised, and code accordingly, with encrypted/authenticated connections between hosts.
- tptacek 11y agoNot that I want to wade into the "don't use D.O." part of this argument, but, in practice, nobody does this. Virtually every deployment environment I've ever seen with more than 4 hosts in it would be fatally compromised by an attacker who could reach any IP address in that environment.
- JoshTriplett 11y agoTrue. I haven't heard folks other than Google explicitly talking about this as a best practice.
- jsmthrowaway 11y agoA VPC is analogous to a physical network, not a subnet. Nobody uses them that way because it's not easy to grok, but you can treat a VPC as a physical network complete with your own numbering and ACL policies. If you're doing that defense in depth on a physical network, I'm impressed by your dedication but would avoid your work for wasting resources.
- Sebguer 11y agoUser-managed kernels are currently supported on new distro releases, and will be supported for all distributions that are released moving forward.
- DanielDent 11y ago"IPv6 support 1. Took forever to implement, and the timetable broke promises to customers. 2. Inferior. Digital Ocean still won't give you a /64 per standards." Far more egregious is that they silently drop port 25 on IPv6. This means that enabling IPv6 will cause mail problems for some destinations (destinations that support IPv6, like Google). When asked they say it's because a /64 is too much address space in the hands of potential (ab)users. This fails to understand that an IPv6 /64 is conceptually similar to an IPv4 /32. (In fact, there are pretty reasonable arguments for assigning IPv6 /56s or /48s with the same semantics as how IPv4 /32s are assigned.)
- joering2 11y agoI can't talk about port 25, but we have 15 mailservers hosted on $5/per pop with DigitalOcean and with properly setup rDNS and SPF including our IPv6 address, we don't see any problems with deliveribility to gmail via subscription port. That is on circa 600,000 emails per month sent to gmail.
- robertcope 11y agoIt sounds like you're talking about sending mail, not receiving it. Port 25 is only an issue if you're trying to receive email.
- mwpmaybe 11y agoIf you want to send mail as a peer (i.e. without using a smart host / relay) you need to be able to connect outbound on port 25, which some providers block.
- DanielDent 11y agoYou send server-to-server messages on port 587? That is... non-standard. But an interesting approach. You are using your users' gmail credentials? Or are you simply delivering over 587 the same as you would over 25?
- 11y ago
- 3pt14159 11y ago> 3. These kernels often contain relevant security vulnerabilities. This is really the only part of the list that I think has merit. I love Digital Ocean in general, but my own gripe is with their API and its lack of proper automatic key management: https://digitalocean.uservoice.com/forums/136585-digitalocean/suggestions/9307569-return-the-droplet-s-ssh-public-key-as-part-of-api https://digitalocean.uservoice.com/forums/136585-digitalocea...
- Sicp 11y agoyou're the joke
- tschellenbach 11y agoOk, no serious project would run their own kernel. What are you hosting that you want to run your own kernel?
- timdorr 11y agoIf you want to maintain your server well and keep the kernel updated against security vulnerabilities, that's a pretty big deal. It would suck to be the victim of a root escalation and have no way of preventing it.
- api 11y agoIf you want a DO competitor check out vultr.com -- I've used them for years and been quite impressed. They seem at least as good if not better and they have some very interesting features like turn-key BGP and AnyCast hosting. I have no affiliation with them.
- DanielDent 11y agoFor the most part, I think Vultr has a better offering than Digital Ocean. However, I had an infuriating experience with Vultr recently (not that I have reason to believe DO would have done any better). Vultr has poorly configured DDoS mitigation equipment. If that equipment believes your IP is under DDoS, it will automatically blackhole your IP from their upstreams. With near-zero traffic on my interface - and for an internally facing VM, the IP address of which has never been published in any DNS, their DDoS 'protection' system decided I was under DDoS attack and blackholed my IP with their upstreams. I investigated and my conclusion is that the most likely cause was that their system mis-idenfied the overlay networking software I was using (which communicates over UDP) as DDoS traffic. I raised this issue with support, who did not manage to help. Amongst other things, they told me that they didn't retain logs long enough to be helpful. I agreed that I would raise the issue with them again should it re-occur. The second time they blackholed my IP, they didn't even bother telling me. And when I filed a support ticket, they took somewhere around an hour to even respond - during the time they've taken my system offline, I expect them to make themselves available. When I tried calling their parent company on the phone, they were downright rude with me. I understand they don't usually do phone support, and that's fine, but if you've told me that (A) you can only help me resolve a problem during an incident because you don't keep logs, and (B) don't make yourself available to me through your regular support channels, don't get pissy with me for phoning you. And especially don't completely refuse to do anything remotely helpful. I ended up terminating the VM and moving that workload elsewhere. I don't have time to be fixing other people's networks, especially when they can't be bothered to participate in the process. It's a shame, because their BGP & AnyCast stuff looked really interesting, and I'd love to explore their offerings more. But I don't currently have confidence I can deploy anything with them in production, because when something goes wrong with their service, they don't appear competent at making themselves available to fix things. (For the record, in some respects I consider one of my companies to be a competitor to both Digital Ocean & Vultr. But I like to be familiar with the competition, so I use them for some things. It also provides us a way to put workloads in locations we wouldn't otherwise be able to justify.)
- manacit 11y agoCustom kernel support is actually available for certain distributions on DigitalOcean, though it's not widely marketed. Right now, Debian 8, Ubuntu 15.04, Ubuntu 15.10, Fedora 23 and FreeBSD all allow you to control your kernel version. The next LTS release of Ubuntu in April, 16.04, will also allow you to customize your kernel.
- andrewsomething 11y agoThere seems to be a bit of confusion about how we handle kernels at DO. Hopefully I can clear things up a bit. On newer distro versions (e.g. Ubuntu >= 15.04, Debian 8, Fedora, CoreOS, FreeBSD) we no longer use "external" kernels. You are free to compile and use custom kernels. We're happy with what we've seen, and with Ubuntu 16.04 around the corner our default distribution will have support for this as well. After that point, we'll be backporting the change to older releases. For Droplets still running with external kernels, we import new ones on a regular basis as they are released. If you happen to need one that hasn't been imported yet, just open a support ticket and the team will do so.
- lobster_johnson 11y agoAre you planning to offer storage options? I find it's ridiculous to have to upgrade to a higher CPU/RAM plan (or add a new node) just because we're running out of disk space. We don't need detachable volumes or anything fancy, just the ability to increase the amount of disk storage.
- andrewsomething 11y agoStorage is also an area we're working on heavily right now, and we should have some good news soon. We definitely recognize that there's a lot of demand for more storage without the need for the corresponding upgrade in compute power. Check out the update from our product team on this UserVoice request, and vote/subscribe to get updates: http://digitalocean.uservoice.com/forums/136585-digitalocean/suggestions/3127077-extra-diskspace http://digitalocean.uservoice.com/forums/136585-digitalocean...
- lobster_johnson 11y agoThanks!
- timfrietas 11y agoProduct manager for Storage at DO here. I generally avoid comment on these kind of things but given the public statements already out there it is safe to say we are launching storage reasonably soon. If you are truly interested in participating in the beta program, email me at tfrietas@digitalocean.com
- Sephr 11y ago> 2. Inferior. Digital Ocean still won't give you a /64 per standards. Huh? I made a droplet just a week ago and it was assigned a whole /64. Maybe this is only a recent change, but you should know that it's here now.
- X-Istence 11y agoYep, but you are allowed to use only 16 of them. Check the configurable address range.
- mwpmaybe 11y ago> Private networking is a joke > 1. Your private IP addresses are accessible by everyone in the same datacenter. This is also the case at Linode. Besides AWS VPC, are there VPS providers that give your hosts their own private VLAN?
- deleted 11y ago[deleted]
- cddotdotslash 11y agoLet's add to this the fact that if you're under a DOS attack with DigitalOcean they disconnect your machine from the internet (making it impossible for you to log in and do log analysis, etc.), send you an email saying "figure out what's happening and stop it", and then reconnect your machine several hours later only to repeat the process practically as soon as it's back online (assuming the DOS continues). I wouldn't trust a side project on DO, let alone my business.
- dayon 11y agoI was about to post this. My project received a DDOS, and I could not access my droplet for over 12 hours.
- andrewthetechie 11y agoJust to clarify, DigitalOcean does not offer any form of DoS mitigation services so they blackhole during a DoS. Its for 3 hours, a lot less than other providers. If you've got a DoS issue, you definitely need a 3rd party DoS protection service. Cloudflare free works pretty well
- cddotdotslash 11y agoIt's not "a lot less than other providers". You know what happens when someone starts DDOSing one of my AWS servers? I get a CloudWatch alert saying "high inbound traffic" and that's it. They don't black hole the thing and cut off all traffic. Then, I can log in, see what's happening, and take my time diagnosing the problem. Even under a fairly heavy DDOS I never feared losing access. With DO, the black hole happens before the email alert. It's a terrible policy and I can't use or recommend DO until it's changed. I can't stick CloudFlare in front of every server I own.