4 ms·

The reason I'm still wary of the cloud is that the abstractions are still too leaky (to borrow Joel Spolsky's turn of phrase). When you start abstracting away c

The reason I'm still wary of the cloud is that the abstractions are still too leaky (to borrow Joel Spolsky's turn of phrase). When you start abstracting away core system calls (like fsync), things work great 99.99% of the time. But when that .01% bites you, it bites hard. We don't expect core system calls (like fsync) to fail. And when they fail, the fact that our code is two or three levels levels of abstraction higher means that we often have no way of fixing the issue. The cloud will be at a disadvantage to "real servers" as long as its abstractions are leaky enough to be distinguishable from real hardware.

wulczer 15y ago
Cloud or not, if you're writing serious software (and by that I mean software that depends on fsync succeeding) you'd better expect system calls to fail and have code to handle that.
- ianburrell 15y ago
  The failure with fsync is not that the system call will fail but that it will succeed but be wrong about writing to durable storage.
wmf 15y ago
In what way does fsync work differently in the cloud?
- josephcooney 15y ago
  It seems that (at least for AWS) EBS might not always do what you expect with fsync() http://teddziuba.com/2011/04/amazon-the-purpose-of-pain.html http://teddziuba.com/2011/04/amazon-the-purpose-of-pain.html
- fleitz 15y ago
  In the way that you have no idea whether fsync actually writes the data to disk, or whether it puts it in memory somewhere (usually a write cache).
  - jrmg 15y ago
    Most modern hard drives don't actually do what most people expect an fsync does anyway. This is an interesting thread about Apple's F_FULLFSYNC ioctl that discusses the problem (this pat is by Apple FS engineer Dominic Giampaolo): http://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072.html http://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072...
- bad_user 15y ago
  Because there's an extra virtual layer between you and the hard-disk, fsync() does not do what it says it does, which means that a database cannot actually guarantee that a transaction was successfully completed. In case your traffic is huge, this can cause big problems that are very hard to fix. And it did cause problems for Reddit, as EBS is notoriously awful in this regard (and others, like big latencies on access). Basically when working at big scales, nothing beats having complete control over your infrastructure. It may be tough and time consuming, but at least you can identify the problem and fix it. It is my opinion that something like Google's search cannot be built on top of Amazon's AWS.
  - asharp 15y ago
    Saying that because fsync() doesn't work in AWS that it won't work 'in the cloud' is much like saying that because you can't take a cheap sedan to the bush that a 4wd can't possibly exist. Google can't exist (well) on top of AWS as AWS is designed for the deployment of (qusi)stateless applications. Saying that you can't build a data intensive application on top of it is like saying that your machinegun sucks at grating cheese. OVM was designed to run a search engine origionally (darkmatr, now defunct due to lack of comparitive profitability), so you could quite easily build google on top of it. It wouldn't surprise me if google would work really well on top of that stack.
asharp 15y ago
Keep in mind that screwing with syscalls is an AWS thing, not a virtualisation thing. They make fsync wierd for good reason though, and they really don't have much they can do.
- quanticle 15y ago
  Right. The question, however, was about working in the "cloud", which usually means that you just have access to the VMs, not to the underlying hardware. VMs are great, but there's nothing like root access to the physical computer to help you diagnose hardware issues.
  - asharp 15y ago
    Yes and no. Remember the interface between the vm and hypervisor is standardised and it is running many other functionaly identical vms. Also remember that platform they are running on is usually homogeneous, very well maintained, monitored, and understood. Also keep in mind that with 'hardware issues' you can reploy onto another machine in minutes, rather then having to wait for say a replacement part from dell/etc. Your standard procedures should come into play in both cases (Failing to a secondary server, bringing a tertiary server into standby, etc.)

wulczer 15y ago

Cloud or not, if you're writing serious software (and by that I mean software that depends on fsync succeeding) you'd better expect system calls to fail and have code to handle that.

ianburrell 15y ago
The failure with fsync is not that the system call will fail but that it will succeed but be wrong about writing to durable storage.

wmf 15y ago

In what way does fsync work differently in the cloud?

josephcooney 15y ago
It seems that (at least for AWS) EBS might not always do what you expect with fsync() http://teddziuba.com/2011/04/amazon-the-purpose-of-pain.html http://teddziuba.com/2011/04/amazon-the-purpose-of-pain.html
fleitz 15y ago
In the way that you have no idea whether fsync actually writes the data to disk, or whether it puts it in memory somewhere (usually a write cache).
- jrmg 15y ago
  Most modern hard drives don't actually do what most people expect an fsync does anyway. This is an interesting thread about Apple's F_FULLFSYNC ioctl that discusses the problem (this pat is by Apple FS engineer Dominic Giampaolo): http://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072.html http://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072...
bad_user 15y ago
Because there's an extra virtual layer between you and the hard-disk, fsync() does not do what it says it does, which means that a database cannot actually guarantee that a transaction was successfully completed. In case your traffic is huge, this can cause big problems that are very hard to fix. And it did cause problems for Reddit, as EBS is notoriously awful in this regard (and others, like big latencies on access). Basically when working at big scales, nothing beats having complete control over your infrastructure. It may be tough and time consuming, but at least you can identify the problem and fix it. It is my opinion that something like Google's search cannot be built on top of Amazon's AWS.
- asharp 15y ago
  Saying that because fsync() doesn't work in AWS that it won't work 'in the cloud' is much like saying that because you can't take a cheap sedan to the bush that a 4wd can't possibly exist. Google can't exist (well) on top of AWS as AWS is designed for the deployment of (qusi)stateless applications. Saying that you can't build a data intensive application on top of it is like saying that your machinegun sucks at grating cheese. OVM was designed to run a search engine origionally (darkmatr, now defunct due to lack of comparitive profitability), so you could quite easily build google on top of it. It wouldn't surprise me if google would work really well on top of that stack.

asharp 15y ago

Keep in mind that screwing with syscalls is an AWS thing, not a virtualisation thing. They make fsync wierd for good reason though, and they really don't have much they can do.

quanticle 15y ago
Right. The question, however, was about working in the "cloud", which usually means that you just have access to the VMs, not to the underlying hardware. VMs are great, but there's nothing like root access to the physical computer to help you diagnose hardware issues.
- asharp 15y ago
  Yes and no. Remember the interface between the vm and hypervisor is standardised and it is running many other functionaly identical vms. Also remember that platform they are running on is usually homogeneous, very well maintained, monitored, and understood. Also keep in mind that with 'hardware issues' you can reploy onto another machine in minutes, rather then having to wait for say a replacement part from dell/etc. Your standard procedures should come into play in both cases (Failing to a secondary server, bringing a tertiary server into standby, etc.)