3 ms·

The article wasn't about the decision of when to write to disk, but how to actually do it. Say you have a point in your program where you have made the decision

The article wasn't about the decision of when to write to disk, but how to actually do it. Say you have a point in your program where you have made the decision that it is necessary to write to disk. How do you actually do that for sure? It's not write() or even necessarily fsync(). The author is finding this frustrating because there are several things to do that can seem arbitrary, random, and counterintuitive. His trust in the OS was damaged and now he or she is understandably grumpy about it.

rdtsc 11y ago
> How do you actually do that for sure? It's not write() or even necessarily fsync(). Right. For decision of "when" you can configure a few kernel parameters, namely dirty writes thresholds. That assumes you just do write()s from your process and kernel decides to flush that data out. That can be configured to be a function of time and or amount of unwritten data. I had to do it a few times. Once it was a realtime-ish system which was recording data to disk, and noticed recording thread would time-out. (Timeouts were in the 10s of seconds and there were noticed by a watchdog system). Thought 10 seconds should be enough for that process. But it turns out because of how priorities were setup and how fast writes went, dirty page flushing was not keeping up with writes. Periodically it would hit the top threshold and at that point any process doing disk writes would be blocked. Was able to get it under control by essentially doing what gp suggested, writing a little bit at a time, but more often. The total throughput probably went went down but performance was smoothed out quite a bit.
- planckscnst 11y ago
  That's still missing the point. Say you're writing a program that stores a log of transactions. People are trading resources and you log those transactions so you can find out who owns which resources. User A just gave 100 units to user B; you need to store this information and tell both users that the transaction is complete. How do you do that? Your program will do something like "write(t_log_fd, t_entry, t_bytes);". Can you now tell the users that the transaction is complete?
  - rdtsc 11y ago
    I guess I have misunderstood, your question about "The article wasn't about the decision of when to write to disk". I thought you are asking how to control in general when your system is writing to disk. Well if you don't use fsync then it decides at some point like I mentioned above. If want to talk about transactions then do a write and fsync, does that not work someetimes fro you? If you are worried about the new file appearing in the directory then fsync the directory as well. If you want more guarantees, you'll have to dig deeper and find out about your specific device, does it have battery power and does it tell lies about it writing data and so on.

rdtsc 11y ago

> How do you actually do that for sure? It's not write() or even necessarily fsync(). Right. For decision of "when" you can configure a few kernel parameters, namely dirty writes thresholds. That assumes you just do write()s from your process and kernel decides to flush that data out. That can be configured to be a function of time and or amount of unwritten data. I had to do it a few times. Once it was a realtime-ish system which was recording data to disk, and noticed recording thread would time-out. (Timeouts were in the 10s of seconds and there were noticed by a watchdog system). Thought 10 seconds should be enough for that process. But it turns out because of how priorities were setup and how fast writes went, dirty page flushing was not keeping up with writes. Periodically it would hit the top threshold and at that point any process doing disk writes would be blocked. Was able to get it under control by essentially doing what gp suggested, writing a little bit at a time, but more often. The total throughput probably went went down but performance was smoothed out quite a bit.

planckscnst 11y ago
That's still missing the point. Say you're writing a program that stores a log of transactions. People are trading resources and you log those transactions so you can find out who owns which resources. User A just gave 100 units to user B; you need to store this information and tell both users that the transaction is complete. How do you do that? Your program will do something like "write(t_log_fd, t_entry, t_bytes);". Can you now tell the users that the transaction is complete?
- rdtsc 11y ago
  I guess I have misunderstood, your question about "The article wasn't about the decision of when to write to disk". I thought you are asking how to control in general when your system is writing to disk. Well if you don't use fsync then it decides at some point like I mentioned above. If want to talk about transactions then do a write and fsync, does that not work someetimes fro you? If you are worried about the new file appearing in the directory then fsync the directory as well. If you want more guarantees, you'll have to dig deeper and find out about your specific device, does it have battery power and does it tell lies about it writing data and so on.