3 ms·

You'll get a lot of mnesia partitions on even the best network with a slightly busy system. It's not very sturdy, and resynchs are really slow.

by scurvy 11y ago

toast0 11y ago
I only see mnesia partitions when the network is having issues. Maybe my network is better than your best network? Maybe network issues a few times a year is a lot? Resynchs are really slow, because "resynch" is copy data from the network. Mostly this goes close to the line speed for your network though; be sure to move the data files out of the way on the node that will be receiving the copies: otherwise mnesia first loads those, then throws them away sigh. It would be pretty nice if there was support for logging changes when a node is down, though.
- lostcolony 11y ago
  Yeah; running Mnesia in production, albeit for pretty low volume systems (maybe 3000 writes a day on average), in multiple clusters, in different locations, running from anywhere between half a year to 2 years, I think I've seen...two netsplits, total, both of them caused by a VM snapshotting process that caused the socket to hang. But we've got 24/7 ops keeping an eye on things in any case, and the data being stored is maybe ~1 gig, so resyncing is fast. It fits our use case.
- scurvy 11y ago
  >I only see mnesia partitions when the network is having issues. Maybe my network is better than your best network?< Doubtful on the network thing. mnesia will partition then resync when the server gets really busy. As others have mentioned, it might have nothing to do with anything that Erlang is doing. It might be something external. It could just be a lot of traffic that causes it to fall behind. Either way, one missed message and then you're forced to fully resync. >Resynchs are really slow, because "resynch" is copy data from the network. Mostly this goes close to the line speed for your network though< Which means your other node has its interface maxed out, causing more service disruptions. I've never run mnesia on a 10gig network, but that definitely was the case with 1gig. I'm not really willing to test or run mnesia in a 10gig/40gig environment. Been burned by it too many times. >Mostly this goes close to the line speed for your network though; be sure to move the data files out of the way on the node that will be receiving the copies: otherwise mnesia first loads those, then throws them away sigh. It would be pretty nice if there was support for logging changes when a node is down, though.< Which again, is another sign that it's not robust enough for Internet application usage. Probably OK for some 1990's phone switching, but not for how distributed systems are built today. How are you going to manually move the files out of the way in today's world of systemd automatically restarting failed daemons? Manual operator intervention? Thought so, and this is why ops teams hate mnesia.

You'll get a lot of mnesia partitions on even the best network with a slightly busy system. It's not very sturdy, and resynchs are really slow.

by scurvy 11y ago

You'll get a lot of mnesia partitions on even the best network with a slightly busy system. It's not very sturdy, and resynchs are really slow.

toast0 11y ago

I only see mnesia partitions when the network is having issues. Maybe my network is better than your best network? Maybe network issues a few times a year is a lot? Resynchs are really slow, because "resynch" is copy data from the network. Mostly this goes close to the line speed for your network though; be sure to move the data files out of the way on the node that will be receiving the copies: otherwise mnesia first loads those, then throws them away sigh. It would be pretty nice if there was support for logging changes when a node is down, though.

lostcolony 11y ago
Yeah; running Mnesia in production, albeit for pretty low volume systems (maybe 3000 writes a day on average), in multiple clusters, in different locations, running from anywhere between half a year to 2 years, I think I've seen...two netsplits, total, both of them caused by a VM snapshotting process that caused the socket to hang. But we've got 24/7 ops keeping an eye on things in any case, and the data being stored is maybe ~1 gig, so resyncing is fast. It fits our use case.
scurvy 11y ago
>I only see mnesia partitions when the network is having issues. Maybe my network is better than your best network?< Doubtful on the network thing. mnesia will partition then resync when the server gets really busy. As others have mentioned, it might have nothing to do with anything that Erlang is doing. It might be something external. It could just be a lot of traffic that causes it to fall behind. Either way, one missed message and then you're forced to fully resync. >Resynchs are really slow, because "resynch" is copy data from the network. Mostly this goes close to the line speed for your network though< Which means your other node has its interface maxed out, causing more service disruptions. I've never run mnesia on a 10gig network, but that definitely was the case with 1gig. I'm not really willing to test or run mnesia in a 10gig/40gig environment. Been burned by it too many times. >Mostly this goes close to the line speed for your network though; be sure to move the data files out of the way on the node that will be receiving the copies: otherwise mnesia first loads those, then throws them away sigh. It would be pretty nice if there was support for logging changes when a node is down, though.< Which again, is another sign that it's not robust enough for Internet application usage. Probably OK for some 1990's phone switching, but not for how distributed systems are built today. How are you going to manually move the files out of the way in today's world of systemd automatically restarting failed daemons? Manual operator intervention? Thought so, and this is why ops teams hate mnesia.