4 ms·

Let's see: Say that one of these storms arrives at a point in time. Then over time, these storms form a stochastic point process. Now it should be fairly cle

by graycat 12y ago

Let's see: Say that one of these storms arrives at a point in time. Then over time, these storms form a stochastic point process. Now it should be fairly clear intuitively that at least roughly increments, that is, arrivals of points, are stationary in time and independent. Presto: It follows that the stochastic point process is a Poisson process about which we know a lot. So, we'd like the arrival rate of this Poisson process, say, in arrivals per year. Okay, we can get a good estimate just by dividing the number of arrivals we have observed over the number of years we have been observing. Then with this estimate of the arrival rate, we can calculate the probability of an arrival in, say, the next 10 years. Since, when was it, 18?? or some such, we've actually had no arrivals at all that hit Earth, the arrival rate is low and the chances of an arrival in the next 10 years is low. Relax! Good news, right? I mean, the sky is not really falling this time after all!

mturmon 12y ago
Bad news for this model: it's not stationary, due to the 11-year solar cycle, and to other behaviors at longer time scales superimposed on the cycle. But if you average out over at least the solar cycle, you could come up with a rate of superstorms per-cycle. There have been about 15 cycles since the 1859 superstorm, which would mean that perhaps you expect ~ 1 chance in 15 of such a storm in the coming cycle.
- graycat 12y ago
  Of course what I mentioned is also called the 'axiomatic' derivation of the Poisson process. In part it's nice because it starts with 'qualitative' assumptions and ends up with the full algebra of the process and its distribution and independence of times between arrivals. Details are in E. Cinlar, 'Introduction to Stochastic Processes' in his chapter on the Poisson process. There he also touches on the non-stationary case, and what he has may (I'm in hurry this morning) be equivalent to your "average out over ... the cycle". Since a finite sum of independent Poisson processes is a Poisson process, your 1 in 15 over the next cycle may be equivalent to 1 in 2014 - 1859 years over the next year. I do the same for asteroids: We have a good ballpark estimate of the rate of killer asteroids, and the rate is low enough to f'get about it.
  - mturmon 12y ago
    Yes, you described a case ("homogeneous P.P.") where the intensity, lambda, is constant over time. In the inhomogeneous P.P., there is a time-varying rate, lambda(t). The derivation of the properties in the inhomogeneous case is almost the same because the key property is independent increments. When you add up the number of catastrophic events over an 11-year cycle, you get a sequence of "numbers of catastrophic CMEs" N1, N2, ..., Nk each of which is Poisson with intensity \int_{cycle} lambda(t) dt [*] If you ignore the larger-than-a-cycle variations in intensity, this integral is the same for each cycle. So the N's above are iid Poisson with some common parameter lambda_0, say. Observing the N's for many cycles would allow you to estimate lambda_0. That's pretty much where the 1 in 15 comes from (indeed, lambda_0 is so low that there are only 0 or 1 events per cycle, so "estimation" is trivial). The "easy" problem is that, even averaging over a cycle, the activity ([*] above) is not constant, e.g. this graph of sunspot numbers vs. time: http://blogs.nicholas.duke.edu/thegreengrok/files/2012/07/sunspotnumbers.jpg http://blogs.nicholas.duke.edu/thegreengrok/files/2012/07/su... The "hard" problem is that the counts are not, in real life, independent, even across weeks-to-months time scales. This would invalidate the Poisson model at the ground level (see the talk referenced by @gone35 above). There are reasons (e.g., Maunder minimum, see the plot above) to believe that the counts are also not independent at the decadal time scale. Lack of independence would be a huge problem for any Poisson model. You'd have to go up to a doubly stochastic process, for example, in which lambda(t) is itself a random process.
    - graycat 12y ago
      You are guilty of letting a lot of really messy data ruin some really nice theory! So, with assumptions of stationary and independent increments, the Maunder minimum shown in the graph would have probability less than zip and zilch of happening in which case we would reject the assumptions. You also are analyzing sun spots while the OP is talking about dangerous CMEs. It's been a while since I studied sun spots: You are saying that a sun spot is just a small case of a CME? Or is a dangerous CME not just a big sun spot but a different animal?
      - mturmon 12y ago
greiskul 12y ago
"Now it should be fairly clear intuitively that at least roughly increments, that is, arrivals of points, are stationary in time and independent." That's not intuitive at all. They are all produced by the same source, the sun, according to some internal cycle. They are definitely NOT independent.
- graycat 12y ago
  Tell you what: I'll give you the history of the solar storms, mass ejections, up-chucks, whatever, and you tell me when the next up-chuck is. If it can't be done, then independence has to hold. If in practice we don't have a clue, then intuitively we have to guess that we have a good first-cut approximation to independence. Here I'm omitting the little derivation in terms of conditional probability -- I've done it too often in the past and am in a hurry this morning. "The same source" doesn't have to mean much. We can flip the same coin and get independent results. The internals of the sun are complicated with, no doubt, a lot of randomness, i.e., roiling and boiling. For a good first cut in practice, at least over time intervals longer than, say, a week, we assume independence. Or, I've got a pot of soup overheated and boiling on the stove, and occasionally it splatters and puts a mess on the stove top. When's the next splatter? Who knows. So, first-cut, independence.
gone35 12y ago
It should be fairly clear intuitively that at least roughly increments, that is, arrivals of points, are stationary in time and independent. Far from it, insofar as the (time-dependent) solar cycle modulates the storm rate [1]. Besides there is some evidence of dependence between the flares themselves --strong enough to manifest itself in a complicated waiting time empirical distribution [1,2]. Be careful with the common --but nefarious-- habit of adducing stationarities and 'nice' distributions unwarrantedly. In the meantime, y-you are not allowed statistics anymore, sorry. [1] http://www.physics.usyd.edu.au/~wheat/talks/fstats_agu_2005.pdf http://www.physics.usyd.edu.au/~wheat/talks/fstats_agu_2005.... [2] http://researchcommons.waikato.ac.nz/handle/10289/1382 http://researchcommons.waikato.ac.nz/handle/10289/1382
- graycat 12y ago
  > Far from it, What? While I'm reading your references, let's be clear on what you are saying: On the one hand, in the context of the OP, we're talking about flairs, as essentially discrete events, 'points' in a 'point process', over time intervals from the present back to at least 1889 or whenever it was, that is, over 100 years, and, if we could get more data, thousands or millions of years and (2) what happens in detail with the sun spots, the 11 year sun spot cycle, the magnetic field of the sun, etc. So, sure, on short time scales, say, within one 'point' there can be a lot of dependence, that is, lack of 'independent increments'. But for these 'points' over years, centuries, or millennia? You are saying that there is a dependence between a flair today and 11 years from now? Assuming that the 'dangerous flairs' are more common when sun spots are in the 11 year cycle, for modeling the stochastic arrival process over several decades, centuries, etc. we should, first-cut, and I saw a lot of first-cutting when I was in physics class, 'smooth out' the 11 year cycle and continue on. Then, tough to believe that you are claiming dependence between points separated by months or years. E.g., I see in your [2] in Figure 2 on page 463 a distribution of waiting times with the horizontal axis a log scale up to 10,000 hours but clearly nearly all the probability mass less than 100 hours, here being sure to note the log scale on the vertical axis. So, in the context of the OP, that 100 hours is within just one particular 'point' in the stochastic point process and essentially irrelevant when considering years, decades, centuries, etc. So, the independent increments assumption seems to continue to hold. And with the smearing out, so does the stationarity. Then, from Cinlar, presto, a Poisson process. Oops: Apparently the Maunder minimum, as in the graph below in this thread, kills off the stationarity assumption, even with the 'smoothing'. So, apparently until we can understand things like the Maunder minimum, we're really short on a good stochastic point process model of sun spots and maybe also dangerous CMEs. Then maybe the best we have is just observe that both humans and life on earth have been around a long time and in some intuitive sense the sun seems fairly 'stable' in some meaningful sense so that we can guess we will still be here a long time. Math fails again!

by graycat 12y ago

mturmon 12y ago

Bad news for this model: it's not stationary, due to the 11-year solar cycle, and to other behaviors at longer time scales superimposed on the cycle. But if you average out over at least the solar cycle, you could come up with a rate of superstorms per-cycle. There have been about 15 cycles since the 1859 superstorm, which would mean that perhaps you expect ~ 1 chance in 15 of such a storm in the coming cycle.

graycat 12y ago
Of course what I mentioned is also called the 'axiomatic' derivation of the Poisson process. In part it's nice because it starts with 'qualitative' assumptions and ends up with the full algebra of the process and its distribution and independence of times between arrivals. Details are in E. Cinlar, 'Introduction to Stochastic Processes' in his chapter on the Poisson process. There he also touches on the non-stationary case, and what he has may (I'm in hurry this morning) be equivalent to your "average out over ... the cycle". Since a finite sum of independent Poisson processes is a Poisson process, your 1 in 15 over the next cycle may be equivalent to 1 in 2014 - 1859 years over the next year. I do the same for asteroids: We have a good ballpark estimate of the rate of killer asteroids, and the rate is low enough to f'get about it.
- mturmon 12y ago
  Yes, you described a case ("homogeneous P.P.") where the intensity, lambda, is constant over time. In the inhomogeneous P.P., there is a time-varying rate, lambda(t). The derivation of the properties in the inhomogeneous case is almost the same because the key property is independent increments. When you add up the number of catastrophic events over an 11-year cycle, you get a sequence of "numbers of catastrophic CMEs" N1, N2, ..., Nk each of which is Poisson with intensity \int_{cycle} lambda(t) dt [*] If you ignore the larger-than-a-cycle variations in intensity, this integral is the same for each cycle. So the N's above are iid Poisson with some common parameter lambda_0, say. Observing the N's for many cycles would allow you to estimate lambda_0. That's pretty much where the 1 in 15 comes from (indeed, lambda_0 is so low that there are only 0 or 1 events per cycle, so "estimation" is trivial). The "easy" problem is that, even averaging over a cycle, the activity ([*] above) is not constant, e.g. this graph of sunspot numbers vs. time: http://blogs.nicholas.duke.edu/thegreengrok/files/2012/07/sunspotnumbers.jpg http://blogs.nicholas.duke.edu/thegreengrok/files/2012/07/su... The "hard" problem is that the counts are not, in real life, independent, even across weeks-to-months time scales. This would invalidate the Poisson model at the ground level (see the talk referenced by @gone35 above). There are reasons (e.g., Maunder minimum, see the plot above) to believe that the counts are also not independent at the decadal time scale. Lack of independence would be a huge problem for any Poisson model. You'd have to go up to a doubly stochastic process, for example, in which lambda(t) is itself a random process.
  - graycat 12y ago
    You are guilty of letting a lot of really messy data ruin some really nice theory! So, with assumptions of stationary and independent increments, the Maunder minimum shown in the graph would have probability less than zip and zilch of happening in which case we would reject the assumptions. You also are analyzing sun spots while the OP is talking about dangerous CMEs. It's been a while since I studied sun spots: You are saying that a sun spot is just a small case of a CME? Or is a dangerous CME not just a big sun spot but a different animal?
    - mturmon 12y ago

greiskul 12y ago

"Now it should be fairly clear intuitively that at least roughly increments, that is, arrivals of points, are stationary in time and independent." That's not intuitive at all. They are all produced by the same source, the sun, according to some internal cycle. They are definitely NOT independent.

graycat 12y ago
Tell you what: I'll give you the history of the solar storms, mass ejections, up-chucks, whatever, and you tell me when the next up-chuck is. If it can't be done, then independence has to hold. If in practice we don't have a clue, then intuitively we have to guess that we have a good first-cut approximation to independence. Here I'm omitting the little derivation in terms of conditional probability -- I've done it too often in the past and am in a hurry this morning. "The same source" doesn't have to mean much. We can flip the same coin and get independent results. The internals of the sun are complicated with, no doubt, a lot of randomness, i.e., roiling and boiling. For a good first cut in practice, at least over time intervals longer than, say, a week, we assume independence. Or, I've got a pot of soup overheated and boiling on the stove, and occasionally it splatters and puts a mess on the stove top. When's the next splatter? Who knows. So, first-cut, independence.

gone35 12y ago

It should be fairly clear intuitively that at least roughly increments, that is, arrivals of points, are stationary in time and independent. Far from it, insofar as the (time-dependent) solar cycle modulates the storm rate [1]. Besides there is some evidence of dependence between the flares themselves --strong enough to manifest itself in a complicated waiting time empirical distribution [1,2]. Be careful with the common --but nefarious-- habit of adducing stationarities and 'nice' distributions unwarrantedly. In the meantime, y-you are not allowed statistics anymore, sorry. [1] http://www.physics.usyd.edu.au/~wheat/talks/fstats_agu_2005.pdf http://www.physics.usyd.edu.au/~wheat/talks/fstats_agu_2005.... [2] http://researchcommons.waikato.ac.nz/handle/10289/1382 http://researchcommons.waikato.ac.nz/handle/10289/1382

graycat 12y ago
> Far from it, What? While I'm reading your references, let's be clear on what you are saying: On the one hand, in the context of the OP, we're talking about flairs, as essentially discrete events, 'points' in a 'point process', over time intervals from the present back to at least 1889 or whenever it was, that is, over 100 years, and, if we could get more data, thousands or millions of years and (2) what happens in detail with the sun spots, the 11 year sun spot cycle, the magnetic field of the sun, etc. So, sure, on short time scales, say, within one 'point' there can be a lot of dependence, that is, lack of 'independent increments'. But for these 'points' over years, centuries, or millennia? You are saying that there is a dependence between a flair today and 11 years from now? Assuming that the 'dangerous flairs' are more common when sun spots are in the 11 year cycle, for modeling the stochastic arrival process over several decades, centuries, etc. we should, first-cut, and I saw a lot of first-cutting when I was in physics class, 'smooth out' the 11 year cycle and continue on. Then, tough to believe that you are claiming dependence between points separated by months or years. E.g., I see in your [2] in Figure 2 on page 463 a distribution of waiting times with the horizontal axis a log scale up to 10,000 hours but clearly nearly all the probability mass less than 100 hours, here being sure to note the log scale on the vertical axis. So, in the context of the OP, that 100 hours is within just one particular 'point' in the stochastic point process and essentially irrelevant when considering years, decades, centuries, etc. So, the independent increments assumption seems to continue to hold. And with the smearing out, so does the stationarity. Then, from Cinlar, presto, a Poisson process. Oops: Apparently the Maunder minimum, as in the graph below in this thread, kills off the stationarity assumption, even with the 'smoothing'. So, apparently until we can understand things like the Maunder minimum, we're really short on a good stochastic point process model of sun spots and maybe also dangerous CMEs. Then maybe the best we have is just observe that both humans and life on earth have been around a long time and in some intuitive sense the sun seems fairly 'stable' in some meaningful sense so that we can guess we will still be here a long time. Math fails again!