9 ms·

No Emails on Saturdays

52 points by latitude 14y ago

birken 14y ago
Thumbtack has an open-source split testing calculator which makes it very easy to see whether or not results like these are significant or not: http://www.thumbtack.com/labs/abba/#Sunday=47%2C389&Tuesday=35%2C378&Wednesday=37%2C400&Saturday=25%2C401&Monday=35%2C379&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true http://www.thumbtack.com/labs/abba/#Sunday=47%2C389&Tues... If we treat "Sunday" as the baseline and use "votes" as the count of a successful email, it appears that you can say with a somewhat high degree of confidence that Sunday is better than Saturday, but nothing else.
- gus_massa 14y ago
  But with the opening rate, there is a totally different story: http://www.thumbtack.com/labs/abba/#Sunday=157%2C389&Tuesday=175%2C378&Wednesday=177%2C400&Saturday=111%2C401&Monday=148%2C379&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true http://www.thumbtack.com/labs/abba/#Sunday=157%2C389&Tue...
andrewcooke 14y ago
are the engagement numbers for sunday correct? it appears to be highest, which doesn't match the conclusions. [edit: maybe i'm not understanding, but engagement on sunday is highest at 47, yet the final recommendation is for "Tuesday or Wednesday" (35 and 39). why wouldn't you choose the day with highest engagement?] [also, incidentally, those numbers are all the same to within a couple of standard deviations (1 sd ~ sqrt(n) ~ 6). so the sample size is too small to draw any really firm conclusions.]
- deleted 14y ago
  [deleted]
- B-Con 14y ago
  FTA: > Not on Sundays or Mondays, but if pressed, pick Sundays. Seems consistent to me...?
- latitude 14y ago
  Yep, Sunday had the highest number of pingbacks. It's actually reasonable. This is an opt-in mailing list and subscribers are generally interested in the project, so if they are hit with a relevant update on their free time, they will respond. (edit) The rationale behind "Tue/Wed" is that I value views more than votes. Agreed on std.dev., yet Saturday still stands out as the least performing day.
- timv 14y ago
  It looks like the conclusion is overly simplistic (but it's a blog entry based on a single experiment, so it's quite reasonable to limit it to simple conclusions) If I understand the data correctly, Sunday has a low open rate, but of those that open, it had very high engagement rate. The author seems to be saying that the open rate is low enough to make Sunday a "bad day", but that really depends on what your email is about. If getting people to read the email is your priority, then avoid Sunday, but if you mostly care about getting them to respond in some way, then Sunday could be good - those that read it will have the free time to actually respond. (Of course, while the limited data provided by this experiment is interesting, it's probably not worth basing any serious decisions on)
  - latitude 14y ago
    What I wanted to share was the fact that there was a noticeable difference to begin with, so it's something worth paying attention to. The exact pattern will depend on the audience. If it's VmWare emailing IT people, then the open rate is probably going to be the same through the workdays and zero on weekends. If it's some hobby stuff, then it'd be in reverse. For my particular project, it just happens that Saturday sucked, which is a good thing to know.
    - timv 14y ago
      I certainly think that your experiment has value - my apologies if it came across any other way. - If nothing else, it shows that there is a difference, and the day you pick for your newsletter matters. - It seems that it would be helpful for other people to repeat your experiment (but stretch it to 7 days) to see what the results are for their readers. - The difference between the open rate and engagement rate is quite interesting.
angelohuang 14y ago
Does this number also apply to Tweets? Maybe someone can tell their experience on tweets.
- bradleyjoyce 14y ago
  We do this for Twitter at http://socialyzerhq.com http://socialyzerhq.com, but rather than just limiting to historical data we try to predict when to send in the future.
mansigandhi 14y ago
Very interesting, thanks for the share.
madnessjames 14y ago
What time of the day you send the email makes a big difference. Especially if your list is large. ISPs do magic with large sends to prevent spamming. If you send on a Sunday evening, and others send Monday morning at 3am for example, most people's email view is reverse chronologically ordered. So on Monday morning, your email is likely to be on the bottom of the pile.
dantiberian 14y ago
There is email campaign software available that tracks the open times on previous emails to each person on your list. The next time you send a newsletter to that person, the email is sent at the optimal time for the person to open, read and interact with it. I forget what the software was but I always thought it was a great idea.
- WillyF 14y ago
  Sailthru does this in addition to lots of other things.
- benihana 14y ago
  While at Bronto software (http://bronto.com/ http://bronto.com/), one of the features I worked on was just that. We called it Send Time Optimization, and from what I understand, it's not an unheard-of feature amongst email software providers.
  - tommccabe 14y ago
    Silverpop has it & also calls it send time optimization. Allows for optimizing the send over 24 hours or 7 days.
  - weeksie 14y ago
    Completely off topic — you worked at Bronto? I did a short (like 1 or 2 day) contract to write a first pass at their Ruby client gem a couple years back. Seemed like nice guys. I get shivers thinking about Ruby and SOAP though. Eesh.
latitude 14y ago
OP's here. A couple of things I forgot to mention in the post. 1. The numbers are based on hits through URLs with tracking information. Some visited and voted from the Twitter feed, some hit the base URL. This affects absolute numbers, obviously, but the %-ages should be about the same. 2. I also tallied up negative and positive votes, thought that perhaps people on Monday were more stressed and negative than on Sunday. But there were too few dislikes, so it's hard to make any conclusions - like !like Sat 22 3 Sun 44 3 Mon 34 1 Tue 34 1 Wed 35 4 EDIT - I added raw emails/opens/votes data at the end of the Tumblr post, just in case.
Mamady 14y ago
Your sample size it way too small for the stats to be meaningful
- latitude 14y ago
  Excellent, I see. So you are saying that if I had 10x the data, then the Saturday opening rate and the Sunday response rate could've been completely different?
  - harshreality 14y ago
    Yes, if you re-ran the experiment 10 times, it could be completely different. The presented data is based on a single campaign. Perhaps something was happening that Saturday, something that the post failed to take into account. Perhaps something happened during those weekdays, causing more engagement than normal.
    - latitude 14y ago
      "I went to kindergarten, I know how the alphabet works!" :) Interestingly enough, nothing was happening on that Saturday. That I actually checked. I obviously don't disagree that there's a lot of variables in play and some vary faster than others. What I meant by "10x" was that if the list were 20k subscribers instead of 2k for this particular experiment - what would've been the chance of the Saturday open rate jumping up.
  - piggity 14y ago
    Because your sample is so small, the results are easily skewed by a few people. As the sample size increases you'll get a more representative picture of what is happening. It doesn't invalidate what you've found, but does mean you should now continue on with your experiment to increase the sample size - and help ensure your data wasn't impacted by something else - e.g. vacations / a conference that was attended by many of your subscribers / etc etc.
  - ihaveajob 14y ago
    Your margin of error is 4.38% [1], which makes the comparison between all non-Saturday meaningless. However, it seems like the numbers for Saturday vs other days are useful. To answer your specific question, if you had 10x more data, your margin of error (with 95% confidence) would drop to 1.39%, so yes, you could reasonably get fairly different results. I was disappointed to read that you skipped Thursday and Friday. In my domain, Mondays are the peak of activity, slowly sliding down to a bottom on Friday/Saturday. I would have liked to see your numbers for every day. Next time! (In that case, your MoE would have been 5.37%, btw.) [1] http://americanresearchgroup.com/moe.html http://americanresearchgroup.com/moe.html
totalrobe 14y ago
Why no Thursday/Friday? Those are slower days at many work places.
dmor 14y ago
One other thing I think you should experiment with is time of day. There was another great post (not able to find the link yet, will update when I do) that talked about open rate being highest within the first few hours of sending regardless of the send time. I've always been a fan of sending between 3-6am, although I think I just took this as a rule of thumb from someone I worked with an never questioned it much. Would love to see a similar study on send times within the optimal range of days.
- Samuel_Michon 14y ago
  MailChimp has a neat page where they list the average open rates for the 5.5 million mailing lists people manage using MailChimp's platform: http://mailchimp.com/resources/research/ http://mailchimp.com/resources/research/ "Most emails are sent between Monday and Friday, with the highest volume on Tuesday and Thursday. More subscribers open email, however, on Wednesday and Thursday." "Subscribers are likely to open email after 12pm, and the most active hours are 2-5pm." "Your newest subscribers are your best subscribers. They’re most engaged immediately after signup, and then trail off over time." Those are averages though, your mileage may vary. For the newsletters I send out, Tuesday morning has proven to be the sweet spot. More information: http://kb.mailchimp.com/article/when-is-the-best-time-to-send-emails http://kb.mailchimp.com/article/when-is-the-best-time-to-sen...
Aardwolf 14y ago
Interesting stats, but I'd love to see this with a bigger sample size, and with thursdays and fridays included!
baby 14y ago
Isn't there a statistic somewhere about facebook, that people go there the most on wednesday 17h?

52 points by latitude 14y ago

Thumbtack has an open-source split testing calculator which makes it very easy to see whether or not results like these are significant or not: http://www.thumbtack.com/labs/abba/#Sunday=47%2C389&Tuesday=35%2C378&Wednesday=37%2C400&Saturday=25%2C401&Monday=35%2C379&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true http://www.thumbtack.com/labs/abba/#Sunday=47%2C389&Tues... If we treat "Sunday" as the baseline and use "votes" as the count of a successful email, it appears that you can say with a somewhat high degree of confidence that Sunday is better than Saturday, but nothing else.

gus_massa 14y ago
But with the opening rate, there is a totally different story: http://www.thumbtack.com/labs/abba/#Sunday=157%2C389&Tuesday=175%2C378&Wednesday=177%2C400&Saturday=111%2C401&Monday=148%2C379&abba%3AintervalConfidenceLevel=0.95&abba%3AuseMultipleTestCorrection=true http://www.thumbtack.com/labs/abba/#Sunday=157%2C389&Tue...

andrewcooke 14y ago

are the engagement numbers for sunday correct? it appears to be highest, which doesn't match the conclusions. [edit: maybe i'm not understanding, but engagement on sunday is highest at 47, yet the final recommendation is for "Tuesday or Wednesday" (35 and 39). why wouldn't you choose the day with highest engagement?] [also, incidentally, those numbers are all the same to within a couple of standard deviations (1 sd ~ sqrt(n) ~ 6). so the sample size is too small to draw any really firm conclusions.]

deleted 14y ago
[deleted]
B-Con 14y ago
FTA: > Not on Sundays or Mondays, but if pressed, pick Sundays. Seems consistent to me...?
latitude 14y ago
Yep, Sunday had the highest number of pingbacks. It's actually reasonable. This is an opt-in mailing list and subscribers are generally interested in the project, so if they are hit with a relevant update on their free time, they will respond. (edit) The rationale behind "Tue/Wed" is that I value views more than votes. Agreed on std.dev., yet Saturday still stands out as the least performing day.
timv 14y ago
It looks like the conclusion is overly simplistic (but it's a blog entry based on a single experiment, so it's quite reasonable to limit it to simple conclusions) If I understand the data correctly, Sunday has a low open rate, but of those that open, it had very high engagement rate. The author seems to be saying that the open rate is low enough to make Sunday a "bad day", but that really depends on what your email is about. If getting people to read the email is your priority, then avoid Sunday, but if you mostly care about getting them to respond in some way, then Sunday could be good - those that read it will have the free time to actually respond. (Of course, while the limited data provided by this experiment is interesting, it's probably not worth basing any serious decisions on)
- latitude 14y ago
  What I wanted to share was the fact that there was a noticeable difference to begin with, so it's something worth paying attention to. The exact pattern will depend on the audience. If it's VmWare emailing IT people, then the open rate is probably going to be the same through the workdays and zero on weekends. If it's some hobby stuff, then it'd be in reverse. For my particular project, it just happens that Saturday sucked, which is a good thing to know.
  - timv 14y ago
    I certainly think that your experiment has value - my apologies if it came across any other way. - If nothing else, it shows that there is a difference, and the day you pick for your newsletter matters. - It seems that it would be helpful for other people to repeat your experiment (but stretch it to 7 days) to see what the results are for their readers. - The difference between the open rate and engagement rate is quite interesting.

angelohuang 14y ago

Does this number also apply to Tweets? Maybe someone can tell their experience on tweets.

bradleyjoyce 14y ago
We do this for Twitter at http://socialyzerhq.com http://socialyzerhq.com, but rather than just limiting to historical data we try to predict when to send in the future.

mansigandhi 14y ago

Very interesting, thanks for the share.

madnessjames 14y ago

What time of the day you send the email makes a big difference. Especially if your list is large. ISPs do magic with large sends to prevent spamming. If you send on a Sunday evening, and others send Monday morning at 3am for example, most people's email view is reverse chronologically ordered. So on Monday morning, your email is likely to be on the bottom of the pile.

dantiberian 14y ago

There is email campaign software available that tracks the open times on previous emails to each person on your list. The next time you send a newsletter to that person, the email is sent at the optimal time for the person to open, read and interact with it. I forget what the software was but I always thought it was a great idea.

WillyF 14y ago
Sailthru does this in addition to lots of other things.
benihana 14y ago
While at Bronto software (http://bronto.com/ http://bronto.com/), one of the features I worked on was just that. We called it Send Time Optimization, and from what I understand, it's not an unheard-of feature amongst email software providers.
- tommccabe 14y ago
  Silverpop has it & also calls it send time optimization. Allows for optimizing the send over 24 hours or 7 days.
- weeksie 14y ago
  Completely off topic — you worked at Bronto? I did a short (like 1 or 2 day) contract to write a first pass at their Ruby client gem a couple years back. Seemed like nice guys. I get shivers thinking about Ruby and SOAP though. Eesh.

latitude 14y ago

OP's here. A couple of things I forgot to mention in the post. 1. The numbers are based on hits through URLs with tracking information. Some visited and voted from the Twitter feed, some hit the base URL. This affects absolute numbers, obviously, but the %-ages should be about the same. 2. I also tallied up negative and positive votes, thought that perhaps people on Monday were more stressed and negative than on Sunday. But there were too few dislikes, so it's hard to make any conclusions - like !like Sat 22 3 Sun 44 3 Mon 34 1 Tue 34 1 Wed 35 4 EDIT - I added raw emails/opens/votes data at the end of the Tumblr post, just in case.

Mamady 14y ago

Your sample size it way too small for the stats to be meaningful

latitude 14y ago
Excellent, I see. So you are saying that if I had 10x the data, then the Saturday opening rate and the Sunday response rate could've been completely different?
- harshreality 14y ago
  Yes, if you re-ran the experiment 10 times, it could be completely different. The presented data is based on a single campaign. Perhaps something was happening that Saturday, something that the post failed to take into account. Perhaps something happened during those weekdays, causing more engagement than normal.
  - latitude 14y ago
    "I went to kindergarten, I know how the alphabet works!" :) Interestingly enough, nothing was happening on that Saturday. That I actually checked. I obviously don't disagree that there's a lot of variables in play and some vary faster than others. What I meant by "10x" was that if the list were 20k subscribers instead of 2k for this particular experiment - what would've been the chance of the Saturday open rate jumping up.
- piggity 14y ago
  Because your sample is so small, the results are easily skewed by a few people. As the sample size increases you'll get a more representative picture of what is happening. It doesn't invalidate what you've found, but does mean you should now continue on with your experiment to increase the sample size - and help ensure your data wasn't impacted by something else - e.g. vacations / a conference that was attended by many of your subscribers / etc etc.
- ihaveajob 14y ago
  Your margin of error is 4.38% [1], which makes the comparison between all non-Saturday meaningless. However, it seems like the numbers for Saturday vs other days are useful. To answer your specific question, if you had 10x more data, your margin of error (with 95% confidence) would drop to 1.39%, so yes, you could reasonably get fairly different results. I was disappointed to read that you skipped Thursday and Friday. In my domain, Mondays are the peak of activity, slowly sliding down to a bottom on Friday/Saturday. I would have liked to see your numbers for every day. Next time! (In that case, your MoE would have been 5.37%, btw.) [1] http://americanresearchgroup.com/moe.html http://americanresearchgroup.com/moe.html

totalrobe 14y ago

Why no Thursday/Friday? Those are slower days at many work places.

dmor 14y ago

One other thing I think you should experiment with is time of day. There was another great post (not able to find the link yet, will update when I do) that talked about open rate being highest within the first few hours of sending regardless of the send time. I've always been a fan of sending between 3-6am, although I think I just took this as a rule of thumb from someone I worked with an never questioned it much. Would love to see a similar study on send times within the optimal range of days.

Samuel_Michon 14y ago
MailChimp has a neat page where they list the average open rates for the 5.5 million mailing lists people manage using MailChimp's platform: http://mailchimp.com/resources/research/ http://mailchimp.com/resources/research/ "Most emails are sent between Monday and Friday, with the highest volume on Tuesday and Thursday. More subscribers open email, however, on Wednesday and Thursday." "Subscribers are likely to open email after 12pm, and the most active hours are 2-5pm." "Your newest subscribers are your best subscribers. They’re most engaged immediately after signup, and then trail off over time." Those are averages though, your mileage may vary. For the newsletters I send out, Tuesday morning has proven to be the sweet spot. More information: http://kb.mailchimp.com/article/when-is-the-best-time-to-send-emails http://kb.mailchimp.com/article/when-is-the-best-time-to-sen...

Aardwolf 14y ago

Interesting stats, but I'd love to see this with a bigger sample size, and with thursdays and fridays included!

baby 14y ago

Isn't there a statistic somewhere about facebook, that people go there the most on wednesday 17h?