7 ms·

When to Hold Out For a Lower Airfare

70 points by rrtwo 12y ago

gk1 12y ago
I regularly book from Kayak, and I like their predictor. A sensitivity setting would be nice. For example, when it says the fare is likely to increase in the next few days, does that mean increase by $40 or increase by $140? I'm more likely to give up a probable saving of $40 to be comfortable in knowing my flight is finally booked, but I would wait if it's $140. I imagine everyone's tolerance is different, which is why the sensitivity setting would be nice.
jsnell 12y ago
I poked around a bit on 538 on the day it relaunched, but hadn't read anything from there since then. Is this representative of the quality of articles? It seems just embarrassing as an example of data-driven journalism. They are evaluating the quality of a "big data" prediction algorithm based on 32 samples. The experimental setup is very odd, with such a short period of price tracking, a bizarre baseline strategy, and ending the trials early (over half right at the beginning!). Or consider this gem: "For the five routes in which there was a financial benefit to waiting, Kayak successfully reduced my fare in each instance. [...] This is a sure sign of intelligence". Yes, of course if you cherry-pick 1/6 of the trials (a whole 5 of them!), the system will look effective. What kind of value is there in "analysis" of this level?
- pdonis 12y ago
  I was underwhelmed as well. In addition to the items you point out, at the end the article says: "...it still might be worth your time and energy to use airfare prediction software. Why? Because following the algorithm isn’t going to cost you more money..." which is simply wrong, since the article explicitly gives examples of following the algorithm costing the author more money.
  - callahad 12y ago
    I think the author was basing that statement on his previous assertion that: > If weighted equally, the prices paid, following Kayak’s algorithm, were 2 percent higher than the initial March 29 prices. Wearing my statistical hat, I’d call that a tie between the two strategies. The author also claims that: > in user surveys that in addition to appealing to quantitative types, another group of regular users said Farecast gave them “peace of mind.” So if the price difference is 2% (basically a wash either way), then waiting would be a superior strategy because it "might actually relieve some of the second-guessing that occurs when you’re left to your own devices," and that peace of mind may be worth more than 2% of the ticket cost. ...or at least, I think that's the author's point.
- jaynos 12y ago
  I've been checking out fivethirtyeight every few days. You are correct. The quality has gone down since relaunching. I believe this is due to the wider focus and the need to pump out articles. They still do some decent sports articles (and I assume the election prediction stuff will still be good), but overall, I'm not impressed.
- w1ntermute 12y ago
  I think that there's a lot of value in amateur data science from an "eyeballs" perspective. Young people want more "intelligent" coverage, but not so dense that they can't understand it. The sort of nitpicking you're engaging in suggests that you're not the target audience. This is why I'm looking forward to more and more journals becoming open access. I think that for people who are dissatisfied with 538, understanding academic publications would not be too big of a leap.
- j2kun 12y ago
  You should check out this analysis of their analysis of their story on Bechdel-test-passing movies. It highlights some of the inadequacies in their methodologies and analysis and gives a call for more openness and higher scientific standards. http://nbviewer.ipython.org/github/brianckeegan/Bechdel/blob/master/Bechdel_test.ipynb http://nbviewer.ipython.org/github/brianckeegan/Bechdel/blob...
- the_watcher 12y ago
  I'm giving it the 6 months that it took Grantland to figure themselves out before making a call on the overall quality - The Value of a Steal series has, so far, been the only really high quality piece I've read, everything else sounds like a proposal to study something interesting (or the abstract of the paper studying something interesting).
  - cwal37 12y ago
    I think one of the major areas they might struggle in is decent prose. I expected the new 538 to have lengthy and interesting prose in addition to data, but most everything has been a slog to read so far. Maybe I've just been spoiled by Grantland, but I really hope it improves.
    - the_watcher 12y ago
      Yea, that's true. Grantland has truly excellent writers, whereas 538 may have been looking for something different than pure writing ability. Still, guys like Kirk Goldsberry could easily write for 538 in terms of technical competence.
- pessimizer 12y ago
  http://krugman.blogs.nytimes.com/2014/03/23/tarnished-silver/ http://krugman.blogs.nytimes.com/2014/03/23/tarnished-silver...
- khafra 12y ago
  > ...ending the trials early Note that early termination rules are pretty common in bayesian statistics, as is taking only enough samples for a reasonable likelihood ratio. I'd rather see samples covering different time periods as well as different routes, but I'm perfectly happy updating my belief in the efficacy of Kayak's price prediction based on these results.
dalek2point3 12y ago
tl;dr version You will probably not save any money following Kayak’s price prediction algorithm and instead buying tickets two weeks ahead of your scheduled departure is a reasonable strategy. You should use prediction algorithms anyway, because they wont lose you money either and help you gain "piece of mind".
michaelochurch 12y ago
Because following the algorithm isn’t going to cost you more money, and it might actually relieve some of the second-guessing that occurs when you’re left to your own devices. Disagree. Airfare pricing is adversarial. It used to be "common knowledge" that buying tickets on Wednesday at 3:05am Eastern was optimal. Then, airlines responded by jacking up prices at that time. You'll also get different prices from an Incognito browser than one with cookies. I'd bet you get hosed if you use your iPhone, for obvious reasons. If something becomes commonly known as "the right way" to get fair prices, it will surely lead to unfair prices as airlines take advantage of the herd. Because it's adversarial, the landscape is constantly changing and there are no guarantees.
- twoodfin 12y ago
  Do you have any documentation on the incognito vs. cookies claim? I'm skeptical that many airlines would do this when their "advertised" price is readily available on so many other sites. ITA's Matrix is my go-to choice. I wonder if they sell search metrics to the airlines.
  - mitchellh 12y ago
    I don't have any documentation, but I've been flying A LOT for many years (last year flew 108 flights for a total over over 150,000 miles). What the parent comment above you is saying is true. It is common knowledge among frequent flyers that you can get better prices sometimes if you're not signed in. For example, if you fly 100,000 miles a year with an airlines, they're pretty confident you're going to fly with them, so why not show you a slightly higher fare? The worst part: it works. I go to HipMunk sometimes and see a fare for $500 dollars less (on the same airline), but I still book through my preferred airlines, because the benefits are worth it for me: no hassle security, no hassle baggage (first on, first off), no hassle boarding, good chance of 1st class upgrade, etc. The good news is that if you do find a lower fare on ITA (and pretty much only ITA), you can usually call the airlines and get that fare. Fares are bucketed into fixed "fare classes" so if there is availability in a cheap bucket, the airline MUST sell it to you. The bad news for frequent flyers is that many benefits do not extend to these cheaper fare buckets. For example, if I want even a chance to be upgraded to business class for free on a transatlantic flight, I have to purchase one of the most expensive economy fare classes. Concrete example: I am flying to Israel in a couple months; for the cheap economy fare it was $1400, but for a chance at an upgrade it was $2,100. I took the $2,100 fare, because being in business to TLV is quite important to me. Its unfortunate, but its how it is.
ape4 12y ago
What about availability. If you buy sooner that's worth something.
- chrismcb 12y ago
  Well the pricing predictor is trying to predict when the prices go up and go down. Availability effects the prices a lot. As prices go up, the availability tends to go down. I'd say one nice thing about buying early is better seat selection.
egypturnash 12y ago
Not included in the cost calculations: several days of remembering to check the cost of the air fare every day, and stressing about whether or not you'll end up paying a ton when the predictor erroneously tells you to wait. The price differentials cited in this article are well worth just buying it and having it out of the way, IMHO.
cmsmith 12y ago
This analysis ignores the fact that it might be beneficial to buy tickets more than 14 days before your trip (an idea which always seemed like common sense to me). The idea that fares rise monotonically for the last two weeks doesn't mean that big data is dead.
deleted 12y ago
[deleted]
lafar6502 12y ago
If Kayak/MSFT uses machine learning for price prediction and airlines are doing the same for predicting their revenue, doesn't it result in both sides quickly reacting to each other moves and countering opponent's actions? After all, they're all using the same ticket sales data. Maybe that's why recommendation engines don't seem to work?
lnanek2 12y ago
Southwest has a no cost flight change feature. So monitoring is worth a lot more for them since you can take advantage of dips after you buy.

70 points by rrtwo 12y ago

gk1 12y ago

I regularly book from Kayak, and I like their predictor. A sensitivity setting would be nice. For example, when it says the fare is likely to increase in the next few days, does that mean increase by $40 or increase by $140? I'm more likely to give up a probable saving of $40 to be comfortable in knowing my flight is finally booked, but I would wait if it's $140. I imagine everyone's tolerance is different, which is why the sensitivity setting would be nice.

jsnell 12y ago

I poked around a bit on 538 on the day it relaunched, but hadn't read anything from there since then. Is this representative of the quality of articles? It seems just embarrassing as an example of data-driven journalism. They are evaluating the quality of a "big data" prediction algorithm based on 32 samples. The experimental setup is very odd, with such a short period of price tracking, a bizarre baseline strategy, and ending the trials early (over half right at the beginning!). Or consider this gem: "For the five routes in which there was a financial benefit to waiting, Kayak successfully reduced my fare in each instance. [...] This is a sure sign of intelligence". Yes, of course if you cherry-pick 1/6 of the trials (a whole 5 of them!), the system will look effective. What kind of value is there in "analysis" of this level?

pdonis 12y ago
I was underwhelmed as well. In addition to the items you point out, at the end the article says: "...it still might be worth your time and energy to use airfare prediction software. Why? Because following the algorithm isn’t going to cost you more money..." which is simply wrong, since the article explicitly gives examples of following the algorithm costing the author more money.
- callahad 12y ago
  I think the author was basing that statement on his previous assertion that: > If weighted equally, the prices paid, following Kayak’s algorithm, were 2 percent higher than the initial March 29 prices. Wearing my statistical hat, I’d call that a tie between the two strategies. The author also claims that: > in user surveys that in addition to appealing to quantitative types, another group of regular users said Farecast gave them “peace of mind.” So if the price difference is 2% (basically a wash either way), then waiting would be a superior strategy because it "might actually relieve some of the second-guessing that occurs when you’re left to your own devices," and that peace of mind may be worth more than 2% of the ticket cost. ...or at least, I think that's the author's point.
jaynos 12y ago
I've been checking out fivethirtyeight every few days. You are correct. The quality has gone down since relaunching. I believe this is due to the wider focus and the need to pump out articles. They still do some decent sports articles (and I assume the election prediction stuff will still be good), but overall, I'm not impressed.
w1ntermute 12y ago
I think that there's a lot of value in amateur data science from an "eyeballs" perspective. Young people want more "intelligent" coverage, but not so dense that they can't understand it. The sort of nitpicking you're engaging in suggests that you're not the target audience. This is why I'm looking forward to more and more journals becoming open access. I think that for people who are dissatisfied with 538, understanding academic publications would not be too big of a leap.
j2kun 12y ago
You should check out this analysis of their analysis of their story on Bechdel-test-passing movies. It highlights some of the inadequacies in their methodologies and analysis and gives a call for more openness and higher scientific standards. http://nbviewer.ipython.org/github/brianckeegan/Bechdel/blob/master/Bechdel_test.ipynb http://nbviewer.ipython.org/github/brianckeegan/Bechdel/blob...
the_watcher 12y ago
I'm giving it the 6 months that it took Grantland to figure themselves out before making a call on the overall quality - The Value of a Steal series has, so far, been the only really high quality piece I've read, everything else sounds like a proposal to study something interesting (or the abstract of the paper studying something interesting).
- cwal37 12y ago
  I think one of the major areas they might struggle in is decent prose. I expected the new 538 to have lengthy and interesting prose in addition to data, but most everything has been a slog to read so far. Maybe I've just been spoiled by Grantland, but I really hope it improves.
  - the_watcher 12y ago
    Yea, that's true. Grantland has truly excellent writers, whereas 538 may have been looking for something different than pure writing ability. Still, guys like Kirk Goldsberry could easily write for 538 in terms of technical competence.
pessimizer 12y ago
http://krugman.blogs.nytimes.com/2014/03/23/tarnished-silver/ http://krugman.blogs.nytimes.com/2014/03/23/tarnished-silver...
khafra 12y ago
> ...ending the trials early Note that early termination rules are pretty common in bayesian statistics, as is taking only enough samples for a reasonable likelihood ratio. I'd rather see samples covering different time periods as well as different routes, but I'm perfectly happy updating my belief in the efficacy of Kayak's price prediction based on these results.

dalek2point3 12y ago

tl;dr version You will probably not save any money following Kayak’s price prediction algorithm and instead buying tickets two weeks ahead of your scheduled departure is a reasonable strategy. You should use prediction algorithms anyway, because they wont lose you money either and help you gain "piece of mind".

michaelochurch 12y ago

Because following the algorithm isn’t going to cost you more money, and it might actually relieve some of the second-guessing that occurs when you’re left to your own devices. Disagree. Airfare pricing is adversarial. It used to be "common knowledge" that buying tickets on Wednesday at 3:05am Eastern was optimal. Then, airlines responded by jacking up prices at that time. You'll also get different prices from an Incognito browser than one with cookies. I'd bet you get hosed if you use your iPhone, for obvious reasons. If something becomes commonly known as "the right way" to get fair prices, it will surely lead to unfair prices as airlines take advantage of the herd. Because it's adversarial, the landscape is constantly changing and there are no guarantees.

twoodfin 12y ago
Do you have any documentation on the incognito vs. cookies claim? I'm skeptical that many airlines would do this when their "advertised" price is readily available on so many other sites. ITA's Matrix is my go-to choice. I wonder if they sell search metrics to the airlines.
- mitchellh 12y ago
  I don't have any documentation, but I've been flying A LOT for many years (last year flew 108 flights for a total over over 150,000 miles). What the parent comment above you is saying is true. It is common knowledge among frequent flyers that you can get better prices sometimes if you're not signed in. For example, if you fly 100,000 miles a year with an airlines, they're pretty confident you're going to fly with them, so why not show you a slightly higher fare? The worst part: it works. I go to HipMunk sometimes and see a fare for $500 dollars less (on the same airline), but I still book through my preferred airlines, because the benefits are worth it for me: no hassle security, no hassle baggage (first on, first off), no hassle boarding, good chance of 1st class upgrade, etc. The good news is that if you do find a lower fare on ITA (and pretty much only ITA), you can usually call the airlines and get that fare. Fares are bucketed into fixed "fare classes" so if there is availability in a cheap bucket, the airline MUST sell it to you. The bad news for frequent flyers is that many benefits do not extend to these cheaper fare buckets. For example, if I want even a chance to be upgraded to business class for free on a transatlantic flight, I have to purchase one of the most expensive economy fare classes. Concrete example: I am flying to Israel in a couple months; for the cheap economy fare it was $1400, but for a chance at an upgrade it was $2,100. I took the $2,100 fare, because being in business to TLV is quite important to me. Its unfortunate, but its how it is.

ape4 12y ago

What about availability. If you buy sooner that's worth something.

chrismcb 12y ago
Well the pricing predictor is trying to predict when the prices go up and go down. Availability effects the prices a lot. As prices go up, the availability tends to go down. I'd say one nice thing about buying early is better seat selection.

egypturnash 12y ago

Not included in the cost calculations: several days of remembering to check the cost of the air fare every day, and stressing about whether or not you'll end up paying a ton when the predictor erroneously tells you to wait. The price differentials cited in this article are well worth just buying it and having it out of the way, IMHO.

cmsmith 12y ago

This analysis ignores the fact that it might be beneficial to buy tickets more than 14 days before your trip (an idea which always seemed like common sense to me). The idea that fares rise monotonically for the last two weeks doesn't mean that big data is dead.

deleted 12y ago

[deleted]

lafar6502 12y ago

If Kayak/MSFT uses machine learning for price prediction and airlines are doing the same for predicting their revenue, doesn't it result in both sides quickly reacting to each other moves and countering opponent's actions? After all, they're all using the same ticket sales data. Maybe that's why recommendation engines don't seem to work?

lnanek2 12y ago

Southwest has a no cost flight change feature. So monitoring is worth a lot more for them since you can take advantage of dips after you buy.