4 ms·

I've worked on ranking in travel before, and you'd be amazed at how terrible ranking is in pretty large companies with a huge incentive to improve things. You'

by splonk 5y ago

I've worked on ranking in travel before, and you'd be amazed at how terrible ranking is in pretty large companies with a huge incentive to improve things. You'd also be amazed at how long it takes to sign a contract with someone that says "we'll increase your conversions by ~15% (and your revenue by literally millions) in exchange for a small portion of your increased profits." Pretty curious about how well you can build a generalized solution and still get uptake from SMBs. I'd think that marketplaces would tend to want to keep that kind of expertise in house, but I guess my experience shows that there are some less eng-focused companies that would pay for that kind of thing. > When we started, we were shocked at how little marketplace companies measure anything. For the travel company mentioned above, our model was built on hotel bookings only. That is, they gave us a list of every booking made on their platform, and then at search time they gave us the parameters of the search (city, dates, incoming flight) and hotel availability, and we were supposed to return the ranked list of hotels. Not in that training set: anything about unconverted searches, what hotels were shown to searchers at any point, or anything about the customers. Again, our model built on super sparse data outperformed their ranking by ~15% over a period of multiple years. (We had even better results over shorter time frames with another customer that never signed a contract.) I kept on telling people that these (Europe-based) companies could have signed a reasonably competent data scientist for like $50k/year, outperformed our models within 6 months or so, and saved themselves 6 figures/year.

andrewyates2020 5y ago
> You'd also be amazed at how long it takes to sign a contract I would not ;) > less eng-focused companies that would pay Actually, our experience has been the opposite. The more sophisticated the engineering team, the more they recognize how big of a pain unified search ranking is to build and maintain, and the more they appreciate what we offer. On the forever "model-bakeoff": our approach is to include all existing models as features into an omni-model. If you are experienced in ML ops, you should be cringing, but we pull it off because from a customer development standpoint, we never want to be competing with some other new technique. Instead, we want to have a big ball of systems and progress is always "add more stuff." Then, the business and product teams can focus on how they want their product to work versus technical details of specific recommendation systems.
- splonk 5y ago
  >> You'd also be amazed at how long it takes to sign a contract > I would not ;) Ha, yeah, that was unclear. My comment was targeted more towards the general HN audience, because _I_ was pretty amazed when I got into that business. "Wait, so you've built the integration, you've built the metrics, you've run A/B tests at various levels for months and months that show beyond a shadow of a doubt that you'll make more money, and you don't want to sign...why?" >> less eng-focused companies that would pay > Actually, our experience has been the opposite. The more sophisticated the engineering team, the more they recognize how big of a pain unified search ranking is to build and maintain, and the more they appreciate what we offer. I see now from the other comments that you're targeting a higher point in the market than I originally thought. Still somewhat surprised that companies at that level would buy rather than build, but I suppose my view of the space is biased since I was only working at places that would want to build that in-house. > our approach is to include all existing models as features into an omni-model Makes sense to me if you're trying to be a generalized solution. I assume there's no crossover between any of your customers and you have to build a unique model for each of them?
  - andrewyates2020 5y ago
    > buy rather than build Candidly, this is our biggest challenge. If we can surmount this, we'll be huge. This is also why care so much about "upmarket brand". Decisions aren't made by numbers alone. We need to show that we're the smarter way to grow faster because other top companies are doing it. This is a big bet. Is there a threshold for engineer TAC in search, discovery, and ads? $1M? 2? More? Hire a dozen? One hundred? I've seen these numbers. They happen because the potential value is there and the VC funding is there, but I haven't always seen the delivered engineering results. Us? We've been there, done that. We'd like to focus on delivering the results, and we know from experience that's going to happen better from the outside. > no crossover between any of your customers The model architecture and infra are the same, but the literal weights in memory are totally independent. Most of the heavy customization is in allocation rules and blender, and we have a DSL for that. https://github.com/promotedai/schema/blob/main/proto/delivery/blender.proto https://github.com/promotedai/schema/blob/main/proto/deliver...
anitil 5y ago
> in exchange for a small portion of your increased profits I've seen this dynamic in sales as well, with people complaining about how much they're paying in commission, seemingly forgetting that the commission comes out of sales that you wouldn't otherwise have
- andrewyates2020 5y ago
  We start by comparing to the price of hiring engineers, which is at least 300k TAC per engineer

by splonk 5y ago

andrewyates2020 5y ago

> You'd also be amazed at how long it takes to sign a contract I would not ;) > less eng-focused companies that would pay Actually, our experience has been the opposite. The more sophisticated the engineering team, the more they recognize how big of a pain unified search ranking is to build and maintain, and the more they appreciate what we offer. On the forever "model-bakeoff": our approach is to include all existing models as features into an omni-model. If you are experienced in ML ops, you should be cringing, but we pull it off because from a customer development standpoint, we never want to be competing with some other new technique. Instead, we want to have a big ball of systems and progress is always "add more stuff." Then, the business and product teams can focus on how they want their product to work versus technical details of specific recommendation systems.

splonk 5y ago
>> You'd also be amazed at how long it takes to sign a contract > I would not ;) Ha, yeah, that was unclear. My comment was targeted more towards the general HN audience, because _I_ was pretty amazed when I got into that business. "Wait, so you've built the integration, you've built the metrics, you've run A/B tests at various levels for months and months that show beyond a shadow of a doubt that you'll make more money, and you don't want to sign...why?" >> less eng-focused companies that would pay > Actually, our experience has been the opposite. The more sophisticated the engineering team, the more they recognize how big of a pain unified search ranking is to build and maintain, and the more they appreciate what we offer. I see now from the other comments that you're targeting a higher point in the market than I originally thought. Still somewhat surprised that companies at that level would buy rather than build, but I suppose my view of the space is biased since I was only working at places that would want to build that in-house. > our approach is to include all existing models as features into an omni-model Makes sense to me if you're trying to be a generalized solution. I assume there's no crossover between any of your customers and you have to build a unique model for each of them?
- andrewyates2020 5y ago
  > buy rather than build Candidly, this is our biggest challenge. If we can surmount this, we'll be huge. This is also why care so much about "upmarket brand". Decisions aren't made by numbers alone. We need to show that we're the smarter way to grow faster because other top companies are doing it. This is a big bet. Is there a threshold for engineer TAC in search, discovery, and ads? $1M? 2? More? Hire a dozen? One hundred? I've seen these numbers. They happen because the potential value is there and the VC funding is there, but I haven't always seen the delivered engineering results. Us? We've been there, done that. We'd like to focus on delivering the results, and we know from experience that's going to happen better from the outside. > no crossover between any of your customers The model architecture and infra are the same, but the literal weights in memory are totally independent. Most of the heavy customization is in allocation rules and blender, and we have a DSL for that. https://github.com/promotedai/schema/blob/main/proto/delivery/blender.proto https://github.com/promotedai/schema/blob/main/proto/deliver...

anitil 5y ago

> in exchange for a small portion of your increased profits I've seen this dynamic in sales as well, with people complaining about how much they're paying in commission, seemingly forgetting that the commission comes out of sales that you wouldn't otherwise have

andrewyates2020 5y ago
We start by comparing to the price of hiring engineers, which is at least 300k TAC per engineer