9 ms·

What Powers Instagram: Hundreds of Instances, Dozens of Technologies

335 points by zio99 14y ago

japhyr 14y ago
I love that companies like Instagram are happy to share this kind of information. I don't have the problem of scaling yet, but it's reassuring to know that when I do, there are good models like this one to follow.
- brlewis 14y ago
  The Instagram founders are exceptionally cool. After the $1B Facebook acquisition Mike Krieger kept right on answering questions on the instagram-api mailing list without missing a beat, as if nothing had happened.
  - antirez 14y ago
    Mike has been an example of good citizenship in the Redis community as well. Always helpful, kind, and providing important feedbacks.
  - citricsquid 14y ago
    I might be wrong, but as far as I know they haven't sold yet (it's still got to clear regulators?) so although he technically could have walked away I don't think it would been sensible as he isn't going to be able to cash out yet?
    - zio99 14y ago
      Some sources (the NY Post for example) say the deal may take up to 6 months with a 50% chance of going through. There's some concerns as to whether Instagram may prevent sharing with social networks other than facebook (flickr for e.g.). I'm already booted off instagram without upgrading to the new app, but fear that I'll be forced to share all my instagram pictures on my facebook timeline.
      - brlewis 14y ago
        To prevent sharing out except to Facebook, Instagram would have to disable its API and stop saving photos to Camera Roll. I don't expect either of those to happen. Even if the deal might fall through, I could easily imagine Mike taking a day or two to celebrate instead of jumping back in and answering questions, particularly on the instagram-api list, since nurturing a slowly-growing ecosystem is not Instagram's best path to a successful exit. Logically, I think he did it because he's a nice guy.
      - joering2 14y ago
        Also worth mention a post than Donna Kline penned: http://www.donnaklinenow.com/investigation/instagram-scam http://www.donnaklinenow.com/investigation/instagram-scam
    - loceng 14y ago
      Indeed - it hasn't cleared, and if it doesn't Facebook still owes them $200 million for the failed buy.
      - riffraff 14y ago
        excuse my ignorance of US corporate law, but why would FB owe them $200M for a failed buy?
        kudos200 14y ago
        It's fairly common in acquisitions for one or both sides to agree to a "break-up" fee to protect against one side getting cold feet, regulatory risks, financing risks, and more. http://blogs.wsj.com/deals/2012/04/25/facebook-ipo-whats-with-that-instagram-breakup-fee/ http://blogs.wsj.com/deals/2012/04/25/facebook-ipo-whats-wit...
        grabeh 14y ago
        I imagine it's not US corporate law per se, but rather that as part of the agreement, even if the merger isn't cleared, FB still has to pay the $200m although at roughly 20% of the fee it seems a little steep to compensate Instagram for the cost of having to reverse the transaction (not that I'm particularly familiar with the deal terms and associated costs!)
        alttab 14y ago
        Its essentially a retainer. What if they go through the motions, and then Facebook backs out at the last second? It could have thrown Instagram for a loop, and sometimes an acquisition takes enough resources that it could cost the company money or disrupt their growth or operation. It essentially could prevent frivolous offers from "attacking" competitors by pretending to buy them out, and then not. I guess you could call it "attempted merger".
        rprasad 14y ago
        It's not a matter of law. It's a "breakup fee" negotiated by Instagram in the event that the acquisition falls through. The purpose of a breakup fee is to compensate the target for costs associated with the failed acquisition and various other losses or expenses. Mostly however, a breakup fee compensates the target for business oppportunities not pursued while the acquisition was being attempted because of the acquisition. In cases where the acquirer is a direct competitor to the target, the breakup fee guarantees that the acquirer can't simply look at the target's books and assets, walk away, and use that knowledge to clone the target's business, without paying for it.
- Heinleinian 14y ago
  Don't forget to copy it locally, in case they (or their acquirers...) ever take it down. The Evernote web-clipper plugin is great for that. I also did the same with all the folks who posted technical details on their experience dealing with the AWS outage. Great stuff to hold on to for future reference.
peterwwillis 14y ago
I don't feel very good about the idea of DR just being duplicate servers in a different zone. You don't know what kind of problem could ripple out and affect more zones, or all of them. A completely different host/cloud/colo provider seems like a safer bet.
- shreyas056 14y ago
  All the zones are geographically apart so its equivalent to putting server in different cloud/colo and probability that all of them gets "blown" away simultaneously is very less.
  - polynomial 14y ago
    And yet all it took was a single AWS availability zone going away for a short while for them to have a major outage.
    - res0nat0r 14y ago
      Major issues in Netflix case per their last blog post due to bugs in their environment not properly failing away from dead ELB's. Also the issues were related to API backups due to everyone rushing to launch new instances in a new AZ, but existing services in other AZ's continued to work fine.
      - polynomial 14y ago
        My reading of the Netflix announcement was that it wasn't just a bug, but that they made the conscious decision to include manual intervention in the process (of releasing dead instances) but grossly underestimated the time required to do this across an entire zone.
- pandemicsyn 14y ago
  When you come down to it earth is a single point of failure, its about how big of a disaster you expect to occur and how big of one you expect to recover from.
mistercow 14y ago
I'm pretty sure that "dozens of technologies" is a meaningless thing to say. Instagram is powered by thousands of technologies; it's just that a few dozen of those happen to have trendy buzzwords attached.
- zio99 14y ago
  True, there's a good blog post regarding that scrappy attitude most hackers have and why we need to simply our architecture: http://www.zemanta.com/fruitblog/i-bet-you-over-engineered-your-startup/ http://www.zemanta.com/fruitblog/i-bet-you-over-engineered-y...
- shoo 14y ago
  Instagram! Powered by the polio vaccine and domestication of wheat.
  - skeletonjelly 14y ago
    Instagram! Powered by a lack of earth destroying quantum vacuum metastability!
    - mistercow 14y ago
      I'm pretty sure that one's not a technology, but just a happy and not very unlikely set of circumstances.
- deleted 14y ago
  [deleted]
yumraj 14y ago
I don't see Amazon SimpleDB in this list, rather Postgres and Redis. Can someone who knows more about various DBs opine on this. Is it better to run your own DB as Instagram seems to be doing or is relying on SimpleDB good enough if you don't need such high performance. Also, as happens with many startups, how easy/difficult is data migration when startups try to scale and need to scale fast.
- dangrossman 14y ago
  SimpleDB is not an RDBMS. It has very specific use-cases where it makes sense, and most database use does not fit into those use-cases. Perhaps you're thinking of Amazon RDS, which is their managed MySQL/Oracle/SQL Server deployments. RDS is very expensive compared to running your own database, even on the very same EC2/EBS platform -- but you get reliability, replication and backups that some employee would have to be managing otherwise. In theory, at least. RDS didn't live up to its promises when I used it, so I moved back to running databases on physical servers at half the cost.
  - genwin 14y ago
    Please elaborate on how RDS didn't live up to its promises?
    - dangrossman 14y ago
      When EBS failed in US-East a year or two ago, my RDS instance did not remain available and did not automatically fail over to its replica, despite my paying for these benefits. Even when it was working correctly, the RDS instances had low capacity compared to physical servers, and there were often spikes of significant latency that would lead to overloaded web servers as DB queries backed up unanswered.
      - genwin 14y ago
        Good to know. I read here on HN that during the storm event in June, the same thing happened (no auto fail over for those who paid for it).
Naushad 14y ago
Well explained, and an awesome way to say "We are hiring a DevOps"
alpb 14y ago
This is a fairly old blog post and it is posted to HN by Kevin Systrom – he's co-founder of Instagram. Here's his post: http://news.ycombinator.com/item?id=3306027 http://news.ycombinator.com/item?id=3306027 (223 points, 220 days ago) so I'd like to say this is a repost. Though, I have no idea how the same link can be reposted to HN.
- joering2 14y ago
  http://news.ycombinator.com/item?id=3306815 http://news.ycombinator.com/item?id=3306815 Guess they figured that one out in less than 221 days.
- achompas 14y ago
  I don't understand how this topic comes up so frequently, but HN is configured to allow reposts after a certain amount of time. That isn't really a problem for old posts with excellent information, so I think a repost is warranted for this.
vegas 14y ago
To sum up: it's gonna fall over at any minute without a continuous injection of money.
- josephcooney 14y ago
  My take-away was "this is why when EC2 when down it wasn't just a case of rebooting the server". My other thought was "and how did all of this help you when EC2 went down?"
  - jbigelow76 14y ago
    That was my thought too, the multiple mentions of "stuff" in different zones had me wondering why the recent EC2 outage was able to get them.
dclowd9901 14y ago
Color me surprised: No NoSQL? Instagram seems to be a prime candidate for such a storage method.
- deleted 14y ago
  [deleted]
- gregburd 14y ago
  You don't consider Amazon/S3 or Redis NoSQL databases?
  - jammmuel 14y ago
    Since when is S3 a 'database'?
    - philjohn 14y ago
      It's got an API, you can store data in it and retreive it later by key, since when was S3 not a database? It's a key/value store that lets you store very large values.
  - pjscott 14y ago
    It's weird that things as different as S3 and Redis all fall under the umbrella of "NoSQL". We should probably replace the SQL-NoSQL dichotomy with terms that more accurately reflect the real differences.
  - dclowd9901 14y ago
    Not really a "distinctive" one (a la mongo, couch, etc.), but I see your point.
- brendoncrawford 14y ago
  They use Redis.
- timaelliott 14y ago
  Redis is a key-value store. S3 is a distributed file system. Can we stop labeling the set of "not a rdbms" data storage mechanisms with the stupid fucking "NoSQL" moniker.
timaelliott 14y ago
> For our geo-search API, we used PostgreSQL for many months, but once our Media entries were sharded, moved over to using Apache Solr. It has a simple JSON interface, so as far as our application is concerned, it’s just another API to consume. Does anyone have particular insight to share on this? Last I checked, Solr's geospatial searching methods are rather inefficient -- haversine across all documents, bounding boxes that rely on haversine and Solr4 was looking into geohashes (better but have some serious edge-case problems where they fall apart). Meanwhile PostgreSQL offers r-tree indexing for spatial queries and is blazing fast. Am I missing some hidden power about Solr's geospatial lookups that make it faster/better than an r-tree implementation?
- awj 14y ago
  It probably was the database sharding. If the Solr setup could handle the geo-search-related data without the need for sharding it probably can beat out Postgres with sharding. Having this exposed through an api that is standardized and maintained by someone else is also nothing to sneeze at. I'd trade a bit of performance for that kind of standardization and turnkey use in the right scenario.
  - rbranson 14y ago
    The reason we use Solr for this specific task is because PostgreSQL cannot efficiently and quickly merge two index queries (time & space). It can do this to a limited degree, but both of these dimensions potentially match 10s of millions of documents, and PG falls over at this.
    - timaelliott 14y ago
      So you make the r-tree 3 dimensions (lat,lng,time). PostgreSQL supports this. I dunno I can't envision Solr being more efficient than a properly designed RDBMS for these situations. If you were integrating a full-text search I'd absolutely believe that to be the case but...
      - 14y ago
deleted 14y ago
[deleted]
deleted 14y ago
[deleted]
deleted 14y ago
[deleted]
deleted 14y ago
[deleted]
deleted 14y ago
[deleted]
deleted 14y ago
[deleted]
rurounijones 14y ago
Do they have any application monitoring stuff a la NewRelic? I saw stuff like munin but that isn't really the same.
deleted 14y ago
[deleted]
resrc 14y ago
"The photos themselves go straight to Amazon S3"... Is there no image resizing involved? It's pretty hard to find stuff about scaling image resizing, It's hugely CPU intensive, that's a sure thing.
- lallysingh 14y ago
  I'd imagine it's easy enough to do before transmission.
  - riobard 14y ago
    Actually they have to do the compression on the mobile clients before uploading, otherwise it will take forever and cost a real fortunate on the mobile data usage.
gagabity 14y ago
Could someone plug these into the AWS monthly calculator and try to figure out the monthly cost, when I try I get around $10,000 per Months, but I think I am wildly underestimating the bandwidth usage. AWS Calculator http://calculator.s3.amazonaws.com/calc5.html http://calculator.s3.amazonaws.com/calc5.html
insaneirish 14y ago
The idea of mdadm on top of EBS volumes makes me cringe. Mark Mayo's thoughts on abstracted block storage are spot on: http://joyeur.com/2011/04/24/magical-block-store-when-abstractions-fail-us/ http://joyeur.com/2011/04/24/magical-block-store-when-abstra...
gsingers 14y ago
A couple of comments as one of the authors of Solr's spatial code: - I don't think any of us have compared it to PostgreSQL, but I can tell you we have clients doing 500 queries per second+ with it including text search. I've yet to see a DB do good text-based fuzzy matching and combining two systems (DB + search) via a join is usually slow. YMMV. - The main goal of the implementation is to add point-based search to text search. It is not a general purpose replacement for an r-tree, etc. - You are not required to use haversine. The distance function is pluggable and we have other options implemented. Also, in many cases, you have other clauses in your query that restrict down the set of documents that need to be scored by distance.

335 points by zio99 14y ago

I love that companies like Instagram are happy to share this kind of information. I don't have the problem of scaling yet, but it's reassuring to know that when I do, there are good models like this one to follow.

brlewis 14y ago
The Instagram founders are exceptionally cool. After the $1B Facebook acquisition Mike Krieger kept right on answering questions on the instagram-api mailing list without missing a beat, as if nothing had happened.
- antirez 14y ago
  Mike has been an example of good citizenship in the Redis community as well. Always helpful, kind, and providing important feedbacks.
- citricsquid 14y ago
  I might be wrong, but as far as I know they haven't sold yet (it's still got to clear regulators?) so although he technically could have walked away I don't think it would been sensible as he isn't going to be able to cash out yet?
  - zio99 14y ago
    Some sources (the NY Post for example) say the deal may take up to 6 months with a 50% chance of going through. There's some concerns as to whether Instagram may prevent sharing with social networks other than facebook (flickr for e.g.). I'm already booted off instagram without upgrading to the new app, but fear that I'll be forced to share all my instagram pictures on my facebook timeline.
    - brlewis 14y ago
      To prevent sharing out except to Facebook, Instagram would have to disable its API and stop saving photos to Camera Roll. I don't expect either of those to happen. Even if the deal might fall through, I could easily imagine Mike taking a day or two to celebrate instead of jumping back in and answering questions, particularly on the instagram-api list, since nurturing a slowly-growing ecosystem is not Instagram's best path to a successful exit. Logically, I think he did it because he's a nice guy.
    - joering2 14y ago
      Also worth mention a post than Donna Kline penned: http://www.donnaklinenow.com/investigation/instagram-scam http://www.donnaklinenow.com/investigation/instagram-scam
  - loceng 14y ago
    Indeed - it hasn't cleared, and if it doesn't Facebook still owes them $200 million for the failed buy.
    - riffraff 14y ago
      excuse my ignorance of US corporate law, but why would FB owe them $200M for a failed buy?
      - kudos200 14y ago
        It's fairly common in acquisitions for one or both sides to agree to a "break-up" fee to protect against one side getting cold feet, regulatory risks, financing risks, and more. http://blogs.wsj.com/deals/2012/04/25/facebook-ipo-whats-with-that-instagram-breakup-fee/ http://blogs.wsj.com/deals/2012/04/25/facebook-ipo-whats-wit...
      - grabeh 14y ago
        I imagine it's not US corporate law per se, but rather that as part of the agreement, even if the merger isn't cleared, FB still has to pay the $200m although at roughly 20% of the fee it seems a little steep to compensate Instagram for the cost of having to reverse the transaction (not that I'm particularly familiar with the deal terms and associated costs!)
      - alttab 14y ago
        Its essentially a retainer. What if they go through the motions, and then Facebook backs out at the last second? It could have thrown Instagram for a loop, and sometimes an acquisition takes enough resources that it could cost the company money or disrupt their growth or operation. It essentially could prevent frivolous offers from "attacking" competitors by pretending to buy them out, and then not. I guess you could call it "attempted merger".
      - rprasad 14y ago
        It's not a matter of law. It's a "breakup fee" negotiated by Instagram in the event that the acquisition falls through. The purpose of a breakup fee is to compensate the target for costs associated with the failed acquisition and various other losses or expenses. Mostly however, a breakup fee compensates the target for business oppportunities not pursued while the acquisition was being attempted because of the acquisition. In cases where the acquirer is a direct competitor to the target, the breakup fee guarantees that the acquirer can't simply look at the target's books and assets, walk away, and use that knowledge to clone the target's business, without paying for it.
Heinleinian 14y ago
Don't forget to copy it locally, in case they (or their acquirers...) ever take it down. The Evernote web-clipper plugin is great for that. I also did the same with all the folks who posted technical details on their experience dealing with the AWS outage. Great stuff to hold on to for future reference.

peterwwillis 14y ago

I don't feel very good about the idea of DR just being duplicate servers in a different zone. You don't know what kind of problem could ripple out and affect more zones, or all of them. A completely different host/cloud/colo provider seems like a safer bet.

shreyas056 14y ago
All the zones are geographically apart so its equivalent to putting server in different cloud/colo and probability that all of them gets "blown" away simultaneously is very less.
- polynomial 14y ago
  And yet all it took was a single AWS availability zone going away for a short while for them to have a major outage.
  - res0nat0r 14y ago
    Major issues in Netflix case per their last blog post due to bugs in their environment not properly failing away from dead ELB's. Also the issues were related to API backups due to everyone rushing to launch new instances in a new AZ, but existing services in other AZ's continued to work fine.
    - polynomial 14y ago
      My reading of the Netflix announcement was that it wasn't just a bug, but that they made the conscious decision to include manual intervention in the process (of releasing dead instances) but grossly underestimated the time required to do this across an entire zone.
pandemicsyn 14y ago
When you come down to it earth is a single point of failure, its about how big of a disaster you expect to occur and how big of one you expect to recover from.

mistercow 14y ago

I'm pretty sure that "dozens of technologies" is a meaningless thing to say. Instagram is powered by thousands of technologies; it's just that a few dozen of those happen to have trendy buzzwords attached.

zio99 14y ago
True, there's a good blog post regarding that scrappy attitude most hackers have and why we need to simply our architecture: http://www.zemanta.com/fruitblog/i-bet-you-over-engineered-your-startup/ http://www.zemanta.com/fruitblog/i-bet-you-over-engineered-y...
shoo 14y ago
Instagram! Powered by the polio vaccine and domestication of wheat.
- skeletonjelly 14y ago
  Instagram! Powered by a lack of earth destroying quantum vacuum metastability!
  - mistercow 14y ago
    I'm pretty sure that one's not a technology, but just a happy and not very unlikely set of circumstances.
deleted 14y ago
[deleted]

yumraj 14y ago

I don't see Amazon SimpleDB in this list, rather Postgres and Redis. Can someone who knows more about various DBs opine on this. Is it better to run your own DB as Instagram seems to be doing or is relying on SimpleDB good enough if you don't need such high performance. Also, as happens with many startups, how easy/difficult is data migration when startups try to scale and need to scale fast.

dangrossman 14y ago
SimpleDB is not an RDBMS. It has very specific use-cases where it makes sense, and most database use does not fit into those use-cases. Perhaps you're thinking of Amazon RDS, which is their managed MySQL/Oracle/SQL Server deployments. RDS is very expensive compared to running your own database, even on the very same EC2/EBS platform -- but you get reliability, replication and backups that some employee would have to be managing otherwise. In theory, at least. RDS didn't live up to its promises when I used it, so I moved back to running databases on physical servers at half the cost.
- genwin 14y ago
  Please elaborate on how RDS didn't live up to its promises?
  - dangrossman 14y ago
    When EBS failed in US-East a year or two ago, my RDS instance did not remain available and did not automatically fail over to its replica, despite my paying for these benefits. Even when it was working correctly, the RDS instances had low capacity compared to physical servers, and there were often spikes of significant latency that would lead to overloaded web servers as DB queries backed up unanswered.
    - genwin 14y ago
      Good to know. I read here on HN that during the storm event in June, the same thing happened (no auto fail over for those who paid for it).

Naushad 14y ago

Well explained, and an awesome way to say "We are hiring a DevOps"

alpb 14y ago

This is a fairly old blog post and it is posted to HN by Kevin Systrom – he's co-founder of Instagram. Here's his post: http://news.ycombinator.com/item?id=3306027 http://news.ycombinator.com/item?id=3306027 (223 points, 220 days ago) so I'd like to say this is a repost. Though, I have no idea how the same link can be reposted to HN.

joering2 14y ago
http://news.ycombinator.com/item?id=3306815 http://news.ycombinator.com/item?id=3306815 Guess they figured that one out in less than 221 days.
achompas 14y ago
I don't understand how this topic comes up so frequently, but HN is configured to allow reposts after a certain amount of time. That isn't really a problem for old posts with excellent information, so I think a repost is warranted for this.

vegas 14y ago

To sum up: it's gonna fall over at any minute without a continuous injection of money.

josephcooney 14y ago
My take-away was "this is why when EC2 when down it wasn't just a case of rebooting the server". My other thought was "and how did all of this help you when EC2 went down?"
- jbigelow76 14y ago
  That was my thought too, the multiple mentions of "stuff" in different zones had me wondering why the recent EC2 outage was able to get them.

dclowd9901 14y ago

Color me surprised: No NoSQL? Instagram seems to be a prime candidate for such a storage method.

deleted 14y ago
[deleted]
gregburd 14y ago
You don't consider Amazon/S3 or Redis NoSQL databases?
- jammmuel 14y ago
  Since when is S3 a 'database'?
  - philjohn 14y ago
    It's got an API, you can store data in it and retreive it later by key, since when was S3 not a database? It's a key/value store that lets you store very large values.
- pjscott 14y ago
  It's weird that things as different as S3 and Redis all fall under the umbrella of "NoSQL". We should probably replace the SQL-NoSQL dichotomy with terms that more accurately reflect the real differences.
- dclowd9901 14y ago
  Not really a "distinctive" one (a la mongo, couch, etc.), but I see your point.
brendoncrawford 14y ago
They use Redis.
timaelliott 14y ago
Redis is a key-value store. S3 is a distributed file system. Can we stop labeling the set of "not a rdbms" data storage mechanisms with the stupid fucking "NoSQL" moniker.

timaelliott 14y ago

> For our geo-search API, we used PostgreSQL for many months, but once our Media entries were sharded, moved over to using Apache Solr. It has a simple JSON interface, so as far as our application is concerned, it’s just another API to consume. Does anyone have particular insight to share on this? Last I checked, Solr's geospatial searching methods are rather inefficient -- haversine across all documents, bounding boxes that rely on haversine and Solr4 was looking into geohashes (better but have some serious edge-case problems where they fall apart). Meanwhile PostgreSQL offers r-tree indexing for spatial queries and is blazing fast. Am I missing some hidden power about Solr's geospatial lookups that make it faster/better than an r-tree implementation?

awj 14y ago
It probably was the database sharding. If the Solr setup could handle the geo-search-related data without the need for sharding it probably can beat out Postgres with sharding. Having this exposed through an api that is standardized and maintained by someone else is also nothing to sneeze at. I'd trade a bit of performance for that kind of standardization and turnkey use in the right scenario.
- rbranson 14y ago
  The reason we use Solr for this specific task is because PostgreSQL cannot efficiently and quickly merge two index queries (time & space). It can do this to a limited degree, but both of these dimensions potentially match 10s of millions of documents, and PG falls over at this.
  - timaelliott 14y ago
    So you make the r-tree 3 dimensions (lat,lng,time). PostgreSQL supports this. I dunno I can't envision Solr being more efficient than a properly designed RDBMS for these situations. If you were integrating a full-text search I'd absolutely believe that to be the case but...
    - 14y ago

deleted 14y ago

[deleted]

deleted 14y ago

[deleted]

deleted 14y ago

[deleted]

deleted 14y ago

[deleted]

deleted 14y ago

[deleted]

deleted 14y ago

[deleted]

rurounijones 14y ago

Do they have any application monitoring stuff a la NewRelic? I saw stuff like munin but that isn't really the same.

deleted 14y ago

[deleted]

resrc 14y ago

"The photos themselves go straight to Amazon S3"... Is there no image resizing involved? It's pretty hard to find stuff about scaling image resizing, It's hugely CPU intensive, that's a sure thing.

lallysingh 14y ago
I'd imagine it's easy enough to do before transmission.
- riobard 14y ago
  Actually they have to do the compression on the mobile clients before uploading, otherwise it will take forever and cost a real fortunate on the mobile data usage.

gagabity 14y ago

Could someone plug these into the AWS monthly calculator and try to figure out the monthly cost, when I try I get around $10,000 per Months, but I think I am wildly underestimating the bandwidth usage. AWS Calculator http://calculator.s3.amazonaws.com/calc5.html http://calculator.s3.amazonaws.com/calc5.html

insaneirish 14y ago

The idea of mdadm on top of EBS volumes makes me cringe. Mark Mayo's thoughts on abstracted block storage are spot on: http://joyeur.com/2011/04/24/magical-block-store-when-abstractions-fail-us/ http://joyeur.com/2011/04/24/magical-block-store-when-abstra...

gsingers 14y ago

A couple of comments as one of the authors of Solr's spatial code: - I don't think any of us have compared it to PostgreSQL, but I can tell you we have clients doing 500 queries per second+ with it including text search. I've yet to see a DB do good text-based fuzzy matching and combining two systems (DB + search) via a join is usually slow. YMMV. - The main goal of the implementation is to add point-based search to text search. It is not a general purpose replacement for an r-tree, etc. - You are not required to use haversine. The distance function is pluggable and we have other options implemented. Also, in many cases, you have other clauses in your query that restrict down the set of documents that need to be scored by distance.