3 ms·

Wow, does that mean we finally get full text search (i.e., LIKE '%%')? Bonus points for OR, JOIN's and multiple inequality filters. Since they forbid these op

by libria 15y ago

Wow, does that mean we finally get full text search (i.e., LIKE '%%')? Bonus points for OR, JOIN's and multiple inequality filters. Since they forbid these operations on the datastore, I'm guessing this will either not scale as well or be significantly more expensive. Since the sign up sheet implies you can combine technologies, I suppose you can use a "hybrid" approach and keep denormalized data in the datastore.

bdonlan 15y ago
Yep, it currently limits you to a 10G database size. Makes sense, though - SQL techniques like LIKE filters or joins really do require all of your data to be in memory with a very high-bandwidth, low latency interconnect - ie, it all has to be in memory on a single box. This is the reason the datastore forbids these operations - because they're extremely difficult to efficiently implement and still scale indefinitely (without making other, potentially very large sacrifices).
- wulczer 15y ago
  What? A join requires all of your data to be in memory? I've sure as hell seen databases that executed queries with joins and the data was not entirely in memory...
  - bdonlan 15y ago
    Well, sure, it doesn't _require_ it. But the alternative is higher query latency. You have to collect a filtered set of the join column from table A, then ship them over to the servers responsible for table B. If your interconnect has high latency or low bandwidth, this can be painful - particularly if the intermediate set of keys is very large. Hence why the datastore simply disallows this - yes, it's _possible_ to make joins work on larger datasets. But it's not possible to make _arbitrary_ joins work well on larger datasets.
- AndrewDucker 15y ago
  Joins on indexed columns don't require anything apart from the index to be in memory in order to do the filtering.
troymc 15y ago
The Google App Engine team has been working on a full-text-search service for GAE projects. They gave a demo at Google I/O back in May. Here's a video of their presentation: http://www.youtube.com/watch?v=7B7FyU9wW8Y http://www.youtube.com/watch?v=7B7FyU9wW8Y
mscarborough 15y ago
Or preferably, Sphinx support. From the announcement I can't see that you can't do it, but not that you can. InnoDB and MyISAM are not good at full text search even on local disk, no reason to make that worse. Somewhat kinda germane post from Percona: http://www.mysqlperformanceblog.com/2009/09/10/what-to-do-with-mysql-full-text-search-while-migrating-to-innodb/ http://www.mysqlperformanceblog.com/2009/09/10/what-to-do-wi...
nl 15y ago
http://www.google.com/events/io/2011/sessions/full-text-search.html http://www.google.com/events/io/2011/sessions/full-text-sear... http://static.googleusercontent.com/external_content/untrusted_dlcp/www.google.com/en//events/io/2011/static/notesfiles/FullTextSearch.pdf http://static.googleusercontent.com/external_content/untrust... I believe this is available in both the Python and Java dev environments. Additionally, http://code.google.com/appengine/docs/python/prospectivesearch/overview.html#Query_Language_Overview http://code.google.com/appengine/docs/python/prospectivesear... might be of interest.

by libria 15y ago

bdonlan 15y ago

Yep, it currently limits you to a 10G database size. Makes sense, though - SQL techniques like LIKE filters or joins really do require all of your data to be in memory with a very high-bandwidth, low latency interconnect - ie, it all has to be in memory on a single box. This is the reason the datastore forbids these operations - because they're extremely difficult to efficiently implement and still scale indefinitely (without making other, potentially very large sacrifices).

wulczer 15y ago
What? A join requires all of your data to be in memory? I've sure as hell seen databases that executed queries with joins and the data was not entirely in memory...
- bdonlan 15y ago
  Well, sure, it doesn't _require_ it. But the alternative is higher query latency. You have to collect a filtered set of the join column from table A, then ship them over to the servers responsible for table B. If your interconnect has high latency or low bandwidth, this can be painful - particularly if the intermediate set of keys is very large. Hence why the datastore simply disallows this - yes, it's _possible_ to make joins work on larger datasets. But it's not possible to make _arbitrary_ joins work well on larger datasets.
AndrewDucker 15y ago
Joins on indexed columns don't require anything apart from the index to be in memory in order to do the filtering.

troymc 15y ago

The Google App Engine team has been working on a full-text-search service for GAE projects. They gave a demo at Google I/O back in May. Here's a video of their presentation: http://www.youtube.com/watch?v=7B7FyU9wW8Y http://www.youtube.com/watch?v=7B7FyU9wW8Y

mscarborough 15y ago

Or preferably, Sphinx support. From the announcement I can't see that you can't do it, but not that you can. InnoDB and MyISAM are not good at full text search even on local disk, no reason to make that worse. Somewhat kinda germane post from Percona: http://www.mysqlperformanceblog.com/2009/09/10/what-to-do-with-mysql-full-text-search-while-migrating-to-innodb/ http://www.mysqlperformanceblog.com/2009/09/10/what-to-do-wi...

nl 15y ago

http://www.google.com/events/io/2011/sessions/full-text-search.html http://www.google.com/events/io/2011/sessions/full-text-sear... http://static.googleusercontent.com/external_content/untrusted_dlcp/www.google.com/en//events/io/2011/static/notesfiles/FullTextSearch.pdf http://static.googleusercontent.com/external_content/untrust... I believe this is available in both the Python and Java dev environments. Additionally, http://code.google.com/appengine/docs/python/prospectivesearch/overview.html#Query_Language_Overview http://code.google.com/appengine/docs/python/prospectivesear... might be of interest.