26 ms·

Searching the web for under $1000/month

456 points by francoismassot 5y ago

artembugara 5y ago
Francois, Adrien, that's a super nice demo. Stateless search engine is something new, for sure. I'd be super interested to see how it evolves over time. We're [1] indexing over 1,000,000 news articles per day. We're using ElasticSearch to index our data. Would be interested to see if there's a way to make a cross-demo? Let me know. [1] https://newscatcherapi.com/ https://newscatcherapi.com/
- fulmicoton 5y ago
  That sounds interesting indeed. Can you schedule a meeting with me? https://calendly.com/paul-quickwit/30min https://calendly.com/paul-quickwit/30min
  - artembugara 5y ago
    Merci
natpat 5y ago
This is super interesting. I've recently also been working on a similar concept: we have a reasonable amount (in the terabytes) of data, that's fairly static, that I need to search fairly infrequently (but sometimes in bulk). A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Random access of a file on S3 is pretty fast, and running in an EC2 instance means latency is almost nil to S3. Cheap, fast and effective. We're using some custom Python code to build a Marisa Trie as our index. I was wondering if there were alternatives to this set up?
- ddorian43 5y ago
  > that I need to search fairly infrequently (but sometimes in bulk). What do you mean by search ? Full-text-search ? Do you need to run custom code on the original data ? > A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Yes, it's like keeping the block-index of a sstable (in rocksdb) in-memory. The next step is to have a local cache on the ec2 node. And the next step one is to have a "distributed" cache on your ec2 nodes, so you don't query S3 for a chunk if it's present in any of your other nodes. Come to think of it, I searched and didn't find a "distributed disk cache with optional replication" that can be used in front of S3 or whatever dataset. You can use nginx/varnish as a reverse-proxy but it doesn't have "distributed". There is Alluxio, but it's single-master.
  - natpat 5y ago
    > What do you mean by search ? Search maybe is too strong a word - "lookup" is probably more correct. I have a couple of identifiers for each document, from which I want to retrieve the full doc. I'm not sure what you mean by running custom code on the data. I usually do some kind of transformation afterwards. I didn't find anything either, which is why I was wondering if I was searching for the wrong thing.
    - ddorian43 5y ago
      How big is each document ? If documents are big, keep each of them as a separate file and store the ids in a database. If documents are small, then you want something like https://github.com/rockset/rocksdb-cloud https://github.com/rockset/rocksdb-cloud for a building block
  - hungnv 5y ago
    > Come to think of it, I searched and didn't find a "distributed disk cache with optional replication" that can be used in front of S3 or whatever dataset. You can use nginx/varnish as a reverse-proxy but it doesn't have "distributed". There is Alluxio, but it's single-master. If you think more about this, it will be like distributed key value store with support both disk and memory access. You can write one using some opensource Raft libraries, or a possible candidate is Tikv from PingCap
    - ddorian43 5y ago
      > If you think more about this, it will be like distributed key value store with support both disk and memory access. You can write one using some opensource Raft libraries, or a possible candidate is Tikv from PingCap My whole point was not building it ;) There's also https://github.com/NVIDIA/aistore https://github.com/NVIDIA/aistore
- looklikean 5y ago
  Combining data-at-rest with some slim index structure coupled with a common access method (like HTTP) was the idea behind a tool a key-value store for JSON I once wrote: https://github.com/miku/microblob https://github.com/miku/microblob I first thought of building a custom index structure, but found that I did not need everything in memory all the time. Using an embedded leveldb works just fine.
- heipei 5y ago
  You could look at AWS Athena, especially if you only query infrequently and can wait a minute on the search results. There are some data layout patterns in your S3 bucket that you can use to optimize the search. Then you have true pay-per-use querying and don't even have to run any EC2 nodes or code yourself.
- giovannibonetti 5y ago
  This is the kind of thing I value in Rails. Active storage [1] has been around for a few years and it solves all of this. All the metadata you care about is in the database - content type, file size, image dimensions, creation date, storage path. [1] https://guides.rubyonrails.org/active_storage_overview.html https://guides.rubyonrails.org/active_storage_overview.html
- fulmicoton 5y ago
  There might be much better alternative but it really depends on the nature of your key. Because the crux of S3 is the latency you can also decide to encode the docs in blocks, and retrieve more data than is actually needed. For this demo, the index from DocID to offset in S3 takes 1.2 bytes per doc. For a log corpus, we end up with 0.2 bytes per doc.
- thejosh 5y ago
  You might want to check out Snowflake for something like this, it makes searching pretty easy, especially as it seems your data is semi-static? We use it pretty extensively at work and it's great. For your usecase it'll be very cheap if you don't access it constantly (you can probably get away with the extra small instances, which you are billed per minute). Not affiliated in anyway, just a suggestion.
- gbrits 5y ago
  Also check out Dremio with parquet files stored on S3
bambax 5y ago
Very interesting! For some reason I find search engines fascinating... How dependent is this on AWS? Can it be ported to another cloud provider?
- fulmicoton 5y ago
  We have a storage abstraction that boils down to being able to perform Range queries. Anything that allows us to do range queries is ok. That includes basically all object storage I know of (Google Cloud Storage, Azure Storage, Minio, you name it), but also HDFS, or even a remote HTTP2 server.
capableweb 5y ago
> which is key as each instance issues a lot of parallel requests to Amazon S3 and tends to be bound by the network I wonder if most of the cost comes from S3, EC2 or the "premium" bandwidth that Amazon charges ridiculously much for. Since it seems to be doing a lot of requests, it wouldn't surprise me if it's the network cost, and if so, I wonder why they would even use AWS at all then.
- ddorian43 5y ago
  > I wonder if most of the cost comes from S3 This current cost comes from the big dataset of storage in S3. > it wouldn't surprise me if it's the network cost Network cost is only outbound. Inside it's free (except multi region etc). Ec2 <-> S3 is free bandwidth (you pay for requests).
heipei 5y ago
This looks really interesting, I wonder how they will monetize it though. As an aside, projects like these are what keep me wondering whether I should switch from cheaper but "dumb" object stores to AWS since on AWS you can use your object store together with things like Athena etc. and get pay-per-use search / grep and a lot of other things, without the egress fees since it's all within AWS.
- fulmicoton 5y ago
  We really need to make this clear in our next blog post. This is not grep here. We are using the same datastructure that are used in Elasticsearch or google. We just adapted them to be object storage friendly. I would not call Object Storage dumb by any mean. They are a very powerful bottom-up abstraction. We do manage to get SSD-like throughput from them. The latency is the big issue. We had to redesign our search to reduce the number of random read in the critical to the bear minimum.
  - heipei 5y ago
    Appreciate the response. I wasn't trying to say this is grep, I fully understand that this is an inverted index which is way more interesting to build on top of S3. I merely wanted to say that by using S3 within AWS you always have the fallback option of brute-force "grep" across your semi-structured "data lake" or whatever it's called thanks to the aggregate bandwidth and Athena.
    - fulmicoton 5y ago
      Ah my bad! Yes, Humio (and Loki) are opting for this approach. This does decouple compute and storage in a trivial manner. There is indeed a realm in which this brute force approach is the best approach. We could probably make a 4D chart with QPS, data size, latency, and retention period and define regions where the elastic/SOLR approach, Humio, and quickwit are the most relevant.
hu3 5y ago
Article title is "Searching the web for < $1000 / month". Despite mentioning Rust once, of course it had to be added to the title on HN as "Search 1B pages on AWS S3 for 1000$ / month, made in Rust and tantivy".
jonatron 5y ago
If you're going for low cost, you could do better: https://www.hetzner.com/dedicated-rootserver/dell/dx181/configurator https://www.hetzner.com/dedicated-rootserver/dell/dx181/conf... Basic configuration in Finland 1 224,91 € 1.92 TB SATA SSD Datacenter Edition 4 95,20 € 320,11 € 320 Euro equals 385.90 United States Dollar
- gallexme 5y ago
  Depending on the requirements https://www.hetzner.com/dedicated-rootserver/ax101/ https://www.hetzner.com/dedicated-rootserver/ax101/ May be an actually better fit Once they available again
  - ddorian43 5y ago
    You'll be waiting 1+ month to get the server above.
    - Fgffhjj 5y ago
      I got a 64gb and server with 2*2tb NVM in 10 minutes. Bought it 2 weeks ago for 60/month. Or are you referencing something like CPU shortage at that model?
      - ddorian43 5y ago
        Referencing the AX101. And the storage line.
  - jiofih 5y ago
    Holy smokes. 8TB SSD + 128GB RAM + Ryzen 9 for 100 euro a month. Can you get anywhere close to this with AWS or even DO?
    - kuschku 5y ago
      Of course not. But that's why the "cloud" (as in the typical DO/AWS/Azure/GCP offerings) are a scam.
visarga 5y ago
Is it a web search engine or an adjective search engine? I'd love to see someone make a deep search engine that goes beyond the 100...1000 limit.
- fulmicoton 5y ago
  It is a web search engine. As explained in the blog post, we made the demo by generating 18k snippets and pushing them to a NLP pipeline that tries to extract the adjective phrase. The tech below is an inverted index.
djdjdjdjdj 5y ago
Hui I wonder why this is not a cost trap. The S3 API request s where relative expensive.
- fulmicoton 5y ago
  The bandwidth is free if you are in the same region.
  - djdjdjdjdj 5y ago
    But you pay for requests.
    - fulmicoton 5y ago
      A normal search experience (displaying a 20 hits search page) requires num segments * (1 + num terms * 2) + 20 GET requests. We have 180 segments for our commoncrawl index. So we can consider a generous upper bound of 1000 requests. The GET request costs adds $0.0004 per commoncrawl search request. Storage costs us $5 per day, so the cost of GET request starts topping storage cost at >10k request per day. Our search engine is meant for searching large datasets, with a low number of queries: Logs, SIEM, e-discovery, exotic big data datasets, etc. These use case have typically a low daily query rate. For high request rate, (1 query per second) like e-commerce, entirely decoupling storage and compute is actually a bad idea. For low request rate (< 1000 per day), using S3 without caring about the GET request cost is perfectly fine. And in the middle, you might probably want to use another object model with a more favorable pricing model.
busymom0 5y ago
Is this reliant on S3 or can it be used on something like minio or digital ocean spaces or backblaze2 too? Backblaze to cloudflare data transfers is free so that can reduce costs a lot plus B2 is much cheaper than S3.
- fulmicoton 5y ago
  It can work on any object storages. I really want to test on Minio to see performance fly :)
marcinzm 5y ago
Interesting although a 15 second response time on certain queries is not a very good user experience.
- cj 5y ago
  On the other hand, under 1.5 seconds on common / basic search terms is pretty good.
  - fulmicoton 5y ago
    The poster was referring to the latency of the demo and is absolutely correct. The demo can reach 30s on some query. Half of it is due to fetch 180k document generation, and half of it is a single threaded python code that has nothing to do with our product :).
- fulmicoton 5y ago
  This demo is not indeed quite misleading. The high response time is due to the fact that we generate 18k snippets to generate the tag cloud. Imagine this is the equivalent of clicking on page 1 to 900 on Google! A "barack obama" phrase query generating 20 snippets runs in less than 2seconds on our 2 cheap servers. I'll set up a "normal 20 results search setting" next week and share it an API to show the latency again.
rossmohax 5y ago
It is a cool project. S3 can be cost efficient, but only if you don't touch data :) Their price calculation doesn't mention cost of S3 requests, which very quickly adds up and is often neglected. It costs $1 for 2.5M GET requests to S3. They have 180 shards, in a general case query seems to fetch all of them. Presumably they don't download full shard per request, but download an index + some relevant ranges. Lets say that is 10 requests per shard. So that would be 1800 S3 GET requests = ~1400 search queries cost them $1. Assuming their service is reasonably popular and serve 1 req/second on average, that would be $1,440 per 30 days in addition to advertised $1,000 spent on EC2 and S3 storage. Seems comparable to AWS ElasticSearch service costs: - 3 nodes m5.2xlarge.elasticsearch = $1,200 - 20TB EBS storage = $1,638
- fulmicoton 5y ago
  I tend to agree :). If we get 1 req/s, even for a dataset of that size, this is not as cost efficient. For that kind of use case, I'd probably start using minio. > Seems comparable to AWS ElasticSearch service costs: > - 3 nodes m5.2xlarge.elasticsearch = $1,200 > - 20TB EBS storage = $1,638 Don't forget S3 includes replication. Also EBS throughput (even with SSD) is not good at all. Also our memory footprint is tiny. This is necessary to make it run on two servers. Finally, cpu-wise, our search engine is almost 2x faster than lucene. If you don't believe us, try to replicate our demo on an elastic search :D. Chatnoir.eu is the only other common crawl cluster we know of. It consists of 120 nodes.
  - rossmohax 5y ago
    > If we get 1 req/s, even for a dataset of that size, this is not as cost efficient. How many req/s do you have in mind for your system to be a viable option? > Also EBS throughput (even with SSD) is not good at all. It is not worse than S3 still, right? > Chatnoir.eu is the only other common crawl cluster we know of. It consists of 120 nodes. I have no deep ES experience. Are you saying, that to host 6TB of indexed data (before replication) you'd need 120 nodes ES cluster? If so, then reducing it to just 2 nodes is the real sales pitch, not S3 usage :)
  - klohto 5y ago
ykevinator3 5y ago
What an amazing project, good luck to you guys and thanks for sharing.
- fulmicoton 5y ago
  Thank you @ykevinator!
karterk 5y ago
Cool demo. Searching for phases like "there was a" and "and there is" take a really long time. I presume that since the words are common, the document IDs mapped to those individual tokens are too long as well, so intersections etc. take longer?
- francoismassot 5y ago
  Thanks! You are totally right. For the demo, we have even banned a few words like "the" because the inverted list contains almost all doc ids...
deleted 5y ago
[deleted]
ryanworl 5y ago
What are you using for metadata storage?
- fulmicoton 5y ago
  There are only 180 splits. For this demo we use a file. For more serious stuff we use postgresql.
  - ryanworl 5y ago
    What does the metadata structure look like?
    - guilload 5y ago
      We store the URI of each shard making up the index and, optionally, partition key and value(s). Along with a few flags, we also store the shard size, creation and last modification time. This additional metadata is not required for the query planning phase and is only useful for managing the life cycle of the shards and debugging/troubleshooting.
Grimm1 5y ago
How are you dealing with the fact common crawl updates their data much less regularly than commercial search engines? And that each update is only a partial refresh? Edit: And I will say your site design is very nice.
- guilload 5y ago
  We indexed Common Crawl only for the purpose of this demo so this is one-time thing, we won't deal with updates.
  - Grimm1 5y ago
    Ah I understand you're showcasing the methodology for the underlying index but you're going to open source the engine. I see, great stuff then, super novel and honestly the rest of the open source search engines can definitely use some competition. Love it!
- francoismassot 5y ago
  Thank you! We did not plan to regularly update the index. But as it takes only 24 hours to index 1B pages, the easiest way would be to reindex everything, upload it to S3 and update the metadata so the search engine will query the right segments.
cardosof 5y ago
Congrats for the project and very cool demo! One point that may help - I've searched the word fast with adjective selected and it didn't show results.
- francoismassot 5y ago
  thanks! I guess you had no luck and the server did not respond, we have a bunch of errors on the python server and it may come from here. It's working now, you can try it and found that the result is "fast and easy": https://common-crawl.quickwit.io/?query=fast&partOfSpeech=ADJECTIVE&position=AFTER https://common-crawl.quickwit.io/?query=fast&partOfSpeech=AD...
ClumsyPilot 5y ago
Seems like you could build a workstation that runs these quesries faster and cheaper than AWS ever could on a RAIDed set of NVME drives. https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-threadripper-pro-workstation/ https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-th...
sam_lowry_ 5y ago
Why use AWS if you are cost-conscious?
- francoismassot 5y ago
  The main reason is that AWS S3 is widely used. We obviously want to make it work on HDFS, MinIO and other relevant storage systems.
- codeecan 5y ago
  Their source data, common crawl has s3 hooks, probably played some part
chris_f 5y ago
Nice! Maybe at one point you can release a general web search engine for the Common Crawl corpus? It seems even simpler than this proof of concept, but potentially more useful for people looking for a true full text web search. There isn't an easy way today to explore or search what is contained in the Common Crawl index.
- hansvm 5y ago
  > There isn't an easy way today to explore or search what is contained in the Common Crawl index. By that you mean searching the full text contents of their crawl, right? The index is super easy to search nowadays -- in pretty much any language you can slap a few lines of code around a get request (using range requests [0] if needed), and explore a columnar representation of the index [1]. [0] https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35 https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec1... [1] https://commoncrawl.org/2018/03/index-to-warc-files-and-urls-in-columnar-format/ https://commoncrawl.org/2018/03/index-to-warc-files-and-urls...
- fulmicoton 5y ago
  That's on my to-do list for next week. :)
ywelsch 5y ago
Interesting! We've built similar support for decoupling compute from storage into Elasticsearch and, as coincidence would have it, just shared some performance numbers today: https://www.elastic.co/blog/querying-a-petabyte-of-cloud-storage-in-10-minutes https://www.elastic.co/blog/querying-a-petabyte-of-cloud-sto... It works just as any regular Elasticsearch index (with full Kibana support etc.). The data being indexed by Lucene allows queries to access index structures and return results orders of magnitude faster than doing a full table scan. It is complemented with various caching layers to make repeat queries fast. We expect this new functionality to be used for less frequently queried data (e.g. operational or security investigations, legal discoveries, or historical performance comparisons on older data), trading query speed for cost. It supports Google Cloud Storage, Azure Blob Storage, Amazon S3 (+ S3 compatible stores), HDFS, and shared file systems.
simonw 5y ago
What does your on-S3 storage format look like? Are you storing relatively large blobs and doing HTTP Range requests against them or are you storing lots of tiny objects and fetching the whole object any time you need it?
- guilload 5y ago
  What we store on S3 is a regular tantivy index and another tiny data structure that we call "turbo index", which makes queries faster on object storages. For this demo, the tantivy indexes are fairly large and we issue HTTP Range requests against them. https://github.com/tantivy-search/tantivy https://github.com/tantivy-search/tantivy
snidane 5y ago
Chaos Search seems to be doing this architecture already and according to the podcast episode [1], it uses a highly optimized storage layout. Never used it, so would be interested if somebody could comment on it. [1] https://www.dataengineeringpodcast.com/chaos-search-with-pete-cheslock-and-thomas-hazel-episode-47/ https://www.dataengineeringpodcast.com/chaos-search-with-pet...
chrisacky 5y ago
Is there a more recent common crawl data set? 2019 is a long time away. Reason I ask is I'm trying to get all subdomain a of a certain domain. So I want a reverse host of unique hostnames under a certain domain.
- guilload 5y ago
  There are more recent versions of the dataset. We used the february/march snapshot from this year and the April snapshot just came out (https://commoncrawl.org/2021/04/april-2021-crawl-archive-now-available/ https://commoncrawl.org/2021/04/april-2021-crawl-archive-now...).
0xbadcafebee 5y ago
> a new breed of full-text search engine The following is a stupid question, so bare with me. I have been using search engines for about... 26 years. I have attempted to make really crappy databases and search engines. I have worked for companies that use search products for internal services and customer products. I'm not a search engineer but I have a decent understanding of them and their issues, I think. And I get why people want full-text search. But is it actually a good idea? Should anyone really be using full text search? I actually work on search products right now. We use Solr as the general full text index. We have separate indexes and algorithms to make context and semantic inferences, and prioritize results based on those, falling back to full text if we don't get anything. The full text sucks. The corpus of relationships of related concepts is what makes the whole thing useful. Are we (all) only using full-text because some users are demanding that it be there? Or shouldn't we all stop this charade of thinking that full-text search of billions of items of data will ever be useful to a human being? Even when I show my coworkers that I can get something done 10x faster with a curated index of content, they still want a search engine that they know doesn't give them the results they want. Is full-text search the junk food of information retrieval?
johnghanks 5y ago
This is an ad.
phendrenad2 5y ago
Searching the web is a fool's errand. Google doesn't even search the web anymore, they just mind-controlled everyone to submit nightly sitemaps to them. Google is more of an index than a search engine nowadays.
ProKevinY 5y ago
Brilliant and interesting project by smart people. Kudos. (the demo is addictive af)
imhoguy 5y ago
Could this be adapted for IPFS? Anyone with stateless client and link to index could search and become part of swarm to speed up trendy queries with redundancy. Then update it with git like diff versioning, utilize IPNS to point to HEAD of the latest chain of the index.
not2b 5y ago
But are you solving the right problem? This sounds like someone has produced a very good and efficient version of AltaVista. Back in the 1990s, if you wanted to do classic keyword searches of the web, and find all pages that had terms A and B but not C, it would give them to you, in a big unsorted pile. The web was still small enough that this was sometimes useful, but until Google came along with tricks to rank pages that are obvious in retrospect, it just wasn't useful for common search terms.
gmiller123456 5y ago
Curious to know if anyone can explain why on-line storage is so expensive? Most places want $100+/mo for 1TB of storage, while a 1TB drive only costs $50. I understand there's management costs, cooling, electricity, physical space, etc. But those would be per drive costs, not per TB, and certainly wouldn't add up to $100/mo. Meanwhile, there's services like Google Drive, etc which costs about $100/yr per TB. Still exorbitant, but not as much so. 3rd party software can mount it as a drive, but only for a short period of time before the token expires. So they seem happy to sell you space at less than 1/10th the cost, as long as it's harder to use. There just seems to be a lot of cash on the table for someone to offer much cheaper storage solutions, but no one is actually doing it.
- cbellew 5y ago
  Don't forget the replication. S3 replicates to at least three other AZs in the same region. I know this still doesn't account for the difference but its worth remembering.
  - gmiller123456 5y ago
    I can see extra features costing more, but a lot of people would be happy with nothing but a network connected drive. If it goes bad, it's bad. Hopefully I have a backup somewhere.
- Dylan16807 5y ago
  The big clouds have premium storage and absolutely gouge on bandwidth. For a more basic offering, B2 is $60 a year and there are other services in the same ballpark. They definitely want it to be convenient for you, and make a modest profit percentage. I haven't experienced any issues with google drive tokens, though I've mostly been using the business tier.
- BugWatch 5y ago
  Well, there are also data integrity issues to take into account... in the simplest terms, you'd actually need 3x the available storage to "guarantee" that there won't be any bit nor data loss if a single drive (our of three) - does... Add a potential (mandatory?) off-site backup, means of restoring it, and... Not even getting into the bandwidth cost can of worms. Again, this is in very simple (and low-scale) terms.

Francois, Adrien, that's a super nice demo. Stateless search engine is something new, for sure. I'd be super interested to see how it evolves over time. We're [1] indexing over 1,000,000 news articles per day. We're using ElasticSearch to index our data. Would be interested to see if there's a way to make a cross-demo? Let me know. [1] https://newscatcherapi.com/ https://newscatcherapi.com/

fulmicoton 5y ago
That sounds interesting indeed. Can you schedule a meeting with me? https://calendly.com/paul-quickwit/30min https://calendly.com/paul-quickwit/30min
- artembugara 5y ago
  Merci

natpat 5y ago

This is super interesting. I've recently also been working on a similar concept: we have a reasonable amount (in the terabytes) of data, that's fairly static, that I need to search fairly infrequently (but sometimes in bulk). A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Random access of a file on S3 is pretty fast, and running in an EC2 instance means latency is almost nil to S3. Cheap, fast and effective. We're using some custom Python code to build a Marisa Trie as our index. I was wondering if there were alternatives to this set up?

ddorian43 5y ago
> that I need to search fairly infrequently (but sometimes in bulk). What do you mean by search ? Full-text-search ? Do you need to run custom code on the original data ? > A solution we came up with was a small , hot, in memory index, that points to the location of the data in a file on S3. Yes, it's like keeping the block-index of a sstable (in rocksdb) in-memory. The next step is to have a local cache on the ec2 node. And the next step one is to have a "distributed" cache on your ec2 nodes, so you don't query S3 for a chunk if it's present in any of your other nodes. Come to think of it, I searched and didn't find a "distributed disk cache with optional replication" that can be used in front of S3 or whatever dataset. You can use nginx/varnish as a reverse-proxy but it doesn't have "distributed". There is Alluxio, but it's single-master.
- natpat 5y ago
  > What do you mean by search ? Search maybe is too strong a word - "lookup" is probably more correct. I have a couple of identifiers for each document, from which I want to retrieve the full doc. I'm not sure what you mean by running custom code on the data. I usually do some kind of transformation afterwards. I didn't find anything either, which is why I was wondering if I was searching for the wrong thing.
  - ddorian43 5y ago
    How big is each document ? If documents are big, keep each of them as a separate file and store the ids in a database. If documents are small, then you want something like https://github.com/rockset/rocksdb-cloud https://github.com/rockset/rocksdb-cloud for a building block
- hungnv 5y ago
  > Come to think of it, I searched and didn't find a "distributed disk cache with optional replication" that can be used in front of S3 or whatever dataset. You can use nginx/varnish as a reverse-proxy but it doesn't have "distributed". There is Alluxio, but it's single-master. If you think more about this, it will be like distributed key value store with support both disk and memory access. You can write one using some opensource Raft libraries, or a possible candidate is Tikv from PingCap
  - ddorian43 5y ago
    > If you think more about this, it will be like distributed key value store with support both disk and memory access. You can write one using some opensource Raft libraries, or a possible candidate is Tikv from PingCap My whole point was not building it ;) There's also https://github.com/NVIDIA/aistore https://github.com/NVIDIA/aistore
looklikean 5y ago
Combining data-at-rest with some slim index structure coupled with a common access method (like HTTP) was the idea behind a tool a key-value store for JSON I once wrote: https://github.com/miku/microblob https://github.com/miku/microblob I first thought of building a custom index structure, but found that I did not need everything in memory all the time. Using an embedded leveldb works just fine.
heipei 5y ago
You could look at AWS Athena, especially if you only query infrequently and can wait a minute on the search results. There are some data layout patterns in your S3 bucket that you can use to optimize the search. Then you have true pay-per-use querying and don't even have to run any EC2 nodes or code yourself.
giovannibonetti 5y ago
This is the kind of thing I value in Rails. Active storage [1] has been around for a few years and it solves all of this. All the metadata you care about is in the database - content type, file size, image dimensions, creation date, storage path. [1] https://guides.rubyonrails.org/active_storage_overview.html https://guides.rubyonrails.org/active_storage_overview.html
fulmicoton 5y ago
There might be much better alternative but it really depends on the nature of your key. Because the crux of S3 is the latency you can also decide to encode the docs in blocks, and retrieve more data than is actually needed. For this demo, the index from DocID to offset in S3 takes 1.2 bytes per doc. For a log corpus, we end up with 0.2 bytes per doc.
thejosh 5y ago
You might want to check out Snowflake for something like this, it makes searching pretty easy, especially as it seems your data is semi-static? We use it pretty extensively at work and it's great. For your usecase it'll be very cheap if you don't access it constantly (you can probably get away with the extra small instances, which you are billed per minute). Not affiliated in anyway, just a suggestion.
gbrits 5y ago
Also check out Dremio with parquet files stored on S3

bambax 5y ago

Very interesting! For some reason I find search engines fascinating... How dependent is this on AWS? Can it be ported to another cloud provider?

fulmicoton 5y ago
We have a storage abstraction that boils down to being able to perform Range queries. Anything that allows us to do range queries is ok. That includes basically all object storage I know of (Google Cloud Storage, Azure Storage, Minio, you name it), but also HDFS, or even a remote HTTP2 server.

capableweb 5y ago

> which is key as each instance issues a lot of parallel requests to Amazon S3 and tends to be bound by the network I wonder if most of the cost comes from S3, EC2 or the "premium" bandwidth that Amazon charges ridiculously much for. Since it seems to be doing a lot of requests, it wouldn't surprise me if it's the network cost, and if so, I wonder why they would even use AWS at all then.

ddorian43 5y ago
> I wonder if most of the cost comes from S3 This current cost comes from the big dataset of storage in S3. > it wouldn't surprise me if it's the network cost Network cost is only outbound. Inside it's free (except multi region etc). Ec2 <-> S3 is free bandwidth (you pay for requests).

heipei 5y ago

This looks really interesting, I wonder how they will monetize it though. As an aside, projects like these are what keep me wondering whether I should switch from cheaper but "dumb" object stores to AWS since on AWS you can use your object store together with things like Athena etc. and get pay-per-use search / grep and a lot of other things, without the egress fees since it's all within AWS.

fulmicoton 5y ago
We really need to make this clear in our next blog post. This is not grep here. We are using the same datastructure that are used in Elasticsearch or google. We just adapted them to be object storage friendly. I would not call Object Storage dumb by any mean. They are a very powerful bottom-up abstraction. We do manage to get SSD-like throughput from them. The latency is the big issue. We had to redesign our search to reduce the number of random read in the critical to the bear minimum.
- heipei 5y ago
  Appreciate the response. I wasn't trying to say this is grep, I fully understand that this is an inverted index which is way more interesting to build on top of S3. I merely wanted to say that by using S3 within AWS you always have the fallback option of brute-force "grep" across your semi-structured "data lake" or whatever it's called thanks to the aggregate bandwidth and Athena.
  - fulmicoton 5y ago
    Ah my bad! Yes, Humio (and Loki) are opting for this approach. This does decouple compute and storage in a trivial manner. There is indeed a realm in which this brute force approach is the best approach. We could probably make a 4D chart with QPS, data size, latency, and retention period and define regions where the elastic/SOLR approach, Humio, and quickwit are the most relevant.

hu3 5y ago

Article title is "Searching the web for < $1000 / month". Despite mentioning Rust once, of course it had to be added to the title on HN as "Search 1B pages on AWS S3 for 1000$ / month, made in Rust and tantivy".

jonatron 5y ago

If you're going for low cost, you could do better: https://www.hetzner.com/dedicated-rootserver/dell/dx181/configurator https://www.hetzner.com/dedicated-rootserver/dell/dx181/conf... Basic configuration in Finland 1 224,91 € 1.92 TB SATA SSD Datacenter Edition 4 95,20 € 320,11 € 320 Euro equals 385.90 United States Dollar

gallexme 5y ago
Depending on the requirements https://www.hetzner.com/dedicated-rootserver/ax101/ https://www.hetzner.com/dedicated-rootserver/ax101/ May be an actually better fit Once they available again
- ddorian43 5y ago
  You'll be waiting 1+ month to get the server above.
  - Fgffhjj 5y ago
    I got a 64gb and server with 2*2tb NVM in 10 minutes. Bought it 2 weeks ago for 60/month. Or are you referencing something like CPU shortage at that model?
    - ddorian43 5y ago
      Referencing the AX101. And the storage line.
- jiofih 5y ago
  Holy smokes. 8TB SSD + 128GB RAM + Ryzen 9 for 100 euro a month. Can you get anywhere close to this with AWS or even DO?
  - kuschku 5y ago
    Of course not. But that's why the "cloud" (as in the typical DO/AWS/Azure/GCP offerings) are a scam.

visarga 5y ago

Is it a web search engine or an adjective search engine? I'd love to see someone make a deep search engine that goes beyond the 100...1000 limit.

fulmicoton 5y ago
It is a web search engine. As explained in the blog post, we made the demo by generating 18k snippets and pushing them to a NLP pipeline that tries to extract the adjective phrase. The tech below is an inverted index.

djdjdjdjdj 5y ago

Hui I wonder why this is not a cost trap. The S3 API request s where relative expensive.

fulmicoton 5y ago
The bandwidth is free if you are in the same region.
- djdjdjdjdj 5y ago
  But you pay for requests.
  - fulmicoton 5y ago
    A normal search experience (displaying a 20 hits search page) requires num segments * (1 + num terms * 2) + 20 GET requests. We have 180 segments for our commoncrawl index. So we can consider a generous upper bound of 1000 requests. The GET request costs adds $0.0004 per commoncrawl search request. Storage costs us $5 per day, so the cost of GET request starts topping storage cost at >10k request per day. Our search engine is meant for searching large datasets, with a low number of queries: Logs, SIEM, e-discovery, exotic big data datasets, etc. These use case have typically a low daily query rate. For high request rate, (1 query per second) like e-commerce, entirely decoupling storage and compute is actually a bad idea. For low request rate (< 1000 per day), using S3 without caring about the GET request cost is perfectly fine. And in the middle, you might probably want to use another object model with a more favorable pricing model.

busymom0 5y ago

Is this reliant on S3 or can it be used on something like minio or digital ocean spaces or backblaze2 too? Backblaze to cloudflare data transfers is free so that can reduce costs a lot plus B2 is much cheaper than S3.

fulmicoton 5y ago
It can work on any object storages. I really want to test on Minio to see performance fly :)

marcinzm 5y ago

Interesting although a 15 second response time on certain queries is not a very good user experience.

cj 5y ago
On the other hand, under 1.5 seconds on common / basic search terms is pretty good.
- fulmicoton 5y ago
  The poster was referring to the latency of the demo and is absolutely correct. The demo can reach 30s on some query. Half of it is due to fetch 180k document generation, and half of it is a single threaded python code that has nothing to do with our product :).
fulmicoton 5y ago
This demo is not indeed quite misleading. The high response time is due to the fact that we generate 18k snippets to generate the tag cloud. Imagine this is the equivalent of clicking on page 1 to 900 on Google! A "barack obama" phrase query generating 20 snippets runs in less than 2seconds on our 2 cheap servers. I'll set up a "normal 20 results search setting" next week and share it an API to show the latency again.

rossmohax 5y ago

It is a cool project. S3 can be cost efficient, but only if you don't touch data :) Their price calculation doesn't mention cost of S3 requests, which very quickly adds up and is often neglected. It costs $1 for 2.5M GET requests to S3. They have 180 shards, in a general case query seems to fetch all of them. Presumably they don't download full shard per request, but download an index + some relevant ranges. Lets say that is 10 requests per shard. So that would be 1800 S3 GET requests = ~1400 search queries cost them $1. Assuming their service is reasonably popular and serve 1 req/second on average, that would be $1,440 per 30 days in addition to advertised $1,000 spent on EC2 and S3 storage. Seems comparable to AWS ElasticSearch service costs: - 3 nodes m5.2xlarge.elasticsearch = $1,200 - 20TB EBS storage = $1,638

fulmicoton 5y ago
I tend to agree :). If we get 1 req/s, even for a dataset of that size, this is not as cost efficient. For that kind of use case, I'd probably start using minio. > Seems comparable to AWS ElasticSearch service costs: > - 3 nodes m5.2xlarge.elasticsearch = $1,200 > - 20TB EBS storage = $1,638 Don't forget S3 includes replication. Also EBS throughput (even with SSD) is not good at all. Also our memory footprint is tiny. This is necessary to make it run on two servers. Finally, cpu-wise, our search engine is almost 2x faster than lucene. If you don't believe us, try to replicate our demo on an elastic search :D. Chatnoir.eu is the only other common crawl cluster we know of. It consists of 120 nodes.
- rossmohax 5y ago
  > If we get 1 req/s, even for a dataset of that size, this is not as cost efficient. How many req/s do you have in mind for your system to be a viable option? > Also EBS throughput (even with SSD) is not good at all. It is not worse than S3 still, right? > Chatnoir.eu is the only other common crawl cluster we know of. It consists of 120 nodes. I have no deep ES experience. Are you saying, that to host 6TB of indexed data (before replication) you'd need 120 nodes ES cluster? If so, then reducing it to just 2 nodes is the real sales pitch, not S3 usage :)
- klohto 5y ago

ykevinator3 5y ago

What an amazing project, good luck to you guys and thanks for sharing.

fulmicoton 5y ago
Thank you @ykevinator!

karterk 5y ago

Cool demo. Searching for phases like "there was a" and "and there is" take a really long time. I presume that since the words are common, the document IDs mapped to those individual tokens are too long as well, so intersections etc. take longer?

francoismassot 5y ago
Thanks! You are totally right. For the demo, we have even banned a few words like "the" because the inverted list contains almost all doc ids...

deleted 5y ago

[deleted]

ryanworl 5y ago

What are you using for metadata storage?

fulmicoton 5y ago
There are only 180 splits. For this demo we use a file. For more serious stuff we use postgresql.
- ryanworl 5y ago
  What does the metadata structure look like?
  - guilload 5y ago
    We store the URI of each shard making up the index and, optionally, partition key and value(s). Along with a few flags, we also store the shard size, creation and last modification time. This additional metadata is not required for the query planning phase and is only useful for managing the life cycle of the shards and debugging/troubleshooting.

Grimm1 5y ago

How are you dealing with the fact common crawl updates their data much less regularly than commercial search engines? And that each update is only a partial refresh? Edit: And I will say your site design is very nice.

guilload 5y ago
We indexed Common Crawl only for the purpose of this demo so this is one-time thing, we won't deal with updates.
- Grimm1 5y ago
  Ah I understand you're showcasing the methodology for the underlying index but you're going to open source the engine. I see, great stuff then, super novel and honestly the rest of the open source search engines can definitely use some competition. Love it!
francoismassot 5y ago
Thank you! We did not plan to regularly update the index. But as it takes only 24 hours to index 1B pages, the easiest way would be to reindex everything, upload it to S3 and update the metadata so the search engine will query the right segments.

cardosof 5y ago

Congrats for the project and very cool demo! One point that may help - I've searched the word fast with adjective selected and it didn't show results.

francoismassot 5y ago
thanks! I guess you had no luck and the server did not respond, we have a bunch of errors on the python server and it may come from here. It's working now, you can try it and found that the result is "fast and easy": https://common-crawl.quickwit.io/?query=fast&partOfSpeech=ADJECTIVE&position=AFTER https://common-crawl.quickwit.io/?query=fast&partOfSpeech=AD...

ClumsyPilot 5y ago

Seems like you could build a workstation that runs these quesries faster and cheaper than AWS ever could on a RAIDed set of NVME drives. https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-threadripper-pro-workstation/ https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-th...

sam_lowry_ 5y ago

Why use AWS if you are cost-conscious?

francoismassot 5y ago
The main reason is that AWS S3 is widely used. We obviously want to make it work on HDFS, MinIO and other relevant storage systems.
codeecan 5y ago
Their source data, common crawl has s3 hooks, probably played some part

chris_f 5y ago

Nice! Maybe at one point you can release a general web search engine for the Common Crawl corpus? It seems even simpler than this proof of concept, but potentially more useful for people looking for a true full text web search. There isn't an easy way today to explore or search what is contained in the Common Crawl index.

hansvm 5y ago
> There isn't an easy way today to explore or search what is contained in the Common Crawl index. By that you mean searching the full text contents of their crawl, right? The index is super easy to search nowadays -- in pretty much any language you can slap a few lines of code around a get request (using range requests [0] if needed), and explore a columnar representation of the index [1]. [0] https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35 https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec1... [1] https://commoncrawl.org/2018/03/index-to-warc-files-and-urls-in-columnar-format/ https://commoncrawl.org/2018/03/index-to-warc-files-and-urls...
fulmicoton 5y ago
That's on my to-do list for next week. :)

ywelsch 5y ago

Interesting! We've built similar support for decoupling compute from storage into Elasticsearch and, as coincidence would have it, just shared some performance numbers today: https://www.elastic.co/blog/querying-a-petabyte-of-cloud-storage-in-10-minutes https://www.elastic.co/blog/querying-a-petabyte-of-cloud-sto... It works just as any regular Elasticsearch index (with full Kibana support etc.). The data being indexed by Lucene allows queries to access index structures and return results orders of magnitude faster than doing a full table scan. It is complemented with various caching layers to make repeat queries fast. We expect this new functionality to be used for less frequently queried data (e.g. operational or security investigations, legal discoveries, or historical performance comparisons on older data), trading query speed for cost. It supports Google Cloud Storage, Azure Blob Storage, Amazon S3 (+ S3 compatible stores), HDFS, and shared file systems.

simonw 5y ago

What does your on-S3 storage format look like? Are you storing relatively large blobs and doing HTTP Range requests against them or are you storing lots of tiny objects and fetching the whole object any time you need it?

guilload 5y ago
What we store on S3 is a regular tantivy index and another tiny data structure that we call "turbo index", which makes queries faster on object storages. For this demo, the tantivy indexes are fairly large and we issue HTTP Range requests against them. https://github.com/tantivy-search/tantivy https://github.com/tantivy-search/tantivy

snidane 5y ago

Chaos Search seems to be doing this architecture already and according to the podcast episode [1], it uses a highly optimized storage layout. Never used it, so would be interested if somebody could comment on it. [1] https://www.dataengineeringpodcast.com/chaos-search-with-pete-cheslock-and-thomas-hazel-episode-47/ https://www.dataengineeringpodcast.com/chaos-search-with-pet...

chrisacky 5y ago

Is there a more recent common crawl data set? 2019 is a long time away. Reason I ask is I'm trying to get all subdomain a of a certain domain. So I want a reverse host of unique hostnames under a certain domain.

guilload 5y ago
There are more recent versions of the dataset. We used the february/march snapshot from this year and the April snapshot just came out (https://commoncrawl.org/2021/04/april-2021-crawl-archive-now-available/ https://commoncrawl.org/2021/04/april-2021-crawl-archive-now...).

0xbadcafebee 5y ago

> a new breed of full-text search engine The following is a stupid question, so bare with me. I have been using search engines for about... 26 years. I have attempted to make really crappy databases and search engines. I have worked for companies that use search products for internal services and customer products. I'm not a search engineer but I have a decent understanding of them and their issues, I think. And I get why people want full-text search. But is it actually a good idea? Should anyone really be using full text search? I actually work on search products right now. We use Solr as the general full text index. We have separate indexes and algorithms to make context and semantic inferences, and prioritize results based on those, falling back to full text if we don't get anything. The full text sucks. The corpus of relationships of related concepts is what makes the whole thing useful. Are we (all) only using full-text because some users are demanding that it be there? Or shouldn't we all stop this charade of thinking that full-text search of billions of items of data will ever be useful to a human being? Even when I show my coworkers that I can get something done 10x faster with a curated index of content, they still want a search engine that they know doesn't give them the results they want. Is full-text search the junk food of information retrieval?

johnghanks 5y ago

This is an ad.

phendrenad2 5y ago

Searching the web is a fool's errand. Google doesn't even search the web anymore, they just mind-controlled everyone to submit nightly sitemaps to them. Google is more of an index than a search engine nowadays.

ProKevinY 5y ago

Brilliant and interesting project by smart people. Kudos. (the demo is addictive af)

imhoguy 5y ago

Could this be adapted for IPFS? Anyone with stateless client and link to index could search and become part of swarm to speed up trendy queries with redundancy. Then update it with git like diff versioning, utilize IPNS to point to HEAD of the latest chain of the index.

not2b 5y ago

But are you solving the right problem? This sounds like someone has produced a very good and efficient version of AltaVista. Back in the 1990s, if you wanted to do classic keyword searches of the web, and find all pages that had terms A and B but not C, it would give them to you, in a big unsorted pile. The web was still small enough that this was sometimes useful, but until Google came along with tricks to rank pages that are obvious in retrospect, it just wasn't useful for common search terms.

gmiller123456 5y ago

Curious to know if anyone can explain why on-line storage is so expensive? Most places want $100+/mo for 1TB of storage, while a 1TB drive only costs $50. I understand there's management costs, cooling, electricity, physical space, etc. But those would be per drive costs, not per TB, and certainly wouldn't add up to $100/mo. Meanwhile, there's services like Google Drive, etc which costs about $100/yr per TB. Still exorbitant, but not as much so. 3rd party software can mount it as a drive, but only for a short period of time before the token expires. So they seem happy to sell you space at less than 1/10th the cost, as long as it's harder to use. There just seems to be a lot of cash on the table for someone to offer much cheaper storage solutions, but no one is actually doing it.

cbellew 5y ago
Don't forget the replication. S3 replicates to at least three other AZs in the same region. I know this still doesn't account for the difference but its worth remembering.
- gmiller123456 5y ago
  I can see extra features costing more, but a lot of people would be happy with nothing but a network connected drive. If it goes bad, it's bad. Hopefully I have a backup somewhere.
Dylan16807 5y ago
The big clouds have premium storage and absolutely gouge on bandwidth. For a more basic offering, B2 is $60 a year and there are other services in the same ballpark. They definitely want it to be convenient for you, and make a modest profit percentage. I haven't experienced any issues with google drive tokens, though I've mostly been using the business tier.
BugWatch 5y ago
Well, there are also data integrity issues to take into account... in the simplest terms, you'd actually need 3x the available storage to "guarantee" that there won't be any bit nor data loss if a single drive (our of three) - does... Add a potential (mandatory?) off-site backup, means of restoring it, and... Not even getting into the bandwidth cost can of worms. Again, this is in very simple (and low-scale) terms.