francoismassot

5 ms·

1.
by francoismassot 1y ago
it's tantivy :)
2.
Tenstorrent Cloud Instances: Unveiling Next-Gen AI Accelerators(koyeb.com)
3 points by francoismassot 2y ago|0 comments
3.
by francoismassot 2y ago
Co-founder of Quickwit here. Seeing our acquisition by Datadog on the HN front page feels like a truly full-circle moment. HN has been interwoven with Quickwit's journey from the very beginning. Looking back, it's striking to see
4.
by francoismassot 2y ago
Latest HN thread on quickwit (Binance built a 100PB log service with Quickwit): https://news.ycombinator.com/item?id=40935701 I also wrote a benchmark on Loki vs. Quickwit: https://quickwit.io/blog/benc
5.
by francoismassot 2y ago
Indeed. They benefit from a discount, but we don't know the discount figure. To further reduce the storage costs, you can use S3 Storage Classes or cheaper object storage like Alibaba for longer retention. Quickwit does not handle that
6.
by francoismassot 2y ago
They have 181 trillion logs
7.
by francoismassot 2y ago
Good question. Let's estimate the costs of compute. For indexing, they need 2800 vCPUs[1], and they are using c6g instances; on-demand hourly price is $0.034/h per vCPU. So indexing will cost them around $70k/month. For searc
8.
by francoismassot 2y ago
But you don’t have fast search on those files stored on object storage.
9.
by francoismassot 2y ago
If you don't need vector search and have very large Elasticsearch deployment, you can have a look at Quickwit, it's a search engine on object storage, it's OSS and works for append-only datasets (like logs, traces, ...) Repo:
10.
by francoismassot 2y ago
One workaround is to use the JSON field, see doc https://github.com/quickwit-oss/tantivy/blob/main/doc/src/js...
11.
by francoismassot 2y ago
Well, MongoDB was under AGPL v3.0 :)
12.
by francoismassot 3y ago
Quickwit is an alternative with a strong focus on scalability (max we have seen is 40PB) with a decoupled compute and storage architecture. But we do only logs and traces for now. Repository: https://github.com/quickwit-oss&
13.
by francoismassot 3y ago
This is awesome; we need this kind of alternative to overpriced software like Splunk. We built and open-sourced Quickwit to see this kind of tool built on top of it. We will follow Tracecat closely. I'm convinced this will impact our r
14.
by francoismassot 3y ago
tantivy, not tantivity!!!!!
15.
by francoismassot 3y ago
Thanks! Quickwit is the distributed engine built on top of tantivy, we basically separated compute and storage for search, I wrote this blog post to introduce the architecture: https://quickwit.io/blog/quickwit-101 PS
16.
by francoismassot 3y ago
Some companies are using it with AWS Lambda to scale to 0.
17.
by francoismassot 3y ago
Building the inverted index is quite CPU-intensive, and we are also merging index files called "splits".
18.
Building a log search service for under $7/mo(quickwit.io)
1 points by francoismassot 3y ago|0 comments
19.
by francoismassot 3y ago
BigQuery is just too costly... Do you know if the dataset is public? We should just offer a cheap alternative and ditch BigQuery.
20.
by francoismassot 3y ago
Oh I forget to add stract is using tantivy too, I really hope this project will take off. https://stract.com/ https://github.com/StractOrg/stract https://news.ycombinator.com/item?id=39
21.
by francoismassot 3y ago
> "Do store the Sonic database on SSD-backed file systems only." From the README, it works only on SSD. All those projects serve different purposes, and several are not actively maintained. - Meilisearch: It provides a search-a
22.
by francoismassot 3y ago
The geocoder is built on top of tantivy which is fast and uses low resources too ( https://github.com/quickwit-oss/tantivy ). I'm curious about the comparison between those two.
23.
Tigris: Globally Distributed Object Storage on Top of Fly.io(tigrisdata.com)
3 points by francoismassot 3y ago|0 comments
24.
by francoismassot 3y ago
I used it with my 10 years old boy for spelling. I like the method. I found the app is still rough on the edges, and now I want to code a small one dedicated to science fields for him :)
25.
An Empirical Evaluation of Columnar Storage Formats [pdf](arxiv.org)
1 points by francoismassot 3y ago|0 comments
26.
by francoismassot 3y ago
Congrats to Qdrant's team, $28M for a Series is really nice. There are a lot of OSS vector search databases out there, we could probably list the main ones: - Qdrant: https://github.com/qdrant/qdrant - Weaviate:
27.
Qdrant, the Vector Search Database, raised $28M in a Series A round(qdrant.tech)
131 points by francoismassot 3y ago|167 comments
28.
by francoismassot 3y ago
Does someone knows how Ceph compares to other object storage engine like MinIO/Garage/...? I would love to see some benchmarks there.
29.
Demystifying the use of Parquet for time series(blog.senx.io)
49 points by francoismassot 3y ago|6 comments
30.
by francoismassot 3y ago
Quickwit is under AGPLv3. Are you saying that AGPLv3 is not FOSS?

More ›