5 ms·

Show HN: Piazza, syncing the web in a vectorized DB

28 points by lcombaldieu 2y ago

We recently released an open source project that syncs wikipedia with a vector database : https://github.com/Piazza-tech/Piazza-Updater https://github.com/Piazza-tech/Piazza-Updater We used Verba, Weaviate and Docker for deployment We'd like to have some feedback on how to continue the project, which data sources would be interesting to vectorize. You can give feedback on our landing page http://piazza.tech http://piazza.tech Please leave a star !

CryptoBanker 2y ago
Demo doesn’t appear to be working
Cheer2171 2y ago
Might want to find a new name. Piazza (https://piazza.com https://piazza.com) is a very popular classroom Q&A and wiki platform.
igorguerrero 2y ago
I'm not sure if I'd call RAG an "advanced technique"...
costco 2y ago
> You've requested a page on a website (chat.piazza.tech) that is on the Cloudflare network. The host (chat.piazza.tech) is configured as an Argo Tunnel, and Cloudflare is currently unable to resolve it.
- marco_z 2y ago
  same for me now.
tejaskumar_ 2y ago
Interesting project! Reminds me of Astra DB who just worked with Wikimedia to vectorize Wikipedia: https://blog.wikimedia.de/2024/09/17/wikidata-and-artificial-intelligence-simplified-access-to-open-data-for-open-source-projects/ https://blog.wikimedia.de/2024/09/17/wikidata-and-artificial... Nice work on the open source approach! Looking forward to seeing how this evolves.
bosky101 2y ago
I see many 3rd party deps, integrations for verba(first stumbling on this rag chat+frontend) Can you give the stack that is suited well for completely self hosting piazza? Will search for those env and docs accordingly
- lcombaldieu 2y ago
  OP here, are you asking about hardware or cloud instance type ? I would recommend using GPUs at least to initialise the DB. You need a lot of compute once at the start to vectorize all the data, and then regularly (but less intensely) to sync the updated changes. You can check out the Weaviate docs here if you want more info : https://weaviate.io/developers/weaviate https://weaviate.io/developers/weaviate

28 points by lcombaldieu 2y ago

CryptoBanker 2y ago

Demo doesn’t appear to be working

Cheer2171 2y ago

Might want to find a new name. Piazza (https://piazza.com https://piazza.com) is a very popular classroom Q&A and wiki platform.

igorguerrero 2y ago

I'm not sure if I'd call RAG an "advanced technique"...

costco 2y ago

> You've requested a page on a website (chat.piazza.tech) that is on the Cloudflare network. The host (chat.piazza.tech) is configured as an Argo Tunnel, and Cloudflare is currently unable to resolve it.

marco_z 2y ago
same for me now.

tejaskumar_ 2y ago

Interesting project! Reminds me of Astra DB who just worked with Wikimedia to vectorize Wikipedia: https://blog.wikimedia.de/2024/09/17/wikidata-and-artificial-intelligence-simplified-access-to-open-data-for-open-source-projects/ https://blog.wikimedia.de/2024/09/17/wikidata-and-artificial... Nice work on the open source approach! Looking forward to seeing how this evolves.

bosky101 2y ago

I see many 3rd party deps, integrations for verba(first stumbling on this rag chat+frontend) Can you give the stack that is suited well for completely self hosting piazza? Will search for those env and docs accordingly

lcombaldieu 2y ago
OP here, are you asking about hardware or cloud instance type ? I would recommend using GPUs at least to initialise the DB. You need a lot of compute once at the start to vectorize all the data, and then regularly (but less intensely) to sync the updated changes. You can check out the Weaviate docs here if you want more info : https://weaviate.io/developers/weaviate https://weaviate.io/developers/weaviate