3 ms·
Knowledge Compressor
- adsharma 1mo agoDid you consider extracting knowledge from the doc into a graph database (while passing the Q&A) and compressing the graph instead?
- adsharma 1mo agohttps://huggingface.co/datasets/ladybugdb/github-knowledge-compressor/tree/main https://huggingface.co/datasets/ladybugdb/github-knowledge-c... These methods could not beat the 480 token result because of the small dataset. But for a sufficiently large corpus with graph reordering and columnar parquet compression, they could be quite competitive. The parquet files are self explanatory. For knowledge.lbdb.zst, first decompress with zstd and then run a cypher such as: lbug> MATCH (a:E)-[t:TRIPLE]->(b:E) RETURN a.name AS subj, t.predicate AS pred, b.name AS obj, t.props AS props, t.scope AS scope;
- Mappleton 1mo agoHow far can documentation be compressed without losing its meaning?
- neoncontrails 1mo agoThis is a good question! The answer is, unfortunately, uncomputable – or so we think. But if you manage to find a more satisfying answer, you'll have solved one of the more beguiling problems of the last century, which is the basis for most of modern Information Theory. You can learn more about it here: https://homepages.cwi.nl/~paulv/kolmogorov.html https://homepages.cwi.nl/~paulv/kolmogorov.html
- idan 1mo agoHello! Idan (from Next) here, and others from the team will drop by the thread if you have questions!