charlie-haley

5 ms·

1.
by charlie-haley 3mo ago
Hey, good question. Marmot is designed to be as generic as possible. An "Asset", whether it's a database, glossary term, topic, API or anything else, has the exact same schema, API endpoint and MCP tool. The MCP server expose
2.
by charlie-haley 6mo ago
I wrote this after spending weeks figuring out how far Postgres could go before reaching for dedicated search indexers or a graph database. Pretty far, as it turns out. pg_trgm can be used for fuzzy matching, there's built-in full-tex
3.
Postgres: One Database to Rule Them All(marmotdata.io)
4 points by charlie-haley 6mo ago|1 comments
4.
by charlie-haley 10mo ago
Hey, that's a good question! At the moment, it treats the latest run as the desired state. So any new changes to a schema will simply overwrite the old version. I'd like to version these so people can navigate schema versions in t
5.
by charlie-haley 10mo ago
Postgres has a lot of features such as trigram-based search which is pretty essential if I don't want to use a dedicated search indexer. It's also much better at handling concurrent writes than SQLite.
6.
by charlie-haley 10mo ago
It can handle discovery within a plugin if the asset types are related. You can also manually add lineage via the UI or use Terraform to create lineage links via IaC. It's pretty complicated to automatically handle discovery of asset l
7.
by charlie-haley 10mo ago
It supports either, I didn't want to restrict people to just one method of getting their catalog populated. The CLI and Plugin system works on needing read credentials to a given Service, it then populates the catalog with those assets
8.
by charlie-haley 10mo ago
Hey, there's some documentation around creating plugins here. It's relatively simple and involves adding a new Go package to the repo. Currently they have to be compiled into the Binary but I'd like to support external plugin
9.
by charlie-haley 10mo ago
That's great to know, I wasn't aware anybody even attempted to used it yet! I'm currently in the process of overhauling the Plugin system, it's been quite hard to test some enterprise closed-source integrations like Tabl
10.
by charlie-haley 10mo ago
It depends on your ecosystem. If everything lives under one vendor their native catalog will probably work really well for you. But most of the time (especially for older orgs) there's usually a huge fragmented ecosystem of data assets
11.
by charlie-haley 10mo ago
Hey HN, I wanted to show off my project Marmot! I decided to build Marmot after discovering a lot of data catalogs can be complex and require many external dependencies such as Kafka, Elasticsearch or an external orchestrator like Airflow.
12.
Show HN: Marmot – Single-binary data catalog (no Kafka, no Elasticsearch)(github.com)
103 points by charlie-haley 10mo ago|21 comments
13.
Show HN: Marmot – Simple data catalog with powerful search and lineage(github.com)
2 points by charlie-haley 1y ago|0 comments