4 ms·

Also see MimicDB [0]. Runs a transparent cache that responds to most S3 API calls locally using Redis to store metadata. Besides the cost savings, it's extremel

Also see MimicDB [0]. Runs a transparent cache that responds to most S3 API calls locally using Redis to store metadata. Besides the cost savings, it's extremely fast. Listing an entire bucket with millions of objects is close to instantaneous. [0] http://mimicdb.com/ http://mimicdb.com/

zimbatm 13y ago
It also adds another piece of infrastructure that needs to be maintained and can go down. Not necessarily the best option for everyone.
- pipeep 13y ago
  While the caching would be more effective with one large centralized instance, I think the intended use case is to have one cache per server. So then it's not really extra infrastructure.
  - nathancahill 13y ago
    Actually, you can do it either way since it's backed by Redis. You can set all servers to connect to the same Redis instance, or run them all individually.
    - fours 13y ago
      Do you use Redis clustering in this case (otherwise how would the cache stay consistent)?
    - cmircea 13y ago
      What if Redis goes down?
      - nathancahill 13y ago
        "What if X goes down?" is my new favorite straw man on HN. Regardless, it's just a caching layer, and requests are passed through to the API in that case.
      - rhizome 13y ago
        There's always a single point of failure.
chaddeshon 13y ago
Is their a SaaS version of mimicdb? I'm ready to signup right now.
- nathancahill 13y ago
  No, and it wouldn't help much if there was. The biggest benefits come from running the metadata cache locally, or at least on the same local network. The cost/time savings come from preventing external API calls.
  - chaddeshon 13y ago
    If I have a bucket with 2 million key, it takes a long time and 2000 requests to get the dir listing. With mimicdb it would only be one fast request.
    - nathancahill 13y ago
      That's true if it was local. A 2 million key response from an externally hosted MimicDB instance would probably have to be split in 2000 requests and take the same amount of time. Send me an email nathan@nathancahill.com and I can show you how to set up a local cache. It's quite simple.

zimbatm 13y ago

It also adds another piece of infrastructure that needs to be maintained and can go down. Not necessarily the best option for everyone.

pipeep 13y ago
While the caching would be more effective with one large centralized instance, I think the intended use case is to have one cache per server. So then it's not really extra infrastructure.
- nathancahill 13y ago
  Actually, you can do it either way since it's backed by Redis. You can set all servers to connect to the same Redis instance, or run them all individually.
  - fours 13y ago
    Do you use Redis clustering in this case (otherwise how would the cache stay consistent)?
  - cmircea 13y ago
    What if Redis goes down?
    - nathancahill 13y ago
      "What if X goes down?" is my new favorite straw man on HN. Regardless, it's just a caching layer, and requests are passed through to the API in that case.
    - rhizome 13y ago
      There's always a single point of failure.

chaddeshon 13y ago

Is their a SaaS version of mimicdb? I'm ready to signup right now.

nathancahill 13y ago
No, and it wouldn't help much if there was. The biggest benefits come from running the metadata cache locally, or at least on the same local network. The cost/time savings come from preventing external API calls.
- chaddeshon 13y ago
  If I have a bucket with 2 million key, it takes a long time and 2000 requests to get the dir listing. With mimicdb it would only be one fast request.
  - nathancahill 13y ago
    That's true if it was local. A 2 million key response from an externally hosted MimicDB instance would probably have to be split in 2000 requests and take the same amount of time. Send me an email nathan@nathancahill.com and I can show you how to set up a local cache. It's quite simple.