6 ms·

CouchDB/PouchDB looks very promising for offline first apps, but I can’t understand how to restrict bad clients. Client potentially could insert document of hug

by yatsyk 7y ago

CouchDB/PouchDB looks very promising for offline first apps, but I can’t understand how to restrict bad clients. Client potentially could insert document of huge size or execute expensive query and degrade experience of other clients on the same server. Is it any way to prevent this?

newfeatureok 7y ago
You resolve that issue the same way you would resolve the same issue if you were using Postgres - you introduce some back-end. For your example specifically I'd use a proxy.
- yatsyk 7y ago
  Custom backend means no synchronisation and no advantages over postgres. Do you propose to create proxy that parses query and estimates complexity? I think this task at least as hard as implementing couchdb myself (actually harder) Is there any secure open source code with pouchdb/couchdb integrations?
  - daleharvey 7y ago
    Your backend can be a reverse proxy that authenticates requests then passes them off to CouchDB (or PouchDB, since that also runs on the server). I have an example up @ https://github.com/daleharvey/noted https://github.com/daleharvey/noted. The server is 200 lines and does signup / email authentication etc.
    - yatsyk 7y ago
      This server can't prevent authenticated user from uploading huge document of running expensive query.
      - daleharvey 7y ago
        I wasnt worried about that since it is a basic proof of concept, adding that would make it ~210 lines of code.
      - Volundr 7y ago
        Any reverse proxy can limit the the size of a document upload. Even just plain NGINX can do that. Just set the client max body size. As for queries, it kind of depends on your model. Mango queries are pretty limited (no joins, no arbitrary filters), so it's not necessarily as easy as you think to write one that hosed performance. A client could of course write one that doesn't use an index, which may or may not be a concern. An easy option if it is though is just don't expose the `_find` endpoint, which effectively limits your users to the map/reduce queries you've written (unless you give them admin they don't have the ability to create their own).
        daleharvey 7y ago
        A popular model is for the clients to run the queries locally, the server doesnt need to expose any query endpoints, only the ones necessary for replication.
        yatsyk 7y ago
        Is it any documents that describes secure couchdb architecture? Most of the articles I find are limited to authentication and basic permissions.
        Volundr 7y ago
        What kind of document are you looking for here? There is [1], but yeah, that covers access controls. As do the MongoDB [2] and Postgres [3] documents. I feel like your thinking about Couch as exposing your entire PostgreSQL DB to the internet, whereas with couch, a common model is to have a single database per user. In the Postgres model, providing the end user with any direct access is a nightmare, because every other users data is in there and I have to keep other users from viewing/modifying it. In Couch, you give them access to their database and only their database, that's how you isolate users. [1]https://docs.couchdb.org/en/stable/intro/security.html https://docs.couchdb.org/en/stable/intro/security.html [2]https://docs.mongodb.com/manual/security/ https://docs.mongodb.com/manual/security/ [3]https://www.postgresql.org/docs/7.0/security.htm https://www.postgresql.org/docs/7.0/security.htm [3]
  - newfeatureok 7y ago
    There are plenty of proxies that do that with some config like nginx. Even if you were using a relational database with a backend you’d still have to solve the same problem.
    - yatsyk 7y ago
      If I use backend I can create all validation logic in application server. But in this case no automatic synchronisation. One of the major selling point of couchdb is replication protocol for client-server data syncing. When you design product with posgress you don't allow to execute raw sql queries from clients without any application server. But looks like it is recommended way to update data in couchdb world if you want to have synchronisation. I can't understand how can this architecture be secure?
      - lhuser123 7y ago
        Couchdb has options for controlling which documents are replicated. This may help depending on your use case.
    - SahAssar 7y ago
      nginx couldn't solve the "execute expensive query" though right, only limit max size. I guess you could do a request timeout + blacklist, but that would also be hard to do right, since at heavy load some proper clients might get blacklisted.
Volundr 7y ago
A couple ways: One you implement validation functions [1] on user databases to control what kind of data can be inserted into couch. These functions can only be changed by database admins, not users, so can act as a security mechanism controlling what goes in. As mentioned by others you can also implement a proxy. This doesn't have to interfere with sync functionality, you just have to make sure you proxy all the endpoints in the replication protocol [2]. Envoy [3] is one such proxy that essentially applies document level permissions to a CouchDB database without interfering with sync. If the goal is just to limit document size, or throttle clients trying to hammer the API, this doesn't even have to be a custom proxy, and reverse proxy with the needed control knobs (such as NGINX) will do. You can of course combine this with validation functions, using validations to ensure the everything that comes in is the right "shape" and using NGINX and it's ilk to apply throttling and sane request limits. At scale there's a decent chance you want a proxy in front of your Couch instance anyway, since Couch is truly multi-master, meaning you probably want to balance your clients across all your nodes anyway. [1] https://docs.couchdb.org/en/stable/ddocs/ddocs.html#validate-document-update-functions https://docs.couchdb.org/en/stable/ddocs/ddocs.html#validate... [2] https://docs.couchdb.org/en/stable/replication/protocol.html# https://docs.couchdb.org/en/stable/replication/protocol.html... [3] https://github.com/cloudant-labs/envoy https://github.com/cloudant-labs/envoy
- yatsyk 7y ago
  Thank you for pointing at validation, I'll check it. It's not completely clear what is it possible to limit not only particular document but database, or how to handle conflict if document changed on pouch, but rejected on couchdb server. I'm not sure about current time but previously it was a problem that couchdb file grow until some limit on filesystem and couchdb just crashed. Start of the envoy readme: it's not battle tested or supported in any way. Also it doesn't do any validation apart from limiting permissions for different users. It's easier to reimplement couchdb than to create smart proxy that will estimate is this query expensive or not. I'm not saying about rate-limiting proxy or load-balancing to different backends which could be implemented on nginx or something else.
  - Volundr 7y ago

by yatsyk 7y ago

newfeatureok 7y ago

You resolve that issue the same way you would resolve the same issue if you were using Postgres - you introduce some back-end. For your example specifically I'd use a proxy.

yatsyk 7y ago
Custom backend means no synchronisation and no advantages over postgres. Do you propose to create proxy that parses query and estimates complexity? I think this task at least as hard as implementing couchdb myself (actually harder) Is there any secure open source code with pouchdb/couchdb integrations?
- daleharvey 7y ago
  Your backend can be a reverse proxy that authenticates requests then passes them off to CouchDB (or PouchDB, since that also runs on the server). I have an example up @ https://github.com/daleharvey/noted https://github.com/daleharvey/noted. The server is 200 lines and does signup / email authentication etc.
  - yatsyk 7y ago
    This server can't prevent authenticated user from uploading huge document of running expensive query.
    - daleharvey 7y ago
      I wasnt worried about that since it is a basic proof of concept, adding that would make it ~210 lines of code.
    - Volundr 7y ago
      Any reverse proxy can limit the the size of a document upload. Even just plain NGINX can do that. Just set the client max body size. As for queries, it kind of depends on your model. Mango queries are pretty limited (no joins, no arbitrary filters), so it's not necessarily as easy as you think to write one that hosed performance. A client could of course write one that doesn't use an index, which may or may not be a concern. An easy option if it is though is just don't expose the `_find` endpoint, which effectively limits your users to the map/reduce queries you've written (unless you give them admin they don't have the ability to create their own).
      - daleharvey 7y ago
        A popular model is for the clients to run the queries locally, the server doesnt need to expose any query endpoints, only the ones necessary for replication.
        yatsyk 7y ago
        Is it any documents that describes secure couchdb architecture? Most of the articles I find are limited to authentication and basic permissions.
        Volundr 7y ago
        What kind of document are you looking for here? There is [1], but yeah, that covers access controls. As do the MongoDB [2] and Postgres [3] documents. I feel like your thinking about Couch as exposing your entire PostgreSQL DB to the internet, whereas with couch, a common model is to have a single database per user. In the Postgres model, providing the end user with any direct access is a nightmare, because every other users data is in there and I have to keep other users from viewing/modifying it. In Couch, you give them access to their database and only their database, that's how you isolate users. [1]https://docs.couchdb.org/en/stable/intro/security.html https://docs.couchdb.org/en/stable/intro/security.html [2]https://docs.mongodb.com/manual/security/ https://docs.mongodb.com/manual/security/ [3]https://www.postgresql.org/docs/7.0/security.htm https://www.postgresql.org/docs/7.0/security.htm [3]
- newfeatureok 7y ago
  There are plenty of proxies that do that with some config like nginx. Even if you were using a relational database with a backend you’d still have to solve the same problem.
  - yatsyk 7y ago
    If I use backend I can create all validation logic in application server. But in this case no automatic synchronisation. One of the major selling point of couchdb is replication protocol for client-server data syncing. When you design product with posgress you don't allow to execute raw sql queries from clients without any application server. But looks like it is recommended way to update data in couchdb world if you want to have synchronisation. I can't understand how can this architecture be secure?
    - lhuser123 7y ago
      Couchdb has options for controlling which documents are replicated. This may help depending on your use case.
  - SahAssar 7y ago
    nginx couldn't solve the "execute expensive query" though right, only limit max size. I guess you could do a request timeout + blacklist, but that would also be hard to do right, since at heavy load some proper clients might get blacklisted.

Volundr 7y ago

A couple ways: One you implement validation functions [1] on user databases to control what kind of data can be inserted into couch. These functions can only be changed by database admins, not users, so can act as a security mechanism controlling what goes in. As mentioned by others you can also implement a proxy. This doesn't have to interfere with sync functionality, you just have to make sure you proxy all the endpoints in the replication protocol [2]. Envoy [3] is one such proxy that essentially applies document level permissions to a CouchDB database without interfering with sync. If the goal is just to limit document size, or throttle clients trying to hammer the API, this doesn't even have to be a custom proxy, and reverse proxy with the needed control knobs (such as NGINX) will do. You can of course combine this with validation functions, using validations to ensure the everything that comes in is the right "shape" and using NGINX and it's ilk to apply throttling and sane request limits. At scale there's a decent chance you want a proxy in front of your Couch instance anyway, since Couch is truly multi-master, meaning you probably want to balance your clients across all your nodes anyway. [1] https://docs.couchdb.org/en/stable/ddocs/ddocs.html#validate-document-update-functions https://docs.couchdb.org/en/stable/ddocs/ddocs.html#validate... [2] https://docs.couchdb.org/en/stable/replication/protocol.html# https://docs.couchdb.org/en/stable/replication/protocol.html... [3] https://github.com/cloudant-labs/envoy https://github.com/cloudant-labs/envoy

yatsyk 7y ago
Thank you for pointing at validation, I'll check it. It's not completely clear what is it possible to limit not only particular document but database, or how to handle conflict if document changed on pouch, but rejected on couchdb server. I'm not sure about current time but previously it was a problem that couchdb file grow until some limit on filesystem and couchdb just crashed. Start of the envoy readme: it's not battle tested or supported in any way. Also it doesn't do any validation apart from limiting permissions for different users. It's easier to reimplement couchdb than to create smart proxy that will estimate is this query expensive or not. I'm not saying about rate-limiting proxy or load-balancing to different backends which could be implemented on nginx or something else.
- Volundr 7y ago