6 ms·

Deprecating the Sync and Datastore APIs

60 points by vdfs 11y ago

gfodor 11y ago
"Sync is hard, use our API" A few years later "Nevermind, you'll need to build sync yourself. We believe in you!" Is this a fair assessment?
- vdfs 11y ago
  Also: "We want to replace you Hard drive" Two years later "Adoption less than expected, you have time to move to something else"
- smarx 11y ago
  Blog post author here. It definitely seems that I worded that part poorly. I hope the comments on the blog help clear this up, but it's a bit hard to talk about in the abstract. If you have an app that's using the Sync API, feel free to reach out to talk specifics.
samwillis 11y ago
Anyone have a recommendation for a Datastore replacement that allows you to share records/datasets between users in the same way datastore does?
- jchrisa 11y ago
  The access control API offered by Couchbase Mobile should be able to express the relationships you are looking for. Overview: http://developer.couchbase.com/mobile/get-started/what-is-sync-gateway/index.html http://developer.couchbase.com/mobile/get-started/what-is-sy...
paulhallett 11y ago
Deprecating an API is a bad way to go, especially if the API is fairly new. This post is telling me two things: 1) Dropbox's API was not designed correctly the first time. It should not just be a representation of their internal stuff exposed publicly. It needs to be designed first. 2) Dropbox is struggling to understand the difference between providing a consumer product (like their normal Dropbox product) and a developer product, like this API.
nubela 11y ago
Damn, thank god I knew better than to use another startup's API. Dropbox reached out to me a while ago when my product got on HN to use their Sync APIs.
- jnks 11y ago
  Ironically, if developers like yourself had adopted the API, they wouldn't be deprecating it now.
  - detaro 11y ago
    Because they cite not enough users as the main reason for deprecating it? Oh, wait, somehow they don't mention that at all...
  - BinaryIdiot 11y ago
    Maybe or maybe not. We don't know what the exact numbers were and what they needed to be to keep them in tact. They may have deprecated them anyway to keep a single SDK (at least that's kind of the impression I got though if the usage was high the timeline would probably be longer and more features would be in the core SDK). It's easy to lambast someone because someone didn't want to rely on another's API but it's a very good point to bring up. Big companies such as Google regularly kill things people use and start-ups are encouraged to think fast, test fast; I have a hard time wanting to rely on either for my business.
  - shawn-furyan 11y ago
    Developers that build products on top of APIs controlled by other companies, particularly recent startups, are routinely abused. The upside of these products is unnaturally clipped due to moral hazard on the part of the API provider. That is, closed APIs incentivize their providers to farm out testing new products based on their data set to others, then reimplement the most successful products themselves, and finally cut off access to the developer because it's now a 'competitor'. This pattern has (allegedly) played out with several of the big closed API providers. This behavior has a double benefit to the API provider, because it potentially turns would be direct competitors who would otherwise work on an alternative to its core technology if not provided with an API into unpaid new product idea validators that leave the API provider the option of crippling their product at will. That's a lot of competitive advantage, especially in markets dominated by network and first mover effects. As a developer, it's prudent to be skeptical of closed API providers, because developer time is valuable, and the incentives of API providers and consumers are not aligned. An API has to be extremely valuable in order to overcome these structural issues, and it appears that developers didn't regard these APIs as sufficiently valuable. So in my mind, the blame for this failure doesn't lie with the developers that failed to use the APIs, but with the company that didn't provide a sufficiently attractive API.
renke1 11y ago
I actually wanted to use the Datastore API for my app. A lot of stuff was happening on the client side so it would have been a perfect match. Now I will probably manage the user's data on the server using RethinkDB.
renke1 11y ago
I actually wanted to use the Datastore API for my app. A lot of stuff was happening on the client side so it would have been a perfect match. Now I will probably manage the user's data on the server using RethinkDB.
jchrisa 11y ago
If you want an open source full featured alternative (with native libraries for most mobile platforms), we've put decades of engineer time into Couchbase Mobile. http://developer.couchbase.com/mobile/ http://developer.couchbase.com/mobile/ Basic feature set: local storage and query, binary attachments, background sync, fine grained access control. Also goodies like webhooks and p2p capabilities.
- tobinharris 11y ago
  Does it still work against directly CouchDB? Had some fun times with the mobile/CouchDB sync. When Sync Gateway came along things got complicated (to me). Mostly because of another component to deploy, monitor etc. Did you consider bundling Sync Gateway into a Couchbase server, even if only for dev/staging environments?
  - jchrisa 11y ago
    Works great with Apache CouchDB, PouchDB, Cloudant, etc. The mobile libraries and Sync Gateway all use the CouchDB sync protocol. We built Sync Gateway because the Apache CouchDB access control model wasn't fine grained enough for most apps, so it adds a data routing function to control sync access. (See link downthread). Thanks for the feedback about simplicity. Our next release makes the Sync Gateway default config a lot more beginner friendly and even today we ship with a dev mode that uses a tiny embedded database instead of a Couchbase Server cluster for storage.
- aikah 11y ago
  The problem is the datastore API worked directly in the browser via javascript. You could deploy an app on a CDN with that,just reference a few api keys and start writing apps using Dropbox as a datastore.
  - jchrisa 11y ago
    For that scenario PouchDB would run in the browser and connect to Sync Gateway using CORS. Of course you'd still have to install Sync Gateway somewhere first, which is more work than outsourcing to a provider like Dropbox, but also more suitable for enterprise deployments.
junto 11y ago
Does anyone know what this means for 1Password? Don't they use the sync api?
- leftnode 11y ago
  It also has a Wifi sync which allows you to sync and keep your passwords off Dropbox.
- smarx 11y ago
  1Password doesn't use the Sync API.
gcr 11y ago
In terms of alternatives: I've worked with Firebase before, which is pretty reasonable after you wrap your head around the learning curve. https://www.firebase.com/ https://www.firebase.com/
- aikah 11y ago
  Does firebase work offline?
  - sararob 11y ago
    [Firebase Developer Advocate] Firebase does work offline if your app loses network connectivity temporarily (details here: https://www.firebase.com/docs/web/guide/offline-capabilities.html https://www.firebase.com/docs/web/guide/offline-capabilities...). All data written while offline is stored in memory and is synced to the Firebase servers when your client reconnects. We’re working on improving offline support when your app goes into the background, and we currently have a beta version of this available for our iOS (https://groups.google.com/d/msg/firebase-talk/dOocEtjQz4w/ssCTAno33AEJ https://groups.google.com/d/msg/firebase-talk/dOocEtjQz4w/ss...) and Android (https://groups.google.com/d/msg/firebase-talk/gYlnLgQ-Yhk/ItowUksVMzsJ https://groups.google.com/d/msg/firebase-talk/gYlnLgQ-Yhk/It...) SDKs.
- bsaul 11y ago
  Be EXTREMELY cautious when using document-based auto-syncing DBs such as firebase. If you're not closely following the best-practice such as using flat document structure instead of nested ones, and dictionaries instead of arrays, you may enter a world of nasty disappointments as you struggle to query and update your local data structure. Data syncing is hard and so far i've only seen two possibilities : roll your own solution using the fact that you have a knowledge of your business that will help you be smart and small, or use a generic solution but take some time to understand deeply its way of working before you build your real product. I actually had to cancel what i thought would be a short angular/firebase project after two days struggling to sync one big nested data structure displayed on a page, and realizing i actually had to rethink my model from scratch to fit the technology (which ended up being completely overkill for my need anyway).
fmkamchatka 11y ago
"The Datastore API is somewhat unique in that it, unlike the Core API, deals with non-file data, so we’re working directly with individual developers to help them migrate to an alternative." I would be interested in what this help means concretely for developers. I imagine Dropbox has limited interest helping small developers who might have used this API.
- smarx 11y ago
  (I'm the blog post's author.) Concretely, it means we emailed everyone by hand with a Datastore API app that had a non-trivial number of users and offered to help. (If anyone didn't get an email, you might only have a couple test users, but feel free to reach out to us yourself.) With most developers who respond, we're digging into their data model and trying to suggest alternatives (possibly files in Dropbox or possibly alternative services like Parse and Firebase).
seanalltogether 11y ago
The nice thing about dropbox's mobile sdks is they have very cleanly solved the problem of how to get data synced to a server despite some users having a very spotty connection. The bad thing about dropbox's mobile sdks is they force all data to be stored in the end users dropbox folder and not in our enterprise folder. I would gladly use their datastore apis if I could avoid having my users go through the hassle of authenticating to dropbox on their own and instead put it into my enterprise storage.
tobinharris 11y ago
Looks like they still need each mobile user to authenticate against DropBox via OAuth popup?
Animats 11y ago
The life of a cloud service seems to be less than five years.
jctrope 11y ago
It's great that they open sourced the implementation of datastore. I can't believe they got Guido van Rossum (creator of python) to write javascript though! https://github.com/dropbox/datastore-js/commit/e75e67eccecfdc76650157c0acb5a984289e284c https://github.com/dropbox/datastore-js/commit/e75e67eccecfd... Why not open sourced the mobile sdks for sync/datastore also?

60 points by vdfs 11y ago

"Sync is hard, use our API" A few years later "Nevermind, you'll need to build sync yourself. We believe in you!" Is this a fair assessment?

vdfs 11y ago
Also: "We want to replace you Hard drive" Two years later "Adoption less than expected, you have time to move to something else"
smarx 11y ago
Blog post author here. It definitely seems that I worded that part poorly. I hope the comments on the blog help clear this up, but it's a bit hard to talk about in the abstract. If you have an app that's using the Sync API, feel free to reach out to talk specifics.

samwillis 11y ago

Anyone have a recommendation for a Datastore replacement that allows you to share records/datasets between users in the same way datastore does?

jchrisa 11y ago
The access control API offered by Couchbase Mobile should be able to express the relationships you are looking for. Overview: http://developer.couchbase.com/mobile/get-started/what-is-sync-gateway/index.html http://developer.couchbase.com/mobile/get-started/what-is-sy...

paulhallett 11y ago

Deprecating an API is a bad way to go, especially if the API is fairly new. This post is telling me two things: 1) Dropbox's API was not designed correctly the first time. It should not just be a representation of their internal stuff exposed publicly. It needs to be designed first. 2) Dropbox is struggling to understand the difference between providing a consumer product (like their normal Dropbox product) and a developer product, like this API.

nubela 11y ago

Damn, thank god I knew better than to use another startup's API. Dropbox reached out to me a while ago when my product got on HN to use their Sync APIs.

jnks 11y ago
Ironically, if developers like yourself had adopted the API, they wouldn't be deprecating it now.
- detaro 11y ago
  Because they cite not enough users as the main reason for deprecating it? Oh, wait, somehow they don't mention that at all...
- BinaryIdiot 11y ago
  Maybe or maybe not. We don't know what the exact numbers were and what they needed to be to keep them in tact. They may have deprecated them anyway to keep a single SDK (at least that's kind of the impression I got though if the usage was high the timeline would probably be longer and more features would be in the core SDK). It's easy to lambast someone because someone didn't want to rely on another's API but it's a very good point to bring up. Big companies such as Google regularly kill things people use and start-ups are encouraged to think fast, test fast; I have a hard time wanting to rely on either for my business.
- shawn-furyan 11y ago
  Developers that build products on top of APIs controlled by other companies, particularly recent startups, are routinely abused. The upside of these products is unnaturally clipped due to moral hazard on the part of the API provider. That is, closed APIs incentivize their providers to farm out testing new products based on their data set to others, then reimplement the most successful products themselves, and finally cut off access to the developer because it's now a 'competitor'. This pattern has (allegedly) played out with several of the big closed API providers. This behavior has a double benefit to the API provider, because it potentially turns would be direct competitors who would otherwise work on an alternative to its core technology if not provided with an API into unpaid new product idea validators that leave the API provider the option of crippling their product at will. That's a lot of competitive advantage, especially in markets dominated by network and first mover effects. As a developer, it's prudent to be skeptical of closed API providers, because developer time is valuable, and the incentives of API providers and consumers are not aligned. An API has to be extremely valuable in order to overcome these structural issues, and it appears that developers didn't regard these APIs as sufficiently valuable. So in my mind, the blame for this failure doesn't lie with the developers that failed to use the APIs, but with the company that didn't provide a sufficiently attractive API.

renke1 11y ago

I actually wanted to use the Datastore API for my app. A lot of stuff was happening on the client side so it would have been a perfect match. Now I will probably manage the user's data on the server using RethinkDB.

renke1 11y ago

jchrisa 11y ago

If you want an open source full featured alternative (with native libraries for most mobile platforms), we've put decades of engineer time into Couchbase Mobile. http://developer.couchbase.com/mobile/ http://developer.couchbase.com/mobile/ Basic feature set: local storage and query, binary attachments, background sync, fine grained access control. Also goodies like webhooks and p2p capabilities.

tobinharris 11y ago
Does it still work against directly CouchDB? Had some fun times with the mobile/CouchDB sync. When Sync Gateway came along things got complicated (to me). Mostly because of another component to deploy, monitor etc. Did you consider bundling Sync Gateway into a Couchbase server, even if only for dev/staging environments?
- jchrisa 11y ago
  Works great with Apache CouchDB, PouchDB, Cloudant, etc. The mobile libraries and Sync Gateway all use the CouchDB sync protocol. We built Sync Gateway because the Apache CouchDB access control model wasn't fine grained enough for most apps, so it adds a data routing function to control sync access. (See link downthread). Thanks for the feedback about simplicity. Our next release makes the Sync Gateway default config a lot more beginner friendly and even today we ship with a dev mode that uses a tiny embedded database instead of a Couchbase Server cluster for storage.
aikah 11y ago
The problem is the datastore API worked directly in the browser via javascript. You could deploy an app on a CDN with that,just reference a few api keys and start writing apps using Dropbox as a datastore.
- jchrisa 11y ago
  For that scenario PouchDB would run in the browser and connect to Sync Gateway using CORS. Of course you'd still have to install Sync Gateway somewhere first, which is more work than outsourcing to a provider like Dropbox, but also more suitable for enterprise deployments.

junto 11y ago

Does anyone know what this means for 1Password? Don't they use the sync api?

leftnode 11y ago
It also has a Wifi sync which allows you to sync and keep your passwords off Dropbox.
smarx 11y ago
1Password doesn't use the Sync API.

gcr 11y ago

In terms of alternatives: I've worked with Firebase before, which is pretty reasonable after you wrap your head around the learning curve. https://www.firebase.com/ https://www.firebase.com/

aikah 11y ago
Does firebase work offline?
- sararob 11y ago
  [Firebase Developer Advocate] Firebase does work offline if your app loses network connectivity temporarily (details here: https://www.firebase.com/docs/web/guide/offline-capabilities.html https://www.firebase.com/docs/web/guide/offline-capabilities...). All data written while offline is stored in memory and is synced to the Firebase servers when your client reconnects. We’re working on improving offline support when your app goes into the background, and we currently have a beta version of this available for our iOS (https://groups.google.com/d/msg/firebase-talk/dOocEtjQz4w/ssCTAno33AEJ https://groups.google.com/d/msg/firebase-talk/dOocEtjQz4w/ss...) and Android (https://groups.google.com/d/msg/firebase-talk/gYlnLgQ-Yhk/ItowUksVMzsJ https://groups.google.com/d/msg/firebase-talk/gYlnLgQ-Yhk/It...) SDKs.
bsaul 11y ago
Be EXTREMELY cautious when using document-based auto-syncing DBs such as firebase. If you're not closely following the best-practice such as using flat document structure instead of nested ones, and dictionaries instead of arrays, you may enter a world of nasty disappointments as you struggle to query and update your local data structure. Data syncing is hard and so far i've only seen two possibilities : roll your own solution using the fact that you have a knowledge of your business that will help you be smart and small, or use a generic solution but take some time to understand deeply its way of working before you build your real product. I actually had to cancel what i thought would be a short angular/firebase project after two days struggling to sync one big nested data structure displayed on a page, and realizing i actually had to rethink my model from scratch to fit the technology (which ended up being completely overkill for my need anyway).

fmkamchatka 11y ago

"The Datastore API is somewhat unique in that it, unlike the Core API, deals with non-file data, so we’re working directly with individual developers to help them migrate to an alternative." I would be interested in what this help means concretely for developers. I imagine Dropbox has limited interest helping small developers who might have used this API.

smarx 11y ago
(I'm the blog post's author.) Concretely, it means we emailed everyone by hand with a Datastore API app that had a non-trivial number of users and offered to help. (If anyone didn't get an email, you might only have a couple test users, but feel free to reach out to us yourself.) With most developers who respond, we're digging into their data model and trying to suggest alternatives (possibly files in Dropbox or possibly alternative services like Parse and Firebase).

seanalltogether 11y ago

The nice thing about dropbox's mobile sdks is they have very cleanly solved the problem of how to get data synced to a server despite some users having a very spotty connection. The bad thing about dropbox's mobile sdks is they force all data to be stored in the end users dropbox folder and not in our enterprise folder. I would gladly use their datastore apis if I could avoid having my users go through the hassle of authenticating to dropbox on their own and instead put it into my enterprise storage.

tobinharris 11y ago

Looks like they still need each mobile user to authenticate against DropBox via OAuth popup?

Animats 11y ago

The life of a cloud service seems to be less than five years.

jctrope 11y ago

It's great that they open sourced the implementation of datastore. I can't believe they got Guido van Rossum (creator of python) to write javascript though! https://github.com/dropbox/datastore-js/commit/e75e67eccecfdc76650157c0acb5a984289e284c https://github.com/dropbox/datastore-js/commit/e75e67eccecfd... Why not open sourced the mobile sdks for sync/datastore also?