5 ms·

> It should be really, really simple and for small-to-medium scale usage (in terms of number of data points stored). To me those tiers are more like tiny-to-sm

by eis 8y ago

> It should be really, really simple and for small-to-medium scale usage (in terms of number of data points stored). To me those tiers are more like tiny-to-small scale :) The problem with these small numbers is that it's trivial to put them into really any DB that you can think of. Even SQLite will handle this amount of data with super ease. I can only see people who know nothing about databases using this service. Or someone who for some reason can't use a DB. 1000 datapoints is at tops maybe 100 kilobytes of data if one uses a very unoptimized format and can easily be less than a kilobyte if done properly. At a million datapoints (your highest tier) we're talking about 1 Megabyte to maybe 100 Megabyte (again, if done very poorly). It's really a trivial amount of data. A standard Postgres or MySQL DB will be able to run queries over this dataset in less than a second and that's using something that's totally not optimized for time series. > It's an interesting idea to charge business users for usage instead of forcing them into a tier. The only issue is that you'll then have to provide them with detailed information on their usage, meaning a bit of work making sure things are logged properly But you already need to log their usage to know if they fall into whatever tiers limits. And you can store this info in... a time series DB :)

rlonn 8y ago
The marketer in me doesn't like the phrase "tiny-to-small scale" It's the same as when you order a latte at Starbucks and they only have various versions of "big" to sell you ;) I think you're missing the point about self-hosted db's though - people will not choose to put their data on pushdata.io because their own database cannot handle the amount of data they have, but because it is more convenient. Many people, even those who do know a little about databases, do not want to maintain another server [instance] and all that comes with doing that (like backups, for instance). They don't want to set it up, they don't want to maintain or be responsible for it. This choice has nothing to do with performance or size of the data set, but with convenience and peace of mind. Hey, it's a really cool idea to log usage data in the time series db! That is real dogfooding if I ever saw it. I will definitely try to do that :)
- eis 8y ago
  > I think you're missing the point about self-hosted db's though - people will not choose to put their data on pushdata.io because their own database cannot handle the amount of data they have, but because it is more convenient. Many people, even those who do know a little about databases, do not want to maintain another server [instance] and all that comes with doing that (like backups, for instance). They don't want to set it up, they don't want to maintain or be responsible for it. This choice has nothing to do with performance or size of the data set, but with convenience and peace of mind. I am not so sure. Most projects already need a DB and it's really trivial to store this small amount of data in there as well. I am not sure if it is easier to maintain another account with a third party, write a HTTP POST instead of SQL INSERT and HTTP GET + parsing + processing code instead of SQL SELECT with processing right in the query. One also already needs to do backups of their main data and those backups would just include this timeseries data as well. They most likely don't need another instance, another DB etc. for this amount of data, they just stick it into their existing one. My point being that yes I agree that people reach for third party services out of convenience and not scalability needs BUT this is such a tiny piece of a bigger puzzle that the overhead it comes with might not be worth it. Removing the need to host the timeseries storage part of a product themselves does not remove the need for a DB for 99.99% of projects. If it is trivial to do something myself, then I will do it myself. But if doing it myself takes considerable effort or requires ongoing maintenance overhead, then I might consider a third party service. The higher the value of your service, the better one can make an argument for oneself to use it. E.g. letting people store much more data, letting them process it etc. Points pro using a third party service: 1. Too much time to develop 2. Too much maintenance overhead 3. Too difficult (to scale, develop, ...) 4. Economically unfeasable (because no economy of scale benefit, no synergies etc) Points contra using a third party service: 1. Giving up control over data 2. Legal problems ("where is the data stored?", "am I allowed to hand this data out?") 3. Overhead of interacting with a third party (account maintenance, payments, integration) 4. Inflexibility ("what if I need to change my existing data?", "what if I need some new feature?") As I see it, the points on "pro" don't apply at this scale. See the tiny scale and featureset is not really about data scale but overall "difficulty to do and keep running". If I offer a logging service with a REST API to store and retrieve up to 1000 lines of text without any way to filter/search/aggregate/process, would you use that because it's more convenient or would you just log things to a local file or syslog and call it a day? Like you said it's v1.0 and I'm sure you'll add more stuff, I'm just suggesting that you'll definitely have to add quite a bit of stuff to catch business users. I hope you can make it, I respect everyone who builds a product themselves and tries to make a business out of it. Best of luck!

by eis 8y ago

rlonn 8y ago

The marketer in me doesn't like the phrase "tiny-to-small scale" It's the same as when you order a latte at Starbucks and they only have various versions of "big" to sell you ;) I think you're missing the point about self-hosted db's though - people will not choose to put their data on pushdata.io because their own database cannot handle the amount of data they have, but because it is more convenient. Many people, even those who do know a little about databases, do not want to maintain another server [instance] and all that comes with doing that (like backups, for instance). They don't want to set it up, they don't want to maintain or be responsible for it. This choice has nothing to do with performance or size of the data set, but with convenience and peace of mind. Hey, it's a really cool idea to log usage data in the time series db! That is real dogfooding if I ever saw it. I will definitely try to do that :)

eis 8y ago
> I think you're missing the point about self-hosted db's though - people will not choose to put their data on pushdata.io because their own database cannot handle the amount of data they have, but because it is more convenient. Many people, even those who do know a little about databases, do not want to maintain another server [instance] and all that comes with doing that (like backups, for instance). They don't want to set it up, they don't want to maintain or be responsible for it. This choice has nothing to do with performance or size of the data set, but with convenience and peace of mind. I am not so sure. Most projects already need a DB and it's really trivial to store this small amount of data in there as well. I am not sure if it is easier to maintain another account with a third party, write a HTTP POST instead of SQL INSERT and HTTP GET + parsing + processing code instead of SQL SELECT with processing right in the query. One also already needs to do backups of their main data and those backups would just include this timeseries data as well. They most likely don't need another instance, another DB etc. for this amount of data, they just stick it into their existing one. My point being that yes I agree that people reach for third party services out of convenience and not scalability needs BUT this is such a tiny piece of a bigger puzzle that the overhead it comes with might not be worth it. Removing the need to host the timeseries storage part of a product themselves does not remove the need for a DB for 99.99% of projects. If it is trivial to do something myself, then I will do it myself. But if doing it myself takes considerable effort or requires ongoing maintenance overhead, then I might consider a third party service. The higher the value of your service, the better one can make an argument for oneself to use it. E.g. letting people store much more data, letting them process it etc. Points pro using a third party service: 1. Too much time to develop 2. Too much maintenance overhead 3. Too difficult (to scale, develop, ...) 4. Economically unfeasable (because no economy of scale benefit, no synergies etc) Points contra using a third party service: 1. Giving up control over data 2. Legal problems ("where is the data stored?", "am I allowed to hand this data out?") 3. Overhead of interacting with a third party (account maintenance, payments, integration) 4. Inflexibility ("what if I need to change my existing data?", "what if I need some new feature?") As I see it, the points on "pro" don't apply at this scale. See the tiny scale and featureset is not really about data scale but overall "difficulty to do and keep running". If I offer a logging service with a REST API to store and retrieve up to 1000 lines of text without any way to filter/search/aggregate/process, would you use that because it's more convenient or would you just log things to a local file or syslog and call it a day? Like you said it's v1.0 and I'm sure you'll add more stuff, I'm just suggesting that you'll definitely have to add quite a bit of stuff to catch business users. I hope you can make it, I respect everyone who builds a product themselves and tries to make a business out of it. Best of luck!