3 ms·

The idea was that the service should not compete directly with the "big data" guys. It should be really, really simple and for small-to-medium scale usage (in t

by rlonn 8y ago

The idea was that the service should not compete directly with the "big data" guys. It should be really, really simple and for small-to-medium scale usage (in terms of number of data points stored). But having said that, I am definitely not sure about 1000 data points being a reasonable free level. We will see I guess. Perhaps you're right that a minimum free tier should let people save at least a couple of days worth of data with 1-minute resolution. I don't know, but was afraid that if I set the paywall too high, noone would upgrade. It is easier to raise a paywall (i.e. become more generous) when you realize it is set too low, than the reverse. And the same goes for the business tiers - I have no idea if they're right or not. I just think that the people who do store billions and billions of data points are much better served going to the "big data" guys, or setting up their own, in-house db. They are not likely customers for pushdata.io even if I let them store a ton more data, because they'll have much higher demands on functionality also. I think pushdata.io is for small-to-medium scale usage, and one hope/belief I have is that people and companies often start out small, when they start collecting data, then their needs grow over time. Initially many probably just want to get started, and want a simple storage solution quickly. If they choose pushdata.io they are then likely to upgrade their account until they outgrow the service, which is when they'll move their data somewhere else. I'm fine with that. Many will never reach that point probably, or pushdata will be developed to support larger-scale usage. This is V1.0, after all :) The system will keep all data that is inserted, but to the user it will appear as a FIFO that deletes the oldest points. If you upgrade your account, however, you will gain access to old data that is stored, that you couldn't see before upgrading. It's an interesting idea to charge business users for usage instead of forcing them into a tier. The only issue is that you'll then have to provide them with detailed information on their usage, meaning a bit of work making sure things are logged properly and then turned into invoices. But it's definitely an interesting idea. And I really appreciate the straight feedback. It is the best kind of feedback. Thanks.

eis 8y ago
> It should be really, really simple and for small-to-medium scale usage (in terms of number of data points stored). To me those tiers are more like tiny-to-small scale :) The problem with these small numbers is that it's trivial to put them into really any DB that you can think of. Even SQLite will handle this amount of data with super ease. I can only see people who know nothing about databases using this service. Or someone who for some reason can't use a DB. 1000 datapoints is at tops maybe 100 kilobytes of data if one uses a very unoptimized format and can easily be less than a kilobyte if done properly. At a million datapoints (your highest tier) we're talking about 1 Megabyte to maybe 100 Megabyte (again, if done very poorly). It's really a trivial amount of data. A standard Postgres or MySQL DB will be able to run queries over this dataset in less than a second and that's using something that's totally not optimized for time series. > It's an interesting idea to charge business users for usage instead of forcing them into a tier. The only issue is that you'll then have to provide them with detailed information on their usage, meaning a bit of work making sure things are logged properly But you already need to log their usage to know if they fall into whatever tiers limits. And you can store this info in... a time series DB :)
- rlonn 8y ago
  The marketer in me doesn't like the phrase "tiny-to-small scale" It's the same as when you order a latte at Starbucks and they only have various versions of "big" to sell you ;) I think you're missing the point about self-hosted db's though - people will not choose to put their data on pushdata.io because their own database cannot handle the amount of data they have, but because it is more convenient. Many people, even those who do know a little about databases, do not want to maintain another server [instance] and all that comes with doing that (like backups, for instance). They don't want to set it up, they don't want to maintain or be responsible for it. This choice has nothing to do with performance or size of the data set, but with convenience and peace of mind. Hey, it's a really cool idea to log usage data in the time series db! That is real dogfooding if I ever saw it. I will definitely try to do that :)
  - eis 8y ago
    > I think you're missing the point about self-hosted db's though - people will not choose to put their data on pushdata.io because their own database cannot handle the amount of data they have, but because it is more convenient. Many people, even those who do know a little about databases, do not want to maintain another server [instance] and all that comes with doing that (like backups, for instance). They don't want to set it up, they don't want to maintain or be responsible for it. This choice has nothing to do with performance or size of the data set, but with convenience and peace of mind. I am not so sure. Most projects already need a DB and it's really trivial to store this small amount of data in there as well. I am not sure if it is easier to maintain another account with a third party, write a HTTP POST instead of SQL INSERT and HTTP GET + parsing + processing code instead of SQL SELECT with processing right in the query. One also already needs to do backups of their main data and those backups would just include this timeseries data as well. They most likely don't need another instance, another DB etc. for this amount of data, they just stick it into their existing one. My point being that yes I agree that people reach for third party services out of convenience and not scalability needs BUT this is such a tiny piece of a bigger puzzle that the overhead it comes with might not be worth it. Removing the need to host the timeseries storage part of a product themselves does not remove the need for a DB for 99.99% of projects. If it is trivial to do something myself, then I will do it myself. But if doing it myself takes considerable effort or requires ongoing maintenance overhead, then I might consider a third party service. The higher the value of your service, the better one can make an argument for oneself to use it. E.g. letting people store much more data, letting them process it etc. Points pro using a third party service: 1. Too much time to develop 2. Too much maintenance overhead 3. Too difficult (to scale, develop, ...) 4. Economically unfeasable (because no economy of scale benefit, no synergies etc) Points contra using a third party service: 1. Giving up control over data 2. Legal problems ("where is the data stored?", "am I allowed to hand this data out?") 3. Overhead of interacting with a third party (account maintenance, payments, integration) 4. Inflexibility ("what if I need to change my existing data?", "what if I need some new feature?") As I see it, the points on "pro" don't apply at this scale. See the tiny scale and featureset is not really about data scale but overall "difficulty to do and keep running". If I offer a logging service with a REST API to store and retrieve up to 1000 lines of text without any way to filter/search/aggregate/process, would you use that because it's more convenient or would you just log things to a local file or syslog and call it a day? Like you said it's v1.0 and I'm sure you'll add more stuff, I'm just suggesting that you'll definitely have to add quite a bit of stuff to catch business users. I hope you can make it, I respect everyone who builds a product themselves and tries to make a business out of it. Best of luck!

by rlonn 8y ago

eis 8y ago

> It should be really, really simple and for small-to-medium scale usage (in terms of number of data points stored). To me those tiers are more like tiny-to-small scale :) The problem with these small numbers is that it's trivial to put them into really any DB that you can think of. Even SQLite will handle this amount of data with super ease. I can only see people who know nothing about databases using this service. Or someone who for some reason can't use a DB. 1000 datapoints is at tops maybe 100 kilobytes of data if one uses a very unoptimized format and can easily be less than a kilobyte if done properly. At a million datapoints (your highest tier) we're talking about 1 Megabyte to maybe 100 Megabyte (again, if done very poorly). It's really a trivial amount of data. A standard Postgres or MySQL DB will be able to run queries over this dataset in less than a second and that's using something that's totally not optimized for time series. > It's an interesting idea to charge business users for usage instead of forcing them into a tier. The only issue is that you'll then have to provide them with detailed information on their usage, meaning a bit of work making sure things are logged properly But you already need to log their usage to know if they fall into whatever tiers limits. And you can store this info in... a time series DB :)

rlonn 8y ago
The marketer in me doesn't like the phrase "tiny-to-small scale" It's the same as when you order a latte at Starbucks and they only have various versions of "big" to sell you ;) I think you're missing the point about self-hosted db's though - people will not choose to put their data on pushdata.io because their own database cannot handle the amount of data they have, but because it is more convenient. Many people, even those who do know a little about databases, do not want to maintain another server [instance] and all that comes with doing that (like backups, for instance). They don't want to set it up, they don't want to maintain or be responsible for it. This choice has nothing to do with performance or size of the data set, but with convenience and peace of mind. Hey, it's a really cool idea to log usage data in the time series db! That is real dogfooding if I ever saw it. I will definitely try to do that :)
- eis 8y ago
  > I think you're missing the point about self-hosted db's though - people will not choose to put their data on pushdata.io because their own database cannot handle the amount of data they have, but because it is more convenient. Many people, even those who do know a little about databases, do not want to maintain another server [instance] and all that comes with doing that (like backups, for instance). They don't want to set it up, they don't want to maintain or be responsible for it. This choice has nothing to do with performance or size of the data set, but with convenience and peace of mind. I am not so sure. Most projects already need a DB and it's really trivial to store this small amount of data in there as well. I am not sure if it is easier to maintain another account with a third party, write a HTTP POST instead of SQL INSERT and HTTP GET + parsing + processing code instead of SQL SELECT with processing right in the query. One also already needs to do backups of their main data and those backups would just include this timeseries data as well. They most likely don't need another instance, another DB etc. for this amount of data, they just stick it into their existing one. My point being that yes I agree that people reach for third party services out of convenience and not scalability needs BUT this is such a tiny piece of a bigger puzzle that the overhead it comes with might not be worth it. Removing the need to host the timeseries storage part of a product themselves does not remove the need for a DB for 99.99% of projects. If it is trivial to do something myself, then I will do it myself. But if doing it myself takes considerable effort or requires ongoing maintenance overhead, then I might consider a third party service. The higher the value of your service, the better one can make an argument for oneself to use it. E.g. letting people store much more data, letting them process it etc. Points pro using a third party service: 1. Too much time to develop 2. Too much maintenance overhead 3. Too difficult (to scale, develop, ...) 4. Economically unfeasable (because no economy of scale benefit, no synergies etc) Points contra using a third party service: 1. Giving up control over data 2. Legal problems ("where is the data stored?", "am I allowed to hand this data out?") 3. Overhead of interacting with a third party (account maintenance, payments, integration) 4. Inflexibility ("what if I need to change my existing data?", "what if I need some new feature?") As I see it, the points on "pro" don't apply at this scale. See the tiny scale and featureset is not really about data scale but overall "difficulty to do and keep running". If I offer a logging service with a REST API to store and retrieve up to 1000 lines of text without any way to filter/search/aggregate/process, would you use that because it's more convenient or would you just log things to a local file or syslog and call it a day? Like you said it's v1.0 and I'm sure you'll add more stuff, I'm just suggesting that you'll definitely have to add quite a bit of stuff to catch business users. I hope you can make it, I respect everyone who builds a product themselves and tries to make a business out of it. Best of luck!