5 ms·

Regarding the free account limit: 1000 data points in time series world is pretty much nothing. What happens if you reach that limit, does it just delete the ol

by eis 8y ago

Regarding the free account limit: 1000 data points in time series world is pretty much nothing. What happens if you reach that limit, does it just delete the oldest point? This would give you a month worth of data with one data point every hour or 16 hours at one point per minute. Though at one point per minute, you'd also exceed the rate limit of 1000 API calls per day. So you're stuck with one data point per hour I guess or maybe one per 5 minute interval and 3 days of data retention. I guess one could argue "but it's free!". There's no way to do any processing of the data like aggregations, min/max, grouping etc. meaning one would have to always fetch the raw data points and then do processing yourself. This kinda works because the amount of datapoints that this supports is so incredibly low. Seeing billions of datapoints over thousands of time series is not that uncommon and you wouldn't be able to use this service for those cases even with the "Business Pro" account for $450/month. When it comes to serious business users, you should ensure that you don't impose any hard limits on them, instead charge them for the usage. You point out how scalable the service is but you don't allow customers to scale their usage of it. Consider why every cloud vendor (AWS, Google, Azure) charges for example per API call, per amount of objects stored etc. They are happy to take your 10 Petabytes of data and charge you handsomely for it. Overall it seems like a service for toy projects or projects with very very low amounts of data. But then again you'd be able to store this amount of data in any SQL DB without problem while retaining control of your data, having much more flexibility, getting more speed out of it and being able to do computations on the data in an easier way. This is just too limited, both in terms of amount of data as well as featureset. I hope this comment doesn't sound too harsh, I intended it as constructive criticism.

rlonn 8y ago
The idea was that the service should not compete directly with the "big data" guys. It should be really, really simple and for small-to-medium scale usage (in terms of number of data points stored). But having said that, I am definitely not sure about 1000 data points being a reasonable free level. We will see I guess. Perhaps you're right that a minimum free tier should let people save at least a couple of days worth of data with 1-minute resolution. I don't know, but was afraid that if I set the paywall too high, noone would upgrade. It is easier to raise a paywall (i.e. become more generous) when you realize it is set too low, than the reverse. And the same goes for the business tiers - I have no idea if they're right or not. I just think that the people who do store billions and billions of data points are much better served going to the "big data" guys, or setting up their own, in-house db. They are not likely customers for pushdata.io even if I let them store a ton more data, because they'll have much higher demands on functionality also. I think pushdata.io is for small-to-medium scale usage, and one hope/belief I have is that people and companies often start out small, when they start collecting data, then their needs grow over time. Initially many probably just want to get started, and want a simple storage solution quickly. If they choose pushdata.io they are then likely to upgrade their account until they outgrow the service, which is when they'll move their data somewhere else. I'm fine with that. Many will never reach that point probably, or pushdata will be developed to support larger-scale usage. This is V1.0, after all :) The system will keep all data that is inserted, but to the user it will appear as a FIFO that deletes the oldest points. If you upgrade your account, however, you will gain access to old data that is stored, that you couldn't see before upgrading. It's an interesting idea to charge business users for usage instead of forcing them into a tier. The only issue is that you'll then have to provide them with detailed information on their usage, meaning a bit of work making sure things are logged properly and then turned into invoices. But it's definitely an interesting idea. And I really appreciate the straight feedback. It is the best kind of feedback. Thanks.
- eis 8y ago
  > It should be really, really simple and for small-to-medium scale usage (in terms of number of data points stored). To me those tiers are more like tiny-to-small scale :) The problem with these small numbers is that it's trivial to put them into really any DB that you can think of. Even SQLite will handle this amount of data with super ease. I can only see people who know nothing about databases using this service. Or someone who for some reason can't use a DB. 1000 datapoints is at tops maybe 100 kilobytes of data if one uses a very unoptimized format and can easily be less than a kilobyte if done properly. At a million datapoints (your highest tier) we're talking about 1 Megabyte to maybe 100 Megabyte (again, if done very poorly). It's really a trivial amount of data. A standard Postgres or MySQL DB will be able to run queries over this dataset in less than a second and that's using something that's totally not optimized for time series. > It's an interesting idea to charge business users for usage instead of forcing them into a tier. The only issue is that you'll then have to provide them with detailed information on their usage, meaning a bit of work making sure things are logged properly But you already need to log their usage to know if they fall into whatever tiers limits. And you can store this info in... a time series DB :)
  - rlonn 8y ago
    The marketer in me doesn't like the phrase "tiny-to-small scale" It's the same as when you order a latte at Starbucks and they only have various versions of "big" to sell you ;) I think you're missing the point about self-hosted db's though - people will not choose to put their data on pushdata.io because their own database cannot handle the amount of data they have, but because it is more convenient. Many people, even those who do know a little about databases, do not want to maintain another server [instance] and all that comes with doing that (like backups, for instance). They don't want to set it up, they don't want to maintain or be responsible for it. This choice has nothing to do with performance or size of the data set, but with convenience and peace of mind. Hey, it's a really cool idea to log usage data in the time series db! That is real dogfooding if I ever saw it. I will definitely try to do that :)
    - eis 8y ago

by eis 8y ago

rlonn 8y ago

The idea was that the service should not compete directly with the "big data" guys. It should be really, really simple and for small-to-medium scale usage (in terms of number of data points stored). But having said that, I am definitely not sure about 1000 data points being a reasonable free level. We will see I guess. Perhaps you're right that a minimum free tier should let people save at least a couple of days worth of data with 1-minute resolution. I don't know, but was afraid that if I set the paywall too high, noone would upgrade. It is easier to raise a paywall (i.e. become more generous) when you realize it is set too low, than the reverse. And the same goes for the business tiers - I have no idea if they're right or not. I just think that the people who do store billions and billions of data points are much better served going to the "big data" guys, or setting up their own, in-house db. They are not likely customers for pushdata.io even if I let them store a ton more data, because they'll have much higher demands on functionality also. I think pushdata.io is for small-to-medium scale usage, and one hope/belief I have is that people and companies often start out small, when they start collecting data, then their needs grow over time. Initially many probably just want to get started, and want a simple storage solution quickly. If they choose pushdata.io they are then likely to upgrade their account until they outgrow the service, which is when they'll move their data somewhere else. I'm fine with that. Many will never reach that point probably, or pushdata will be developed to support larger-scale usage. This is V1.0, after all :) The system will keep all data that is inserted, but to the user it will appear as a FIFO that deletes the oldest points. If you upgrade your account, however, you will gain access to old data that is stored, that you couldn't see before upgrading. It's an interesting idea to charge business users for usage instead of forcing them into a tier. The only issue is that you'll then have to provide them with detailed information on their usage, meaning a bit of work making sure things are logged properly and then turned into invoices. But it's definitely an interesting idea. And I really appreciate the straight feedback. It is the best kind of feedback. Thanks.

eis 8y ago
> It should be really, really simple and for small-to-medium scale usage (in terms of number of data points stored). To me those tiers are more like tiny-to-small scale :) The problem with these small numbers is that it's trivial to put them into really any DB that you can think of. Even SQLite will handle this amount of data with super ease. I can only see people who know nothing about databases using this service. Or someone who for some reason can't use a DB. 1000 datapoints is at tops maybe 100 kilobytes of data if one uses a very unoptimized format and can easily be less than a kilobyte if done properly. At a million datapoints (your highest tier) we're talking about 1 Megabyte to maybe 100 Megabyte (again, if done very poorly). It's really a trivial amount of data. A standard Postgres or MySQL DB will be able to run queries over this dataset in less than a second and that's using something that's totally not optimized for time series. > It's an interesting idea to charge business users for usage instead of forcing them into a tier. The only issue is that you'll then have to provide them with detailed information on their usage, meaning a bit of work making sure things are logged properly But you already need to log their usage to know if they fall into whatever tiers limits. And you can store this info in... a time series DB :)
- rlonn 8y ago
  The marketer in me doesn't like the phrase "tiny-to-small scale" It's the same as when you order a latte at Starbucks and they only have various versions of "big" to sell you ;) I think you're missing the point about self-hosted db's though - people will not choose to put their data on pushdata.io because their own database cannot handle the amount of data they have, but because it is more convenient. Many people, even those who do know a little about databases, do not want to maintain another server [instance] and all that comes with doing that (like backups, for instance). They don't want to set it up, they don't want to maintain or be responsible for it. This choice has nothing to do with performance or size of the data set, but with convenience and peace of mind. Hey, it's a really cool idea to log usage data in the time series db! That is real dogfooding if I ever saw it. I will definitely try to do that :)
  - eis 8y ago