11 ms·

Show HN: Pushdata.io – Ultra simple time series data storage

108 points by rlonn 8y ago

rlonn 8y ago
Hi HN, I built this thing! The idea came a few years back when me and an angel investor in the company I had founded were involved in a BI project, defining and collecting some basic KPIs for the business. At one point when we were thinking about where to store the very limited set of time series data that was our complete set of KPI metrics, the investor said “why isn’t there some service where you can just send a few numeric data points for storage, and get them back, without a lot of hassle?”. I really liked it as I saw it as a very simple idea, easy to implement but which still felt like it may be new/unique. The new thing here, of course, is not the ability to store time series data. The new/unique thing is the low barrier to entry. The API is extremely small and simple to use and learn, and there is no registration required to start using the service. I tried to make onboarding as frictionless as possible, and I think it’s hard to make it simpler than it is now. All feedback is greatly appreciated! Some perhaps useful links: Site: https://pushdata.io https://pushdata.io API docs: https://speca.io/ragnarlonn/pushdata-io https://speca.io/ragnarlonn/pushdata-io Blog article about making pushdata.io: https://bit.ly/2RGuIxc https://bit.ly/2RGuIxc
- gravypod 8y ago
  It might be a good idea to add more RESTful handling of ingestion. Something similar to the following POST /{account}/{series} [ { metrics: { "a": 10, "b": 20, "c": [1, 2, 3, 4] }, occurred: "2018-01-01 01:01:01" }, { metrics: { "a": 20, "b": 30, "c": [2, 3, 4, 5] }, occurred: "2018-01-01 01:01:02" } ] The overhead of an HTTP call of every data point submission might now work well in low bandwith situations. Also in cases where you have intermittent internet connections supplying a timestamp for when an event happens is helpful.
  - rlonn 8y ago
    There is an /api/timeseries end point where you can POST multiple data points with one single call. while putting that end point under /{email}/{series} may be more intuitive, I wanted to make it obvious that it is separate, as that end point is a premium feature. See https://speca.io/ragnarlonn/pushdata-io#extended-api https://speca.io/ragnarlonn/pushdata-io#extended-api
jajo 8y ago
Looks marvellous!
alex_112 8y ago
Seems like a fantastic service!
nissehulth 8y ago
I liked the part about replacing the React code. Small is beautiful.
- rlonn 8y ago
  Sometimes I worry that I'm too old-school, doing things like throwing out frameworks everyone else is using, and I see young(er) developers being incredibly productive when using those frameworks plus components, libraries, etc but then I also see that a lot of what they create looks the same and works the same, contains the same bugs, because (IMO) they're sometimes over-using external code and using cannons to shoot flies. I'm probably a bit too much the other way around, but it seems to work for me. I tend to be quite focused on minimizing external dependencies and only use what is absolutely necessary.
  - tnr23 8y ago
    I absolutely agree! Built a 6-figure-MRR SaaS with having exactly this attitude.
    - rlonn 8y ago
      My attitude, or the other one? :)
      - tnr23 8y ago
        yours :)
  - ex_amazon_sde 8y ago
    ...looking incredibly productive rather than being. Complexity always come back and get you. All this frameworks focus on throwing out code in weeks and make stuff impossible to maintain in few years - that's the tradeoff. It's no surprise FAANG and especially Amazon focus so much on simplifying their infra and write stuff from scratch so often.
MungoBBQ 8y ago
I like that I get a finished cURL example right away - so easy to start using. :)
jphelps 8y ago
This will be immediately useful for all of my little projects. Looking forward to metrics-without-headache. Thanks!
snicker7 8y ago
Quite a few comments praising this service seem to be from users with only 1 comment.
- rlonn 8y ago
  Maybe a mistake to tell everyone I know about it and that I'd make a launch post on HN today... Sweden is awake right now, so probably over-representation by locals but it should correct itself once the US wakes up. I would like some useful criticism.
- sho 8y ago
  At least one of them is 4+ years old, and rlonn is a veteran. I'm inclined to give the benefit of the doubt in this case. Bit of a lesson here for "show HN" hopefuls though to not have too many greentext usernames piling on...
rajadigopula 8y ago
Nice idea. Easy to plugin and use with IOT devices.
jermaustin1 8y ago
This reminds me a lot of another product called Metric Board: https://metricboard.io/how-does-it-work https://metricboard.io/how-does-it-work
- rlonn 8y ago
  That's cool, I hadn't seen that one. It's very similar, but with more functionality and seems to require signup (you have to have an API key to store data points)
kareemm 8y ago
Love the idea, it was head-smackingly obvious when I saw your Show HN. Handling this is a PITA in an app and I’d much rather pay for a 3rd party service to handle it. Some feedback: - the main benefit is that I don’t need to write or maintain code around storing, retrieving, and hopefully visualizing time series data - pricing looks fine. Maybe even a little low for Business. But I’d have to try it out to understand if the usage thresholds are too high / low. May want to have a plan between 75 and 450 - that’s a big, painful jump - May want to think about spending some time editing your copy so it’s crisper. A lot of it can be put into an eg FAQ (why not to use a temp email, for example). - some examples in other languages would be useful (Ruby, Python, JS, etc) - I like that I can create an account with just a curl POST. Good stuff. - one concern is how fast is retrieval? If I’m retrieving data in a request thread I want it to be FAST. One factor here is server performance. If you’re using eg AWS to host that would give me some confidence that you can scale up beyond a hamster wheel if necessary - no way to try this out on mobile, which is I suspect where a lot of people will first see this. How can you remind them to come back when they’re at a desktop? Collect an email and remind them maybe? Haven’t tried the service yet but assuming it works as advertised it’s a good start. Congratulations!
- rlonn 8y ago
  This is fantastic feedback - thanks! The thresholds are pretty much guesswork - they're probably a bit off but I'll adjust them when I know more about how people use the service. One thing that might not be immediately obvious is the amount of data you can store at the different price points. The number of time series and the number of points per series go up, which means a lot more potential data storage each time you go up a level. More storage means more transfers, and so I set the levels and pricing based on worst-case calculations for AWS bandwidth usage costs. And yes, it's hosted on AWS at the moment. Virginia. It should be reasonable fast for US-based users, but if it turns out to be viable I might look at distributing it more. As for scalability it is designed to be scalable, but currently there is a bottleneck in the form of a lookup database that is hit for each request for timeseries data. If I add caching of those lookups, it should scale well horizontally - I just add more API servers and timeseries databases I'll definitely be adding code examples. Perhaps also documentation/howtos for interfacing via IFTTT, Zapier etc? Oh, and the copy etc - I wanted to minimize the number of pages, so I put all the most useful info on the front page :) I definitely agree that e.g. an FAQ section would be nice. Thanks again for some solid feedback!
santypk4 8y ago
Really cool!! I'm looking forward to use this on my next project
GRiMe2D 8y ago
I love the idea but some points: - Why do one POST a data in URL? I've tried to insert a "_" value and got homepage as HTML with 200 status code. I suggest using HTTP body for that. - GET requests seem don't respect "Accept: " HTTP header. What if if want to get XML structured response? "Accept: application/xml" - It's unclear how to sign your requests and is there any authentication. If someone knows my email and/or name of db, can they insert their arbitrary data? Is there any private key or token exchange to verify POST requests? Edit 1: Formatting
- rlonn 8y ago
  Uhm, that POST should probably return something other than 200. The default is to serve index.html if the server doesn't recognize the URL, but in this case it is pretty obvious that you should get an error code back (400 - Bad Request seems logical). And yeah, JSON is the only supported encoding right now. I originally had the system output plain text actually, with an option to get JSON instead, but people who saw that thought it was very weird so the plain text support was ropped in a rewrite from Python -> Go. You're right that other common formats, like XML or CSV, should be supported. I'll put it on the issues list. Authentication: you post a data point to register an account. An account that has just been created has no API key, no security: anyone can access it and post or fetch data to/from it. To secure the account you have to confirm it, which is done by opening the URL you get in your confirmation email. When that URL is accessed, an API key will be created for you and the account is confirmed. Thinking about it, it might be better to just send out the API key, and then consider the account confirmed once the first request using the API key comes in. Hmm. I didn't want to send the key over email, but it is easy to change it on the site, so maybe that's not a big issue. I'll have to think about that. I understand the current process can be confusing because it is not standard. Thanks for the feedback!
eis 8y ago
Regarding the free account limit: 1000 data points in time series world is pretty much nothing. What happens if you reach that limit, does it just delete the oldest point? This would give you a month worth of data with one data point every hour or 16 hours at one point per minute. Though at one point per minute, you'd also exceed the rate limit of 1000 API calls per day. So you're stuck with one data point per hour I guess or maybe one per 5 minute interval and 3 days of data retention. I guess one could argue "but it's free!". There's no way to do any processing of the data like aggregations, min/max, grouping etc. meaning one would have to always fetch the raw data points and then do processing yourself. This kinda works because the amount of datapoints that this supports is so incredibly low. Seeing billions of datapoints over thousands of time series is not that uncommon and you wouldn't be able to use this service for those cases even with the "Business Pro" account for $450/month. When it comes to serious business users, you should ensure that you don't impose any hard limits on them, instead charge them for the usage. You point out how scalable the service is but you don't allow customers to scale their usage of it. Consider why every cloud vendor (AWS, Google, Azure) charges for example per API call, per amount of objects stored etc. They are happy to take your 10 Petabytes of data and charge you handsomely for it. Overall it seems like a service for toy projects or projects with very very low amounts of data. But then again you'd be able to store this amount of data in any SQL DB without problem while retaining control of your data, having much more flexibility, getting more speed out of it and being able to do computations on the data in an easier way. This is just too limited, both in terms of amount of data as well as featureset. I hope this comment doesn't sound too harsh, I intended it as constructive criticism.
- rlonn 8y ago
  The idea was that the service should not compete directly with the "big data" guys. It should be really, really simple and for small-to-medium scale usage (in terms of number of data points stored). But having said that, I am definitely not sure about 1000 data points being a reasonable free level. We will see I guess. Perhaps you're right that a minimum free tier should let people save at least a couple of days worth of data with 1-minute resolution. I don't know, but was afraid that if I set the paywall too high, noone would upgrade. It is easier to raise a paywall (i.e. become more generous) when you realize it is set too low, than the reverse. And the same goes for the business tiers - I have no idea if they're right or not. I just think that the people who do store billions and billions of data points are much better served going to the "big data" guys, or setting up their own, in-house db. They are not likely customers for pushdata.io even if I let them store a ton more data, because they'll have much higher demands on functionality also. I think pushdata.io is for small-to-medium scale usage, and one hope/belief I have is that people and companies often start out small, when they start collecting data, then their needs grow over time. Initially many probably just want to get started, and want a simple storage solution quickly. If they choose pushdata.io they are then likely to upgrade their account until they outgrow the service, which is when they'll move their data somewhere else. I'm fine with that. Many will never reach that point probably, or pushdata will be developed to support larger-scale usage. This is V1.0, after all :) The system will keep all data that is inserted, but to the user it will appear as a FIFO that deletes the oldest points. If you upgrade your account, however, you will gain access to old data that is stored, that you couldn't see before upgrading. It's an interesting idea to charge business users for usage instead of forcing them into a tier. The only issue is that you'll then have to provide them with detailed information on their usage, meaning a bit of work making sure things are logged properly and then turned into invoices. But it's definitely an interesting idea. And I really appreciate the straight feedback. It is the best kind of feedback. Thanks.
dawnerd 8y ago
Wait so I can post data using any email with no security by default? That seems like a really bad idea. I get you’re trying to be clever but damn. Can’t possibly see that causing a problem.
- rlonn 8y ago
  What is the threat scenario you see?
  - gitgud 8y ago
    Spamming the server with millions of unique but unverified email addresses, it will store anything you throw at it...
    - rlonn 8y ago
      I wrote about that just now: there are rate limits in place to prevent it. You'd have to go to quite a bit of trouble to be able to use the system for any kind of effective spamming, coming from different unique client IPs and only send a few emails from each of them, and not send too many emails to the same domains (e.g. spamming gmail.com would be impossible).
StavrosK 8y ago
This is very interesting, good job. I had made something a bit simpler back in the day, it's a single server binary you can run (and I run a demo server you can use) but I think yours has a few more features. Maybe you can use it for inspiration, it's called Gweet: https://github.com/skorokithakis/gweet https://github.com/skorokithakis/gweet
- rlonn 8y ago
  Ah, and built in Go also. I'll check it out :)
- zoom6628 8y ago
  That is actually pretty neat.
lozzo 8y ago
my feedback. Excellent work. I hope you make sure that you get high in google ranking. A while back (about a year ago) I had the need for something like this and kept searching on google for such a service. Surely somebody must have written it, I thought. Well, I ended up accepting that either it did not exist or if it existed it was impossible to find (which is the same thing from my angle). So again hope people find you once the hackernews show traffic fades away... cheers
- rlonn 8y ago
  Yeah, we'll see. I'd need inlinks and without a ton of publicly accessible content, that can be hard to get. I'm thinking maybe IoT communities would be interested and may link to the site, as the product could be quite useful to hobby IoT hackers with little or no budget for data storage, and not much interest in being db maintainers.
fudged71 8y ago
I have been looking for a service exactly like this for storing/retrieving data for Siri Shortcuts! Integrations with Firebase, Google Docs, etc are too difficult, but this API is very easily supported by Shortcuts. In fact without needing a user to register, you've opened up a huge group of users who can just download a shortcut and use it without ever needing to touch your service. Fantastic job with this, I can't wait to try it
- rlonn 8y ago
  That sounds like an interesting use case. If you want, I'll give you a free premium subscription if I can do a case study on your problem and the solution.
  - fudged71 8y ago
    Yes absolutely! How should I contact you?
    - rlonn 8y ago
      Send me something on hello at pushdata. io and I'll respond from my regular email.
vaidhy 8y ago
Maybe I am missing something.. I am curious how this is different from multiple other hosted graphite/grafana solutions. All of them allow for time-series to be stored and retrieved/graphed.
- rlonn 8y ago
  pushdata.io is simpler. You don't need to set up an account, the API is very, very small and simple, and the visualization is also much simpler than using Grafana. It is meant to be as frictionless as possible to get started with.
jpmoyn 8y ago
I've noticed with a lot of these "get started in seconds" services, you inevitably run into challenges down the road when the complications of trying to use them for a project come full swing. Not to mention the shaky nature of cloud hosting with a company that could easily be out of business in 6 months.
- rlonn 8y ago
  I agree, but often the alternative to using a service that offers painless onboarding, is to not do anything. I.e. you want to do X, and you're trying to find out how. You find product/service/component A, B and C that offers frictionless onboarding, and it allows you to get started on X. If, on the other hand, you find A, B and C that all require you to spend a couple of days learning their APIs etc, then you may never get started on X because the task becomes too big. There is a lot to be said for getting started on something. Components can always be replaced.
gitgud 8y ago
It says that an account is automatically created when you push to a url with an email not seen before. Then it goes on to say that there's no security, its not recommended etc... Why not make sure the email address has been validated before allowing it to be automatically created? Because right now someone could spam the sites servers with millions of unverified emails...
- rlonn 8y ago
  There are rate limits in place to prevent email spamming. The system stops creating new accounts for a domain where the percentage of unconfirmed accounts is too high. There is also rate limiting on client IP, disallowing client IPs to create lots of unconfirmed accounts. I am also planning to, but haven't got around to it yet, add exponential backoff to email sendouts, so that an increase in the sendout rate causes sendouts to be slower and slower (until manual intervention to increase limits). So, in short: yes, the system could be used for email spamming, but not very effectively. There are better options if all you want to do is spam people.

108 points by rlonn 8y ago

Hi HN, I built this thing! The idea came a few years back when me and an angel investor in the company I had founded were involved in a BI project, defining and collecting some basic KPIs for the business. At one point when we were thinking about where to store the very limited set of time series data that was our complete set of KPI metrics, the investor said “why isn’t there some service where you can just send a few numeric data points for storage, and get them back, without a lot of hassle?”. I really liked it as I saw it as a very simple idea, easy to implement but which still felt like it may be new/unique. The new thing here, of course, is not the ability to store time series data. The new/unique thing is the low barrier to entry. The API is extremely small and simple to use and learn, and there is no registration required to start using the service. I tried to make onboarding as frictionless as possible, and I think it’s hard to make it simpler than it is now. All feedback is greatly appreciated! Some perhaps useful links: Site: https://pushdata.io https://pushdata.io API docs: https://speca.io/ragnarlonn/pushdata-io https://speca.io/ragnarlonn/pushdata-io Blog article about making pushdata.io: https://bit.ly/2RGuIxc https://bit.ly/2RGuIxc

gravypod 8y ago
It might be a good idea to add more RESTful handling of ingestion. Something similar to the following POST /{account}/{series} [ { metrics: { "a": 10, "b": 20, "c": [1, 2, 3, 4] }, occurred: "2018-01-01 01:01:01" }, { metrics: { "a": 20, "b": 30, "c": [2, 3, 4, 5] }, occurred: "2018-01-01 01:01:02" } ] The overhead of an HTTP call of every data point submission might now work well in low bandwith situations. Also in cases where you have intermittent internet connections supplying a timestamp for when an event happens is helpful.
- rlonn 8y ago
  There is an /api/timeseries end point where you can POST multiple data points with one single call. while putting that end point under /{email}/{series} may be more intuitive, I wanted to make it obvious that it is separate, as that end point is a premium feature. See https://speca.io/ragnarlonn/pushdata-io#extended-api https://speca.io/ragnarlonn/pushdata-io#extended-api

jajo 8y ago

Looks marvellous!

alex_112 8y ago

Seems like a fantastic service!

nissehulth 8y ago

I liked the part about replacing the React code. Small is beautiful.

rlonn 8y ago
Sometimes I worry that I'm too old-school, doing things like throwing out frameworks everyone else is using, and I see young(er) developers being incredibly productive when using those frameworks plus components, libraries, etc but then I also see that a lot of what they create looks the same and works the same, contains the same bugs, because (IMO) they're sometimes over-using external code and using cannons to shoot flies. I'm probably a bit too much the other way around, but it seems to work for me. I tend to be quite focused on minimizing external dependencies and only use what is absolutely necessary.
- tnr23 8y ago
  I absolutely agree! Built a 6-figure-MRR SaaS with having exactly this attitude.
  - rlonn 8y ago
    My attitude, or the other one? :)
    - tnr23 8y ago
      yours :)
- ex_amazon_sde 8y ago
  ...looking incredibly productive rather than being. Complexity always come back and get you. All this frameworks focus on throwing out code in weeks and make stuff impossible to maintain in few years - that's the tradeoff. It's no surprise FAANG and especially Amazon focus so much on simplifying their infra and write stuff from scratch so often.

MungoBBQ 8y ago

I like that I get a finished cURL example right away - so easy to start using. :)

jphelps 8y ago

This will be immediately useful for all of my little projects. Looking forward to metrics-without-headache. Thanks!

snicker7 8y ago

Quite a few comments praising this service seem to be from users with only 1 comment.

rlonn 8y ago
Maybe a mistake to tell everyone I know about it and that I'd make a launch post on HN today... Sweden is awake right now, so probably over-representation by locals but it should correct itself once the US wakes up. I would like some useful criticism.
sho 8y ago
At least one of them is 4+ years old, and rlonn is a veteran. I'm inclined to give the benefit of the doubt in this case. Bit of a lesson here for "show HN" hopefuls though to not have too many greentext usernames piling on...

rajadigopula 8y ago

Nice idea. Easy to plugin and use with IOT devices.

jermaustin1 8y ago

This reminds me a lot of another product called Metric Board: https://metricboard.io/how-does-it-work https://metricboard.io/how-does-it-work

rlonn 8y ago
That's cool, I hadn't seen that one. It's very similar, but with more functionality and seems to require signup (you have to have an API key to store data points)

kareemm 8y ago

Love the idea, it was head-smackingly obvious when I saw your Show HN. Handling this is a PITA in an app and I’d much rather pay for a 3rd party service to handle it. Some feedback: - the main benefit is that I don’t need to write or maintain code around storing, retrieving, and hopefully visualizing time series data - pricing looks fine. Maybe even a little low for Business. But I’d have to try it out to understand if the usage thresholds are too high / low. May want to have a plan between 75 and 450 - that’s a big, painful jump - May want to think about spending some time editing your copy so it’s crisper. A lot of it can be put into an eg FAQ (why not to use a temp email, for example). - some examples in other languages would be useful (Ruby, Python, JS, etc) - I like that I can create an account with just a curl POST. Good stuff. - one concern is how fast is retrieval? If I’m retrieving data in a request thread I want it to be FAST. One factor here is server performance. If you’re using eg AWS to host that would give me some confidence that you can scale up beyond a hamster wheel if necessary - no way to try this out on mobile, which is I suspect where a lot of people will first see this. How can you remind them to come back when they’re at a desktop? Collect an email and remind them maybe? Haven’t tried the service yet but assuming it works as advertised it’s a good start. Congratulations!

rlonn 8y ago
This is fantastic feedback - thanks! The thresholds are pretty much guesswork - they're probably a bit off but I'll adjust them when I know more about how people use the service. One thing that might not be immediately obvious is the amount of data you can store at the different price points. The number of time series and the number of points per series go up, which means a lot more potential data storage each time you go up a level. More storage means more transfers, and so I set the levels and pricing based on worst-case calculations for AWS bandwidth usage costs. And yes, it's hosted on AWS at the moment. Virginia. It should be reasonable fast for US-based users, but if it turns out to be viable I might look at distributing it more. As for scalability it is designed to be scalable, but currently there is a bottleneck in the form of a lookup database that is hit for each request for timeseries data. If I add caching of those lookups, it should scale well horizontally - I just add more API servers and timeseries databases I'll definitely be adding code examples. Perhaps also documentation/howtos for interfacing via IFTTT, Zapier etc? Oh, and the copy etc - I wanted to minimize the number of pages, so I put all the most useful info on the front page :) I definitely agree that e.g. an FAQ section would be nice. Thanks again for some solid feedback!

santypk4 8y ago

Really cool!! I'm looking forward to use this on my next project

GRiMe2D 8y ago

I love the idea but some points: - Why do one POST a data in URL? I've tried to insert a "_" value and got homepage as HTML with 200 status code. I suggest using HTTP body for that. - GET requests seem don't respect "Accept: " HTTP header. What if if want to get XML structured response? "Accept: application/xml" - It's unclear how to sign your requests and is there any authentication. If someone knows my email and/or name of db, can they insert their arbitrary data? Is there any private key or token exchange to verify POST requests? Edit 1: Formatting

rlonn 8y ago
Uhm, that POST should probably return something other than 200. The default is to serve index.html if the server doesn't recognize the URL, but in this case it is pretty obvious that you should get an error code back (400 - Bad Request seems logical). And yeah, JSON is the only supported encoding right now. I originally had the system output plain text actually, with an option to get JSON instead, but people who saw that thought it was very weird so the plain text support was ropped in a rewrite from Python -> Go. You're right that other common formats, like XML or CSV, should be supported. I'll put it on the issues list. Authentication: you post a data point to register an account. An account that has just been created has no API key, no security: anyone can access it and post or fetch data to/from it. To secure the account you have to confirm it, which is done by opening the URL you get in your confirmation email. When that URL is accessed, an API key will be created for you and the account is confirmed. Thinking about it, it might be better to just send out the API key, and then consider the account confirmed once the first request using the API key comes in. Hmm. I didn't want to send the key over email, but it is easy to change it on the site, so maybe that's not a big issue. I'll have to think about that. I understand the current process can be confusing because it is not standard. Thanks for the feedback!

eis 8y ago

Regarding the free account limit: 1000 data points in time series world is pretty much nothing. What happens if you reach that limit, does it just delete the oldest point? This would give you a month worth of data with one data point every hour or 16 hours at one point per minute. Though at one point per minute, you'd also exceed the rate limit of 1000 API calls per day. So you're stuck with one data point per hour I guess or maybe one per 5 minute interval and 3 days of data retention. I guess one could argue "but it's free!". There's no way to do any processing of the data like aggregations, min/max, grouping etc. meaning one would have to always fetch the raw data points and then do processing yourself. This kinda works because the amount of datapoints that this supports is so incredibly low. Seeing billions of datapoints over thousands of time series is not that uncommon and you wouldn't be able to use this service for those cases even with the "Business Pro" account for $450/month. When it comes to serious business users, you should ensure that you don't impose any hard limits on them, instead charge them for the usage. You point out how scalable the service is but you don't allow customers to scale their usage of it. Consider why every cloud vendor (AWS, Google, Azure) charges for example per API call, per amount of objects stored etc. They are happy to take your 10 Petabytes of data and charge you handsomely for it. Overall it seems like a service for toy projects or projects with very very low amounts of data. But then again you'd be able to store this amount of data in any SQL DB without problem while retaining control of your data, having much more flexibility, getting more speed out of it and being able to do computations on the data in an easier way. This is just too limited, both in terms of amount of data as well as featureset. I hope this comment doesn't sound too harsh, I intended it as constructive criticism.

rlonn 8y ago
The idea was that the service should not compete directly with the "big data" guys. It should be really, really simple and for small-to-medium scale usage (in terms of number of data points stored). But having said that, I am definitely not sure about 1000 data points being a reasonable free level. We will see I guess. Perhaps you're right that a minimum free tier should let people save at least a couple of days worth of data with 1-minute resolution. I don't know, but was afraid that if I set the paywall too high, noone would upgrade. It is easier to raise a paywall (i.e. become more generous) when you realize it is set too low, than the reverse. And the same goes for the business tiers - I have no idea if they're right or not. I just think that the people who do store billions and billions of data points are much better served going to the "big data" guys, or setting up their own, in-house db. They are not likely customers for pushdata.io even if I let them store a ton more data, because they'll have much higher demands on functionality also. I think pushdata.io is for small-to-medium scale usage, and one hope/belief I have is that people and companies often start out small, when they start collecting data, then their needs grow over time. Initially many probably just want to get started, and want a simple storage solution quickly. If they choose pushdata.io they are then likely to upgrade their account until they outgrow the service, which is when they'll move their data somewhere else. I'm fine with that. Many will never reach that point probably, or pushdata will be developed to support larger-scale usage. This is V1.0, after all :) The system will keep all data that is inserted, but to the user it will appear as a FIFO that deletes the oldest points. If you upgrade your account, however, you will gain access to old data that is stored, that you couldn't see before upgrading. It's an interesting idea to charge business users for usage instead of forcing them into a tier. The only issue is that you'll then have to provide them with detailed information on their usage, meaning a bit of work making sure things are logged properly and then turned into invoices. But it's definitely an interesting idea. And I really appreciate the straight feedback. It is the best kind of feedback. Thanks.

dawnerd 8y ago

Wait so I can post data using any email with no security by default? That seems like a really bad idea. I get you’re trying to be clever but damn. Can’t possibly see that causing a problem.

rlonn 8y ago
What is the threat scenario you see?
- gitgud 8y ago
  Spamming the server with millions of unique but unverified email addresses, it will store anything you throw at it...
  - rlonn 8y ago
    I wrote about that just now: there are rate limits in place to prevent it. You'd have to go to quite a bit of trouble to be able to use the system for any kind of effective spamming, coming from different unique client IPs and only send a few emails from each of them, and not send too many emails to the same domains (e.g. spamming gmail.com would be impossible).

StavrosK 8y ago

This is very interesting, good job. I had made something a bit simpler back in the day, it's a single server binary you can run (and I run a demo server you can use) but I think yours has a few more features. Maybe you can use it for inspiration, it's called Gweet: https://github.com/skorokithakis/gweet https://github.com/skorokithakis/gweet

rlonn 8y ago
Ah, and built in Go also. I'll check it out :)
zoom6628 8y ago
That is actually pretty neat.

lozzo 8y ago

my feedback. Excellent work. I hope you make sure that you get high in google ranking. A while back (about a year ago) I had the need for something like this and kept searching on google for such a service. Surely somebody must have written it, I thought. Well, I ended up accepting that either it did not exist or if it existed it was impossible to find (which is the same thing from my angle). So again hope people find you once the hackernews show traffic fades away... cheers

rlonn 8y ago
Yeah, we'll see. I'd need inlinks and without a ton of publicly accessible content, that can be hard to get. I'm thinking maybe IoT communities would be interested and may link to the site, as the product could be quite useful to hobby IoT hackers with little or no budget for data storage, and not much interest in being db maintainers.

fudged71 8y ago

I have been looking for a service exactly like this for storing/retrieving data for Siri Shortcuts! Integrations with Firebase, Google Docs, etc are too difficult, but this API is very easily supported by Shortcuts. In fact without needing a user to register, you've opened up a huge group of users who can just download a shortcut and use it without ever needing to touch your service. Fantastic job with this, I can't wait to try it

rlonn 8y ago
That sounds like an interesting use case. If you want, I'll give you a free premium subscription if I can do a case study on your problem and the solution.
- fudged71 8y ago
  Yes absolutely! How should I contact you?
  - rlonn 8y ago
    Send me something on hello at pushdata. io and I'll respond from my regular email.

vaidhy 8y ago

Maybe I am missing something.. I am curious how this is different from multiple other hosted graphite/grafana solutions. All of them allow for time-series to be stored and retrieved/graphed.

rlonn 8y ago
pushdata.io is simpler. You don't need to set up an account, the API is very, very small and simple, and the visualization is also much simpler than using Grafana. It is meant to be as frictionless as possible to get started with.

jpmoyn 8y ago

I've noticed with a lot of these "get started in seconds" services, you inevitably run into challenges down the road when the complications of trying to use them for a project come full swing. Not to mention the shaky nature of cloud hosting with a company that could easily be out of business in 6 months.

rlonn 8y ago
I agree, but often the alternative to using a service that offers painless onboarding, is to not do anything. I.e. you want to do X, and you're trying to find out how. You find product/service/component A, B and C that offers frictionless onboarding, and it allows you to get started on X. If, on the other hand, you find A, B and C that all require you to spend a couple of days learning their APIs etc, then you may never get started on X because the task becomes too big. There is a lot to be said for getting started on something. Components can always be replaced.

gitgud 8y ago

It says that an account is automatically created when you push to a url with an email not seen before. Then it goes on to say that there's no security, its not recommended etc... Why not make sure the email address has been validated before allowing it to be automatically created? Because right now someone could spam the sites servers with millions of unverified emails...

rlonn 8y ago
There are rate limits in place to prevent email spamming. The system stops creating new accounts for a domain where the percentage of unconfirmed accounts is too high. There is also rate limiting on client IP, disallowing client IPs to create lots of unconfirmed accounts. I am also planning to, but haven't got around to it yet, add exponential backoff to email sendouts, so that an increase in the sendout rate causes sendouts to be slower and slower (until manual intervention to increase limits). So, in short: yes, the system could be used for email spamming, but not very effectively. There are better options if all you want to do is spam people.