4 ms·
BigQuery public datasets would be a better hosting platform for this kind of data. I worry they are not anticipating the security & budgeting issues of hosting
by tonymet 3y ago
BigQuery public datasets would be a better hosting platform for this kind of data. I worry they are not anticipating the security & budgeting issues of hosting a real-time API.
With PUblic Datasets, the account making queries pays for the queries. NPS only pays for the storage (which is minimal).
With this API, NPS has to pay for every call to the API. That’s not cheap.
- capital_guy 3y agoYes, and the flip side of that is you require people to have and use Google accounts to access public data? That’s not exactly ideal.
- tonymet 3y agoFine host it as CSV as a backup for the luddites.
- tonymet 3y agoAnd at what cost. Why are we paying for customers to query this data in realtime? So someone can host a ripoff NPS app on the App Store and taxpayers now pay for content hosting?
- dylan604 3y agoThe USPS provides a free service for address validation. You have to register an account and receive an access token. If they feel your token is using too much, they can handle it as necessary. Why this same concept couldn't be done in the same way is just lack of imagination. You can access for free, but if you abuse or break the TOS your access is revoked. Done
- tonymet 3y agoUSPS is a corporation with profit and loss. I still consider this irresponsible but theoretically I'm not paying for their largesse
- tonymet 3y agoWith their API they have to write a bunch of boilerplate code to transform from their SQL db to REST. Authentication, throttling, threat prevention, encoding, etc etc. With BigQuery they just copy the data in via CSV and Big Query handles the indexing & query engine.
- mlhpdx 3y agoWho says they have a SQL DB? This looks to be almost entirely static data, occasionally updated.
- tonymet 3y agoWhatever storage format they have, they are writing boilerplate to transform it into REST . Regardless, it will be cheaper to just ingest into BigQuery
- jimmyrocks 3y agoIt’s Apache Solr. Most of the data is static, but alerts and events get frequent updates.
- tonymet 3y agoSame concern about unnecessary code and compute stands.
- dragonwriter 3y ago> With their API they have to write a bunch of boilerplate code to transform from their SQL db to REST. Open source tools that will present a simple, read-only REST API over an SQL db with little to no custom code exist (so do proprietary tools, often from DB vendors and sometimes as part of SQL db products.) Same with NoSQL or sorta-SQL storage solutions. The idea that they have to write a bunch of boilerplate code to do this is false. They might choose to do that, but its defintely not necessary. > Authentication, throttling, threat prevention, encoding, etc etc. Again, open source canned solutions that take a little bit of configuration exist for many of those, and some of them are likely shared services that they just point at whatever service needs them.
- ethbr1 3y agoRequiring use of a private party to access public data is usually something we discourage.
- tonymet 3y agoAt what cost?
- tonymet 3y agoThey’re hosting on AWS . So either taxpayer pays for hosting, or the customer pays .
- tonymet 3y agoPrivate parties that the customer needs to pay access this NPS public data: * AWS * Comcast for their internet service * Apple for their laptop * A number of software providers for their development tools. But asking the customer to pay google to query the data is crossing the line?
- AdieuToLogic 3y agoThis argument is nonsensical. Site hosting is not a customer cost. The rest you list are costs orthogonal to this service. > But asking the customer to pay google to query the data is crossing the line? Yes. Why are you arguing for a US government agency to require its citizens to pay for access to data which they have already paid for by funding said agency?
- huytersd 3y agoAre you suggesting the government put their public access API behind a paywall?
- tonymet 3y ago[flagged]
- dragonwriter 3y ago> They should choose a solution that makes the customers pay for accessing the data. They are. The NPS's customers are the American public, and they are paying for the data via taxes (approximately; government finance is more complex than that, and taxes don't really pay for spending that's a model that really is only true when working with commodity and not fiat currency, but, its reasonable enough for this purpose.) What you want isn't for the current customers to pay for the data, but for the consumers of data to be viewed as customers and then charged for the data. but that potentially makes things more expensive for the NPS's customers, for instance, if one of the significant consumers turns out to be other public agencies, who then are paying a direct cost which pays for both the access to the data and the additional costs of billing and account management that a consumer-pays model imposes, and paying the overhead for the payer-side costs associated with payments, as well as the actual amount of the payments, then you end up ultimately with largely the same customers paying, but paying a whole lot of additional overhead.
- jdksmdbtbdnmsm 3y agoLooking at the data made available by this API, I think it's safe to say this is fine.
- AdieuToLogic 3y ago> BigQuery public datasets would be a better hosting platform for this kind of data. I worry they are not anticipating the security & budgeting issues of hosting a real-time API. Then use their API to populate a BigQuery public dataset and make available to all. Otherwise, perhaps we, as outside observers, need to consider the possibility that those whom made the decisions to provide this service as such did so for reasons which we may not be aware.
- tonymet 3y ago[flagged]
- AdieuToLogic 3y ago> Are you new here? Yes, I am new here. So why don't you tell me why you haven't answered this question I posited earlier: Why are you arguing for a US government agency to require its citizens to pay for access to data which they have already paid for by funding said agency?
- dylan604 3y agoThere's a difference from paying for the information, and paying for the expense of delivering that information. You have to pay for official copies of documents like birth/marriage/death certificates. To get copies of public court documents costs as well even though you can read the data for free. Toner/paper isn't free, fancy paper for certs aren't free. You can have the data, but you gotta pay for the copy.
- tonymet 3y agowell said
- tonymet 3y agook good suggestion here you go nps-public-data.nps_public_data
- dragonwriter 3y ago> NPS has to pay for every call to the API. That’s not cheap. I am perfectly fine with it being considered part of the basic, taxpayer-supported functions of government agencies to be providing the public with relevant data. If there is a concrete abuse or wildly disproportionate cost problem in particular areas, that may need to be addressed with charges for particular kinds or patterns of access.
- tonymet 3y agoAt what cost? Rest APIs are very expensive ways for the government to make CSV data available to the public.
- dragonwriter 3y agoThey are a whole lot less expensive than tracking customer usage and billing for it, and a whole lot more useful to the public than having the data nominally publicly accessible but only "on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying Beware of the Leopard." [0] [0] Douglas Adams, Hitchhiker's Guide to the Galaxy
- tonymet 3y agonot in this case that's my whole point . they are choosing the most expensive (and riskiest) way to make csv files available to the public
- everforward 3y agoThere's likely some truth that CSV would work well here, and would likely be cheaper to operate. I wouldn't be surprised if a lot of clients are or could be doing full transfers of the data and doing their own queries. I'd be pretty happy with sqlite dumps too. I don't really have an issue with the REST, though. I wouldn't be surprised if this was just a standard and cheap to set up Django+REST libraries stack. Yeah, the compute costs are higher than transferring static files, but I'd be shocked if this was taking enough QPS for the difference in cost to make a meaningful difference. I get wanting the government to be responsible, but this veers a bit too far into Brutalist architecture as an organizational principle.
- whakim 3y agoREST APIs are extremely battle-tested, easy to integrated with, and far more mainstream than BigQuery public datasets or any other niche technology that may or may not exist at some point in the future. If cost is truly an issue, perhaps the solution is to properly fund the NPS so it can make smart technology decisions.
- tonymet 3y agoat what cost?
- whakim 3y agoI’m not sure if you’re serious (given the spammy nature of your posts, I’m inclined to believe not), but given that REST is the de facto standard for exchange of information between machines across the internet, I think the onus is on you to estimate how much money you think the NPS stands to save by doing it your way. Then the rest of us can evaluate whether that’s a good tradeoff.
- tonymet 3y agoWe only know based on the information we have that they are sharing CSV files via REST on AWS/EC2. That’s the most expensive way to share it, and also risky . What’s spammy about my post? I have asked people to focus on costs when they make general statements like “all govt data should have a REST api”.
- whakim 3y agoDo you actually know that the only source of the data is a static CSV? Or is that just speculation? I think we can dispense with the risky argument, because this API has existed for years without issue. When you make the argument that "X is too expensive," the onus is on you to prove it's expensive in a relative sense, not simply in an absolute sense. Saving $100 matters if you're spending $1000; it probably doesn't matter if you're spending $10m. Feel free to convince us: make some estimates, crunch some numbers, and look at existing NPS IT spending and see if they seem ballpark reasonable. Otherwise you're just banging on about a left-field solution that almost no one wants because it's putatively cheaper (but by how much, you can't say).
- bennyg 3y agoThis is a fascinating thread under this comment. Everyone is keying off of one part of the comment (querier pays) and not the more critical issue IMO - anticipating security and budgeting issues of hosting a real-time API. You suggested an alternative and everyone is pitting the status quo against that alternative instead of maybe looking for other alternatives that help address the issue. People here clearly don’t like a querier pays model and that’s fine. But should NPS still reinvent the wheel across the SDLC to serve this data? I think there’s a compelling argument in there.
- tonymet 3y agoYes thank you for noticing that. My bigger concern is NPS paying for expensive auto-scale resources for what is basically CSV files that could be hosted cheaply and securely. REST API compute is very expensive when you include compute costs, transfer fees and admin costs to keep it up. Not to mention the cost to implement a bespoke API and deal with security issues. All to make CSV available!
- karpatic 3y agoI'd consider it a public transit service. We wouldn't be upset about people using shuttle busses to get to the parks, would we? I think long term footing the bill for an open platform with principle beneficiaries who use it is fine so long as it provides a net benefit.
- tonymet 3y agoIf you have to pay for REST API OR shuttle busses which one gets funded?
- CaptWillard 3y agoOn the list of alarming or even questionable things our taxes pay for, this doesn't even make the top 100.
- tonymet 3y agostart a thread on one of those let's discuss