8 ms·
You can serve static data over HTTP
- alecst 3y agoThanks, I think I can actually apply this idea actually. I'm writing a chat app for fun and one of the (admittedly small) issues I'm dealing with is cleanly storing the local state of the chat so that it appears correctly on refresh. There's some complexity involved with how messages are served, received, etc. with notifications, dings, animations, whatever -- but maybe saving that information to a file (as an alternative to localStorage or database read) could be a robust alternative to whatever I was trying to do. I'll give it a try.
- nesarkvechnep 3y agoDatabase reads all the time for your use-case.
- est 3y agothat's why I really, really hope ActivityPub or Nostr protocol could be designed as static-website compatible. But nope.
- schemescape 3y agoRSS seems closest to that.
- nl 3y agoI think you should be able to to a read-only, unauthenticated implementation of ActivityPub just using ActivityStreams files. Best as I can tell the part that might be non-compliant here is that new followers should be added to the follower collection.
- jeroenhd 3y agoParts of ActivityPub will work with static files: https://justingarrison.com/blog/2022-12-06-mastodon-files-instance/ https://justingarrison.com/blog/2022-12-06-mastodon-files-in... You need something to deal with follows (cryptographically verify that the follow attempts actually originate from the servers they claim to) and submitting posts to your followers (push based, not pull based in practice) but all the metadata and user lookup stuff can be done with JSON files in the right place.
- faizshah 3y agoNot really in the spirit of the post but I learned recently that you can also back your API Gateway apis in AWS with an s3 bucket: https://docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-s3.html https://docs.aws.amazon.com/apigateway/latest/developerguide... Makes it super simple to implement mocks of your api design before building the api.
- graypegg 3y agoRails, by default, will put a “.json” suffix on json routes. This is so supremely useful if you have a Rails api app, speaking to some over complicated front end that might need a quick and dirty mock for something. Just serve your /api/thing.json from somewhere. Only issue is with HTTP verbs other than GET, but usually at that point I need a lot more than this 15second mock. Good for testing little things though.
- Alifatisk 3y agoI was about to say this!
- Lio 3y agoThis also lends itself to easy caching with a versioning key and proxy servers, just like with static files. …I guess that was kind of the point of REST in the first place but it’s easy to forget. REST has a wonderful simplicity to it. I’m yet to hear about an equivalent caching solution for GrqphQL APIs for example.
- nesarkvechnep 3y agoLet me tell you about hypermedia controls and discuss simplicity then? CRUD data API’s are simple, until some level of complexity.
- userbinator 3y agoNot from a modern developer. You need to jump down the rabbit hole of the history of solid and proven solutions. Indeed. I've noticed an increasing number of people who call themselves "developers" and appear to create software, but all they can do is follow step-by-step tutorials to glue together some massively bloated thing that they have absolutely no understanding of, much less the skills to debug it when something doesn't work perfectly as expected.
- gofreddygo 3y ago> but all they can do is follow step-by-step tutorials Lego's dream has come true.
- keyle 3y agoAt some point, the plane flew so high above the clouds, and it hasn't seen the ground for so long, the passengers might think the clouds is the ground.
- turtleyacht 3y agoWhen "Hello World" is a cloud instance spun up and billed on your tuition. That won't really happen, right? No Computer Science (CS) professor would designate special cloud Interactive Development Environments (IDEs) to students. On the other hand, it would really resolve a lot of "doesn't work on my machine" issues.
- samarthr1 3y agoI think that would be a great way to make a weeder course? Setup some obscure software, with poor documentation, build something nice on top of it, then rewrite it because some fundamental assumption is broken, sounds like this kind of first year of cs course we should have. The idea being weeding out those not as passionate about computers.
- dllthomas 3y agoIt's been a while, but when I was in school I think things had to work on the campus ultrasparc systems.
- chrisshroba 3y agoWhat if the data is dynamic but small and low qps? What are the current easiest projects and services for authed data CRUD? Is firebase still good?
- firecall 3y agoI’ve used it a bit recently and it doesn’t seem terrible! But only for prototyping and testing the auth and database tools. Haven’t really made a decision on it vs rolling my own in Rails.
- dkh 3y agoSupabase[1] is a popular, open-source[2], and very modern alternative with a great free tier! [1] https://supabase.com/ https://supabase.com/ [2] https://github.com/supabase/supabase https://github.com/supabase/supabase
- villgax 3y agoWas this is a joke or something? You could have always done this with whatever framework you were using including Express, FastAPI, Flask & everything else....
- LVB 3y agoIt was a sarcastic reminder to KISS.
- oneeyedpigeon 3y agoYou could even do it without a framework altogether!
- Zaheer 3y agoThis is essentially how Levels.fyi operated for several years: https://www.levels.fyi/blog/scaling-to-millions-with-google-sheets.html https://www.levels.fyi/blog/scaling-to-millions-with-google-...
- firecall 3y agoI’ve got some JSON being served to an iOS App from the cheapest of hosting which is ~5 lines of PHP to get the data from MySQL and return JSON. It’s been running unattended for a decade….
- Alifatisk 3y agoThe good ol’ days
- deleted 3y ago[deleted]
- chii 3y agoAnd serving it over plain http is how you can get Man-In-the-Middled!
- marginalia_nu 3y agoTo be fair, you can get MITMd with HTTPS too. How do you think Cloudflare inspects web traffic?
- grokkedit 3y agonot sure that MITM definition applies here: cloudflare is serving the data, they're not in the middle, they're either alice or bob
- marginalia_nu 3y agoThey do host some data, but they delegate a lot of connections as well. One of their biggest products is bot mitigation, where they'll inspect incoming connections/traffic and pass them through if they don't seem to be bots. This is essentially MITM-as-a-service.
- chii 3y agoBut i think they terminate the SSL connection, and then re-establish a new one right? It's not like your backend server didnt know it's been MITM'ed. I suppose they should've named MITM as Surprise-MITM.
- marginalia_nu 3y agoIt's still a MITM, and a MITM for a good chunk of all the web's traffic. Given the Snowden revelations, it's pretty much a given they're snooping.
- intelVISA 3y agoPlease, don't gender Cloudflare they prefer 'CFitM'!
- etoxin 3y agoIt's pretty common for cicd pipelines to generate static json like this to track environment info.
- fiddlerwoaroof 3y agoThis technique of producing an “API” from directories of json generated by batch jobs and creative use of symlinks can be surprisingly effective. I wrote a simple program to transform a list of RSS feeds into a directory tree that I’ve been using for years as my feed reader API to great effect.
- captn3m0 3y agoWe do this at https://endoflife.date https://endoflife.date API, and it works quite well.
- jarofgreen 3y agoI really like endoflife! It's great for looking up dates with a consistent UI and also codenames - I could never learn that Debian Bullseye was 11 or all the other silly codenames so I look them up here. Thank you! We also do something similar on Datatig, which is a tool I'm writing for the general situation of crowdsourcing data in Git repos. It makes it easier for people to submit data and then creates usable data and handy websites. The static website output has an API which is just static JSON files. https://pypi.org/project/DataTig/ https://pypi.org/project/DataTig/ or https://datatig.readthedocs.io/en/latest/ https://datatig.readthedocs.io/en/latest/
- oneeyedpigeon 3y agoYeah, I've always tried to generate RSS feeds when needed rather than on-demand — it strikes me that the ratio of requests to updates must be so enormous that it doesn't make sense doing anything else.
- 0393134149 3y ago[flagged]
- mkleczek 3y agoKISS is good... until it isn't. At my work I struggle with the opposite: all problems are being squeezed into "let's put it into static JSON on the CDN" - which ends up with a complex custom JSON based language (schema) to support sharing information between apps, subsetting information (ie. search) etc - ie. implementing an ad-hoc one-file database. Ahh... and don't forget about complex CI/CD pipelines and Git setup that allow business users to manipulate data in these files. I found "just use Postgres for everything" a much saner default in the end.
- xlii 3y ago> don’t forget about complex CI/CD ... So it’s simple or complex because I got confused ;) KISS is ALWAYS good. But simplicity doesn’t mean easy or naïve solutions. Naïve solutions get very complex since the initial input is low and they don’t cover edge cases by design making it tangled mess. Simple solutions are often very difficult to create, require a lot of input to be simple, but they cover a lot of cases behind the curtains and are easy to describe. Your “use Postgres” strategy is more KISS than doing complex mocking. Sure, initial effort is higher, but using well known, powerful yet ergonomic singular entity is as simple as one can get on some levels. It’s equivalent of “just use Excel for it”. I agree with you, but I think tagline is misplaced.
- Freedom2 3y agoWhat is KISS? Apologies, I'm unfamiliar with some acronyms.
- justinclift 3y agohttps://en.wikipedia.org/wiki/KISS_principle https://en.wikipedia.org/wiki/KISS_principle
- deleted 3y ago[deleted]
- re-thc 3y ago[flagged]
- 3y ago
- kokanee 3y agoI do this when possible. Caching can be a pain to get right though. If you’re set up to do hashed filenames for JS and CSS already, you can leverage that for cache busting your json, and then you need to inject the hashed filename into the app somehow.
- 2143 3y agoI like your website. Nice and clean. Even the underlying HTML is clean. Well done.
- mholt 3y agocaddy file-server does this, but more production-ready, honors cache headers, sets Etag, etc. Also: caddy respond can be used to hard-code specific responses if you're testing an HTTP client. It can even spin up on a whole port range and supports templates in the response text right there on the command line: caddy respond --listen :2000-2004 "I'm server {{.N}} on port {{.Port}}" Here's an example maintenance page: cat maintenance.html | caddy respond \ --listen :80 \ --status 503 \ --header "Content-Type: text/html" https://caddyserver.com/docs/command-line https://caddyserver.com/docs/command-line
- Alifatisk 3y agoIs caddy that easy to setup and run?
- yjftsjthsd-h 3y agoI'd say "easy to setup and run" is arguably caddy's entire point:) Single binary file, yes running is that easy for simple stuff, and honestly it's not exactly complicated for more complex stuff. It got a lot of its popularity because it made HTTPS, including automatic cert provisioning, utterly trivial.
- iainmerrick 3y agoIt’s pretty easy, but I find the config file format rather confusing. In the classic Unix style, you’re learning yet another little language, so there’s a learning curve. Once you learn it, it’s fairly powerful and expressive, but the docs aren’t very clear.
- e12e 3y agoYes. Caddy excels at making simple [1] use-cases trivial - either with no config, or 3-5 lines of config. It might seem jarring comming from Apache or nginx - but i think a lot of it comes from the developers having destilled a lot of common use-cases and carefully chosen reasonable defaults. [1] Simple to articulate; not necessarily simple to execute - eg letsencrypt works out of the box. Caddy will give you https if you ask (or rather unless you go out of your way to avoid it). It can be a great stop-gap in running an ssl proxy in front of a legacy application server stuck on some horrible old distro, for example - since the binary just needs a Linux kernel (and can ignore old ssl libraries).
- praveen9920 3y agoI was working on content heavy web app few years ago and realised that lot of database reads are happening to very few "popular" posts then it hit me that we can just host that content as json files using Nginx without API getting involved. That worked like a charm. Decreased usage of database, decrease in latency. etc. In hindsight we could have just used CDN but for the shoestring budgets we had, we achieved lot of scalability with single nginx server.
- justinclift 3y agoA common approach in between these two is to use something like memcached (or redis) to cache the results of common requests. That also tends to provide results faster than the database layer, so better latency and less database load. It's not as fast as serving static content via nginx or a CDN though. :)
- Attummm 3y agoThe python server is not meant for production, serving files from it is incredibly slow and resource intensive. It also is not full webserver. There are better options for serving files
- brabel 3y ago[flagged]
- psnehanshu 3y agoOfcourse, this was just an example. One of the prod solutions can be Nginx.
- Attummm 3y agoAdding that to the blog post would strengthen the argument. We shouldn't assume the knowledge level of the reader. Examples become production all the time. Using files with unix philosophy is a tried and tested way to be really productive.
- jgaa 3y ago> You can serve static data over HTTP Yes. Just like you can put letters together and form words. It's kind of what it was designed to do.
- fredrikholm 3y agoYou'd be surprised talking to the younger crowd.
- FabHK 3y agoSome time ago, a website was posted here (forgot the topic) which consisted of .md files, and a JavaScript library which would then, on the client, render the markdown on the fly. Why on earth the markdown was not pushed through a renderer so that then the server could serve static HTML is probably a question that only occurs to old timers like us.
- Krutonium 3y agoThat specific case sounds like a case of "Because I can and I think it's neat" more than anything.
- Terretta 3y agohttps://caddyserver.com/docs/caddyfile/directives/templates https://caddyserver.com/docs/caddyfile/directives/templates https://github.com/caddyserver/website/blob/master/src/docs/markdown/caddyfile/directives/templates.md https://github.com/caddyserver/website/blob/master/src/docs/...
- oneeyedpigeon 3y agoI also initially misread the tone of the title. It's heavily dripping with sarcasm.
- vermon 3y agoI built and API for an internet radio station archive. Initially I tried to build an app but instead I realised I can run a scheduled Github Action that runs every hour, scrapes the archive, builds a Json file and deploys it to Github Pages for free: https://bd.maido.io/api.json https://bd.maido.io/api.json
- willwinger 3y agoHad a smile reading through this. So many web developers out there who have no idea about history of HTTP. That it was designed for static document serving and since been twisted to serve applications.
- Roark66 3y agoHere I was expecting this will be about not needing HTTPS for everything static...
- shinycode 3y agoThere is https://github.com/typicode/json-server https://github.com/typicode/json-server that is really useful for that matter
- SenHeng 3y agoFor a 'fake' API, I can't overstate just how powerful this is. I once spent an entire month building a proper API server out of this. Then one day while tweaking some DTOs for some POST endpoints, I realised what I was doing and switched to Postgres.
- pSYoniK 3y agoThis is great advice and its most of the time everything you need for a personal website. A generation that grew up with Wordpress et al. will have dificulties getting that really all there is to a website is some html. I switched from a Spring Boot web app that used Thymeleaf for the templating engine, with a MySQL db on a Linux vps and Heroku (I did it to learn/practice) to a Hugo blog that I publish locally then scp to a VPS with Caddy setup to serve the static content. Nothing to hack/maintain/worry about. As an extra point I would even complete the suggestion with adding direct HTTP not S, support. Your page will load even faster and you're transmitting some static content anyway.
- bladelessninja2 3y agoI hope the author means HTTPS, not HTTP - otherwise, I don't think this is a good idea. Anyway, as everything in software development: it all depends - serving a simple JSON over an existing "full blown web application back-end" is usually more convenient and less confusing for developers than configuring hosting for static content. Although, if you JUST want to serve a JSON then, yeah, you can add it as a static resource.
- edwinjm 3y ago> It's not illegal to use static files and it does work. No, it doesn't. Maybe for your personal homepage with three hits per month, but even on a low traffic server, static files gave me headaches. 1) When two visitors write to the file at the same time, the data of one of the visitors will be gone. This can be solved by using lock files. This is a topic on its own (what happens when two processes both create a lockfile at the same time?). Also, when there are a lot of writes happening, multiple processes will wait for the lockfiles, creating load on the server and wait times for the visitor. 2) When a file is read during while another process is doing a write, the file is only read partially. There are workarounds for this as well, you can first write to another file and when done, move this other file to the correct file. As the number of visitor increases or the file size increases, problems will get harder. Nowadays it's not hard to add a database. Whether it's on your server or in the cloud. It's performant, it works and it will grow with increasing demand. It's probably easier to set up than a file based solution. Sometimes what seems to be simple isn't really.
- iofiiiiiiiii 3y agoSounds like your files are not static, therefore your problem is not the same as the problem the article is addressing.
- pjc50 3y agoIf there's writing going on, you're not using static files.
- edwinjm 3y agoThen I'm mistaken.
- KomoD 3y agoDoesn't sound static to me.
- ygouzerh 3y agoFor mocks, it's a good idea. For the rest, even for a prototype, it seems a lot of trouble to insert, update and delete data.
- nulbyte 3y agoActually, text editors are pretty good these days. But on a(n only slightly) more serious note, maybe the data is generated outside of the web server. Different projects have different needs, but there are plenty of them that just need to serve some mostly static data.
- ThatPlayer 3y agoI've taken this to an extreme myself with 10MB+ JSON files. I would recommend enabling gzip compression, but also not letting it get to that point. I'm considering switching to sqlite as that has been ported to WASM and allow pages
- politelemon 3y agoAlong similar lines. If you're starting out with feature toggles for a web application, in many cases you don't need a separate third party application that manages the flags for you. You can just serve a .json file.
- mpweiher 3y agoYes, so much in programming is so much simpler than we think it is. I first came across this when creating a demo for a smartpen that was supposed to have live interactivity features, which I was going to implement by putting a little HTTP server on the pen. When it came to simulating my solution, I took away features until I came up with the "serving static files over HTTP" trick the post describes. Then I realised that I could do the whole thing by just putting the demo files in a directory and changing the base URL in the client from https:// https://... to file:///... Another time we built a backend system for feed processing and delayed putting in a database until we discovered we didn't actually need it.[1]. The standard operating procedure at the time was to create an Oracle DB for everything, including a system that needed a total of 6 data items. Wow. My boss at Wunderlist used to brag that the database we used for the iOS/macOS clients was just a bunch of JSON files in the filesystem. Worked wonders and was really, really fast. These and many other experiences led me to come up with Storage Combinators[2]. Once you try them, you'll wonder why things always have to be so complicated. And yes, I agree with the idea that "gluing complex libraries together that we don't fully understand" is a problem. And also with the notion that it's an achievement that we can and should be proud of, at least partially, because that used to be something we couldn't do at all[3]. Nowadays we can at least do it badly[4]. My take on this is that we don't have the right kinds of glue, particularly procedural/functional abstraction is often no the right glue, and OOP notwithstanding that's really all we got. [5][6] [1] https://link.springer.com/chapter/10.1007/978-1-4614-9299-3_11 https://link.springer.com/chapter/10.1007/978-1-4614-9299-3_... [2] https://2019.splashcon.org/details/splash-2019-Onward-papers/7/Storage-Combinators https://2019.splashcon.org/details/splash-2019-Onward-papers... [3] https://blog.metaobject.com/2021/07/glue-code-is-success-condition.html https://blog.metaobject.com/2021/07/glue-code-is-success-con... [4] https://blog.metaobject.com/2021/06/glue-dark-matter-of-software.html https://blog.metaobject.com/2021/06/glue-dark-matter-of-soft... [5] https://blog.metaobject.com/2019/02/why-architecture-oriented-programming.html https://blog.metaobject.com/2019/02/why-architecture-oriente... [6] https://objective.st https://objective.st
- bhanu423 3y agoI usually just use `npx http-server` which gives a server that supports concurrent streams for faster downloads over the python server(which I use on devices where python is present and node isn't). I often use it transfer file over wifi from a device to other.
- billyhoffman 3y agoReminds me of a funny take from @marginalia: The Static File Startup: > We’re going to disrupt Internet hosting and serve static files in a novel way through a cloud based SaaS. https://www.marginalia.nu/log/51-the-file-startup/ https://www.marginalia.nu/log/51-the-file-startup/
- marginalia_nu 3y agoLooks like first few lines the post got a bit mangled when I switched blog systems :-/
- andrewmcwatters 3y agoIt’s funny but it hurts. I read the whole thing amused but with disdain on my face.
- dolmen 3y agoContent-Type?
- intelVISA 3y agothe heck's an octet-stream?
- slim 3y agoThis is called PCM (path controller mapping), it's very new, I'm going to give a talk about it at next aws summit /s