4 ms·

They're using SQLite which is single threaded and single user by design. So this website will not be able to service that much traffic.

tyingq 6y ago
Should be fine for reads. Sqlite.org is dynamic, pulling from sqlite data for ~20% of the pages, and it does fine with HN piling on. The single threaded would be an issue for writes, but I don't see why they would be doing writes for a page view. See https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html
- pfraze 6y ago
  Also, caching layers. Presumably a service uses SQLite to simplify their ops. So long as the caching layer is equally simple to maintain, then SQLite continues to make sense to me
  - tyingq 6y ago
    Their repo shows the "DummyCache" (no caching) as the default for the Python/Django setup: https://github.com/cmu-db/dbdb.io/blob/master/dbdb/settings.py#L103 https://github.com/cmu-db/dbdb.io/blob/master/dbdb/settings.... I wonder if that's how it is in production.
    - kevincox 6y ago
      For a mostly static site you would want to cache "over" Django, probably in a caching proxy or CDN. Of course there are a lot of details, such as many CDNs will always go to the origin on a edge miss instead of locating another copy in the CDN. Of course running a caching nginx on the same box as the Django is probably way more performant than caching "under" Django
      - tyingq 6y ago
        It's intermittently working now. Shows the server as Apache 2.4.18 / Ubuntu. Apache can page cache, but I assume they aren't using it. I don't see any typical CDN headers either. Also, somewhat odd, they are bounding the browser-side cache to 10 minutes: Date: Wed, 16 Sep 2020 18:19:13 GMT Expires: Wed, 16 Sep 2020 18:29:14 GMT
  - tbran 6y ago
    For this kind of content site, I'd convert the Django site to a static site with django-bakery [1] and stick it on Netlify. I suppose there would be a bunch of rewriting to do searching (lunr.js maybe) and filtering client side. Buuuuuut, at this point, just turning on caching would be easier. [1]: https://github.com/datadesk/django-bakery https://github.com/datadesk/django-bakery
- remram 6y ago
  Well it looks down now.
- ehsankia 6y ago
  Well it clearly wasn't :) Site is down already
  - tyingq 6y ago
    There's several layers. It's Python/Django. I don't think we know what the issue really is. They could be, for example, logging visits to Sqlite. Or other issues unrelated to Sqlite.
goostavos 6y ago
I've survived a few HN front pages on SQlite + $5 DO droplet. Don't disparage my primary tech stack like this ^_^ The traffic from a HN frontpage is relatively low per second tbh.
- deleted 6y ago
  [deleted]
simonw 6y ago
SQLite isn't single threaded - and it can support multiple readers very well. The limitation with SQLite is that it doesn't support concurrent writes well - it needs to take a lock on the entire database to perform a write. Writes are crazy fast (a few ms) so this often isn't a problem - but it does mean you wouldn't want to use it to build a site that has many people writing at once, like Hacker News for example. For a site that has low (or no) writes, SQLite works really well even at a much larger scale - 100s of requests a second.
- the_duke 6y ago
  Enabling the write ahead log makes sqlite behaviour much, much better under (write) contention: > PRAGMA journal_mode=WAL;
  - simonw 6y ago
    Yeah I was getting occasional "database is locked" read errors on a project that had crons writing to the SQLite file which I solved by switching on WAL mode. It still doesn't let you have concurrent writes but it does mean that reads won't error if a write is going on at the same time.
bob1029 6y ago
There is zero reason a carefully engineered application utilizing SQLite cannot completely saturate the IO capabilities of the host it resides on. Going even further, there are no competing technologies (i.e. hosted SQL solutions) which, when running in single node/instance mode, are competitive with the performance of well-tuned SQLite. PRAGMA journal_mode=WAL makes all the difference in the universe.
kreetx 6y ago
.. and it does appear to be very slow at the moment.

tyingq 6y ago

Should be fine for reads. Sqlite.org is dynamic, pulling from sqlite data for ~20% of the pages, and it does fine with HN piling on. The single threaded would be an issue for writes, but I don't see why they would be doing writes for a page view. See https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html

pfraze 6y ago
Also, caching layers. Presumably a service uses SQLite to simplify their ops. So long as the caching layer is equally simple to maintain, then SQLite continues to make sense to me
- tyingq 6y ago
  Their repo shows the "DummyCache" (no caching) as the default for the Python/Django setup: https://github.com/cmu-db/dbdb.io/blob/master/dbdb/settings.py#L103 https://github.com/cmu-db/dbdb.io/blob/master/dbdb/settings.... I wonder if that's how it is in production.
  - kevincox 6y ago
    For a mostly static site you would want to cache "over" Django, probably in a caching proxy or CDN. Of course there are a lot of details, such as many CDNs will always go to the origin on a edge miss instead of locating another copy in the CDN. Of course running a caching nginx on the same box as the Django is probably way more performant than caching "under" Django
    - tyingq 6y ago
      It's intermittently working now. Shows the server as Apache 2.4.18 / Ubuntu. Apache can page cache, but I assume they aren't using it. I don't see any typical CDN headers either. Also, somewhat odd, they are bounding the browser-side cache to 10 minutes: Date: Wed, 16 Sep 2020 18:19:13 GMT Expires: Wed, 16 Sep 2020 18:29:14 GMT
- tbran 6y ago
  For this kind of content site, I'd convert the Django site to a static site with django-bakery [1] and stick it on Netlify. I suppose there would be a bunch of rewriting to do searching (lunr.js maybe) and filtering client side. Buuuuuut, at this point, just turning on caching would be easier. [1]: https://github.com/datadesk/django-bakery https://github.com/datadesk/django-bakery
remram 6y ago
Well it looks down now.
ehsankia 6y ago
Well it clearly wasn't :) Site is down already
- tyingq 6y ago
  There's several layers. It's Python/Django. I don't think we know what the issue really is. They could be, for example, logging visits to Sqlite. Or other issues unrelated to Sqlite.

goostavos 6y ago

I've survived a few HN front pages on SQlite + $5 DO droplet. Don't disparage my primary tech stack like this ^_^ The traffic from a HN frontpage is relatively low per second tbh.

deleted 6y ago
[deleted]

simonw 6y ago

SQLite isn't single threaded - and it can support multiple readers very well. The limitation with SQLite is that it doesn't support concurrent writes well - it needs to take a lock on the entire database to perform a write. Writes are crazy fast (a few ms) so this often isn't a problem - but it does mean you wouldn't want to use it to build a site that has many people writing at once, like Hacker News for example. For a site that has low (or no) writes, SQLite works really well even at a much larger scale - 100s of requests a second.

the_duke 6y ago
Enabling the write ahead log makes sqlite behaviour much, much better under (write) contention: > PRAGMA journal_mode=WAL;
- simonw 6y ago
  Yeah I was getting occasional "database is locked" read errors on a project that had crons writing to the SQLite file which I solved by switching on WAL mode. It still doesn't let you have concurrent writes but it does mean that reads won't error if a write is going on at the same time.

bob1029 6y ago

There is zero reason a carefully engineered application utilizing SQLite cannot completely saturate the IO capabilities of the host it resides on. Going even further, there are no competing technologies (i.e. hosted SQL solutions) which, when running in single node/instance mode, are competitive with the performance of well-tuned SQLite. PRAGMA journal_mode=WAL makes all the difference in the universe.

kreetx 6y ago

.. and it does appear to be very slow at the moment.