4 ms·

There is a github of the factbook for anyone that just wants JSON or markdown files:=> https://github.com/factbook https://github.com/factbook "A cache for dat

There is a github of the factbook for anyone that just wants JSON or markdown files:=> https://github.com/factbook https://github.com/factbook "A cache for datasets for the country profiles from the World Factbook in the original (1:1) format from the cia.gov website" https://github.com/factbook/cache.factbook.json https://github.com/factbook/cache.factbook.json

MilkMp 8mo ago
Hi there, thanks for linking this! My GitHub and website both link to and use this source! I just thought putting it in a SQL database and making the entire 1990-2025 queryable was needed since I couldn't find one anywhere :)
- genewitch 8mo ago
  it is a lot of fun and rewarding to do this! I've done it several times for medium-sized datasets, like wikipedia dumps, the entire geospatial dataset to mapreduce it (pgsql). The wikipedia one was great, i had it set up to query things like "show me all ammunition manufactured after 1950 that is between .30 and .40" and it could just return it nearly instantly. The wikimedia dumps keep the infoboxes and relations intact, so you can do queries like this easily.
  - 3eb7988a1663 8mo ago
    Do you have a write-up of this somewhere? When I last looked at the Wikipedia dumps, they looked like a mess to parse. How were you getting structured information?
    - genewitch 8mo ago
      unfortunately, i consider it proprietary.
    - iamacyborg 8mo ago
      You'd presumably have to run some part of the transclusion pipeline to properly handle template/module/page transclusion.
sovietmudkipz 8mo ago
Ahhhh yes thank you for this link! I'm working on a project and my agent referred me to the factbook. It said all this great savory information was made available on the factbook website. I thought great but when I went to check it was all very basic. Truly I thought my agent was hallucinating! It wasn't! Note to self: also check archive.org in case there is an internet archive for any sites an agent might reference. I checked out this repo -- it has the information useful to my project. Thanks for sharing this!

MilkMp 8mo ago

Hi there, thanks for linking this! My GitHub and website both link to and use this source! I just thought putting it in a SQL database and making the entire 1990-2025 queryable was needed since I couldn't find one anywhere :)

genewitch 8mo ago
it is a lot of fun and rewarding to do this! I've done it several times for medium-sized datasets, like wikipedia dumps, the entire geospatial dataset to mapreduce it (pgsql). The wikipedia one was great, i had it set up to query things like "show me all ammunition manufactured after 1950 that is between .30 and .40" and it could just return it nearly instantly. The wikimedia dumps keep the infoboxes and relations intact, so you can do queries like this easily.
- 3eb7988a1663 8mo ago
  Do you have a write-up of this somewhere? When I last looked at the Wikipedia dumps, they looked like a mess to parse. How were you getting structured information?
  - genewitch 8mo ago
    unfortunately, i consider it proprietary.
  - iamacyborg 8mo ago
    You'd presumably have to run some part of the transclusion pipeline to properly handle template/module/page transclusion.

sovietmudkipz 8mo ago

Ahhhh yes thank you for this link! I'm working on a project and my agent referred me to the factbook. It said all this great savory information was made available on the factbook website. I thought great but when I went to check it was all very basic. Truly I thought my agent was hallucinating! It wasn't! Note to self: also check archive.org in case there is an internet archive for any sites an agent might reference. I checked out this repo -- it has the information useful to my project. Thanks for sharing this!