4 ms·
I was actually playing around those files today. I know this will be a typical HN comment and I'm sorry but, if it takes you an hour to load that data into sqli
by fallingsquirrel 2y ago
I was actually playing around those files today. I know this will be a typical HN comment and I'm sorry but, if it takes you an hour to load that data into sqlite, you're probably doing something very wrong.
$ sqlite3 x.db 'create table foo(a,b,c)'
$ time for year in {1880..2023}; do
sqlite3 x.db '.import yob'$year'.txt foo --csv';
done
real 0m1.359s
user 0m1.013s
sys 0m0.323s
$ sqlite3 x.db 'select count(*) from foo'
2117219
(Numbers are from Linux/tmpfs/i9-12900k)
Obviously that needs tweaked to store the year in a column, but it shouldn't run for more than a few seconds. Those csv files are the furthest thing from "big data".
- runnr_az 2y agoRight on. Outta curiosity, whatcha up to with the name data? Fair. I'm sure there are waaaay more efficient ways to load that into the db. That said, it's one of those things - like, where do I wanna spend my time -- sorting that out, something I only gotta do once a year and I can basically push "go" and walk away -- or something else. It's not like I gotta keep turning the crank. ... and, of course, PRs are welcome :)
- fallingsquirrel 2y agoFair enough on the perf, I get it lol I'm hunting for names that had resurgence in popularity because of TV shows/movies. The name that inspired the project is Mabel. It disappeared in the 1960s, then reappeared the year after Gravity Falls came out and it's still climbing. https://www.behindthename.com/name/mabel/top https://www.behindthename.com/name/mabel/top I want to find as many examples of that as I can.
- runnr_az 2y agoInteresting. So are you just looking at names with a dramatic slope / climb after a given year, then trying to figure out why?