3 ms·

It's slow on the first load as you download a 100mb parquet file with all the IMDb data. It'll be cached on the second load though (using browser's indexeddb to

It's slow on the first load as you download a 100mb parquet file with all the IMDb data. It'll be cached on the second load though (using browser's indexeddb to cache). Not sure what you mean by scraping in parallel.

cranberryturkey 2y ago
oh i thought you were convert html to sql. nevermind.
maleldil 2y ago
100 Mb is a lot of data to download. Maybe you should be upfront about it and, hopefully, require some user input before you do it.
netsharc 2y ago
Groan... Are you hosting this 100MB (and not mb)? If yes, I hope the bill will teach a lesson. If not, 100 visitors is already 10GB of bandwidth, that's quite abusive. Maybe I'm just an old fart, and blowing 10GB of data is cheap nowadays?
- Jach 2y ago
  OVH and Hetzner offer cheap unmetered VPS instances, Cloudflare CDN doesn't have any bandwidth restrictions... Yes 10GB of data is cheap. If they were serving off EC2, then after the first 100 GB of the month it's 9 cents/GB or 90 cents per 10GB, which is absurd, but still not terribly expensive in the grand scheme of things: what would be a surprise lesson-teaching bill, $90 after a TB / 10,000 visitors? I'm surprised some users think it's a lot on an individual scale. Consider an ideal persona for this submission: query some movies, then go stream several GBs for one. The 100 MB payload isn't much in comparison. I admit it is kind of bad form for mobile users who might be on metered data plans, and a warning and trigger for manual action for those devices would be kinder. One thing that might help reset your notion of what a lot of bandwidth is would be to browse around with your network tab in the developer console open for a day. nytimes home page is 14 MB and they get a ton of traffic. Even a corporate blog on the HN front page right now that could just be the tiny compressed text is 2 MB. Single image loads on many pages can be 1 MB or more. Glancing at the submission, the response headers indicate it seems they're serving from S3 Cloudfront, which is free for the first TB per month, though after that it gets back to absurd pricing. AWS is not price competitive.
jafitc 2y ago
I think you should consider trimming that file. Exclude movies with very low number of rating or potentially very low scores too. The long tail reduction would be significant
- afiodorov 2y ago
  I initially loved looking for obscure stuff, e.g. setting region to soviet union. It surely is the case that 99% of the users want 10% of the data at most. I'll have to work ability to select the file and download & cache it only if the relevant query is asking for it.
thimabi 2y ago
I have never imagined that simply by using your app it would download and cache a 100 MB file on my phone. If I hadn’t read the discussion here, I wouldn’t even know about it. That’s an irresponsible waste of both bandwidth and storage. You should have really made that clear in the website.
- afiodorov 2y ago
  A 100 mb is not much - about 2 mins on youtube
smusamashah 2y ago
Is that data file also in your github repo or is it a third party source?
- afiodorov 2y ago
  The data was produced using this script: https://github.com/afiodorov/imdb-sql/blob/main/notebooks/imdb.ipynb https://github.com/afiodorov/imdb-sql/blob/main/notebooks/im... It is on hosted on http://www.imdb-sql.com/imdb01-11-2024.parquet http://www.imdb-sql.com/imdb01-11-2024.parquet In fact the main reason this project exists is a) I wrote a jupyter notebook ages ago that'd join the raw data into a queryable form. b) I committed it and forgot about it after the initial viewing of top movies/series. c) For the Halloween I wanted to find a good-rated horror movie, a genre I don't watch much. d) I found my notebook but it was a drag to get it running again, first pandas would keep throwing OOM errors so I had to migrate to polars. Secondly I had to find a spare working laptop since iPad is my primary off-work computing device as of late. Finally the schema is not so intuitive - I need to look up things like region/language codes. I want something I can just leave unattended but still be able to use a few months later without going through a dev-env set-up. A static website with all the data is one potential way of avoiding maintaining it as it should keep running, just like an exe would keep running on Windows in the past. I avoid maintaining personal projects as my software job requires me to maintain software and I don't find it enjoyable to do the same in my spare time.

cranberryturkey 2y ago

oh i thought you were convert html to sql. nevermind.

maleldil 2y ago

100 Mb is a lot of data to download. Maybe you should be upfront about it and, hopefully, require some user input before you do it.

netsharc 2y ago

Groan... Are you hosting this 100MB (and not mb)? If yes, I hope the bill will teach a lesson. If not, 100 visitors is already 10GB of bandwidth, that's quite abusive. Maybe I'm just an old fart, and blowing 10GB of data is cheap nowadays?

Jach 2y ago
OVH and Hetzner offer cheap unmetered VPS instances, Cloudflare CDN doesn't have any bandwidth restrictions... Yes 10GB of data is cheap. If they were serving off EC2, then after the first 100 GB of the month it's 9 cents/GB or 90 cents per 10GB, which is absurd, but still not terribly expensive in the grand scheme of things: what would be a surprise lesson-teaching bill, $90 after a TB / 10,000 visitors? I'm surprised some users think it's a lot on an individual scale. Consider an ideal persona for this submission: query some movies, then go stream several GBs for one. The 100 MB payload isn't much in comparison. I admit it is kind of bad form for mobile users who might be on metered data plans, and a warning and trigger for manual action for those devices would be kinder. One thing that might help reset your notion of what a lot of bandwidth is would be to browse around with your network tab in the developer console open for a day. nytimes home page is 14 MB and they get a ton of traffic. Even a corporate blog on the HN front page right now that could just be the tiny compressed text is 2 MB. Single image loads on many pages can be 1 MB or more. Glancing at the submission, the response headers indicate it seems they're serving from S3 Cloudfront, which is free for the first TB per month, though after that it gets back to absurd pricing. AWS is not price competitive.

jafitc 2y ago

I think you should consider trimming that file. Exclude movies with very low number of rating or potentially very low scores too. The long tail reduction would be significant

afiodorov 2y ago
I initially loved looking for obscure stuff, e.g. setting region to soviet union. It surely is the case that 99% of the users want 10% of the data at most. I'll have to work ability to select the file and download & cache it only if the relevant query is asking for it.

thimabi 2y ago

I have never imagined that simply by using your app it would download and cache a 100 MB file on my phone. If I hadn’t read the discussion here, I wouldn’t even know about it. That’s an irresponsible waste of both bandwidth and storage. You should have really made that clear in the website.

afiodorov 2y ago
A 100 mb is not much - about 2 mins on youtube

smusamashah 2y ago

Is that data file also in your github repo or is it a third party source?

afiodorov 2y ago
The data was produced using this script: https://github.com/afiodorov/imdb-sql/blob/main/notebooks/imdb.ipynb https://github.com/afiodorov/imdb-sql/blob/main/notebooks/im... It is on hosted on http://www.imdb-sql.com/imdb01-11-2024.parquet http://www.imdb-sql.com/imdb01-11-2024.parquet In fact the main reason this project exists is a) I wrote a jupyter notebook ages ago that'd join the raw data into a queryable form. b) I committed it and forgot about it after the initial viewing of top movies/series. c) For the Halloween I wanted to find a good-rated horror movie, a genre I don't watch much. d) I found my notebook but it was a drag to get it running again, first pandas would keep throwing OOM errors so I had to migrate to polars. Secondly I had to find a spare working laptop since iPad is my primary off-work computing device as of late. Finally the schema is not so intuitive - I need to look up things like region/language codes. I want something I can just leave unattended but still be able to use a few months later without going through a dev-env set-up. A static website with all the data is one potential way of avoiding maintaining it as it should keep running, just like an exe would keep running on Windows in the past. I avoid maintaining personal projects as my software job requires me to maintain software and I don't find it enjoyable to do the same in my spare time.