4 ms·
To scrape, start here http://metmuseum.org/api/collection/collectionlisting?offset=0&pageSize=0&perPage=100&sortBy=Relevance&sortOrder=asc http://metmuseum.org
by gregsadetsky 10y ago
To scrape, start here
http://metmuseum.org/api/collection/collectionlisting?offset=0&pageSize=0&perPage=100&sortBy=Relevance&sortOrder=asc http://metmuseum.org/api/collection/collectionlisting?offset...
and increase 'offset' by 100. The JSON output contains image URLs. The total number of results is 441048, so finding another endpoint that doesn't enforce a limit of 100 on the 'perPage' argument would be great.
---
EDIT: thanks to spitfare, I updated the perPage argument to 100. The site doesn't allow larger values, but that's definitely a great start! (about 4k requests to get everything)
- deleted 10y ago[deleted]
- scosman 10y agoThey published an index on Github. It would likely be a more responsible way to access this versus hammering their API. https://github.com/metmuseum/openaccess https://github.com/metmuseum/openaccess
- gregsadetsky 10y agoUnfortunately, as stated in that repo's readme: `Images are not included and are not part of the dataset.` The repo doesn't include the images, which is understandable; however, the CSV file doesn't even include links to the images.
- kjell 10y agoExcluding the images themselves from the dataset is one thing, but there's an open issue to include a link to the images: https://github.com/metmuseum/openaccess/issues/2 https://github.com/metmuseum/openaccess/issues/2
- DenisM 10y agoAre there any docs for this API? I wish I could filter by artist...
- lostmsu 10y agoIf you finish scraping, can you still make a .torrent?
- awqrre 10y agoedit: it doesn't download the highest available resolution... This Perl script should get most of the images (but it would probably be better to save them in the original sub-directories): use strict; use warnings; use LWP::Simple; use JSON qw(from_json); my $url = "http://metmuseum.org/api/collection/collectionlisting?offset="; my $args = "&pageSize=0&perPage=100&sortBy=Relevance&sortOrder=asc"; my $offset = 0; while ($offset < 5000) { my $decoded = from_json(get($url.$offset.$args)); $offset = $offset + 100; my @results = @{ $decoded->{'results'} }; foreach my $i ( @results ) { my $filename = $i->{"largeImage"}; my $title = $i->{"title"}; print "Status: ".getstore("http://images.metmuseum.org/CRDImages/".$filename,$title.".jpg")." ".$filename. "\n"; } }