3 ms·

This is what I've assumed, but it's in a pretty uncomfortable place right now as e.g. the documentation often refers to export credits with a broken link. The

This is what I've assumed, but it's in a pretty uncomfortable place right now as e.g. the documentation often refers to export credits with a broken link. The API is somewhat unsuitable for exporting large volumes because it seems remarkably unstable as to ordering, it suggests that you can do paginated requests but the second page tends to have 30% overlap with the first page. I 100% understand the product motive to move large exports to an "Enterprise" feature, but it's rather disappointing because as a small-scale independent operation I don't expect to be able to afford it, and that would go for a lot of productive people in security research. But then, that's capitalism.

achillean 6y ago
I decided that a broken link is better than having people spend money on something that will be deprecated. We're obviously working on cleaning up those broken links but it's an easy thing to explain if anybody emails support@shodan.io The ordering is based on timestamp and it can happen that new results were indexed in between your 1st request and 2nd request which creates an overlapping result. A 30% overlap is unusual and sounds like it's for a query with many results. Finally, most researchers don't actually need to download data. They could just use our free API and facet queries to get the information without downloading the actual data. This entire website is powered by a free API key that uses facets: https://exposure.shodan.io/#/ https://exposure.shodan.io/#/ I think a lot of researchers go into the default mode of "I want to have the data" but using facets is way easier, faster and doesn't cost any money at all. And you can navigate the available facets using our new beta website (another area we're trying to make things a bit clearer). For example: https://beta.shodan.io/search/facet?query=http&facet=http.component https://beta.shodan.io/search/facet?query=http&facet=http.co... Note that we provide free upgrades to universities/ students/ professors as well as routinely work together with researchers so we're not trying to push them into the enterprise product. We also let universities monitor up to ~120k IPs for free using Shodan Monitor (https://monitor.shodan.io https://monitor.shodan.io). But the use cases for researchers are few and we figure that if you need lots of data then you can send us an email.

achillean 6y ago

I decided that a broken link is better than having people spend money on something that will be deprecated. We're obviously working on cleaning up those broken links but it's an easy thing to explain if anybody emails support@shodan.io The ordering is based on timestamp and it can happen that new results were indexed in between your 1st request and 2nd request which creates an overlapping result. A 30% overlap is unusual and sounds like it's for a query with many results. Finally, most researchers don't actually need to download data. They could just use our free API and facet queries to get the information without downloading the actual data. This entire website is powered by a free API key that uses facets: https://exposure.shodan.io/#/ https://exposure.shodan.io/#/ I think a lot of researchers go into the default mode of "I want to have the data" but using facets is way easier, faster and doesn't cost any money at all. And you can navigate the available facets using our new beta website (another area we're trying to make things a bit clearer). For example: https://beta.shodan.io/search/facet?query=http&facet=http.component https://beta.shodan.io/search/facet?query=http&facet=http.co... Note that we provide free upgrades to universities/ students/ professors as well as routinely work together with researchers so we're not trying to push them into the enterprise product. We also let universities monitor up to ~120k IPs for free using Shodan Monitor (https://monitor.shodan.io https://monitor.shodan.io). But the use cases for researchers are few and we figure that if you need lots of data then you can send us an email.