2 ms·

Something I've always wondered: when you sign up for a proprietary "intelligence management" platform like Palantir Gotham, do they give you access to a bunch o

by derefr 4y ago

Something I've always wondered: when you sign up for a proprietary "intelligence management" platform like Palantir Gotham, do they give you access to a bunch of OSINT datasets they scrape and manage themselves, to combine with your proprietary intelligence dataset for enhanced understanding? Sort of like how BigQuery has "BigQuery public datasets"? If not, seems like a missed opportunity for value-add / lock-in. And if they do provide such data, it'd be nice if one such platform would make either the scrapers, or the scraped data, open-source. It's all just cleaned+normalized forms of already-public data, after all. (Like how the Yellow Pages could always be seen as a cleaned+normalized form of phone numbers listed in public view on shop windows.) This is on my mind recently after noticing that there's actually no public dataset of "all WHOIS data for every domain on the Internet", despite WHOIS data being something explicitly designed for public querying. Almost certainly, intelligence agencies compile such a dataset themselves in order to fill in some gaps in cybercrime network-analysis graphs. Almost certainly multiple of them do, such that the number of crawlers doing it probably hurts the WHOIS services. Almost certainly the WHOIS services would be better off partnering with one of them to act as an "official crawler" to build such an index for everyone's use.

schroeding 4y ago
A public (historical) WHOIS dataset is a nightmare due to GDPR et al., though, isn't it? You couldn't opt-out of WHOIS (some TLDs just didn't allow for WHOIS privacy / proxy registrations) in the past, and it's IMO quite a hard sell that the public interest in the historic WHOIS data of some personal website should have any weight at all in this case.
- posterboy 4y ago
  That's an interesting way to put it. There is some interest, no doubt, and there is some potential abusing that information. Since the information was committed knowingly and willingly to the public domain, though I do not know much about it to begin with, I don't see how GDPR applies at all.
  - schroeding 4y ago
    The GDPR allows you to force companies to delete data that was knowingly and willingly commited, though, via article 21 and 17c GDPR [1], except if there are "overriding legitimate grounds for the processing" of the data. I doubt there are, at least for low-profile, personal websites and historical WHOIS data. You could IMO also make an argument that the collection and scraping of WHOIS data without consent was unlawful processing as a whole, as the data was not provided to the domain NIC with explicit consent for third-parties to collect and save them forever, which would even make any overriding legitimate grounds for processing moot. 17a may also apply: The reason for WHOIS is to find out who owns a domain now, right? There is no version history. Historical WHOIS data does not serve this purpose anymore. It's quite the can of worms. [1] https://gdpr-text.com/de/read/article-17/ https://gdpr-text.com/de/read/article-17/
jpleger 4y ago
Some do, some don't. The big thing that Palantir provides you is a structured analysis platform more than anything. The idea is that your intel analysts are doing the same thing and structuring their artifacts in a consistent way. It isn't so much about pulling in feeds (though obviously most pull these in), but as structured process. On something like whois, there are commercial vendors that provide history/change reporting, but its a pain in the butt to parse/normalize whois records and that is before you even get to the fact that many providers have shut down their whois feeds "for gdpr compliance".
hammock 4y ago
> when you sign up for a proprietary "intelligence management" platform like Palantir Gotham, do they give you access to a bunch of OSINT datasets they scrape and manage themselves, to combine with your proprietary intelligence dataset for enhanced understanding? I don’t know about OSINT, but in the consumer data landscape (Acxiom, Datalogix, etc) this is the standard MO
deleted 4y ago
[deleted]

by derefr 4y ago

schroeding 4y ago

A public (historical) WHOIS dataset is a nightmare due to GDPR et al., though, isn't it? You couldn't opt-out of WHOIS (some TLDs just didn't allow for WHOIS privacy / proxy registrations) in the past, and it's IMO quite a hard sell that the public interest in the historic WHOIS data of some personal website should have any weight at all in this case.

posterboy 4y ago
That's an interesting way to put it. There is some interest, no doubt, and there is some potential abusing that information. Since the information was committed knowingly and willingly to the public domain, though I do not know much about it to begin with, I don't see how GDPR applies at all.
- schroeding 4y ago
  The GDPR allows you to force companies to delete data that was knowingly and willingly commited, though, via article 21 and 17c GDPR [1], except if there are "overriding legitimate grounds for the processing" of the data. I doubt there are, at least for low-profile, personal websites and historical WHOIS data. You could IMO also make an argument that the collection and scraping of WHOIS data without consent was unlawful processing as a whole, as the data was not provided to the domain NIC with explicit consent for third-parties to collect and save them forever, which would even make any overriding legitimate grounds for processing moot. 17a may also apply: The reason for WHOIS is to find out who owns a domain now, right? There is no version history. Historical WHOIS data does not serve this purpose anymore. It's quite the can of worms. [1] https://gdpr-text.com/de/read/article-17/ https://gdpr-text.com/de/read/article-17/

jpleger 4y ago

Some do, some don't. The big thing that Palantir provides you is a structured analysis platform more than anything. The idea is that your intel analysts are doing the same thing and structuring their artifacts in a consistent way. It isn't so much about pulling in feeds (though obviously most pull these in), but as structured process. On something like whois, there are commercial vendors that provide history/change reporting, but its a pain in the butt to parse/normalize whois records and that is before you even get to the fact that many providers have shut down their whois feeds "for gdpr compliance".

hammock 4y ago

> when you sign up for a proprietary "intelligence management" platform like Palantir Gotham, do they give you access to a bunch of OSINT datasets they scrape and manage themselves, to combine with your proprietary intelligence dataset for enhanced understanding? I don’t know about OSINT, but in the consumer data landscape (Acxiom, Datalogix, etc) this is the standard MO

deleted 4y ago

[deleted]