4 ms·

How Palantir pushed into policing

78 points by mcone 9y ago

randcraw 9y ago
Nice article. Palantir's practices seem to follow Thiel's master plan for world domination to a T -- establish monopoly then protect it by any means necessary: obfuscation, proprietary APIs, overly complex user interfaces, and finally claims of sensitive secret company technology that must be kept secret or risk compromise, perhaps causing harm to the US military and lowering our defenses against terrorism. What a crock. There's little doubt that 99% of Palantir's infrastructure is commodity open source, trivial to access, monitor, and replace. It's absurd that any customer would pay a premium for proprietary hardware and system support, much less be told that the system disallows even the shallowest oversight of the company's data and algorithms. Unlike military contracts, police intel must be transparent to oversight and subpoena, or we risk further abrogation of our 4th, 5th, and 1st amendment rights. As an aide to the justice system, Palantir has a lot to explain, not just in court cases but on a continuing daily basis. Anything else violates the very fabric of America's principles.
- wavefunction 9y ago
  >>establish monopoly then protect it by any means necessary rent-seeker P Thiel
doytch 9y ago
I don't really get this article. It doesn't really say anything new and continually throws out statements designed to get the dear reader on the opposite side. "Trump supporter Peter Thiel", complaining about how only Palantir can maintain their systems, tossing around how many millions of license plates it can store, complaining about how it doesn't operate well with other systems...this reads like a hit piece. Yeah, Palantir's ecosystem is a closed ecosystem. But that's kind of becoming de rigeur these days for market leaders. Yeah, it sucks and ideally wouldn't exist, but don't paint it as some novel way for capitalists to fuck you over. That disingenuous approach does more harm than good imo. I'm most definitely not a fan of Palantir ever since digging into them and bailing out of the hiring process ~five years ago bc it made me squeamish, but this article just doesn't really seem to say anything new or even show an even-handed introduction to the company.
- wavefunction 9y ago
  >only Palantir can maintain their systems, tossing around how many millions of license plates it can store, complaining about how it doesn't operate well with other systems...this reads like a hit piece Those actually sound like completely valid concerns when considering whether Palantir should be offered a concession. Even better is to consider whether this concession should be offered in the first place in a supposedly free country.
- shubb 9y ago
  It's worth considering that palantir, while big and spooky from a startup perspective, are a new kid on a block with some even spookier bruisers. Palantirs competitors are folks like IBM, Lockheed Martin, and BAE. We know a little about thier technology and what spooky stuff it can be used for because they had to publicly market it, because they have been trying to break into markets. These other guys have comparable products. They have big data and network analysis tools. And more - IBM have some natural language analysis tech and curated data others can't touch for instance, and warehouses full of phds driving it. But they also have better contacts, shady revolving door sales arrangements with budget holders, and moats of unintegratable software already installed. Would you bet your well paid post retirement consultancy job on the continued survival of Palantir or Lockheed? I remember when the cooperation between hb garry and palantir on a tech demo came out in the aftermath of the anonymous hacks. They were scraping Facebook. If Lockheed were doing a demo like that, it would have a million dollar budget paid by darpa and use some secret bulk intercept database from our nightmares, and the demo would be to a high level audience at a military base, not at a tech conference. I don't like palantir, or the kind of things they are doing, but I like to remember they are a shrill little kid and the real scary stuff is behind the curtain. Imagine the most impressive deepmind demo done on a dataset of every email the nsa holds.
leroy_masochist 9y ago
The opening anecdote of this article is not a great look for Palantir. 100% consistency with the underlying database ACLs has long been one of the major selling points of the Gotham platform; perhaps the most important one. The fact that (according to the FOIA emails, per the article) the gang detective was asking for a fix for the ACL on a specific record and not hearing back for months while the record remained accessible to other officers who should not have seen it is especially surprising, and bad.
- whatnotests 9y ago
  Any "data" company must have full chain-of-custody for each one of their data points (where did we get it, how did we clean it, how would we update it, what was it previously, what has changed since X time, etc). Otherwise they're just making accusations without any defense from repudiation.
pcsanwald 9y ago
Some of the wording in the article, like "data trove", implies that Palantir has a central repo for all their data. From what I know, this isn't the case; a lot of government contracts mandate on-prem deployments, often on air gapped networks. The Police data might be an exception, though. I'm curious if anyone knows if this is the case? The rest of the article paints the process in a light that has been mentioned many times: They are a very successful consulting business.
- avs733 9y ago
  conscious that this probably sounds glib... mandates and has are (in my experience) very different things in military systems. I would suspect that the police systems, as are a lot of military systems, far less likely to be airgapped mostly for purposes of convienience I think the article is making the point that they are 'successful' in an economic sense but posing the question of whether this is an appropriate place for that type of success or the pursuit of that type of success. The potential for malfeasance seems particularly high.
  - pcsanwald 9y ago
    that might be true, but my experience is that generally, these kind of software deployments are really actually on-prem and aren't using AWS (or similar) under the covers. This was the certainly the case for my last employer, that sold very similar software. My remark at the end, I should have added emphasis on "consulting". my point was not so much that they are successful, but rather, they seem to operate as a consulting company, whilst portraying themselves as a seller of software.
    - avs733 9y ago
      fair and agreed. I have to be careful with Palantir that my personal disgust with them doesn't leak into my logic.
    - deleted 9y ago
      [deleted]
wolfspider 9y ago
Are these systems really that sought after? Recently my team and I finished a responsive hazmat application which uses mapbox, elasticsearch, and some .Net for real time DB replication (about 5 seconds). Documents and data are ingested onsite and displayed on a map for public records request with secure bulk file download. Records date back to the mid-nineties up until now. So with some ML I've created a Palantir?
- ousta 9y ago
  you did, but think about the cost of having competent senior software teams + it + product manager + project manager + ML engineers + data scientist and for a gigantic company buying palantir can make sense. for a software centric company it makes much less sense.
- pcsanwald 9y ago
  The devil is always in the details: - You'd have to go through through a ton of red tape to sell your software to various govt agencies, in terms of getting approved as a vendor. - Deployment complexity for various intelligence organizations is quite complex: air gapped networks, firewalls, and extremely locked down machines where getting a port opened is a bureaucratic nightmare. - Data ingestion is a huge piece: don't forget that Palantir promises support for tons of different data types. Word docs, email, chat, GIS data, etc etc. hammering all this stuff into a relational schema and resolving entities (same person, different email) and de-duping is a nightmare. - Have to support the inevitable waves of features that come in if basic functionality is successful. The basic technical pieces, from everything I've ever seen with regards to palantir, are fairly simple. used to be Oracle, some Java services, some good frontend work with very good visuals. The real reason they've been so successful is less about innovative technology and more about managing operational complexity very very well. Note that all this is secondhand information, I've never worked for Palantir and only know their product because we shared a lot of clients. I'd love to hear from someone at Palantir where I'm wrong.
blueyes 9y ago
Peter Thiel's hypocrisy takes my breath away. He pushes to elect Trump and buys himself citizenship in New Zealand. He calls himself a libertarian and builds a company at the service of the surveillance state. He gets two degrees from Stanford and tells kids they shouldn't go to college. The guy's a fake, a phony, and he's doing harm to the country and ecosystem that allowed him to succeed.

78 points by mcone 9y ago

Nice article. Palantir's practices seem to follow Thiel's master plan for world domination to a T -- establish monopoly then protect it by any means necessary: obfuscation, proprietary APIs, overly complex user interfaces, and finally claims of sensitive secret company technology that must be kept secret or risk compromise, perhaps causing harm to the US military and lowering our defenses against terrorism. What a crock. There's little doubt that 99% of Palantir's infrastructure is commodity open source, trivial to access, monitor, and replace. It's absurd that any customer would pay a premium for proprietary hardware and system support, much less be told that the system disallows even the shallowest oversight of the company's data and algorithms. Unlike military contracts, police intel must be transparent to oversight and subpoena, or we risk further abrogation of our 4th, 5th, and 1st amendment rights. As an aide to the justice system, Palantir has a lot to explain, not just in court cases but on a continuing daily basis. Anything else violates the very fabric of America's principles.

wavefunction 9y ago
>>establish monopoly then protect it by any means necessary rent-seeker P Thiel

doytch 9y ago

I don't really get this article. It doesn't really say anything new and continually throws out statements designed to get the dear reader on the opposite side. "Trump supporter Peter Thiel", complaining about how only Palantir can maintain their systems, tossing around how many millions of license plates it can store, complaining about how it doesn't operate well with other systems...this reads like a hit piece. Yeah, Palantir's ecosystem is a closed ecosystem. But that's kind of becoming de rigeur these days for market leaders. Yeah, it sucks and ideally wouldn't exist, but don't paint it as some novel way for capitalists to fuck you over. That disingenuous approach does more harm than good imo. I'm most definitely not a fan of Palantir ever since digging into them and bailing out of the hiring process ~five years ago bc it made me squeamish, but this article just doesn't really seem to say anything new or even show an even-handed introduction to the company.

wavefunction 9y ago
>only Palantir can maintain their systems, tossing around how many millions of license plates it can store, complaining about how it doesn't operate well with other systems...this reads like a hit piece Those actually sound like completely valid concerns when considering whether Palantir should be offered a concession. Even better is to consider whether this concession should be offered in the first place in a supposedly free country.
shubb 9y ago
It's worth considering that palantir, while big and spooky from a startup perspective, are a new kid on a block with some even spookier bruisers. Palantirs competitors are folks like IBM, Lockheed Martin, and BAE. We know a little about thier technology and what spooky stuff it can be used for because they had to publicly market it, because they have been trying to break into markets. These other guys have comparable products. They have big data and network analysis tools. And more - IBM have some natural language analysis tech and curated data others can't touch for instance, and warehouses full of phds driving it. But they also have better contacts, shady revolving door sales arrangements with budget holders, and moats of unintegratable software already installed. Would you bet your well paid post retirement consultancy job on the continued survival of Palantir or Lockheed? I remember when the cooperation between hb garry and palantir on a tech demo came out in the aftermath of the anonymous hacks. They were scraping Facebook. If Lockheed were doing a demo like that, it would have a million dollar budget paid by darpa and use some secret bulk intercept database from our nightmares, and the demo would be to a high level audience at a military base, not at a tech conference. I don't like palantir, or the kind of things they are doing, but I like to remember they are a shrill little kid and the real scary stuff is behind the curtain. Imagine the most impressive deepmind demo done on a dataset of every email the nsa holds.

leroy_masochist 9y ago

The opening anecdote of this article is not a great look for Palantir. 100% consistency with the underlying database ACLs has long been one of the major selling points of the Gotham platform; perhaps the most important one. The fact that (according to the FOIA emails, per the article) the gang detective was asking for a fix for the ACL on a specific record and not hearing back for months while the record remained accessible to other officers who should not have seen it is especially surprising, and bad.

whatnotests 9y ago
Any "data" company must have full chain-of-custody for each one of their data points (where did we get it, how did we clean it, how would we update it, what was it previously, what has changed since X time, etc). Otherwise they're just making accusations without any defense from repudiation.

pcsanwald 9y ago

Some of the wording in the article, like "data trove", implies that Palantir has a central repo for all their data. From what I know, this isn't the case; a lot of government contracts mandate on-prem deployments, often on air gapped networks. The Police data might be an exception, though. I'm curious if anyone knows if this is the case? The rest of the article paints the process in a light that has been mentioned many times: They are a very successful consulting business.

avs733 9y ago
conscious that this probably sounds glib... mandates and has are (in my experience) very different things in military systems. I would suspect that the police systems, as are a lot of military systems, far less likely to be airgapped mostly for purposes of convienience I think the article is making the point that they are 'successful' in an economic sense but posing the question of whether this is an appropriate place for that type of success or the pursuit of that type of success. The potential for malfeasance seems particularly high.
- pcsanwald 9y ago
  that might be true, but my experience is that generally, these kind of software deployments are really actually on-prem and aren't using AWS (or similar) under the covers. This was the certainly the case for my last employer, that sold very similar software. My remark at the end, I should have added emphasis on "consulting". my point was not so much that they are successful, but rather, they seem to operate as a consulting company, whilst portraying themselves as a seller of software.
  - avs733 9y ago
    fair and agreed. I have to be careful with Palantir that my personal disgust with them doesn't leak into my logic.
  - deleted 9y ago
    [deleted]

wolfspider 9y ago

Are these systems really that sought after? Recently my team and I finished a responsive hazmat application which uses mapbox, elasticsearch, and some .Net for real time DB replication (about 5 seconds). Documents and data are ingested onsite and displayed on a map for public records request with secure bulk file download. Records date back to the mid-nineties up until now. So with some ML I've created a Palantir?

ousta 9y ago
you did, but think about the cost of having competent senior software teams + it + product manager + project manager + ML engineers + data scientist and for a gigantic company buying palantir can make sense. for a software centric company it makes much less sense.
pcsanwald 9y ago
The devil is always in the details: - You'd have to go through through a ton of red tape to sell your software to various govt agencies, in terms of getting approved as a vendor. - Deployment complexity for various intelligence organizations is quite complex: air gapped networks, firewalls, and extremely locked down machines where getting a port opened is a bureaucratic nightmare. - Data ingestion is a huge piece: don't forget that Palantir promises support for tons of different data types. Word docs, email, chat, GIS data, etc etc. hammering all this stuff into a relational schema and resolving entities (same person, different email) and de-duping is a nightmare. - Have to support the inevitable waves of features that come in if basic functionality is successful. The basic technical pieces, from everything I've ever seen with regards to palantir, are fairly simple. used to be Oracle, some Java services, some good frontend work with very good visuals. The real reason they've been so successful is less about innovative technology and more about managing operational complexity very very well. Note that all this is secondhand information, I've never worked for Palantir and only know their product because we shared a lot of clients. I'd love to hear from someone at Palantir where I'm wrong.

blueyes 9y ago

Peter Thiel's hypocrisy takes my breath away. He pushes to elect Trump and buys himself citizenship in New Zealand. He calls himself a libertarian and builds a company at the service of the surveillance state. He gets two degrees from Stanford and tells kids they shouldn't go to college. The guy's a fake, a phony, and he's doing harm to the country and ecosystem that allowed him to succeed.