7 ms·

Having worked at Palantir in the past, I’d like to again take the opportunity to let everyone know what it is they actually do, because they really aren’t this

Having worked at Palantir in the past, I’d like to again take the opportunity to let everyone know what it is they actually do, because they really aren’t this all-seeing boogeyman of tech that everyone enjoys making them out to be (such rumors do bolster their reputation, however, so they aren’t likely to correct them, and paranoid responses like some here just lend them more credence). They build a federated search engine with a simple UI to make it accessible to a non-technical audience. That’s it. Have a whole lot of data in different databases/data lakes? Palantir will let you search across all of it quickly and simply. They don’t source the data, they even hold the data, they install their servers on premise with where the data is already being stored, and import it. It’s a specialized Google with very robust and configurable access controls that you can buy and run because you’ve collected more data than you know what to do with. Now you may still be legitimately unhappy with them because of who they work with or the implication of making those entities more efficient in processing data, but given that HN is a technical audience I’d really hope that we can get over this trend of thinking Palantir has some magical data summoning abilities.

tweetle_beetle 6y ago
With great power comes great responsibility, including moral. The isn't the first time technology can have far reaching positive and negative applications, and it certainly won't be the last. Palantir has made commercial choices that are hard to stomach and they should have to deal with being considered a boogeyman of tech, even it's not the technology's "fault". It may not be "all-seeing" but it appears to be happy to look at any data for any reason. > In 1937 IBM's tabulating equipment enabled organizations to process huge amounts of data. Its clients included the U.S. Government, during its first effort to maintain the employment records for 26 million people pursuant to the Social Security Act, and Hitler's Third Reich, for the tracking of Jews and other persecuted groups, largely through the German subsidiary Dehomag. The social security-related business gave an 81% increase in revenue from 1935 to 1939. https://en.wikipedia.org/wiki/IBM#History https://en.wikipedia.org/wiki/IBM#History
- RcouF1uZ4gsC 6y ago
  The IBM example gets brought up a lot, but the real solution to the Nazis was nation states going to war against them and defeating them . In a counterfactual world where IBM did not sell them the equipment, the Nazis still would have figured out a way to mass murder people. The Japanese and Stalin and Mao did not have access to IBM tech, but they still mass murdered people.
  - tweetle_beetle 6y ago
    You're absolutely right, but if the technology you sell is not even truly required by an unethical client like those mentioned, then there is even less reason to engage with them. The parent post from a former employee suggests that Palantir is in a similar position: > They build a federated search engine with a simple UI to make it accessible to a non-technical audience. That’s it. That's not a value proposition that is beyond the realms of possibility for other companies to develop. So why does Palantir appear to engage with more unethical projects that other companies if it's not by having a low ethical standard? (Obviously I can only comment on what is public knowledge.)
    - RcouF1uZ4gsC 6y ago
      > but if the technology you sell is not even truly required by an unethical client like those mentioned, then there is even less reason to engage with them. If the technology is not truly required by the unethical client, then you are doing a good, by taking their money and providing technology they don’t need. Money they give is money they can’t spend for other stuff that they truly need.
      - tweetle_beetle 6y ago
        The idea of ethics is based on having some level of shared values. If we act collectively we can promote these values. In this case, if we collectively disagree with something then we can "punish" it by not engaging with it commercially. I'm not sure how the abstract sabotage suggested makes any difference if the reasoning behind it isn't expressed - giving someone exactly what they want is hardly a great long term motivator for change.
lrvick 6y ago
So Palantir does digestion, interpretation, search, and display of data to non technical users who don't understand the error prone nature of such processes. It just so happens that some of this modeling is done to optimize indexing and searching data on humans our government would prefer dead, and no specific work has ever been done by Palantir employees to optimize anything for this use case, right? You produce a search engine that helps identify who to kill, but don't pull the trigger so you have no ethical responsibility, right? This must be how arms dealers think when they sell to dictators.
- taywrobel 6y ago
  I get that opinions differ on how to handle situations like these, and I respect that disagreement, but I’m personally of the opinion that if you don’t like what government does, vote to stop them from doing it. Don’t blame those that build the software they use. To me that makes every open source project a potential ethical minefield since they could all be getting used to do immoral acts without even knowing about it. As a concrete example, most of Palantir’s indexing and searching back when I was there was done using lucene. Should the lucene authors feel some moral conflict over this? If we’re shifting blame/responsibility down the tech stack why stop at the productized form? Why not the libraries? Why not the language? I understand that the case is different since Palantir is more purpose built for this type of work, but you have to admit the lines get blurry pretty quickly.
  - detaro 6y ago
    More purpose build for this type of work and actively promoting doing it among governments, actively hampering the efforts to get to the point of vote to stop them from doing it or implementing alternative approaches.
  - lrvick 6y ago
    I don't buy the "you can't blame anyone except the person that pulls the trigger" argument. That type of whataboutisim is at the heart of every major engineering effort of silicon valley and surveillance capitalisim that is used abusively on the masses. We can say no. We -must- say no. Making alcohol should be legal. Specifically selling it to a kid is not legal, because then you are specifically helping them commit a crime. Likewise I don't blame steel mills for producing steel that is later used in guns wielded by dictators on their own people. I sure do blame the people that sell the guns to said dictators with full knowledge of their likely use case though. We as engineers have an ethical responsibility if we make a nuke with full knowledge the buyer probably intends to drop it on a city. I once built a company that produced a very effective social data aggregator, classifier, and sentiment analysis engine with an easy search interface. It quickly identified sentiment trends across a dozen networks and could of been amazing for brand managers or stock traders. My investors decided they wanted to repurpose it for political propaganda and market it to a specific party. I could of said "well this is really cool tech, and I am making good money... Not my problem what they do with it". I thought that for a bit. I even helped build an MVP for political candidate tracking. It felt wrong, so I quit. I didn't help them replace me. The company ended and all my years of work and research with it. So be it. I don't always get it right, but looking back, -not- continuing that project remains one of the proudest choices of my life.
birdyrooster 6y ago
No one thinks they are magical and the fact that you had to trot that straw man out shows you are arguing in bad faith. People take issue with who they have contracts with and not the tech stack. Excuse my hyperbole but it reads like: “Oh I didn’t actually source the data for the nazis I just helped them search their data lakes”
- taywrobel 6y ago
  Maybe I’m old fashioned, but I’d rather address the nazis themselves than the tools they use. People still drive Volkswagens and wear Hugo Boss. Edit: A bit surprised to see this comment flagged. What about HN's guidelines were violated by this response to the parent comment's allusion to Nazis?
  - Nowado 6y ago
    Each part of this mutually beneficial relationship is equally replaceable and necessary.
  - chopin24 6y ago
    You should read “ IBM and the Holocaust: The Strategic Alliance between Nazi Germany and America's Most Powerful Corporation.” It’s not either/or, it’s both/and. This fucking matters. Letting corporations off the hook because they were just following orders enables genocide.
  - lrvick 6y ago
    Since you brought it up... Some of the Nazis that setup the gas chambers likely said "I don't like this but I have to do it because my boss told me to. They have all the responsibility." It sounds a lot like you are trying to justify work you are not particularly proud of to me.
    - taywrobel 6y ago
      I’m not a fan of invoking Godwin’s law, I only mentioned the nazis because the commenter I responded to make the allusion. And you’re entitled to your opinion, of course, but I wouldn’t recommend trying to presume anyone’s intentions.
0goel0 6y ago
> It’s a specialized Google ...used to kill innocent people
hshshs2 6y ago
This isn’t the argument anyone is making. People don’t like Palantir because they willingly facilitate extrajudicial murder as part of the military industrial complex.
fortydegrees 6y ago
You're getting some heat but I just want to say that I found this super interesting and valuable, and makes complete sense to me. Thank you.
- taywrobel 6y ago
  It’s okay, I’ve tried to explain what Palantir does on a few threads and always get a decent amount of pushback. So it goes with politicized issues. So I get a lot of downvotes and lose some imaginary internet points; that’s fine. I’m not attacking anyone, and for the most part nobody is attacking me, so I’d still call this healthy if contentious debate.
  - lrvick 6y ago
    Upvoted this because even though I think Palantir is dangerous and that you have argued for an ethical compass I greatly disagree with, I respect that you are willing to share an unpopular view to facilitate discussion of a complex issue. You are a person I would gladly grab a beer with when such things are a thing again.
    - ryandrake 6y ago
      Great point. The problem doesn’t go away if you avoid discussion and downvote the messenger. Insider knowledge and perspectives, even (especially) when it comes to the ethically gray area companies, is one of the reasons this site is great!
Bakary 6y ago
Did you get some direct insight on Palantir's marketing strategy? Did they directly plan to construct a shadowy reputation or was it more an accident they capitalized on? Your post confirms some suspicions about the company and its media coverage I entertained but had no way of verifying.
- taywrobel 6y ago
  Disclaimer that I left in 2014, but at that time they didn’t lean into their reputation at all. With that said, <speculation>I’d guess that they’re leaning into it and capitalizing on it because any press is good press. </speculation> They were actually incredibly open with what they did technically, and a lot of who they worked with it’s just that nobody seemed to care at the time. They had tech talks showing the full system architecture on YouTube that didn’t get more than 10k or so views. A quick search indicates that a lot of those are gone now (maybe just outdated) and the channel is more marketing heavy now, but just as low trafficed. They also open source a lot of their core technologies. The main database they used was “AtlasDB”, a relational layer on top of Cassandra. It’s open sourced and actively developed, and again, nobody seems to care, with it at a measly 700ish stars - https://github.com/palantir/atlasdb https://github.com/palantir/atlasdb People just like a good boogie man story, and after 2 decades of being labeled as such eventually you just stop trying to fight it I guess.
alasdair_ 6y ago
My understanding is that some of the people that work at Palantir have security clearances and work on classified work. Do you know what those people worked on? I've read a lot of things in the past that strongly suggest Palantir helps configure there systems for specific use cases, including the "killing people" ones mentioned in the article - is this erroneous? Is there no customer-specific configuration and assistance and help with whatever goals the customer has in mind?
- taywrobel 6y ago
  Good news/bad news. Good news is that I was one of those people that had a clearance, so can confirm that those individuals exist at least. Bad news is because I had a clearance, I can’t say anything more than that. Way too much liability to even speak vaguely about it, even tho I no longer have it. The US government doesn’t play around with that type of stuff.
mgh2 6y ago
Palantir's tech misunderstandings, including its ethical concerns have long been explored before: https://news.ycombinator.com/item?id=25542949 https://news.ycombinator.com/item?id=25542949 https://news.ycombinator.com/item?id=24740791 https://news.ycombinator.com/item?id=24740791
IG_Semmelweiss 6y ago
Thank you for that assesment. However i disagree with your conclusion. Palantir has some magical data summoning activities.here is why Circa 1990, if you said there was a way you could get any answer you wanted in 10 seconds or less,you would have been called a charlatan. Then google happened. Google is not worth billions because it just has a simple UI and is accesible. No. They have figured out how to make the web searchable. Palantir is the same. Just because search is not romantic anymore, doesnt mean they dont have magic to them. If aggregating intelligence data was such a no brainer , google or many of the others may have taken a stab at the problem. Here lies what i believe is a contradiction. If palantir does not have "magic", then why do you mention one of the most valuable companies on earth to describe palantir's function? In short, just because the u2 has wings and turbines, it doesn't mean it was just another plane when it came out.
erikbye 6y ago
Based on this: https://www.youtube.com/watch?v=jgj6FYghPh8 https://www.youtube.com/watch?v=jgj6FYghPh8 They do a lot more than what you claim. Did you forget e.g. blu/red force tracking and routes? Mission planning?

tweetle_beetle 6y ago

With great power comes great responsibility, including moral. The isn't the first time technology can have far reaching positive and negative applications, and it certainly won't be the last. Palantir has made commercial choices that are hard to stomach and they should have to deal with being considered a boogeyman of tech, even it's not the technology's "fault". It may not be "all-seeing" but it appears to be happy to look at any data for any reason. > In 1937 IBM's tabulating equipment enabled organizations to process huge amounts of data. Its clients included the U.S. Government, during its first effort to maintain the employment records for 26 million people pursuant to the Social Security Act, and Hitler's Third Reich, for the tracking of Jews and other persecuted groups, largely through the German subsidiary Dehomag. The social security-related business gave an 81% increase in revenue from 1935 to 1939. https://en.wikipedia.org/wiki/IBM#History https://en.wikipedia.org/wiki/IBM#History

RcouF1uZ4gsC 6y ago
The IBM example gets brought up a lot, but the real solution to the Nazis was nation states going to war against them and defeating them . In a counterfactual world where IBM did not sell them the equipment, the Nazis still would have figured out a way to mass murder people. The Japanese and Stalin and Mao did not have access to IBM tech, but they still mass murdered people.
- tweetle_beetle 6y ago
  You're absolutely right, but if the technology you sell is not even truly required by an unethical client like those mentioned, then there is even less reason to engage with them. The parent post from a former employee suggests that Palantir is in a similar position: > They build a federated search engine with a simple UI to make it accessible to a non-technical audience. That’s it. That's not a value proposition that is beyond the realms of possibility for other companies to develop. So why does Palantir appear to engage with more unethical projects that other companies if it's not by having a low ethical standard? (Obviously I can only comment on what is public knowledge.)
  - RcouF1uZ4gsC 6y ago
    > but if the technology you sell is not even truly required by an unethical client like those mentioned, then there is even less reason to engage with them. If the technology is not truly required by the unethical client, then you are doing a good, by taking their money and providing technology they don’t need. Money they give is money they can’t spend for other stuff that they truly need.
    - tweetle_beetle 6y ago
      The idea of ethics is based on having some level of shared values. If we act collectively we can promote these values. In this case, if we collectively disagree with something then we can "punish" it by not engaging with it commercially. I'm not sure how the abstract sabotage suggested makes any difference if the reasoning behind it isn't expressed - giving someone exactly what they want is hardly a great long term motivator for change.

lrvick 6y ago

So Palantir does digestion, interpretation, search, and display of data to non technical users who don't understand the error prone nature of such processes. It just so happens that some of this modeling is done to optimize indexing and searching data on humans our government would prefer dead, and no specific work has ever been done by Palantir employees to optimize anything for this use case, right? You produce a search engine that helps identify who to kill, but don't pull the trigger so you have no ethical responsibility, right? This must be how arms dealers think when they sell to dictators.

taywrobel 6y ago
I get that opinions differ on how to handle situations like these, and I respect that disagreement, but I’m personally of the opinion that if you don’t like what government does, vote to stop them from doing it. Don’t blame those that build the software they use. To me that makes every open source project a potential ethical minefield since they could all be getting used to do immoral acts without even knowing about it. As a concrete example, most of Palantir’s indexing and searching back when I was there was done using lucene. Should the lucene authors feel some moral conflict over this? If we’re shifting blame/responsibility down the tech stack why stop at the productized form? Why not the libraries? Why not the language? I understand that the case is different since Palantir is more purpose built for this type of work, but you have to admit the lines get blurry pretty quickly.
- detaro 6y ago
  More purpose build for this type of work and actively promoting doing it among governments, actively hampering the efforts to get to the point of vote to stop them from doing it or implementing alternative approaches.
- lrvick 6y ago
  I don't buy the "you can't blame anyone except the person that pulls the trigger" argument. That type of whataboutisim is at the heart of every major engineering effort of silicon valley and surveillance capitalisim that is used abusively on the masses. We can say no. We -must- say no. Making alcohol should be legal. Specifically selling it to a kid is not legal, because then you are specifically helping them commit a crime. Likewise I don't blame steel mills for producing steel that is later used in guns wielded by dictators on their own people. I sure do blame the people that sell the guns to said dictators with full knowledge of their likely use case though. We as engineers have an ethical responsibility if we make a nuke with full knowledge the buyer probably intends to drop it on a city. I once built a company that produced a very effective social data aggregator, classifier, and sentiment analysis engine with an easy search interface. It quickly identified sentiment trends across a dozen networks and could of been amazing for brand managers or stock traders. My investors decided they wanted to repurpose it for political propaganda and market it to a specific party. I could of said "well this is really cool tech, and I am making good money... Not my problem what they do with it". I thought that for a bit. I even helped build an MVP for political candidate tracking. It felt wrong, so I quit. I didn't help them replace me. The company ended and all my years of work and research with it. So be it. I don't always get it right, but looking back, -not- continuing that project remains one of the proudest choices of my life.

birdyrooster 6y ago

No one thinks they are magical and the fact that you had to trot that straw man out shows you are arguing in bad faith. People take issue with who they have contracts with and not the tech stack. Excuse my hyperbole but it reads like: “Oh I didn’t actually source the data for the nazis I just helped them search their data lakes”

taywrobel 6y ago
Maybe I’m old fashioned, but I’d rather address the nazis themselves than the tools they use. People still drive Volkswagens and wear Hugo Boss. Edit: A bit surprised to see this comment flagged. What about HN's guidelines were violated by this response to the parent comment's allusion to Nazis?
- Nowado 6y ago
  Each part of this mutually beneficial relationship is equally replaceable and necessary.
- chopin24 6y ago
  You should read “ IBM and the Holocaust: The Strategic Alliance between Nazi Germany and America's Most Powerful Corporation.” It’s not either/or, it’s both/and. This fucking matters. Letting corporations off the hook because they were just following orders enables genocide.
- lrvick 6y ago
  Since you brought it up... Some of the Nazis that setup the gas chambers likely said "I don't like this but I have to do it because my boss told me to. They have all the responsibility." It sounds a lot like you are trying to justify work you are not particularly proud of to me.
  - taywrobel 6y ago
    I’m not a fan of invoking Godwin’s law, I only mentioned the nazis because the commenter I responded to make the allusion. And you’re entitled to your opinion, of course, but I wouldn’t recommend trying to presume anyone’s intentions.

0goel0 6y ago

> It’s a specialized Google ...used to kill innocent people

hshshs2 6y ago

This isn’t the argument anyone is making. People don’t like Palantir because they willingly facilitate extrajudicial murder as part of the military industrial complex.

fortydegrees 6y ago

You're getting some heat but I just want to say that I found this super interesting and valuable, and makes complete sense to me. Thank you.

taywrobel 6y ago
It’s okay, I’ve tried to explain what Palantir does on a few threads and always get a decent amount of pushback. So it goes with politicized issues. So I get a lot of downvotes and lose some imaginary internet points; that’s fine. I’m not attacking anyone, and for the most part nobody is attacking me, so I’d still call this healthy if contentious debate.
- lrvick 6y ago
  Upvoted this because even though I think Palantir is dangerous and that you have argued for an ethical compass I greatly disagree with, I respect that you are willing to share an unpopular view to facilitate discussion of a complex issue. You are a person I would gladly grab a beer with when such things are a thing again.
  - ryandrake 6y ago
    Great point. The problem doesn’t go away if you avoid discussion and downvote the messenger. Insider knowledge and perspectives, even (especially) when it comes to the ethically gray area companies, is one of the reasons this site is great!

Bakary 6y ago

Did you get some direct insight on Palantir's marketing strategy? Did they directly plan to construct a shadowy reputation or was it more an accident they capitalized on? Your post confirms some suspicions about the company and its media coverage I entertained but had no way of verifying.

taywrobel 6y ago
Disclaimer that I left in 2014, but at that time they didn’t lean into their reputation at all. With that said, <speculation>I’d guess that they’re leaning into it and capitalizing on it because any press is good press. </speculation> They were actually incredibly open with what they did technically, and a lot of who they worked with it’s just that nobody seemed to care at the time. They had tech talks showing the full system architecture on YouTube that didn’t get more than 10k or so views. A quick search indicates that a lot of those are gone now (maybe just outdated) and the channel is more marketing heavy now, but just as low trafficed. They also open source a lot of their core technologies. The main database they used was “AtlasDB”, a relational layer on top of Cassandra. It’s open sourced and actively developed, and again, nobody seems to care, with it at a measly 700ish stars - https://github.com/palantir/atlasdb https://github.com/palantir/atlasdb People just like a good boogie man story, and after 2 decades of being labeled as such eventually you just stop trying to fight it I guess.

alasdair_ 6y ago

My understanding is that some of the people that work at Palantir have security clearances and work on classified work. Do you know what those people worked on? I've read a lot of things in the past that strongly suggest Palantir helps configure there systems for specific use cases, including the "killing people" ones mentioned in the article - is this erroneous? Is there no customer-specific configuration and assistance and help with whatever goals the customer has in mind?

taywrobel 6y ago
Good news/bad news. Good news is that I was one of those people that had a clearance, so can confirm that those individuals exist at least. Bad news is because I had a clearance, I can’t say anything more than that. Way too much liability to even speak vaguely about it, even tho I no longer have it. The US government doesn’t play around with that type of stuff.

mgh2 6y ago

Palantir's tech misunderstandings, including its ethical concerns have long been explored before: https://news.ycombinator.com/item?id=25542949 https://news.ycombinator.com/item?id=25542949 https://news.ycombinator.com/item?id=24740791 https://news.ycombinator.com/item?id=24740791

IG_Semmelweiss 6y ago

Thank you for that assesment. However i disagree with your conclusion. Palantir has some magical data summoning activities.here is why Circa 1990, if you said there was a way you could get any answer you wanted in 10 seconds or less,you would have been called a charlatan. Then google happened. Google is not worth billions because it just has a simple UI and is accesible. No. They have figured out how to make the web searchable. Palantir is the same. Just because search is not romantic anymore, doesnt mean they dont have magic to them. If aggregating intelligence data was such a no brainer , google or many of the others may have taken a stab at the problem. Here lies what i believe is a contradiction. If palantir does not have "magic", then why do you mention one of the most valuable companies on earth to describe palantir's function? In short, just because the u2 has wings and turbines, it doesn't mean it was just another plane when it came out.

erikbye 6y ago

Based on this: https://www.youtube.com/watch?v=jgj6FYghPh8 https://www.youtube.com/watch?v=jgj6FYghPh8 They do a lot more than what you claim. Did you forget e.g. blu/red force tracking and routes? Mission planning?