4 ms·
When I interned at Palantir (summer 2014) their business was mostly in data ingestion, visualization, and correlation. A typical workflow for a Palantir custom
by Manuel_D 2y ago
When I interned at Palantir (summer 2014) their business was mostly in data ingestion, visualization, and correlation.
A typical workflow for a Palantir customer was that Palantir would come in and dump a ton of data out of old crufty databases and into Palantir's datastore. Then, they'd establish connections between that data. This is all sounds kind of hand-wavy, but the gist of it is that a lot of government agencies have data that lives in separate databases and they can't easily correlate data between those two databases. Once the data was in Palantir's system, they could do queries against all their data, and make connections and correlations that they wouldn't otherwise be able to find when the data was previously siloed.
One of the sample use cases was identifying people filling prescriptions for schedule II drugs multiple times on the same day, and correlating that with pharmacies run by people connected to known drug traffickers. Previously, this was hard to do because the database of prescription purchases was disconnected from the database of drug convictions.
- hammock 2y agoSo it’s hygiene and structure
- Manuel_D 2y agoThat, and a really powerful visualization suite. In the example I gave above, you could plot the prescription purchases on a map and see that people were driving along the highway and hitting up pharmacies along the interstate. Better yet, you could drop into Google Street view in front of one of the pharmacies, and look at it from the street level and see that it doesn't even have signage indicating it's a pharmacy.
- swells34 2y agoI used it quite a bit early on during military operations. The ability to see the timing component was key; not only would you plot the purchase locations, but you could play the timeframe of records, work out the timing so you knew the order in which they visited the locations, where they must have stopped for gas along the route. In a classic workflow, you'd then investigate the gas stations, attach them to the event with confidence intervals, pull CCTV footage, see if you can get a payment receipt, and enter all of that data back into palantir. A few days of doing this, and you can build up all a map of every aspect of the drug run; the who what when where and why. It's a fantastic organization system.
- lapphi 2y agoI appreciate the technical achievements here. However, I wonder how long before it’s standard practice to track all peoples movement, not just those suspected of a crime. I know of at least one YouTube channel that is always recording all traffic camera streams in Washington so there must be some State entities doing the same. Back in 2020 there was a twitch channel that would play a 9x9 grid of all the livestream footage from the George Floyd protests. I’m sure an archive of that exists somewhere on a LE server.
- beeboobaa3 2y agonsa is storing everything
- fijiaarone 2y agoVisualization is a fancy package. Nobody looks at visualizations, but that's what makes people buy.
- vundercind 2y agoI’ve known companies to spend stupid amounts of money on fake, fancy “war rooms” they staff with people doing nothing useful, filled with “big board” style maps and shit, big graphs and visualizations that aren’t used anywhere else, just as a sales tool. Walk the visiting CEO through, let them pretend what they’re involved in is way cooler and more interesting and important than it really is, and I guess that assists sales so much that such endeavors make way more money than they cost. I connect this with comments I heard from several major management consulting firm folks stating bluntly that the best way to communicate effectively with execs is to approach them like young children. Life is super weird. Who knew imaginative play would be such a big thing for “serious” adults? I’d never have imagined, but it’s kinda everywhere.
- red-iron-pine 2y agoI used to get paid to develop those war room monitoring solutions. literally just crafting dashboards that no one would ever look at directly, but just sorta had around. > execs is to approach them like young children. lots of images. bright colors. no more than 3 bulletpoints per slide. no more than 4 minutes to get to the point, and be unambiguous about what and why.
- anitil 2y ago> approach them like young children To take a generous go at this - my guess is that they have multiple urgent issues they're dealing with at any one time, and so the cognitive bandwidth they're able to dedicate to 'random presentation number 3 for the day' is quite low But I do agree that a lot of day-to-day work is play acting at being cooler than our actual work.
- danudey 2y agoIIRC part of it is that the software itself can make connections between separate data sets. You're not just ingesting data about purchasing information and drug convictions and so on, you're getting automatic relationship detection. For example, figuring out that the cust_ss_num field in one dataset correlates to the conv_ssn_full field in another dataset, and knowing that those fields are the "SSN" field from a third dataset, and being able to automatically give you a view where those three datasets are correlated. This saves people having to go through every data set and manually map each field to each other equivalent field in each other related dataset. I could be mistaken, but I think this is how it was explained to me originally.
- browningstreet 2y agoIn many of the enterprise orgs I've worked in, the two tech teams that are chronically understaffed are 1) info sec, 2) DBA/ data architecture/ data science. I'm lumping those 3 together on purpose, because they're always understaffed and typically not empowered to build anything.
- hitekker 2y agoYou're right to group Data teams together. They seem to share a common plight. In my experience, internal employees outside Data have a funny relationship with Data. They hate to manage it but they love to blame it, especially in analytical / decision-making scenarios. Teams that "own" the data usually get the blame, on top of having to deal with a mass of rotting pipes and noncompliant teams, while also losing out on credit when non-Data teams report big wins. Based on what the GP says, it sounds like Palantir knows how to exploit common internal politics around Data. They build up technical & social expertise in ETL'ing disparate data sources, and they can avoid blame by being hired by executives as an external third party.
- vundercind 2y agoThis is exactly what I thought TFA was getting at when it brought up politics being a problem at companies and in sectors Palantir engages with, but instead it went a much more general direction.
- hitekker 2y agoHe talks about it a little: > Why is data integration so hard? The data is often in different formats that aren’t easily analyzed by computers – PDFs, notebooks, Excel files (my god, so many Excel files) and so on. But often what really gets in the way is organizational politics: a team, or group, controls a key data source, the reason for their existence is that they are the gatekeepers to that data source, and they typically justify their existence in a corporation by being the gatekeepers of that data source (and, often, providing analyses of that data). [3] This politics can be a formidable obstacle to overcome, and in some cases led to hilarious outcomes – you’d have a company buying an 8-12 week pilot, and we’d spend all 8-12 weeks just getting data access, and the final week scrambling to have something to demo. I think he's seen more companies without talented Data experts than companies with that talent.
- thimkerbell 2y agoSo if they are dumping data out of old crufty databases and into Palantir's datastore, which one is the active database going forward? In 2024.
- vundercind 2y agoAll of them, plus whatever next vendor they “migrate” to in three years (I’m being generous).
- UltraSane 2y ago[dead]
- sroerick 2y agoPeople dismiss this type of work as no big deal, but in my experience this is the actual hard work of producing something useful for companies, and what 90% of SaaS resellers will never be able to deliver on.
- jeltz 2y agoYes, it is very hard. But does Palantir succeed? Or do they like some other companies just trick customers with big wallets to buy?
- trenchgun 2y agoTo me it seems they do https://logicmag.io/commons/enter-the-dragnet/ https://logicmag.io/commons/enter-the-dragnet/
- osrec 2y agoWe used them at a bulge bracket investment bank and they failed miserably...
- vundercind 2y agoThe impression I get from their involvement at one company I know of is that it’s very much the latter. I was pretty surprised to see them behaving and performing about the same as any parasitic enterprise software vendor with an integration services arm. One wonders how different they really are, and if maybe they just have very good PR and marketing. Chalk it up as yet another case of some famous one-would-suppose impressive entity, or strata of a company hierarchy, or whatever, turning out to be pretty average, or even below average. You’d think I’d stop being surprised by now. Then again, maybe I was just seeing their B-team.
- insane_dreamer 2y agoI’ve heard you often get the A Team coming up with the plan and making the sale and then the B Team doing the actual implementation which surprise! doesn’t live up to the A Team hype. Not specific to Palantir.
- LarsDu88 2y agoSo basically data warehousing, and making it possible to do joins? Super boring, but super important stuff, which I've seen neglected at far too many places I've worked. Sounds like data engineering with a dash of ML.
- throwaway2037 2y agoThis is a good post to explain the value proposition. It sounds like "Big Data" from the 1990s, but a very good salesperson was able to infiltrate some US gov orgs to sell the idea.
- rsynnott 2y agoThis all just sounds like any other consulting company, really?
- anitil 2y agoPerhaps with the additional level of security clearance that government entities are able to hand over very sensitive data sets.