9 ms·
How Palantir Is Taking Over New York City
- someone7x 10y agoThis gives me such mixed feelings. Using technology to increase productivity? Good Committing resources to quality of life improvements? Good 75% of enforcement done in neighborhoods of "color"? Yikes CIA-backed data analysis firm Palantir Technologies? Dear god
- TeMPOraL 10y agoMixed feelings for me, too. I'd really love to see that "all things city knows about this location" tablet app. It's just amazing to have this information easily available. But then the pesky issues of privacy and human conflict pop up, and we can't have nice things :/.
- akozak 10y agoIn-Q-Tel is a VC firm that makes loads of tech investments (https://en.wikipedia.org/wiki/In-Q-Tel#Investments https://en.wikipedia.org/wiki/In-Q-Tel#Investments) and I'm not aware of evidence that they play an active role in the company.
- blackbagboys 10y agoWould "75% of enforcement done in neighborhoods of color" still be an issue if 75% of crime occurs in those same neighborhoods?
- mildbow 10y agoBut how can one figure that out? Isn't enforcement of a law (through an arrest for example) the first indication that a crime might have occurred? If so, how to figure out if the 75% attribution is fair or just a result of selection/confirmation bias?
- blackbagboys 10y agoThere are numerous reasonable proxies; locations of murders, reported assaults or robberies, calls to the police department reporting a possible crime, etc.
- dangerlibrary 10y agoIs that what is actually being used, though? I suspect there is no way for the public to know.
- dangerlibrary 10y agoThis argument is so seductive, but it bugs me that people assume that "crime rates" are based on how often people are committing crimes. It's how often crimes are reported / how often criminals are caught. Given the greater prevalence of things like stop and frisk in neighborhoods with large minority populations, is it any surprise more people are caught for things like drug possession? That's going to further skew the stats, leading to more enforcement in those neighborhoods (since they are "high crime"). Drug use is actually higher in young white populations than it is among young black populations, but because of where law enforcement spends their time the incarceration rates differ wildly.
- blackbagboys 10y agoLook at the distribution of violent crime.
- dangerlibrary 10y agoYes, that would be one of many possible corrective measures.
- pc86 10y agoIt seems like you're making an argument that crime statistics should not be used as a metric to focus law enforcement efforts. What other metrics should we use? If you simply assign patrol routes based on population you are going to under-serve areas with higher crime and over-police areas that don't need it.
- dangerlibrary 10y agoI'm arguing that it's a more complicated statistical problem than "we see more criminal 'events' in this neighborhood, so we should send more cops there."
- searine 10y agoOffering criticism without offering an alternative isn't helpful. Of course it's complicated, and of course crime stats are a simplification. Most stats are. The point is, stats are much better than going by biased "gut feelings". Yes there are underlying issues with stats such as underreporting, but the solution isn't to get rid of the stats. It's to improve the underlying cause of the bias in the data, such as improving police/public relations.
- ritchiea 10y agoThis isn't nuanced, it doesn't add much to the discussion. It's "I think people of color commit more crimes" wrapped in an acceptable veneer.
- steveeq1 10y agoAnd what if "people of color commit more crimes" is statistically true?
- aljones 10y agoSince pigmentation doesn't cause crime, we should find the root cause.
- danieltillett 10y agoYou actually don't know that pigmentation does not cause more crime or the alternative an absence of pigment decreases crime. I am not saying that pigmentation is a causative factor, just that nobody knows if it is not.
- Cyph0n 10y agoWhat an enlightening comment.
- danieltillett 10y agoI assume you are being sarcastic, but there are plausible hypothesis that could be tested to answer this question. For example the genes that have caused a reduction in human pigmentation in some populations have occurred very recently (within historical times) and under very strong selection. Genes that influence the propensity for impulsiveness or violence may have been carried along with the selection for low pigmentation. Another possible hypothesis is that since melanin and dopamine have an overlapping biochemical pathway that the genes influencing melanin production may also influence dopamine levels. A third possible hypothesis is that since different populations have historically lived at different densities this may have influenced how well people interact peacefully at high population densities (i.e. how domesticated people are in the same way cows are much calmer than wild cattle). All of these are testable hypothesis that no sane scientist will ever study - in the end we just don't know the answer and most likely never will.
- mtgx 10y agoIt's important to keep in mind that in a "racist society" machine learning will also tend to be racist (but much more so). After all, the "AI" is learning from human data, so it's certainly not as objective as you would think a "machine" to be. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing https://www.propublica.org/article/machine-bias-risk-assessm... https://enterprisersproject.com/article/2016/9/beware-biases-machine-learning-one-cto-explains-why-it-happens https://enterprisersproject.com/article/2016/9/beware-biases... http://www.nytimes.com/2016/06/23/us/backlash-in-wisconsin-against-using-data-to-foretell-defendants-futures.html http://www.nytimes.com/2016/06/23/us/backlash-in-wisconsin-a...
- ebfe 10y agoAnd similarly, in a society in which biological differences exist between races, an objective, unbiased AI will be tend to be racist.
- TuringNYC 10y agoIt is a question of a crime taking place vs one which results in an arrest, convictions and statistics. I worked on Wall St for many years -- there are a lot of drugs on Wall St, more than i've seen anywhere else. Yet the rules are rarely enforced as they are in other neighborhoods. So of course, Wall St wont make it into the statistics. It is a vicious cycle.
- Torn 10y agoIt's also an interesting business model. Core platform tech and then consultancy. A LOT of hand-cranking integrations and massaging data, presumably that's where they make most of their money - billing hours to agencies with deep pockets.
- FLUX-YOU 10y agoIt can't be that easy. Not for a million a month. Even the most bass-ackwards database designs can be reverse engineered by bright people in a day or two (we know Palantir can attract these people) especially if they have some scars dealing with dumb database design. And it's probably not likely that the person who built that database and the program that uses it went to great lengths to obfuscate anything. I'll even go a step further and say that most data sources are probably really simple programs. Throw in any kind of documentation as a bonus and the learning curve just goes down. What the hell could they be charging that much for?
- specialp 10y agoPalantir likes to promote these minimal links to the CIA quietly to increase the feeling that they give to their "forward deployed engineers" that they are some Seal Team 6 of developers. Their office in the Meatpacking district of NYC is kinda reminiscent to a cult headquarters. Rooftop bar/DJ booth unlimited* vacation time, bring your pets to work, free gym membership right nearby, every desk has a "go bag"... Why would anyone want to leave?
- visitedpalnyc 10y agoI've visited the NYC office for dinner a few times. There's no DJ booth, nor a rooftop bar, nor free gym memberships (according to a friend), nor go bags. There's a rooftop patio, as many buildings in the area have, and gym subsidies as most tech companies have. I've been told they have emergency readiness kits (which are advocated by every level of government in every jurisdiction), but they're definitely not at the desks (or they are, but invisible). How are any of these criticisms distinct from Google, which owns an entire street block (no joke street to street, Ave to Ave)? I've had more than a few meals there as well and it's pretty much the same situation as far as I can tell.
- binarytransform 10y agoMongoDB is In-Q-Tel backed - so by that sensationalized metric - anyone who runs a MEAN stack is thus "CIA Backed" as well. Hilarious to see the kneejerky lack of intellectual rigor around HN these days.
- bogomipz 10y agoAlmost as hilarious as your strawman argument. Palantir was started with the explicit mission of aiding the surveillance-industrial complex. Mongo Inc was started with explicit mission of selling cloud databases(PaaS) to the general public. Source: http://fortune.com/2014/09/04/peter-thiels-contrarian-strategy/ http://fortune.com/2014/09/04/peter-thiels-contrarian-strate... “Basically,” Thiel explains, “I thought that some of the approaches that PayPal had used to fight fraud”—which at one point posed an existential threat to PayPal—“could be extended into other contexts, like fighting terrorism.” For contrast: Mongo Inc's Eliot Horowitz: https://medium.com/s-c-a-l-e/mongodb-co-creator-explains-why-nosql-came-to-be-and-why-open-source-mastery-is-an-elusive-goal-3a138480b9cd#.wksy3ond5 https://medium.com/s-c-a-l-e/mongodb-co-creator-explains-why...
- binarytransform 10y agoWhat argument? You mean *[argument rest of internet always makes re: Palantir]?
- FLUX-YOU 10y agoGuess we found out why MongoDB had insecure defaults tinfoil hat
- ufmace 10y agoIt sounds a little odd to me that people are objecting to enforcement of the law in "neighborhoods of color". Yes, enforcement of drug possession is probably out of whack, but this isn't about that. We passed those laws, are we saying that the laws are bad, or that the laws are good, but "people of color" don't deserve to live under them? Do they not deserve to live in neighborhoods without broken windows and graffiti? Do they not deserve to not worry about getting robbed on the way home from work? Do they not deserve to live and work in buildings are properly maintained? It seems like it's kind of a trap. If you agree with the frame of the article, then you have to conclude that either 1. We passed those laws to make our neighborhoods better, but minorities and the lower classes don't deserve that, so we should ignore some set of laws there, and that totally won't become apartheid or 2. Maybe all of these nanny-state laws made by wannabe social engineers aren't such a good idea after all, since they give the Government too much arbitrary power to be enforced on whoever they don't like this week, so we should repeal them and tell people to mind their own business when they come up with this stuff.
- jonathankoren 10y agoOr 3, you want the laws applied equally in all places and equally to all people. The problem isn't the law. The problem is the selective enforcement. It's the selective suspicion. It's the selective police state. Do you honestly believe that some cop in white suburbia is going to care about loose cigarettes? No way.
- ufmace 10y agoDo you honestly believe that anybody is selling loose cigarettes in suburbia? I wouldn't say that it's never happened, but I'd bet it's rare enough that even if cops made it a top priority, they'd have a tough time finding anybody doing it. But that doesn't really matter anyways. I thought we were talking about not throwing people in jail and giving them criminal records for petty crimes. If a black guy gets in trouble for, say, selling loose cigarettes, is he supposed to feel better knowing that if a white guy in a different neighborhood did the same thing, that guy would be getting in trouble too? Is that really the best we can do here? I'd rather try and find a way to not throw him in jail in the first place.
- panic 10y agoThe City Hall official discussed the city’s use of the data-mining technology on background, and declined to provide the full list of data sources or describe what is contained in the datasets. Presumably this technology is supposed to be helping the people of NYC. Shouldn't these people know what data is being collected about them so they can decide whether or not they actually want it?
- dmix 10y agoI believe this is why FOIA was so important...
- hprotagonist 10y ago"they are not all accounted for, the lost seeing stones ..." why anyone thought it was a great idea to name their company after the remote sensing device guaranteed to lie to you and make humans suicidally depressed has always been beyond me.
- TrevorJ 10y agoProbably the same people who designed the NRO spy satellite mission patch: http://b-i.forbesimg.com/kashmirhill/files/2013/12/Satellite-logo-for-spying.jpg http://b-i.forbesimg.com/kashmirhill/files/2013/12/Satellite... If anything, they are both a little too on the money.
- stonogo 10y agoSeems perfect to me, since the stones in the books were not inherently evil, but only when a malevolent superpower controlled them...
- maxander 10y ago> Co-founded in 2004 by Peter Thiel and Alex Karp, Palantir... It is a continuous marvel that Peter Thiel, nominally an outspoken and prominent libertarian, is partially responsible for one of the most insidious powers that the U.S. government has over its people.
- evgen 10y agoThiel is not a libertarian, he is a feudalist. He wants to be lord of the manor and governments have a habit of getting in his way when he wants to make you his serf.
- mempko 10y agoin some ways big 'L' libertarians are petty feudalists.
- __jal 10y agoYep. After several years of pointing out that feudalism was the natural outcome of his political sympathies, I finally convinced a friend that he wasn't a libertarian a while back. I say this as someone who is very sympathetic in principle to a lot of libertarian thought. In practice, that's just not how humans work.
- int_19h 10y agoThere is a way to salvage the term (and preserve its distinctive meaning from "liberal" etc). Most libertarians already speak of minimal government, as opposed to no government - ancaps are a vocal but small minority. So the question simply becomes, what exactly constitutes "minimal". And that is very much a subjective assessment. I posit that there is a broader category, which is distinct from what people usually mean when they speak of or self-identify as libertarians, that adheres to the same principle. Basically, the idea is that government is always, by its very nature ("all power comes from the barrel of a gun"), an intrusion on some liberties - and so any extension of government requires a solid justification and thorough vetting. However, some freedoms and liberties have to be intruded upon in order to maintain others. Again, most bona fide libertarians would agree - say, the freedom to violently coerce other people is clearly not the one that you want. But once you get into this mode of thinking, and ditch ideological stereotypes, there are many other limitations that appear perfectly reasonable. More importantly, you realize that whether some limitation is justifiable or not depends on your [inherently subjective] assessment of what is good and what isn't - but that is orthogonal to the minimal government principle. In other words, there are many different kinds of libertarians, who all agree on that basic principle, but disagree on what outcome they desire (and hence on how much government is "just enough"). So you can be a libertarian, but still consider public welfare programs to be a good way to spend money, because the alternative would be worse, in terms of overall individual liberties.
- bogomipz 10y agoThere's a few other recent developments of the "Big Data" city that New York is aspiring to that also give some residents mixed feelings. In chronological order: http://www.nytimes.com/2016/09/20/nyregion/cellphone-alerts-used-in-search-of-manhattan-bombing-suspect.html http://www.nytimes.com/2016/09/20/nyregion/cellphone-alerts-... http://www.theatlantic.com/technology/archive/2016/04/linknyc-new-york-wifi-privacy-security/477696/ http://www.theatlantic.com/technology/archive/2016/04/linkny... http://www.nyclu.org/content/automatic-license-plate-readers http://www.nyclu.org/content/automatic-license-plate-readers
- spunker540 10y agoThe article is implying there's more to be worried about than there actually is - they didn't mention anything actually "disturbing" but want us all to be concerned about the vague threat of city surveillance nonetheless. What does Palantir do? “integrate[s] disparate data sets and conduct[s] rich, multifaceted analysis across the entire range of data.” How does NYC use it? Tax fraud, fire code violations, fake security guards, fake IDs, fake cigarettes, fake marijuana. So the data already existed in NYC databases and the crimes they're enforcing already existed. And yet: "the potential for that kind of outright abuse is less disturbing than the ways in which Palantir’s tech is already being used. The city’s embrace of Palantir, outside of law enforcement, has quietly ushered in an era of civil surveillance so ubiquitous as to be invisible." -- total hyperbole! If anything the most telling part of this article to me, was the small sums of money being made by Palantir which is frequently lauded as one of the most elite, selective startups for software engineering positions. It seems to operate in small change relative to all the hype.
- guelo 10y agoIf NYC has data that has not previously been used for surveillance and Palantir is enabling them to use it for surveillance ("you're violating the fire code!") then it is ushering in an era of civil surveillance. I don't see what the hyperbole is.
- noamyoungerm 10y agoI would argue that the surveillance isn't in how the data is used, but rather in the fact that NYC has that data at all. Limits on acceptable data collection are more straightforward, enforceable, and fair than limits on acceptable data analysis. What would you say if a human had reached the same conclusion as Palantir by analyzing the same data?
- deleted 10y ago[deleted]
- LeifCarrotson 10y ago
- polskibus 10y agoDoes anyone know what differentiates Palantir from a typical OLAP + ETL stack like SASS + SSIS?
- qaq 10y agohourly rates they charge :)
- rudolf0 10y ago"""Proprietary algorithms"""
- blackbagboys 10y agoIt has a user interface that civil servants and law enforcement analysts actually have a prayer of learning.
- w1ntermute 10y agoThe difference is that the executives buying Palantir's products don't know or care what any of those acronyms are, and they're willing to pay a lot of money so they don't have to find out.
- Animats 10y agoAt first I thought this was about Palantir leasing vast amounts of office space, as they did in Palo Alto. But NYC? They're not that big. Take a look at the top 10 US government contractors.[1] Most of the top 10 make weapons systems. But two are in information processing: Leidos (used to be SAIC), and L-3 Communications. Palantir isn't even in the top 100. Maybe they're more into state and local customers. There's lots of potential for innovation in the state and local government space. A smartphone app for building inspectors, for example. One that involves lots of picture taking and GPS tagging. There are building inspector apps, but they're basically paper forms reworked for tablets. An ambitious project would be a system which takes the video and audio from a cop's body cam and does most of the paperwork. Show it a driver's license or a face, and it's in the record and understood by the system. Cops hate paperwork, yet have to document much of what they do. Automate that and cops will be glad to wear a cam. Difficult and controversial, but useful. It might be easier to sell in countries where local government is more standardized. In the US, you'd have to customize a system for every police department. [1] https://en.wikipedia.org/wiki/Top_100_Contractors_of_the_U.S._federal_government https://en.wikipedia.org/wiki/Top_100_Contractors_of_the_U.S...
- pritianka 10y agoHaha, my first thought was about Palantir leasing all of PA downtown as well. But your point is very interesting. I agree that there's a lot of value a company like Palantir can deliver to the traditionally luddite government.
- bane 10y agoSince Palantir's revenue is secret, and we may not actually end up seeing all obligated dollars, it's very hard to make a general comparison.
- dannylandau 10y agoWow, just a few million dollars in contracts from NYC agencies. That is peanuts. Not sure how they are able to justify such a high valuation. Seems like a lot of hype at Palantr
- luminiferous 10y agoThis is just Palantir's foot in the door. Other municipalities will want to follow in NYC's steps, as Palantir essentially expands the smart city market. When the market is big enough, then Palantir will start pricing in a manner that actually makes them money. At least in theory. The similarity to a typical startup (low price to build demand, growth drives high valuation) is uncanny.
- deleted 10y ago[deleted]
- Spooky23 10y agoArticles like this are so obnoxious. When the government calls it "Open Data" (ie. https://nycopendata.socrata.com https://nycopendata.socrata.com, https://data.ny.gov https://data.ny.gov) it's lauded as an inspirational embracing of transparency and seen as just awesome. When the government uses it to make itself more efficient and effective, there's this implication that something nefarious is going on. If you want to be offended by government software purchases, check out the POs for Oracle Enterprise whatever or the IBM Passport Advantage agreement that NYC has issued in the last year. Chances are, the city doesn't even know wtf half of the products they license are!
- oneloop 10y agoDoes any one know how Palantir started? I.e. from what I can remember it seemed to come out of nowhere. How did they manage to get such huge contracts from the start?
- home_boi 10y agoThiel
- jhanschoo 10y agoIIRC Palantir had difficulty getting clients early on. The tech evolved from Paypal's efforts to combat fraud via data analysis, esp. against the Russian mafia trafficking on their platform. The CIA was one of its first clients, which eventually led to other government agencies trusting them.
- ChuckMcM 10y agoI'm sure if your number is up, they will find you. :-) There seems to be a lot of momentum in the "canvas large data sets" space. It has always been on the wish list for the authorities (see many RFP's for the DARPA "Total Information Awareness" initiative) and the challenge has always been storage and algorithm development. Storage is becoming a non-issue when you can have a petabyte in M.2 class SSDs available across a 10Gbit network of processors. The challenge is the needle-in-haystack finding activity.
- sfifs 10y agoInteresting that all this data is primarily being used to catch tax violations and improving property prices. So from a city manager point of view, this may be paying out for itself.
- contingencies 10y ago"Quality of life violation"? Newspeak is truly here.