4 ms·

Are relationships discovered automatically across DBs? Does anyone have any insight as to how this might work technically?

by jawn 16y ago

slashclee 16y ago
Depends on the kind of relationship you're talking about, but generally speaking, someone has to map the data from the original databases into Palantir entities/properties/links. But—and this is kind of a big deal—the Palantir product does not have a fixed set of entities/properties/links/etc. So if you're more interested in tracking, say, computer systems, networks, and packet traffic than terrorists and bank transactions, it's fairly straightforward to define a new ontology that only contains the things you care about. https://devzone.palantirtech.com/display/pgdz/Integrating+Structured+Data+Into+Palantir+With+Groovy https://devzone.palantirtech.com/display/pgdz/Integrating+St... has a pretty useful overview of how you can take raw data from an existing source (granted, this source is XML data, but it could just as easily come directly from a regular database) and map it into a Palantir stack.
- jawn 16y ago
  Thanks for the links and info. Great stuff. So basically, the heavy lifting palantir does is in the presentation and data abstraction. With a person still needed to define mappings across data sources. Is this a correct statement? With (seemingly) limited off the shelf mapping abilities I'm surprised that they are looking to sell more as an appliance, and less as a turn key service with consulting engagements etc... Neat product.
  - slashclee 16y ago
    I should probably mention that I work at Palantir, but [insert boilerplate disclaimer here]. Yes, you definitely need a person to define the mappings across data sources. However, this is basically a one-time setup task; once the mappings have been established, new data from the existing databases can be continuously imported. I don't have any stats on how long this initial setup phase takes at real customer sites, and if I did I'm sure I couldn't share them publicly, but the goal is definitely for it to be something that doesn't require an entire team of consultants to deploy and babysit. If you want to see the actual government product, and how it works, you can sign up at https://analyzethe.us/ https://analyzethe.us/ for an account. This is a real live Palantir instance with real data from data.gov and it's open to the public.
    - jawn 16y ago
      Awesome. Thank you.
    - elblanco 16y ago
      At the dozen or sites I'm aware of, only a couple are where I'd say are in "full operational mode" after 7-9 months. I've known a couple that have been in a protracted (even by that standard) deployment stage for >12 months. Every place I've seen has a 2-4 Palantir engineers working F/T there during the deployment phase and at least 1 or 2 FTEs to maintain the ontology as mission requirements change ("oh, now we need VIN numbers associated with vehicles!"). I'd imagine it's getting faster as staff gets trained and various little ETL tools are built to funnel data into the backend. But it's been a very frustrating process for most of the customers to drop the money on the deployment and not have an operational tool for the better part of their first year, and the maintenance tail has been reported to be some percentage higher than the original purchase (that may have changed, I've noticed the pricing structure has been changing quite a bit over the last year). Of course, I also understand that when the customer says "pump data source X into Palantir" and data source X requires a 6 month approval process to hook up to, it can just take a while.
elblanco 16y ago
They map data from different sources onto a common semantic ontology. From that, relationships can be derived. It works better with nicely structured data, but people can be leveraged to manually tag entities in documents and add those to the appropriate part of the ontology.

Are relationships discovered automatically across DBs? Does anyone have any insight as to how this might work technically?

by jawn 16y ago

Are relationships discovered automatically across DBs? Does anyone have any insight as to how this might work technically?

slashclee 16y ago

Depends on the kind of relationship you're talking about, but generally speaking, someone has to map the data from the original databases into Palantir entities/properties/links. But—and this is kind of a big deal—the Palantir product does not have a fixed set of entities/properties/links/etc. So if you're more interested in tracking, say, computer systems, networks, and packet traffic than terrorists and bank transactions, it's fairly straightforward to define a new ontology that only contains the things you care about. https://devzone.palantirtech.com/display/pgdz/Integrating+Structured+Data+Into+Palantir+With+Groovy https://devzone.palantirtech.com/display/pgdz/Integrating+St... has a pretty useful overview of how you can take raw data from an existing source (granted, this source is XML data, but it could just as easily come directly from a regular database) and map it into a Palantir stack.

jawn 16y ago
Thanks for the links and info. Great stuff. So basically, the heavy lifting palantir does is in the presentation and data abstraction. With a person still needed to define mappings across data sources. Is this a correct statement? With (seemingly) limited off the shelf mapping abilities I'm surprised that they are looking to sell more as an appliance, and less as a turn key service with consulting engagements etc... Neat product.
- slashclee 16y ago
  I should probably mention that I work at Palantir, but [insert boilerplate disclaimer here]. Yes, you definitely need a person to define the mappings across data sources. However, this is basically a one-time setup task; once the mappings have been established, new data from the existing databases can be continuously imported. I don't have any stats on how long this initial setup phase takes at real customer sites, and if I did I'm sure I couldn't share them publicly, but the goal is definitely for it to be something that doesn't require an entire team of consultants to deploy and babysit. If you want to see the actual government product, and how it works, you can sign up at https://analyzethe.us/ https://analyzethe.us/ for an account. This is a real live Palantir instance with real data from data.gov and it's open to the public.
  - jawn 16y ago
    Awesome. Thank you.
  - elblanco 16y ago
    At the dozen or sites I'm aware of, only a couple are where I'd say are in "full operational mode" after 7-9 months. I've known a couple that have been in a protracted (even by that standard) deployment stage for >12 months. Every place I've seen has a 2-4 Palantir engineers working F/T there during the deployment phase and at least 1 or 2 FTEs to maintain the ontology as mission requirements change ("oh, now we need VIN numbers associated with vehicles!"). I'd imagine it's getting faster as staff gets trained and various little ETL tools are built to funnel data into the backend. But it's been a very frustrating process for most of the customers to drop the money on the deployment and not have an operational tool for the better part of their first year, and the maintenance tail has been reported to be some percentage higher than the original purchase (that may have changed, I've noticed the pricing structure has been changing quite a bit over the last year). Of course, I also understand that when the customer says "pump data source X into Palantir" and data source X requires a 6 month approval process to hook up to, it can just take a while.

elblanco 16y ago

They map data from different sources onto a common semantic ontology. From that, relationships can be derived. It works better with nicely structured data, but people can be leveraged to manually tag entities in documents and add those to the appropriate part of the ontology.