3 ms·
I would love to know an alternative to that superbly integrated stack . [be aware that none of those tools require coding knowledge, nor any DB administration,
by lolive 1y ago
I would love to know an alternative to that superbly integrated stack .
[be aware that none of those tools require coding knowledge, nor any DB administration, nor scalability consideration.
But for me the most important part is that the data workflow is not dataFrame driven, but semantic entity/relationship driven. Think of that as if you had an ORM magically wrapping your Spark cluster, and make it look like a graph database. Oh, and the sync with ElaticSearch is built-in for a no-brainer semantic search feature on your objects]
- belter 1y ago> but semantic entity/relationship driven This is marketing speak who means nothing. What do you mean? Something like this?: https://www.ibm.com/products/information-governance-catalog https://www.ibm.com/products/information-governance-catalog
- deleted 1y ago[deleted]
- lolive 1y agoYour data in Foundry are available both as tabular content and as a graph at the same time. Here is a tool to visually explore your graph: https://www.palantir.com/docs/foundry/vertex/graphs-explore https://www.palantir.com/docs/foundry/vertex/graphs-explore And here is the search engine to search for your objects (free-text and/or types): https://www.palantir.com/docs/foundry/object-explorer/getting-started/ https://www.palantir.com/docs/foundry/object-explorer/gettin... And here is the tool to manage the ontology (the types and relationships of the graph, plus the mapping of the graph to the tabular datasets): https://www.palantir.com/docs/foundry/ontology-manager/overview/ https://www.palantir.com/docs/foundry/ontology-manager/overv... The UI designer based on the graph data is called Workshop: https://www.palantir.com/docs/foundry/workshop/overview/ https://www.palantir.com/docs/foundry/workshop/overview/ I still have to find a stack where this duality is as integrated as in Foundry.
- belter 1y ago> I would love to know an alternative to that superbly integrated stack This one has 10 years of precedence: https://www.ibm.com/products/information-server https://www.ibm.com/products/information-server
- belter 1y ago> be aware that none of those tools require coding knowledge, nor any DB administration, nor scalability consideration. These are slogans with marketing grade truth, not engineering truth. The moment you leave the happy path wizards you run straight into Python, SQL, JVM knobs, and cluster limits. The visual Pipeline Builder drops you into a Code node the second you need anything beyond basic joins/filters. Here is an example: https://www.palantir.com/docs/foundry/code-workbook/getting-started https://www.palantir.com/docs/foundry/code-workbook/getting-... >> nor scalability consideration. Foundry Spark jobs expose tunable variables, you are expected to tweak them for performance and cost: https://www.palantir.com/docs/foundry/optimizing-pipelines/spark-concepts https://www.palantir.com/docs/foundry/optimizing-pipelines/s...
- lolive 1y ago99% of our pipelineBuilders are purely based on the available building blocks [that cover most of PySpark functions: https://www.palantir.com/docs/foundry/pipeline-builder/functions-index/ https://www.palantir.com/docs/foundry/pipeline-builder/funct...]. Most notably we do join/filters/explode/group, Dozens of them, realigning data from different parts of the company. Having most of those PySpark functions exposed visually as building blocks with debug points available at each step simply outperforms a codeWorkbook in term of maintainability and readability.
- phendrenad2 1y agoYeah as a general rule, when something requires "no coding" I question if they're too zoomed-in. In reality, you can just hire a programmer instead of a box-clicker. ...Except in this case, there might be a lot of value in letting James Bond or Boris Badenov whip up a quick spy pipeline without putting up a job ad for a Python coder.
- lolive 1y agoOnce you have used a visual block-based version of PySpark, going back to regular Python code will really look like going back to prehistory.