5 ms·

what do you do instead?

worthless-trash 7y ago
Not OP, but I imagine he's bought into the Hadoop and other big data stories. Most data processing of large data sets can probably be done with standard UNIX tools. There is nothing new under the sun, it's all just rebranded.
alkonaut 7y ago
If you do "object manipulation" e.g. columns/records/objects etc then at least Powershell offers some sanity.
- hnlmorg 7y ago
  Shameless plug but this is what I wrote murex for. Murex is a "UNIX" shell that's keeps enough similarities with POSIX that you can use it like a Bash REPL with (hopefully) minimal disruption but it breaks from POSIX compatibility where it makes sense. And one significant area it does break compatibility is how it handles structured data formats like JSON, S-Expressions, CSV's, and other tabulated data (to name a few). It's designed from the ground up to support object manipulation while still retaining compatibility with the UNIX pipeline. I do this by building a suite of builtin tools that are aware of structured data files (primarily because that information is passed down the pipeline as a data-type) but it still breaks into normal pipeline when forking an external executable. https://github.com/lmorg/murex https://github.com/lmorg/murex
oweiler 7y ago
We use Groovy or Kotlin with KScript.
- zmmmmm 7y ago
  Groovy is particularly nice I find. Completely cross platform and supports command line args similar to Perl to enable inline / pipe style processing easily, but if you are using Java in your back end you can throw any of your business logic in there too and use those.
kuschku 7y ago
Output data as JSON, manipulate and show it with jq
- bryanrasmussen 7y ago
  not all data comes from sources that you control and have chosen how to output.
  - BaltoRouberol 7y ago
    https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc can come pretty handy there
    - bryanrasmussen 7y ago
      thanks!
    - enriquto 7y ago
      oh my god, why?! just why!? What the world needs is the inverse program of "jc", where an unparseable json string is expanded into a flat list of lines all of the form "field.subfield=value"
      - bryanrasmussen 7y ago
        ok well I can understand not wanting json when all you need is something simpler, but I'm not sure if I understand unparseable - I mean if it is JSON then it is parseable.
        enriquto 7y ago
        You can only parse json easily by using json libraries. Plain text, or "field=value" pairs, you can easily cut(1) or grep(1), or sed(1) to your pleasure. This is what I mean by parseable. Parseable trivially by tools that do not understand the format. I can also sed and awk json files, and I do, but it is extremely painful; and more often than not these files are nothing more than simple lists of variables with values, for which the use of json is a ridiculous overkill.
seemslegit 7y ago
Use any modern programming language.
- spurdoman77 7y ago
  The gnu tools are super fast and do the job. Ive seen people spinning their own solutions which end up being super slow and arguable take more time to develop. With any gnu tools it is great that they will stay there for your life and are usually by default installed on every system
  - seemslegit 7y ago
    Yeah, except anything that is not trivial ends up being write-only
- matvore 7y ago
  Modern programming language features are overrated. I prefer to choose the tools that are the most portable and involve the least social friction (i.e. something that is widely understood by people I work with). For "fun" projects (stuff I'm not paid for) and workflow optimization, I just care about portability, which means C (with heavy use of the C stream library, a simple collections library of about 200 LOC, and occasionally POSIX syscalls) and shell scripting. After spending a lot of time learning languages as a hobby, I just don't believe the dark corners and warts of C and shell scripting are any worse than other languages.

Not OP, but I imagine he's bought into the Hadoop and other big data stories. Most data processing of large data sets can probably be done with standard UNIX tools. There is nothing new under the sun, it's all just rebranded.

alkonaut 7y ago

If you do "object manipulation" e.g. columns/records/objects etc then at least Powershell offers some sanity.

hnlmorg 7y ago
Shameless plug but this is what I wrote murex for. Murex is a "UNIX" shell that's keeps enough similarities with POSIX that you can use it like a Bash REPL with (hopefully) minimal disruption but it breaks from POSIX compatibility where it makes sense. And one significant area it does break compatibility is how it handles structured data formats like JSON, S-Expressions, CSV's, and other tabulated data (to name a few). It's designed from the ground up to support object manipulation while still retaining compatibility with the UNIX pipeline. I do this by building a suite of builtin tools that are aware of structured data files (primarily because that information is passed down the pipeline as a data-type) but it still breaks into normal pipeline when forking an external executable. https://github.com/lmorg/murex https://github.com/lmorg/murex

oweiler 7y ago

We use Groovy or Kotlin with KScript.

zmmmmm 7y ago
Groovy is particularly nice I find. Completely cross platform and supports command line args similar to Perl to enable inline / pipe style processing easily, but if you are using Java in your back end you can throw any of your business logic in there too and use those.

kuschku 7y ago

Output data as JSON, manipulate and show it with jq

bryanrasmussen 7y ago
not all data comes from sources that you control and have chosen how to output.
- BaltoRouberol 7y ago
  https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc can come pretty handy there
  - bryanrasmussen 7y ago
    thanks!
  - enriquto 7y ago
    oh my god, why?! just why!? What the world needs is the inverse program of "jc", where an unparseable json string is expanded into a flat list of lines all of the form "field.subfield=value"
    - bryanrasmussen 7y ago
      ok well I can understand not wanting json when all you need is something simpler, but I'm not sure if I understand unparseable - I mean if it is JSON then it is parseable.
      - enriquto 7y ago
        You can only parse json easily by using json libraries. Plain text, or "field=value" pairs, you can easily cut(1) or grep(1), or sed(1) to your pleasure. This is what I mean by parseable. Parseable trivially by tools that do not understand the format. I can also sed and awk json files, and I do, but it is extremely painful; and more often than not these files are nothing more than simple lists of variables with values, for which the use of json is a ridiculous overkill.

seemslegit 7y ago

Use any modern programming language.

spurdoman77 7y ago
The gnu tools are super fast and do the job. Ive seen people spinning their own solutions which end up being super slow and arguable take more time to develop. With any gnu tools it is great that they will stay there for your life and are usually by default installed on every system
- seemslegit 7y ago
  Yeah, except anything that is not trivial ends up being write-only
matvore 7y ago
Modern programming language features are overrated. I prefer to choose the tools that are the most portable and involve the least social friction (i.e. something that is widely understood by people I work with). For "fun" projects (stuff I'm not paid for) and workflow optimization, I just care about portability, which means C (with heavy use of the C stream library, a simple collections library of about 200 LOC, and occasionally POSIX syscalls) and shell scripting. After spending a lot of time learning languages as a hobby, I just don't believe the dark corners and warts of C and shell scripting are any worse than other languages.