4 ms·

Blows my mind. I am a backend programmer and a semi-decent sysadmin and I would have immediately told you: "make a ZFS or BCacheFS pool with 20-30% redundancy b

Blows my mind. I am a backend programmer and a semi-decent sysadmin and I would have immediately told you: "make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs, I know dozens that work on CSV and XML, what's the problem?". And I am not a specialized data scientist. But with time I am wondering if such a thing even exists... being a good backender / sysadmin and knowing a lot of CLI tools has always seemed to do the job for me just fine (though granted I never actually managed a data lake, so I am likely over-simplifying it).

WesolyKubeczek 2y ago
> just go wild with CLI programs, I know dozens that work on CSV and XML ...or put it into SQLite for extra blazing fastness! No kidding.
- pdimitar 2y ago
  That's included in CLI tools. Also duckdb and clickhouse-local are amazing.
  - WesolyKubeczek 2y ago
    I need to learn more about the latter for some log processing...
    - fijiaarone 2y ago
      Log files aren’t data. That’s your first problem. But that’s the only thing that most people have that generates more bytes than can fit on screen in a single spreadsheet.
      - thfuran 2y ago
        Of course they are. They just aren't always structured nicely.
      - WesolyKubeczek 2y ago
        Everything is data if you are brave enough.
  - c0brac0bra 2y ago
    clickhouse-local had been astonishingly fast for operating on many GB of local CSVs. I had a heck of a time running the server locally before I discovered the CLI.
nevi-me 2y ago
To be fair on candidates, CLI programs create technical debt the moment they're written. A good answer that strikes a balance between size of data, latency and frequency requirements is a candidate who is able to show that they can choose the right tool that the next person will be comfortable with.
- pdimitar 2y ago
  True on the premise, yep, though I'm not sure how using CLI programs like LEGO blocks creates a tech debt?
  - ImPostingOnHN 2y ago
    I remember replacing a CLI program built like Lego blocks. It was 90-100 LEGO blocks, written over the course of decades, in: Cobol; Fortran; C; Java; Bash; and Perl, and the Legos "connected" with environmental variables. Nobody wanted to touch it lest they break it. Sometimes it's possible to do things too smartly. Apache Spark runs locally (and via CLI).
    - pdimitar 2y ago
      No no, I didn't mean that at all. I meant a script using well-known CLI programs. Obviously organically grown Frankenstein programs are a huge liability, I think every reasonable techie agrees on that.
      - actionfromafar 2y ago
        Well your little CLI-query is suddenly in production and then... it easily escalates.
        pdimitar 2y ago
        I already said I never managed a data lake and simply got stuff when it was needed but if you need to criticize then by all means, go wild.
ImPostingOnHN 2y ago
> But with time I am wondering if such a thing even exists Check out "data science at the command line": https://jeroenjanssens.com/dsatcl/ https://jeroenjanssens.com/dsatcl/
apwell23 2y ago
> make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door as someone who i cannot trust data with.
- zaphar 2y ago
  What about his answer prevents any of that? As stated the question didn't require any of what you outline here. ZFS will probably do a better job of protecting your data than almost any other filesystem out there so it's not a bad foundation to start with if you want to protect data. Your entire post reeks of "I'm smarter than you" smugness while at the same time revealing no useful information or approaches. Near as I can tell no one should trust you with any data.
  - apwell23 2y ago
    > Your entire post reeks of "I'm smarter than you" unlike "blows my mind" ? > As stated the question didn't require any of what you outline here. Right. OP mentioned it was "tricky question" . What makes it tricky is that all those attributes are implicitly assumed. I wouldn't interview at google and tell them my "stack" is "load it on your laptop". I would never say that in an interview even if I think that's the right "stack" .
    - zaphar 2y ago
      "blows my mind" is similar in tone yes. But I wasn't replying to the OP. Further the OP actually goes into some detail about how he would approach the problem. You do not. You are assuming you know what the OP meant by tricky question. And your assumption contradicts the rest of the OP's post regarding what he considered good answers to the question and why.
      - apwell23 2y ago
        [dead]

WesolyKubeczek 2y ago

> just go wild with CLI programs, I know dozens that work on CSV and XML ...or put it into SQLite for extra blazing fastness! No kidding.

pdimitar 2y ago
That's included in CLI tools. Also duckdb and clickhouse-local are amazing.
- WesolyKubeczek 2y ago
  I need to learn more about the latter for some log processing...
  - fijiaarone 2y ago
    Log files aren’t data. That’s your first problem. But that’s the only thing that most people have that generates more bytes than can fit on screen in a single spreadsheet.
    - thfuran 2y ago
      Of course they are. They just aren't always structured nicely.
    - WesolyKubeczek 2y ago
      Everything is data if you are brave enough.
- c0brac0bra 2y ago
  clickhouse-local had been astonishingly fast for operating on many GB of local CSVs. I had a heck of a time running the server locally before I discovered the CLI.

nevi-me 2y ago

To be fair on candidates, CLI programs create technical debt the moment they're written. A good answer that strikes a balance between size of data, latency and frequency requirements is a candidate who is able to show that they can choose the right tool that the next person will be comfortable with.

pdimitar 2y ago
True on the premise, yep, though I'm not sure how using CLI programs like LEGO blocks creates a tech debt?
- ImPostingOnHN 2y ago
  I remember replacing a CLI program built like Lego blocks. It was 90-100 LEGO blocks, written over the course of decades, in: Cobol; Fortran; C; Java; Bash; and Perl, and the Legos "connected" with environmental variables. Nobody wanted to touch it lest they break it. Sometimes it's possible to do things too smartly. Apache Spark runs locally (and via CLI).
  - pdimitar 2y ago
    No no, I didn't mean that at all. I meant a script using well-known CLI programs. Obviously organically grown Frankenstein programs are a huge liability, I think every reasonable techie agrees on that.
    - actionfromafar 2y ago
      Well your little CLI-query is suddenly in production and then... it easily escalates.
      - pdimitar 2y ago
        I already said I never managed a data lake and simply got stuff when it was needed but if you need to criticize then by all means, go wild.

ImPostingOnHN 2y ago

> But with time I am wondering if such a thing even exists Check out "data science at the command line": https://jeroenjanssens.com/dsatcl/ https://jeroenjanssens.com/dsatcl/

apwell23 2y ago

> make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door as someone who i cannot trust data with.

zaphar 2y ago
What about his answer prevents any of that? As stated the question didn't require any of what you outline here. ZFS will probably do a better job of protecting your data than almost any other filesystem out there so it's not a bad foundation to start with if you want to protect data. Your entire post reeks of "I'm smarter than you" smugness while at the same time revealing no useful information or approaches. Near as I can tell no one should trust you with any data.
- apwell23 2y ago
  > Your entire post reeks of "I'm smarter than you" unlike "blows my mind" ? > As stated the question didn't require any of what you outline here. Right. OP mentioned it was "tricky question" . What makes it tricky is that all those attributes are implicitly assumed. I wouldn't interview at google and tell them my "stack" is "load it on your laptop". I would never say that in an interview even if I think that's the right "stack" .
  - zaphar 2y ago
    "blows my mind" is similar in tone yes. But I wasn't replying to the OP. Further the OP actually goes into some detail about how he would approach the problem. You do not. You are assuming you know what the OP meant by tricky question. And your assumption contradicts the rest of the OP's post regarding what he considered good answers to the question and why.
    - apwell23 2y ago
      [dead]