4 ms·

> Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door a

> Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door as someone who i cannot trust data with. No need to act smug and superior, especially since nothing about OP's plan here actually precludes having all the nice things you mentioned, or even having them inside $your_favorite_enterprise_environment. You risk coming across as a person who feels threatened by simple solutions, perhaps someone who wants to spend $500k in vendor subscriptions every year for simple and/or imaginary problems... exactly the type of thing TFA talks about. But I'll ask the question.. why do you think safety, auditablity, access control, and knowledge sharing are incompatible with CLI tools and a specific choice of file system? What's your preferred alternative? Are you sticking with that alternative regardless of how often the work load runs, how often it changes, and whether the data fits in memory or requires a cluster?

apwell23 2y ago
> No need to act smug and superior I responded with the same tone that gp responded with. "blows my mind" ( that people can be so stupid) .
- photonthug 2y ago
  Another comment mentions this classic meme: > Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000. A lot of industry work really does fall into this category, and it's not controversial to say that going the wrong way on this thing is mind-blowing. More than not being controversial, it's not confrontational, because his comment was essentially re: the industry, whereas your comment is directed at a person. Drive by sniping where it's obvious you don't even care to debate the tech itself might get you a few "sick burn, bro" back-slaps from certain crowds, or the FUD approach might get traction with some in management, but overall it's not worth it. You don't sound smart or even professional, just nervous and afraid of every approach that you're not already intimately familiar with.
  - apwell23 2y ago
    i repurposed the parent comment "not understanding the scale of "real" big data was a no-go in my eyes when hiring." , "real winner" ect. But yea you are right. I shouldn't have directed it at commenter. I was miffed at interviewers who use "tricky questions" and expect people to read their minds and come up with their preconceived solution.
- pdimitar 2y ago
  The classic putting words in people's mouths technique it is then. The good old straw man. If you really must know: I said "blows my mind [that people don't try simpler and proven solutions FIRST]". I don't know what do you have to gain to come here and pretend to be in my head. Now here's another thing that blows my mind.
  - apwell23 2y ago
    > that people don't try simpler and proven solutions FIRST Well why don't people do that according to you ? Its not 'mind blowing' to me because you can never guess what angle interviewer is coming at you. Especially when they use the words like ' data stack'.
    - pdimitar 2y ago
      I don't know why and this is why I said it's mind-blowing. Because to me trying stuff that can work on most laptops comes naturally in my head as the first viable solution. As for interviews, sure, they have all sorts of traps. It really depends on the format and the role. Since I already disclaimed that I am not actual data scientist and just a seasoned dev who can make some magic happen without a dedicated data team (if/when the need arises) then I wouldn't even be in a data scientist interview in the first place. ¯\_(ツ)_/¯
      - apwell23 2y ago
        Thats fair. My comment wasn't directed at you. I was trying to be smart and write an inverse of original comment. Where I as an interviewer was looking for a proper 'data stack' and interviewee responded with a bespoke solution. "not understanding the scale of "real" big data was a no-go in my eyes when hiring."
        pdimitar 2y ago
        Sure, okay, I get it. My point was more like "Have you tried this obvious thing first that a lot of devs can do for you without too much hassle?". If I were to try for a dedicated data scientist position then I'd have done homework.
HelloNurse 2y ago
Abstractly, "safety, auditablity, access control, knowledge sharing" are about people reading and writing files: simplifying away complicated management systems improves security. The operating system should be good enough.

apwell23 2y ago

> No need to act smug and superior I responded with the same tone that gp responded with. "blows my mind" ( that people can be so stupid) .

photonthug 2y ago
Another comment mentions this classic meme: > Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000. A lot of industry work really does fall into this category, and it's not controversial to say that going the wrong way on this thing is mind-blowing. More than not being controversial, it's not confrontational, because his comment was essentially re: the industry, whereas your comment is directed at a person. Drive by sniping where it's obvious you don't even care to debate the tech itself might get you a few "sick burn, bro" back-slaps from certain crowds, or the FUD approach might get traction with some in management, but overall it's not worth it. You don't sound smart or even professional, just nervous and afraid of every approach that you're not already intimately familiar with.
- apwell23 2y ago
  i repurposed the parent comment "not understanding the scale of "real" big data was a no-go in my eyes when hiring." , "real winner" ect. But yea you are right. I shouldn't have directed it at commenter. I was miffed at interviewers who use "tricky questions" and expect people to read their minds and come up with their preconceived solution.
pdimitar 2y ago
The classic putting words in people's mouths technique it is then. The good old straw man. If you really must know: I said "blows my mind [that people don't try simpler and proven solutions FIRST]". I don't know what do you have to gain to come here and pretend to be in my head. Now here's another thing that blows my mind.
- apwell23 2y ago
  > that people don't try simpler and proven solutions FIRST Well why don't people do that according to you ? Its not 'mind blowing' to me because you can never guess what angle interviewer is coming at you. Especially when they use the words like ' data stack'.
  - pdimitar 2y ago
    I don't know why and this is why I said it's mind-blowing. Because to me trying stuff that can work on most laptops comes naturally in my head as the first viable solution. As for interviews, sure, they have all sorts of traps. It really depends on the format and the role. Since I already disclaimed that I am not actual data scientist and just a seasoned dev who can make some magic happen without a dedicated data team (if/when the need arises) then I wouldn't even be in a data scientist interview in the first place. ¯\_(ツ)_/¯
    - apwell23 2y ago
      Thats fair. My comment wasn't directed at you. I was trying to be smart and write an inverse of original comment. Where I as an interviewer was looking for a proper 'data stack' and interviewee responded with a bespoke solution. "not understanding the scale of "real" big data was a no-go in my eyes when hiring."
      - pdimitar 2y ago
        Sure, okay, I get it. My point was more like "Have you tried this obvious thing first that a lot of devs can do for you without too much hassle?". If I were to try for a dedicated data scientist position then I'd have done homework.

HelloNurse 2y ago

Abstractly, "safety, auditablity, access control, knowledge sharing" are about people reading and writing files: simplifying away complicated management systems improves security. The operating system should be good enough.