4 ms·

I... am... just like baffled at why Docker is used. Are any of these like not normal unix utilities? What dependencies are you pulling in after GNU/Linux... whi

I... am... just like baffled at why Docker is used. Are any of these like not normal unix utilities? What dependencies are you pulling in after GNU/Linux... which is where CONTAINERS come from? I'm even only even putting "GNU" here because it's literally every fucking Linux distro on the planet containing the utilities referenced. Is it common place now-a-days to bloat a docker image with text data or something? Is docker only for the csvlookup tool? Not trying to be a supermassive asshole but I do feel like I'm losing my mind reading this... In order to use GNU/Linux one must install Linux, setup a container runtime, and then! Use the same utilities available to them in the runtime... from a container? I feel like if this is the state of a tutorial for someone new to these utilities we're all pretty fucked and should probably take a step back to re-evaluate the mess we've made. I'll go crawl back into my hole now to do just that.

larusso 6y ago
I was also wondering why it starts with docker. But since this post is targeted for data science or becoming data science folk maybe the idea is to also support windows users?
- Maha-pudma 6y ago
  Would it not work in WSL though? If you wanted windows support.
  - larusso 6y ago
    Sure. It was just an idea why docker was choosen :)
  - dr_zoidberg 6y ago
    Yep, they all come with WSL, since you basically install Ubuntu (or another WSL-compatible distro).
- II2II 6y ago
  If you are interested in platform independence, why even bother with the Unix utilities? You can accomplish the same with Python, have results that are more readable to the non-Unix crowd, and not rely upon sometimes esoteric knowledge of the Unix utilities (and the inconsistencies between vendors). That isn't to say that the Unix utilities are without merit. I have used them extensively for similar tasks. That being said, it is in environments where everyone used some version of Unix (typically Linux, Solaris, or Mac OS X).
  - IanCal 6y ago
    > If you are interested in platform independence, why even bother with the Unix utilities? You can accomplish the same with Python, Because then you're also looking at getting people using the same version of python and your dependencies - and it's likely slower. > That being said, it is in environments where everyone used some version of Unix (typically Linux, Solaris, or Mac OS X). If only there was a technology aimed at sharing an exact same version for easy use ;)
inimino 6y ago
I was also baffled. However, if you follow the link to the "why we use docker" post you'll see the author is using a Mac and had difficulty before while following instructions written for Linux. Data scientists using Macs for everything and sometimes needing to run Linux in a container to get work done is more common than you might think.
- isoprophlex 6y ago
  brew install coreutils Lot less complexity than running things in docker.
  - leokennis 6y ago
    Was just about to say. What text editing powerhouse from Linux (sed, awk, grep etc.) are not available on Mac? The answer is: none.
    - isoprophlex 6y ago
      "Able to install packages" Hmmm no that don't look as good on my resume as "autodidact docker wizard" There's something to be said for the cool-new-tech driven approach, when it comes to resume building.
    - howlin 6y ago
      There are some rather archaic behaviors of the versions of these tools that come by default on MasOSX. For instance, on a mac the 'sed' command doesn't understand the difference between space and tab https://stackoverflow.com/questions/40311339/using-sed-on-mac-os-10-to-replace-tab-character-in-text-files/40312229 https://stackoverflow.com/questions/40311339/using-sed-on-ma...
      - leokennis 6y ago
        Oh, very likely true. But nothing stands in your way to just replace them with the GNU versions. Still way easier than running a Docker container...
scottlocklin 6y ago
csvlook/csvkit, apparently; something I never heard of before, which is actually not installed by default in ubuntu. He also included the data itself, which is pretty helpful, but which could have more usefully gone into a git repo somewhere. Also "kids these days" all reach for docker. Otherwise any sane person would have just used cat instead of csvlook. Upvoted the kid's article, because command line text cleaning is of TOWERING IMPORTANCE to any DS person worth their salt. But he is definitely an inexperienced kid.
- mdifrgechd 6y ago
  I think the csvlook utility comes from the Command Line Data Science book the author cites. I read some of that book, there was good info in there, but ultimately it was a bit annoying because it's author based a lot of things on utilities he had written himself (like csvlook) which in many cases just did trivial things you could easily have done with shell commands or coreutils.
rgoulter 6y ago
Docker 'can' be used as a way to provide a shell which has a consistent set of programs installed. (As a bonus, using Docker this way let's you avoid 'polluting' the host OS with various packages). Sure, this case where it's just coreutils + csvlookup, the benefit for the added abstractions isn't so great. Using docker like this makes more sense if there are more dependencies. (Maybe it's just easier to say "use docker" than to explain how to install a dependency on different OSs?). In general, I think it's fine for people to be able to solve problems without having a full understanding of the mechanisms involved in getting the tool to work. If they need to know it later, they'll learn it then. This is a top-down approach -- But, naturally, there'll be some awkward solutions on the way.
- rubyn00bie 6y ago
  No. I’m sorry but I disagree in this case. These tools are literally installed on every posix machine on the planet. The whole point here is that there are no dependencies this is the “system.” It’s like trying to explain how an engine works by only looking at the turbo, “Oh yeah, gotta have a turbo or your car won’t drive.” ... You mean I gotta have an engine or I can’t even use a turbo? The whole point of my post is that running a container otecshstration system to accomplish any of this, for someone new, is insane. It is absolutely 100% only adding complexity and making things more confusing. How many people will not realize these utilities are on every computer they use? How long until someone poisons these repacked uselessly opaque versions? This makes things easier for no one and only furthers to complicate things by adding unnecessary complexity. If someone is on a Mac, then they are now running a Linux VM, then running containers, then logging into said fucked up environment and then finallly... running... awk? macOS -> Linux -> Container Runtime -> SSH-> finally use awk, sed, head, or anything else. Why stop there? We can probably then start up Qemu from a container, boot MacOS, and then simply repeat the process for maximum usefulness.
  - IanCal 6y ago
    > If someone is on a Mac, then they are now running a Linux VM, then running containers, then logging into said fucked up environment and then finallly... running... awk? Err, no, they are not doing this. That's not how it's working for a start (there's no SSH here) but the user experience is literally Install a standard application, downloaded like any other. Open terminal. Run "docker run ..." That's it. > It is absolutely 100% only adding complexity and making things more confusing. How many people will not realize these utilities are on every computer they use? They aren't on every computer, and they are different on different computers. A point the author not only talks about in the linked explanation but describes how that bit them.
IanCal 6y ago
This is a very strong response to what seems like an obvious point clearly laid out at the start. This is a tutorial and set of examples, and it comes with a fully working environment with example data exactly setup to work. No difference between gsort and sort, no version issues or whether someone is using brew or macports or anything else. No python environment problems or pip Vs pip3 or whatever other random crap you need to deal with when trying to do something simple with lots of varied setups people have. You are entirely free to not use docker. > In order to use GNU/Linux one must Nope. You can. You don't have to.
- rubyn00bie 6y ago
  > This is a very strong response to what seems like an obvious point clearly laid out at the start. The point is invalid and clearly untrue. > This is a tutorial and set of examples, and it comes with a fully working environment with example data exactly setup to work. No difference between gsort and sort, no version issues or whether someone is using brew or macports or anything else. No python environment problems or pip Vs pip3 or whatever other random crap you need to deal with when trying to do something simple with lots of varied setups people have. I really don't see how having to get Docker going is alleviating any of the above headaches but I'm far removed from these tools being "new to me." Setting up virtualization to run a process on my local machine which I can run without virtualization or containerization the very same way seems crazy to me. It seems like an awful amount of complexity to push onto beginners for what is likely to be zero benefit and gives them only what they had at the start... > You are entirely free to not use docker. >> In order to use GNU/Linux one must > Nope. You can. You don't have to. Yeah that's my point exactly... you don't have to at all in fact, Docker offers most literally nothing but opaque complexity. I was saying you MUST have Linux if you want containers because containers are a Linux feature and DNE on MacOS but all them sweet utilities are already there save the weird csv tool. I don't think the complexity of Docker is warranted for simply handling a CSV tool and I find it to be a disservice to anyone who is learning. /shrug
  - IanCal 6y ago
    What point is invalid? That it's a prepackaged setup for all users to avoid having these kinds of problems? It's pointed out right at the beginning and then a longer description is linked to about how the writer hit exactly this kind of problem because mac has different basic utilities and their sed worked differently. > I really don't see how having to get Docker going is alleviating any of the above headaches Because it is a single thing to get working that then leaves all users of your tutorial on all major platforms dealing with the exact same environment. Have you ever tried to work with getting end users up to speed only to discover that one is using python2.7, another has anaconda but pip is pointing to the wrong place, pip3 is required for someone else and one user has mac so needs to install brew then replace every standard call to sort with gsort. Then dateutils appears as some conflicted dependency... > Setting up virtualization to run a process on my local machine which I can run without virtualization or containerization the very same way seems crazy to me Then don't, you don't have to. Go and setup an identical environment. For those that have docker this becomes a single command to run, and those that don't can get setup quickly. If they're heading into data science they're likely to benefit from having at least a basic understanding of docker. > It seems like an awful amount of complexity to push onto beginners for what is likely to be zero benefit and gives them only what they had at the start... It doesn't give zero benefit, it removes a whole bunch of differences in setup that are entirely irrelevant to the tutorial. > Docker offers most literally nothing but opaque complexity. It offers a standard environment that puts all users in the same place. > but all them sweet utilities are already there save the weird csv too No they are not, not on a mac and not on windows. > I don't think the complexity of Docker is warranted for simply handling a CSV tool What complexity? 'docker run --rm -it X bash' is literally all I need to do.

larusso 6y ago

I was also wondering why it starts with docker. But since this post is targeted for data science or becoming data science folk maybe the idea is to also support windows users?

Maha-pudma 6y ago
Would it not work in WSL though? If you wanted windows support.
- larusso 6y ago
  Sure. It was just an idea why docker was choosen :)
- dr_zoidberg 6y ago
  Yep, they all come with WSL, since you basically install Ubuntu (or another WSL-compatible distro).
II2II 6y ago
If you are interested in platform independence, why even bother with the Unix utilities? You can accomplish the same with Python, have results that are more readable to the non-Unix crowd, and not rely upon sometimes esoteric knowledge of the Unix utilities (and the inconsistencies between vendors). That isn't to say that the Unix utilities are without merit. I have used them extensively for similar tasks. That being said, it is in environments where everyone used some version of Unix (typically Linux, Solaris, or Mac OS X).
- IanCal 6y ago
  > If you are interested in platform independence, why even bother with the Unix utilities? You can accomplish the same with Python, Because then you're also looking at getting people using the same version of python and your dependencies - and it's likely slower. > That being said, it is in environments where everyone used some version of Unix (typically Linux, Solaris, or Mac OS X). If only there was a technology aimed at sharing an exact same version for easy use ;)

inimino 6y ago

I was also baffled. However, if you follow the link to the "why we use docker" post you'll see the author is using a Mac and had difficulty before while following instructions written for Linux. Data scientists using Macs for everything and sometimes needing to run Linux in a container to get work done is more common than you might think.

isoprophlex 6y ago
brew install coreutils Lot less complexity than running things in docker.
- leokennis 6y ago
  Was just about to say. What text editing powerhouse from Linux (sed, awk, grep etc.) are not available on Mac? The answer is: none.
  - isoprophlex 6y ago
    "Able to install packages" Hmmm no that don't look as good on my resume as "autodidact docker wizard" There's something to be said for the cool-new-tech driven approach, when it comes to resume building.
  - howlin 6y ago
    There are some rather archaic behaviors of the versions of these tools that come by default on MasOSX. For instance, on a mac the 'sed' command doesn't understand the difference between space and tab https://stackoverflow.com/questions/40311339/using-sed-on-mac-os-10-to-replace-tab-character-in-text-files/40312229 https://stackoverflow.com/questions/40311339/using-sed-on-ma...
    - leokennis 6y ago
      Oh, very likely true. But nothing stands in your way to just replace them with the GNU versions. Still way easier than running a Docker container...

scottlocklin 6y ago

csvlook/csvkit, apparently; something I never heard of before, which is actually not installed by default in ubuntu. He also included the data itself, which is pretty helpful, but which could have more usefully gone into a git repo somewhere. Also "kids these days" all reach for docker. Otherwise any sane person would have just used cat instead of csvlook. Upvoted the kid's article, because command line text cleaning is of TOWERING IMPORTANCE to any DS person worth their salt. But he is definitely an inexperienced kid.

mdifrgechd 6y ago
I think the csvlook utility comes from the Command Line Data Science book the author cites. I read some of that book, there was good info in there, but ultimately it was a bit annoying because it's author based a lot of things on utilities he had written himself (like csvlook) which in many cases just did trivial things you could easily have done with shell commands or coreutils.

rgoulter 6y ago

Docker 'can' be used as a way to provide a shell which has a consistent set of programs installed. (As a bonus, using Docker this way let's you avoid 'polluting' the host OS with various packages). Sure, this case where it's just coreutils + csvlookup, the benefit for the added abstractions isn't so great. Using docker like this makes more sense if there are more dependencies. (Maybe it's just easier to say "use docker" than to explain how to install a dependency on different OSs?). In general, I think it's fine for people to be able to solve problems without having a full understanding of the mechanisms involved in getting the tool to work. If they need to know it later, they'll learn it then. This is a top-down approach -- But, naturally, there'll be some awkward solutions on the way.

rubyn00bie 6y ago
No. I’m sorry but I disagree in this case. These tools are literally installed on every posix machine on the planet. The whole point here is that there are no dependencies this is the “system.” It’s like trying to explain how an engine works by only looking at the turbo, “Oh yeah, gotta have a turbo or your car won’t drive.” ... You mean I gotta have an engine or I can’t even use a turbo? The whole point of my post is that running a container otecshstration system to accomplish any of this, for someone new, is insane. It is absolutely 100% only adding complexity and making things more confusing. How many people will not realize these utilities are on every computer they use? How long until someone poisons these repacked uselessly opaque versions? This makes things easier for no one and only furthers to complicate things by adding unnecessary complexity. If someone is on a Mac, then they are now running a Linux VM, then running containers, then logging into said fucked up environment and then finallly... running... awk? macOS -> Linux -> Container Runtime -> SSH-> finally use awk, sed, head, or anything else. Why stop there? We can probably then start up Qemu from a container, boot MacOS, and then simply repeat the process for maximum usefulness.
- IanCal 6y ago
  > If someone is on a Mac, then they are now running a Linux VM, then running containers, then logging into said fucked up environment and then finallly... running... awk? Err, no, they are not doing this. That's not how it's working for a start (there's no SSH here) but the user experience is literally Install a standard application, downloaded like any other. Open terminal. Run "docker run ..." That's it. > It is absolutely 100% only adding complexity and making things more confusing. How many people will not realize these utilities are on every computer they use? They aren't on every computer, and they are different on different computers. A point the author not only talks about in the linked explanation but describes how that bit them.

IanCal 6y ago

This is a very strong response to what seems like an obvious point clearly laid out at the start. This is a tutorial and set of examples, and it comes with a fully working environment with example data exactly setup to work. No difference between gsort and sort, no version issues or whether someone is using brew or macports or anything else. No python environment problems or pip Vs pip3 or whatever other random crap you need to deal with when trying to do something simple with lots of varied setups people have. You are entirely free to not use docker. > In order to use GNU/Linux one must Nope. You can. You don't have to.

rubyn00bie 6y ago
> This is a very strong response to what seems like an obvious point clearly laid out at the start. The point is invalid and clearly untrue. > This is a tutorial and set of examples, and it comes with a fully working environment with example data exactly setup to work. No difference between gsort and sort, no version issues or whether someone is using brew or macports or anything else. No python environment problems or pip Vs pip3 or whatever other random crap you need to deal with when trying to do something simple with lots of varied setups people have. I really don't see how having to get Docker going is alleviating any of the above headaches but I'm far removed from these tools being "new to me." Setting up virtualization to run a process on my local machine which I can run without virtualization or containerization the very same way seems crazy to me. It seems like an awful amount of complexity to push onto beginners for what is likely to be zero benefit and gives them only what they had at the start... > You are entirely free to not use docker. >> In order to use GNU/Linux one must > Nope. You can. You don't have to. Yeah that's my point exactly... you don't have to at all in fact, Docker offers most literally nothing but opaque complexity. I was saying you MUST have Linux if you want containers because containers are a Linux feature and DNE on MacOS but all them sweet utilities are already there save the weird csv tool. I don't think the complexity of Docker is warranted for simply handling a CSV tool and I find it to be a disservice to anyone who is learning. /shrug
- IanCal 6y ago
  What point is invalid? That it's a prepackaged setup for all users to avoid having these kinds of problems? It's pointed out right at the beginning and then a longer description is linked to about how the writer hit exactly this kind of problem because mac has different basic utilities and their sed worked differently. > I really don't see how having to get Docker going is alleviating any of the above headaches Because it is a single thing to get working that then leaves all users of your tutorial on all major platforms dealing with the exact same environment. Have you ever tried to work with getting end users up to speed only to discover that one is using python2.7, another has anaconda but pip is pointing to the wrong place, pip3 is required for someone else and one user has mac so needs to install brew then replace every standard call to sort with gsort. Then dateutils appears as some conflicted dependency... > Setting up virtualization to run a process on my local machine which I can run without virtualization or containerization the very same way seems crazy to me Then don't, you don't have to. Go and setup an identical environment. For those that have docker this becomes a single command to run, and those that don't can get setup quickly. If they're heading into data science they're likely to benefit from having at least a basic understanding of docker. > It seems like an awful amount of complexity to push onto beginners for what is likely to be zero benefit and gives them only what they had at the start... It doesn't give zero benefit, it removes a whole bunch of differences in setup that are entirely irrelevant to the tutorial. > Docker offers most literally nothing but opaque complexity. It offers a standard environment that puts all users in the same place. > but all them sweet utilities are already there save the weird csv too No they are not, not on a mac and not on windows. > I don't think the complexity of Docker is warranted for simply handling a CSV tool What complexity? 'docker run --rm -it X bash' is literally all I need to do.