5 ms·

I am supportive of clear explanations of some of the building blocks, but I worry repeatedly describing things as being grade school math level gives the wrong

• lettergram 10y ago
  I setup an automated script to setup a AWS g2 instance, train my neural net using tensorflow, copy my model to my personal computer, and spin down. It costs like $5-$10 to train/test most neural network models. My most expensive model cost like $100 and required a ton of time and resources. It took like 4 days or something. You really dont need $2k workstation.. Of course, for personal use I do have a gtx 1080 because I like to game and play tensorflow/caffe
  • danieltillett 10y ago
    Would you mind sharing this script? I am thinking of investing in the 2K workstation, but this sounds like a much more efficient way to go.
    • lettergram 10y ago
      I tried to get it released before, but was shut down by the "open source office". So I can't give you the exact script. However, h2o has a script that launches a cluster, that's very similar: https://github.com/ledell/h2oEnsemble-benchmarks/blob/master/ec2/h2o-cluster-launch-instances.py https://github.com/ledell/h2oEnsemble-benchmarks/blob/master... You can also do stuff like following in python to execute some code: os.system("scp script_demo.py " + sys.argv[1] + ":/home/ubuntu/") os.system("ssh " + sys.argv[1] + " sudo chmod 777 /home/ubuntu/script_demo.py") os.system("ssh " + sys.argv[1] + " './script_demo.py'") os.system("scp " + sys.argv[1] + ":/home/ubuntu/output/* ./output/") Then it will execute, after it executes - shut it down.
      • viraptor 10y ago
        Even if it may not matter in this case specifically, this is a terrible example. "sudo chmod 777" in public code is basically "I don't know what I'm doing, but go on, do the same thing yourself" :( You don't need sudo, because it's your file. You need just "chmod 755", not "777". And you don't need chmod in the first place - just run "python script_demo.py".
        • oh_sigh 10y ago
          Yes. On the other hand, there isn't a strong overlap with machine learning experts and security experts(I'm sure there will be in coming years though, as security experts start using NNs to detect anomalous/dangerous behavior).
          • viraptor 10y ago
            I know what you mean, but I disagree with "security experts". You don't need to be a security expert, just know the basics of how privileges work on your system. And if people don't, we just need to keep calling it out, because otherwise code like this ends up in production one day, where it actually matters. Simpler bash equivalent for the record: scp script_demo.py "$1" ssh "$1" python script_demo.py scp "$1":output/* ./output/
            • anantzoid 10y ago
              Exactly. If you're a ML person, you must be working with Linux in most cases. And in such case, you need to be familiar with the types of permissions, cause that's relevant even during the installation of various libraries, doing ssh etc.
        • danieltillett 10y ago
          These things come about because people learn the simple way to solve all permission problems is just do everything as su (or root) and chmod everything to 777. The problem is it does solve all problems (well sweep them under the rug) - I guess when the only tool you have is a hammer everything looks like a nail.
  • mrfusion 10y ago
    What kind of models do y'all make?
  • anantzoid 10y ago
    Spot instances cost way cheaper. The only downside is you need to create an AMI everytime before termination. But, also, AWS g2 has NVIDIA Grid K50 with 4GB memory, so it's not very good with performance.
    • alien_ 10y ago
      The AMI creation is only needed if you store data on the machines, which you shouldn't do anyway, not even with on-demand ones. You should always try to keep the instances stateless and store any data outside the instances, such as on S3 or EFS.
• ionforce 10y ago
  It's a better start than having no visibility into the topic at all.