4 ms·

Think it depends but it does happen. We’ve been able to carve ourselves out of IT with our own infrastructure and our data scientists specifically focus on rese

Think it depends but it does happen. We’ve been able to carve ourselves out of IT with our own infrastructure and our data scientists specifically focus on research and analysis. The only issue is when we try to get anything to prod and we hit IT.

mlthoughts2018 8y ago
> “The only issue is when we try to get anything to prod and we hit IT.” Which inevitably means the one person on the data science team who is good with linux and docker suddenly becomes the IT wizard, and their time gets sucked up by having to find ways to go around utterly stupid barriers and tactics used by IT to avoid doing work to help you. Source: I am currently this person for my team.
- deleted 8y ago
  [deleted]
- tixocloud 8y ago
  I feel your pain. I am in it too. Would love to get your thoughts some time - I'm building a product to make life a little bit easier (3 step deploy model as API) but with a vision towards more broaded deployment usecases. Your advice will be valuable and much appreciated!
- logosmonkey 8y ago
  As someone who is about to be tasked with building this sort of infrastructure for a hospital system what would be your dream architecture? I hate IT hurdles and I am hoping I can avoid building them into our infrastructure.
  - mlthoughts2018 8y ago
    The number one thing is to make containers a first class deployment and provisioning artifact. As long as dev teams can control their own containers, they will be able to do what they need no matter how arbitrary or assumption-breaking. Do not ever require dev teams to go through IT to get their chosen tools deployed to the right places or with the right resources provisioned. Never. This is the root of all evil with infra teams: if they see themselves or their mandate as being gatekeepers of provisioned resources, then dev teams have lost and you as a data scientist / ML engineer, you’ll never get your work done. As an ML engineer, I want to define the entire runtime and development environments of any analytics artifacts or web services that I create, and to change these environments as needed, as indicated by what’s required to get the job done. Let me define containers, hook them up in whatever CI tools are used, push and pull them from some internal container repository, and describe the configuration for the resources they need. Offer that as the contract to dev teams and then infra’s job is to maintain the underlying data center that physically supports running the containers and occasional hand holding for special exceptions, networking, secrets management, and cost tracking.
    - logosmonkey 8y ago
      Good, that's my current plan! I appreciate your input.
    - tixocloud 8y ago
      Very well said. I can’t stress the container environment enough not to mention the ability to control the toolset.
- Xcelerate 8y ago
  Used to be this person at a previous company. I didn't run molecular dynamics simulations on supercomputers so I could explain to the CISO why a machine learning library isn't a virus.
telchar 8y ago
Don't know why you're getting downvoted for this - we have the same pain. We're not allowed to have a sysadmin or administer our own servers but IT is busy supporting thousands of non-computery-users. To them we're a fly in the ointment.
- tixocloud 8y ago
  I find it quite interesting as technically speaking both IT and data science are technical. Pardon the pun. The real value of data science teams comes from great infrastructure in addition to a solid team so it is a bit of a pity sometimes.

mlthoughts2018 8y ago

> “The only issue is when we try to get anything to prod and we hit IT.” Which inevitably means the one person on the data science team who is good with linux and docker suddenly becomes the IT wizard, and their time gets sucked up by having to find ways to go around utterly stupid barriers and tactics used by IT to avoid doing work to help you. Source: I am currently this person for my team.

deleted 8y ago
[deleted]
tixocloud 8y ago
I feel your pain. I am in it too. Would love to get your thoughts some time - I'm building a product to make life a little bit easier (3 step deploy model as API) but with a vision towards more broaded deployment usecases. Your advice will be valuable and much appreciated!
logosmonkey 8y ago
As someone who is about to be tasked with building this sort of infrastructure for a hospital system what would be your dream architecture? I hate IT hurdles and I am hoping I can avoid building them into our infrastructure.
- mlthoughts2018 8y ago
  The number one thing is to make containers a first class deployment and provisioning artifact. As long as dev teams can control their own containers, they will be able to do what they need no matter how arbitrary or assumption-breaking. Do not ever require dev teams to go through IT to get their chosen tools deployed to the right places or with the right resources provisioned. Never. This is the root of all evil with infra teams: if they see themselves or their mandate as being gatekeepers of provisioned resources, then dev teams have lost and you as a data scientist / ML engineer, you’ll never get your work done. As an ML engineer, I want to define the entire runtime and development environments of any analytics artifacts or web services that I create, and to change these environments as needed, as indicated by what’s required to get the job done. Let me define containers, hook them up in whatever CI tools are used, push and pull them from some internal container repository, and describe the configuration for the resources they need. Offer that as the contract to dev teams and then infra’s job is to maintain the underlying data center that physically supports running the containers and occasional hand holding for special exceptions, networking, secrets management, and cost tracking.
  - logosmonkey 8y ago
    Good, that's my current plan! I appreciate your input.
  - tixocloud 8y ago
    Very well said. I can’t stress the container environment enough not to mention the ability to control the toolset.
Xcelerate 8y ago
Used to be this person at a previous company. I didn't run molecular dynamics simulations on supercomputers so I could explain to the CISO why a machine learning library isn't a virus.

telchar 8y ago

Don't know why you're getting downvoted for this - we have the same pain. We're not allowed to have a sysadmin or administer our own servers but IT is busy supporting thousands of non-computery-users. To them we're a fly in the ointment.

tixocloud 8y ago
I find it quite interesting as technically speaking both IT and data science are technical. Pardon the pun. The real value of data science teams comes from great infrastructure in addition to a solid team so it is a bit of a pity sometimes.