4 ms·

Lots of data scientists, ex-statisticians, analysts, etc. are pressured with "you should use git". It's generally a noble thing, and the right one -- these folk

Lots of data scientists, ex-statisticians, analysts, etc. are pressured with "you should use git". It's generally a noble thing, and the right one -- these folks often otherwise don't use git enough for their code. It does however come with this risk, that they let data, or Jupyter notebook output or etc. get committed. I'm not saying what happened here is excusable -- it's not, it's ridiculously bad. But I can see a few ways how it'd happen. To me the equal or better question is why does this person have PII in the first place? It's not necessary to do analysis. Someone should have masked or removed it before this person ever got their hands on it. There's no way a PII dataset is needed for training.

shadowgovt 5y ago
Correct. This is a common failure mode... The easiest way to build a training set for a system is to take some live data, salami slice it, and store it in a static location. ... that's also a short path to dropping a user's PII in the clear.
bborud 5y ago
> these folks often otherwise don't use git enough for their code. Or at all. This is actually a fairly common problem among certain types of research scientists who think software engineering is "peasant work". I remember a friend of mine working at a research organization where several researchers lost months of work due to an unscheduled server reboot. Turns out when you log into ephemeral containers and pretend they are VMs, things go poof.
- mynameisvlad 5y ago
  I read a post on Reddit's legal advice subreddit of a data scientist having their apartment robbed, and part of it was saying they'd lost all their code and currently working on projects. As horrible as the robbery was, I was screaming inside "WHAT ABOUT VERSION CONTROL?!??!"
  - 999900000999 5y ago
    That’s a really sad story. If he doesn’t come from a programming background version can seem foreign. But like ya know… Dropbox. I’d be more concerned with my hard drive corrupting than anything else
  - alpaca128 5y ago
    The worst story I read was of someone working on his PhD thesis for 2 years, and then he forgot his laptop in a bus and everything was gone because in 2 years he didn't store the files anywhere else. I personally met a teacher who stored the only copy of her students' final exam submissions on her everyday thumb drive without backup. For many people technology really is magic.
- dharmab 5y ago
  I worked at a company which had a bunch of VMs for their data science teams. They replaced them with containers, then suddenly panicked when they realized that the containers would lose state when the admins of the container hosts applied security patches and rebooted or removed the outdated nodes. (The old VMs were rarely, if ever, patched after the initial boot, which became a compliance problem.) It turned into a monstrous issue where the data scientists wanted 6 months of continuity, the data platform SREs wanted 30 days and the container host admins wanted 15 minutes. I think they might still be deadlocked to this day.
  - 999900000999 5y ago
    Wouldn’t it be possible to automatically backup the data before reboot ? Docker offers persistent volumes, so my 30 second solution would be that
    - dharmab 5y ago
      I believe that was asked for by the SREs- e.g. Tensorflow supports checkpointing to disk and restoring progress- but the ML training software used by the data scientists did not have this feature.
    - bborud 5y ago
      That assumes a willingness to learn enough about Docker to understand how you configure your container to do what you want. That can, in some organizations, be too much to ask of researchers. That's the core of the problem.
  - pgwhalen 5y ago
    I love hearing these kinds of stories just to remind myself how not dysfunctional my own company actually is.
  - icedchai 5y ago
    I know plenty of companies that rarely apply any patches or do OS upgrades. I'm talking servers with 4 to 5 year uptimes, still running Ubuntu 16.04 or worse. They don't want to reboot because some dude who left 4 years ago set everything up, often in a non-standard way, and nobody is quite sure how it works. They certainly don't want to be blamed when production goes down. I did a contract job for a very large company that was running a 6 year old distro on a "critical" server and was afraid to do anything beyond changing a password. It's easier to have an outside person do it, so they can blame them when it gets screwed up.
ipaddr 5y ago
Use git but don't use github.
- raziel2p 5y ago
  git alone doesn't protect you against accidentally deleting your code directory and whatnot, and GitHub is the most accessible remote git offering right now.
  - number6 5y ago
    I was thinking: what about just some different PC where you got ssh access. Yes this might not be accessible enough. So the next thing could be gitlab. They offer free private repos
    - deleted 5y ago
      [deleted]
  - bcrosby95 5y ago
    Isn't this what backups are for?
  - ipaddr 5y ago
    Your operating system might. But it doesn't protect against your machine dying. That's a different problem with different solutions. You could self host on a different machine... Is github the most accessible? It is the most popular and best for discovery but a discussion around if gitlabs or bitbucket being more accessible could be had.
Dylan16807 5y ago
> It does however come with this risk, that they let data, or Jupyter notebook output or etc. get committed. You make that sound like it's generically a bad thing. Having the data you used and tested with is good in many ways. Having it in the same git branch and repo as everything else is suboptimal but usually a lot better than not safekeeping it at all.

shadowgovt 5y ago

Correct. This is a common failure mode... The easiest way to build a training set for a system is to take some live data, salami slice it, and store it in a static location. ... that's also a short path to dropping a user's PII in the clear.

bborud 5y ago

> these folks often otherwise don't use git enough for their code. Or at all. This is actually a fairly common problem among certain types of research scientists who think software engineering is "peasant work". I remember a friend of mine working at a research organization where several researchers lost months of work due to an unscheduled server reboot. Turns out when you log into ephemeral containers and pretend they are VMs, things go poof.

mynameisvlad 5y ago
I read a post on Reddit's legal advice subreddit of a data scientist having their apartment robbed, and part of it was saying they'd lost all their code and currently working on projects. As horrible as the robbery was, I was screaming inside "WHAT ABOUT VERSION CONTROL?!??!"
- 999900000999 5y ago
  That’s a really sad story. If he doesn’t come from a programming background version can seem foreign. But like ya know… Dropbox. I’d be more concerned with my hard drive corrupting than anything else
- alpaca128 5y ago
  The worst story I read was of someone working on his PhD thesis for 2 years, and then he forgot his laptop in a bus and everything was gone because in 2 years he didn't store the files anywhere else. I personally met a teacher who stored the only copy of her students' final exam submissions on her everyday thumb drive without backup. For many people technology really is magic.
dharmab 5y ago
I worked at a company which had a bunch of VMs for their data science teams. They replaced them with containers, then suddenly panicked when they realized that the containers would lose state when the admins of the container hosts applied security patches and rebooted or removed the outdated nodes. (The old VMs were rarely, if ever, patched after the initial boot, which became a compliance problem.) It turned into a monstrous issue where the data scientists wanted 6 months of continuity, the data platform SREs wanted 30 days and the container host admins wanted 15 minutes. I think they might still be deadlocked to this day.
- 999900000999 5y ago
  Wouldn’t it be possible to automatically backup the data before reboot ? Docker offers persistent volumes, so my 30 second solution would be that
  - dharmab 5y ago
    I believe that was asked for by the SREs- e.g. Tensorflow supports checkpointing to disk and restoring progress- but the ML training software used by the data scientists did not have this feature.
  - bborud 5y ago
    That assumes a willingness to learn enough about Docker to understand how you configure your container to do what you want. That can, in some organizations, be too much to ask of researchers. That's the core of the problem.
- pgwhalen 5y ago
  I love hearing these kinds of stories just to remind myself how not dysfunctional my own company actually is.
- icedchai 5y ago
  I know plenty of companies that rarely apply any patches or do OS upgrades. I'm talking servers with 4 to 5 year uptimes, still running Ubuntu 16.04 or worse. They don't want to reboot because some dude who left 4 years ago set everything up, often in a non-standard way, and nobody is quite sure how it works. They certainly don't want to be blamed when production goes down. I did a contract job for a very large company that was running a 6 year old distro on a "critical" server and was afraid to do anything beyond changing a password. It's easier to have an outside person do it, so they can blame them when it gets screwed up.

ipaddr 5y ago

Use git but don't use github.

raziel2p 5y ago
git alone doesn't protect you against accidentally deleting your code directory and whatnot, and GitHub is the most accessible remote git offering right now.
- number6 5y ago
  I was thinking: what about just some different PC where you got ssh access. Yes this might not be accessible enough. So the next thing could be gitlab. They offer free private repos
  - deleted 5y ago
    [deleted]
- bcrosby95 5y ago
  Isn't this what backups are for?
- ipaddr 5y ago
  Your operating system might. But it doesn't protect against your machine dying. That's a different problem with different solutions. You could self host on a different machine... Is github the most accessible? It is the most popular and best for discovery but a discussion around if gitlabs or bitbucket being more accessible could be had.

Dylan16807 5y ago

> It does however come with this risk, that they let data, or Jupyter notebook output or etc. get committed. You make that sound like it's generically a bad thing. Having the data you used and tested with is good in many ways. Having it in the same git branch and repo as everything else is suboptimal but usually a lot better than not safekeeping it at all.