4 ms·

I once worked somewhere with kind of old-school sysadmin guys. I had to spend so much time asking and badgering one to add disk space/allocate me more disk spac

I once worked somewhere with kind of old-school sysadmin guys. I had to spend so much time asking and badgering one to add disk space/allocate me more disk space on a computing cluster to load a model. The cost of the excess space was probably around $1-5, averaged out of the cost of a large disk. In a cloud environment I could do this myself instantly or not even have to do it at all. Especially as the cloud becomes more popular, and devops/modern tooling begins to be adopted in house even for companies that aren’t “on the cloud”, I see the role of sysadmins who aren’t also software engineers going away.

caro_douglos 7y ago
What was the iops you eventually got or needed? Since you mentioned model and cluster I’m curious as to wether or not others were also on said cluster.
- opportune 7y ago
  The way this computing cluster worked, each person had some default disk/memory space they were allocated out of a large total. I don’t know exactly how it was set up. People were running things like Hadoop jobs, Spark, Jupyter notebooks. I think the default disk space allowed was very small, like 5GB, and I had some large NLP model that was 8GB (probably the entire library and the part I needed was smaller). I needed to save the NLP library to my user account’s disk so I would only have to download it once. I think I could redownload it every job, but that would take a pretty long time, and had other issues. So it wasn’t even IOPS, it was just the disk space I was allocated on my user account. Probably a one line command by the sysadmin guy, unless he also needed to go get another hard drive and plug it in, in which case it would be 10 minutes and then a one line command.
  - deleted 7y ago
    [deleted]
pmoriarty 7y ago
"The cost of the excess space was probably around $1-5, averaged out of the cost of a large disk." So what do you do when that disk dies and you lose all your data? Oh, you want backups? Well, maybe you could do them yourself. Oh, you don't want to do the backups yourself but want the sysadmins to do your backups for you, do you? Or what happens when the network that's hosting the disk becomes inaccessible? You want to get to your data anyway, so that means adding network redundancy. And you also probably don't want downtime if the server hosting the disk goes down, or the server is up but the disk dies, so that means RAID or maybe a distributed filesystem. And what about making sure the data is secure and the servers hosting it meet compliance requirements? And when something goes wrong you want someone in ops to troubleshoot it, right? Maybe you want six 9's of availability, and you want us to wear a pager so we can work on fixing it at any time of the day or night, right? All of this will take planning and time and more hardware, configuration and monitoring, perhaps a bigger headcount since sysadmin teams are often already running short-staffed and don't have the time to dedicate to supporting even more infrastructure than they already do. You see, it's not always so simple as "just add another disk".
- opportune 7y ago
  Or I just use the cloud and configure all that myself without an arbitrary gatekeeper I didn’t even care about any of that stuff. I just needed to be able to work on my own model using some other model that I could always redownload if there was an issue. If I had been empowered to just do it all myself it would have been a 0-10 minute task.

caro_douglos 7y ago

What was the iops you eventually got or needed? Since you mentioned model and cluster I’m curious as to wether or not others were also on said cluster.

opportune 7y ago
The way this computing cluster worked, each person had some default disk/memory space they were allocated out of a large total. I don’t know exactly how it was set up. People were running things like Hadoop jobs, Spark, Jupyter notebooks. I think the default disk space allowed was very small, like 5GB, and I had some large NLP model that was 8GB (probably the entire library and the part I needed was smaller). I needed to save the NLP library to my user account’s disk so I would only have to download it once. I think I could redownload it every job, but that would take a pretty long time, and had other issues. So it wasn’t even IOPS, it was just the disk space I was allocated on my user account. Probably a one line command by the sysadmin guy, unless he also needed to go get another hard drive and plug it in, in which case it would be 10 minutes and then a one line command.
- deleted 7y ago
  [deleted]

pmoriarty 7y ago

"The cost of the excess space was probably around $1-5, averaged out of the cost of a large disk." So what do you do when that disk dies and you lose all your data? Oh, you want backups? Well, maybe you could do them yourself. Oh, you don't want to do the backups yourself but want the sysadmins to do your backups for you, do you? Or what happens when the network that's hosting the disk becomes inaccessible? You want to get to your data anyway, so that means adding network redundancy. And you also probably don't want downtime if the server hosting the disk goes down, or the server is up but the disk dies, so that means RAID or maybe a distributed filesystem. And what about making sure the data is secure and the servers hosting it meet compliance requirements? And when something goes wrong you want someone in ops to troubleshoot it, right? Maybe you want six 9's of availability, and you want us to wear a pager so we can work on fixing it at any time of the day or night, right? All of this will take planning and time and more hardware, configuration and monitoring, perhaps a bigger headcount since sysadmin teams are often already running short-staffed and don't have the time to dedicate to supporting even more infrastructure than they already do. You see, it's not always so simple as "just add another disk".

opportune 7y ago
Or I just use the cloud and configure all that myself without an arbitrary gatekeeper I didn’t even care about any of that stuff. I just needed to be able to work on my own model using some other model that I could always redownload if there was an issue. If I had been empowered to just do it all myself it would have been a 0-10 minute task.