3 ms·

My post was too long so I am breaking it down. Part 1 So... This is where I got my start in tech. I worked there and and supported and racked many of said br

My post was too long so I am breaking it down. Part 1 So... This is where I got my start in tech. I worked there and and supported and racked many of said bread rack servers. I guess I could be wrong and they could be speaking of another company but there was only one company I ever knew to do this at it was rackshack, before it became ev1servers, and then later theplanet and now softlayer. There were a few others in Houston at the time trying to get this working too, namely redhost (I think, or was it alphared or something like that. ) which I had the chance to visit once. This place was much worse they cooled the iles the customers would walk down meanwhile the servers got no AC. They also shimmied servers into parking lot storage rooms. But that is not likely the place the article was about. While what described in the article sounded horrible it was not all that bad. But then again this was also the early 2000s. While each rack did not have a dedicated firewall they did provide some niceties globally to all servers such as basic dos attack prevention and some interesting but easily avoidable hosting panel exploit prevention. It really was the wild wild west of hosting. You had the tech support named webtech who was normally off site and basically was a job for anybody who could spell linux. They did in house training to support the most common linux related issues. When I was there there was a guy named Bill Badoux. This guy ran training and had a fairly comprehensive training class for all incoming techs, but none of that really mattered, most of the real work always filtered down to a handful of knowledgeable people. What was important was somebody answered the phone and was there to listen to the customer -- mostly complaining. On the other end of things we had DC techs. They were mostly about rebooting servers as none of the whitebox servers on bread racks supported any sort of remote reboot. At the time you would phone in your reboot, or submit a ticket in this POS software called "pipeline" which was a total shitstorm in house ticketing system, that would more likely get you fired for it doing something different than you thought it might do than actually help you do anything. But us techs made the best of it and often resorted to IRC to actually get the real work done. The NOC was a separate entity -- I think their paychecks even came from a different org. They mostly sat around and watched graphs -- and again would call somebody who knew what to do when "the graphs looked bad". I know one guy nicked named Brain at the time who was one of the knowledgeable guys. He ended up writing some software to be able to display the graphs in real time. This really helped the techs spot things much faster. I was there about ~4 years. Man there were stories. One time I saw a DC tech drop a handful of hard drives on the way to a server to replace drives that had just failed. He picked the dropped hard drive up off the floor, dusted it off and proceeded to replace a failed dive with the drive he just dropped. By chance the server booted up and entered a FSCK -- we might be okay! -- but atlas the tech just powered the server off and powered it back one and it never saw the light of day again. Back then and probably still interrupting a FSCK was most defiantly doom to the FS. Slightly NFSW but a true story. Rackshack hosted 2 things - Porn, and Jesus sites. Neither were all that nice to work with. The Jesus guys would always find out or suspect their server, hard drive, or IP was used to host porn in a previous life and demand we change one or all of the things they suspected had porn on it. This more than often resulted in your request causing your server to be shut down for a few hours and then turned back on and a note updating the ticket from DC stating the hard drive or whatever that had previously had contact with porn was swapped out. In reality the server just sat powered off for a few hours. Request for IP changes did happen and were actually carried out if they could, but again this is was the early days so that often required the customer to migrate their site to a new server in the data center. But back to the porn. One time I got a call from a guy wanting to know if his site was up. He ask me to navigate to a obvious porn site -- sure enough on our network with the less than 1 ms ping time. He proceeded to give me login information and navigate me to a specific video and asked me to click on it. It played instantly. After a few seconds I reported back to the customer and he asked a few question about the speed of the site. Then he proceeded to tell me the video was of him and his wife.... Now at that point I kept good notes in the POS ticketing system named pipeline. Those notes included the login credentials to the site for future testing -- I was a noob for sure. The next day I walk in to work and it was clear everybody had found the login credentials to the site and that was the first time I caused a policy change of do not record such things as the techs will abuse it for their personal pleasure.

mbrumlow 9y ago
Part 2 On another occasion I had to calm a customer down who was clearly about to have a aneurysm on the phone -- over a faster reboot request. At the time I was working in webtech. I had no control over reboots and could only forward the request to the DC techs to take care of when they could. At the time it was like 1 hour reboot request. This guy demanded to speak to my supervisor and was yelling and screaming. My supervisor was already listening in on the phone (shout out to Steve!) and had told me in IRC to just get him off the phone, there was nothing we could do and stick to the normal script of "Your server reboot request has been made and will be processed in the order it was received". And of course this was not acceptable. After a few more screams of rage from the guy and me asking him to calm down I told him I could not continue the call and that he should call back once he had a chance to cool off -- and then hung up the phone. Once when I was working in what was known as DC2 the newest addition at the time to the data centers located in a area of town we lovingly called "guns point" because of the prolific amount of gun violence that accorded at "guns point mall". Side note, I was actually at the mall once when gun shots went off. Side note two, I shook the hand of a man who later washed me car, set a building on fire and then proceeded to go guns point mall to work out, attempt to rape a woman, and then shot the lady and him self. But atlas, we are talking about DC2. Somebody had taken what I can only describe as a "baby sized" dump. By baby sized I mean the size of a new born baby. It towered out of the water with a impressive girth. Nobody wanted to deal with it so the bathroom which happened to be the woman's bathroom sat shut for about two weeks. It was mostly fine but then the complaints form the women in the office started to roll in about the obvious issues; seat left up, pee on the seat, having to wait too long, smells. Being the problem solver type I am I decided I would rectify this issue. The entire office (about 8 people) were all integrated and decided to help me. We grabbed several paper towel rolls from the men's bathroom, and proceeded to wrap me up like a mummy -- head to toe, just tiny slits left for my eyes. Armed with a plunger they opened the door to the woman's bathroom and I shimmied in with them closing the door behind me. I told them not to open the door back until there was a proper flush. I proceeded to battle this baby sized dump for the next few minutes. It took about 6 flushes to get the entire thing to go down without flooding the place. The blame for the baby sized shit always landed on the smallest girl in the office :/ -- note nobodies feelings were hurt, it was all fun and games, for fucks sake we were making like $7 a hour. This has gotten much longer and I have only just given a few of the crazy stories about this place, but it is almost lunch time so I will leave you with this one last story. During the DC2 bring up I volunteered to work on the wiring crew. I thought it would be a good opportunity to learn more about networking and make a little extra money. There I learned how to crimp network cable and the art of zip ties. We were on a super tight deadline so some times corners were cut, or should I say not cut. This one rack me and my coworker were wiring we failed to properly measure the correct cable length. Or goal was to work on both ends at the same time. The end result was we had a HUGE amount of cable left over. So with the art of zip ties we bundled it up, about 20 cables -- in to a big burrito and placed it on the top center of the bread rack -- out of the way and all... Little did I realize this rack was right in view of one of 2 network cameras that broadcast the DC live on the main page of the companies web site. The next day I was called to Pete's desk, the guy running the wiring crew (also know for having a box fan pointed directly at his junk while at his desk, and responsible for ordering copper power cables that would not really work for powering the types of servers we planned on powering.) Pete proceeded to pull up the website and point at the web cam picture (at the time updated once every 1 min) and asked me what was wrong with that picture. I did not see it at first, and said nothing. Pete then asked me about the huge blue burrito on top of the rack :(. I spent that day rewiring that rack. Of the stories missing are the make shift data recovery operations on failed drives, platter swaps, incompetent employees who refused to use computers and required paper print outs, limo rides, foosball, bike building, tails of hacking, and the FBI. All and all the the place was a awesome place to work with unlimited freedom to learn or do anything you wanted so long as it appeared to maybe help the business and it did not interfere with your daily duties. This is the place I learned to code, and got serious about computers hardware and software. I often look back at the things I learned here and how they help me do my job today. Shout out to Brain, David (both of them), Sammy, Steve, Todd shorty pants, Bill, Jason, Curtis, Patrick, Bridget, Phil, Samantha, Cory, Fox, Brandon, Ken, Jen, and many others I am sure I have not forgotten just at a loss for names. TLDR; Severs on bread racks, people paid crap, fun place to work.

mbrumlow 9y ago

Part 2 On another occasion I had to calm a customer down who was clearly about to have a aneurysm on the phone -- over a faster reboot request. At the time I was working in webtech. I had no control over reboots and could only forward the request to the DC techs to take care of when they could. At the time it was like 1 hour reboot request. This guy demanded to speak to my supervisor and was yelling and screaming. My supervisor was already listening in on the phone (shout out to Steve!) and had told me in IRC to just get him off the phone, there was nothing we could do and stick to the normal script of "Your server reboot request has been made and will be processed in the order it was received". And of course this was not acceptable. After a few more screams of rage from the guy and me asking him to calm down I told him I could not continue the call and that he should call back once he had a chance to cool off -- and then hung up the phone. Once when I was working in what was known as DC2 the newest addition at the time to the data centers located in a area of town we lovingly called "guns point" because of the prolific amount of gun violence that accorded at "guns point mall". Side note, I was actually at the mall once when gun shots went off. Side note two, I shook the hand of a man who later washed me car, set a building on fire and then proceeded to go guns point mall to work out, attempt to rape a woman, and then shot the lady and him self. But atlas, we are talking about DC2. Somebody had taken what I can only describe as a "baby sized" dump. By baby sized I mean the size of a new born baby. It towered out of the water with a impressive girth. Nobody wanted to deal with it so the bathroom which happened to be the woman's bathroom sat shut for about two weeks. It was mostly fine but then the complaints form the women in the office started to roll in about the obvious issues; seat left up, pee on the seat, having to wait too long, smells. Being the problem solver type I am I decided I would rectify this issue. The entire office (about 8 people) were all integrated and decided to help me. We grabbed several paper towel rolls from the men's bathroom, and proceeded to wrap me up like a mummy -- head to toe, just tiny slits left for my eyes. Armed with a plunger they opened the door to the woman's bathroom and I shimmied in with them closing the door behind me. I told them not to open the door back until there was a proper flush. I proceeded to battle this baby sized dump for the next few minutes. It took about 6 flushes to get the entire thing to go down without flooding the place. The blame for the baby sized shit always landed on the smallest girl in the office :/ -- note nobodies feelings were hurt, it was all fun and games, for fucks sake we were making like $7 a hour. This has gotten much longer and I have only just given a few of the crazy stories about this place, but it is almost lunch time so I will leave you with this one last story. During the DC2 bring up I volunteered to work on the wiring crew. I thought it would be a good opportunity to learn more about networking and make a little extra money. There I learned how to crimp network cable and the art of zip ties. We were on a super tight deadline so some times corners were cut, or should I say not cut. This one rack me and my coworker were wiring we failed to properly measure the correct cable length. Or goal was to work on both ends at the same time. The end result was we had a HUGE amount of cable left over. So with the art of zip ties we bundled it up, about 20 cables -- in to a big burrito and placed it on the top center of the bread rack -- out of the way and all... Little did I realize this rack was right in view of one of 2 network cameras that broadcast the DC live on the main page of the companies web site. The next day I was called to Pete's desk, the guy running the wiring crew (also know for having a box fan pointed directly at his junk while at his desk, and responsible for ordering copper power cables that would not really work for powering the types of servers we planned on powering.) Pete proceeded to pull up the website and point at the web cam picture (at the time updated once every 1 min) and asked me what was wrong with that picture. I did not see it at first, and said nothing. Pete then asked me about the huge blue burrito on top of the rack :(. I spent that day rewiring that rack. Of the stories missing are the make shift data recovery operations on failed drives, platter swaps, incompetent employees who refused to use computers and required paper print outs, limo rides, foosball, bike building, tails of hacking, and the FBI. All and all the the place was a awesome place to work with unlimited freedom to learn or do anything you wanted so long as it appeared to maybe help the business and it did not interfere with your daily duties. This is the place I learned to code, and got serious about computers hardware and software. I often look back at the things I learned here and how they help me do my job today. Shout out to Brain, David (both of them), Sammy, Steve, Todd shorty pants, Bill, Jason, Curtis, Patrick, Bridget, Phil, Samantha, Cory, Fox, Brandon, Ken, Jen, and many others I am sure I have not forgotten just at a loss for names. TLDR; Severs on bread racks, people paid crap, fun place to work.