4 ms·

Everybody who pays @95th dreams of doing something like this the very first time that 95th billing is explained to them. I mean Everbody. What the Article fai

Everybody who pays @95th dreams of doing something like this the very first time that 95th billing is explained to them. I mean Everbody. What the Article fails to explain, is that there is both a port cost, and a minimum commit. For much less than what it would have cost them for the port, and minimum commit on a 1 Gig connection, they could have simply mailed their data via USPS and had it there in three days. Also, if we do the math - and presume that they only ran 1 gigabit for 5% of the time - 1 gigabit/second at 36 hours (5% of a month) in terabytes = 16 Terabytes - which barely makes it into the "10s of terabytes" category, and well within the "Just mail an AIT-3 tape. [Edit - cheaper yet, just load up the hard drives that you are going to install anyways in your destination] Edit 2: Doh - http://www.mkomo.com/cost-per-gigabyte http://www.mkomo.com/cost-per-gigabyte shows that the largest hard drives back then were 20 Gigabytes. We're talking a minimum of 1000 hard drives for "10s" of Terabytes. Maybe that 1 GigCircuit Hack wasn't such a bad idea after all.

mctx 14y ago
You'd trust your hard drives with USPS? Edit: I'm not implying that USPS is necessarily unreliable, I'm curious to hear if this is common practice.
- tempestn 14y ago
  When they're adequately backed up, as in this circumstance, why not? Send the data in triplicate if you want, just to avoid the hassle of re-sending any that get damaged. Regardless, the cost would be trivial, even for young-Google.
- timdorr 14y ago
  RAID = Redundant Array of Independent Deliveries
- notatoad 14y ago
  You've always got to expect a certain amount of packet loss over any connection.
- ghshephard 14y ago
  Presumably I'm not sending the originals. Also, powered down, hard drives are pretty robust - their read/write heads are locked up. Remember - those drives have been bouncing around a LOT to get to you through retail/wholesale channels. [edit] Also - they obviously had to buy hard drives for the destination, right? Presumably, early stage google was all about the open-chassis on plywood slam drives in approach, and wouldn't have been averse to loading up 30 or 40 drives, mailing them out, and simply diffing and resyncing as needed if 1 or 2 drives had issues, which would have been unlikely. Also, presuming you needed to have one of your people go to the remote data center, there is always the old, move the datacenter with hard drives in my luggage trick. Even in 2000 you could put a lot of data into carry-on.
- dredmorbius 14y ago
  You could make up any small number of botched drives with over-the-wire transfers. If it were a sufficient number of drives, you could simply re-image and re-transmit the data.
jamesaguilar 14y ago
In the year 2000 storage cost about $12/GB [1]. This article claims 10s of TB, so let's say 20 TB, which is the most conservative possible estimate. 20 TB = 20000 GB. At $12/GB, that comes out to $240,000. That doesn't include retransmission cost (for busted drives) or shipping. I'm not sure the analysis is correct here. Presumably their minimum commit was a lot lower than $240k, because otherwise they wouldn't have shrunk from just using the connection. They probably saved a lot of money by doing it their way. [1] http://www.mkomo.com/cost-per-gigabyte http://www.mkomo.com/cost-per-gigabyte * I wasn't around then so I don't actually know. * * It also depends what part of the year they needed to do the transfer. By [1], costs had dropped to $8/GB by the end of the year.
- ghshephard 14y ago
  Right - but they had to buy those hard drives anyways, so it didn't cost them anything more than the transportation cost of the drives to buy them at the source, load them up, and then ship them.
  - jamesaguilar 14y ago
    It would be hell to get the data off before imaging the drives. What do you transfer it to? The shuffling would require a lot of technical work, which also costs money. Also, we don't know if they were on homogenous hardware by then -- they may have still been using budget pcs and refurb'd parts. So it's hard to know if your plan could work. The investment of inventing a process for mass data transfer versus using the normal process but only during a certain time period is also significantly higher. I have to think someone ran the numbers. It's not like the folks in 2000-google were dummies.
    - ghshephard 14y ago
      Admittedly, it would have been a lot of drives. Back in circa 2000, according to http://www.mkomo.com/cost-per-gigabyte http://www.mkomo.com/cost-per-gigabyte, 20 Gigabyte Drives were in vogue @$200/pop. So, a Terabyte would have been 50 drives, 10 Terabytes @500 Drives. 10s of terabytes at a minimum 1000 drives. I'm beginning to question how "simple" an operation this might have been after all....
deleted 14y ago
[deleted]
harshreality 14y ago
Your 16.2TB estimate is for one month. The article says this continued for multiple months.
fallous 14y ago
I can assure you that there were drives larger than 20G in 2000. I had a server with 4 23G Seagate Elites at that time, and the Elite was an _old_ 5 1/4 FH SCSI drive. 18G Seagate Barracudas were certainly alive and kicking since 1997 at minimum (the year I started building real production servers).

mctx 14y ago

You'd trust your hard drives with USPS? Edit: I'm not implying that USPS is necessarily unreliable, I'm curious to hear if this is common practice.

tempestn 14y ago
When they're adequately backed up, as in this circumstance, why not? Send the data in triplicate if you want, just to avoid the hassle of re-sending any that get damaged. Regardless, the cost would be trivial, even for young-Google.
timdorr 14y ago
RAID = Redundant Array of Independent Deliveries
notatoad 14y ago
You've always got to expect a certain amount of packet loss over any connection.
ghshephard 14y ago
Presumably I'm not sending the originals. Also, powered down, hard drives are pretty robust - their read/write heads are locked up. Remember - those drives have been bouncing around a LOT to get to you through retail/wholesale channels. [edit] Also - they obviously had to buy hard drives for the destination, right? Presumably, early stage google was all about the open-chassis on plywood slam drives in approach, and wouldn't have been averse to loading up 30 or 40 drives, mailing them out, and simply diffing and resyncing as needed if 1 or 2 drives had issues, which would have been unlikely. Also, presuming you needed to have one of your people go to the remote data center, there is always the old, move the datacenter with hard drives in my luggage trick. Even in 2000 you could put a lot of data into carry-on.
dredmorbius 14y ago
You could make up any small number of botched drives with over-the-wire transfers. If it were a sufficient number of drives, you could simply re-image and re-transmit the data.

jamesaguilar 14y ago

In the year 2000 storage cost about $12/GB [1]. This article claims 10s of TB, so let's say 20 TB, which is the most conservative possible estimate. 20 TB = 20000 GB. At $12/GB, that comes out to $240,000. That doesn't include retransmission cost (for busted drives) or shipping. I'm not sure the analysis is correct here. Presumably their minimum commit was a lot lower than $240k, because otherwise they wouldn't have shrunk from just using the connection. They probably saved a lot of money by doing it their way. [1] http://www.mkomo.com/cost-per-gigabyte http://www.mkomo.com/cost-per-gigabyte * I wasn't around then so I don't actually know. * * It also depends what part of the year they needed to do the transfer. By [1], costs had dropped to $8/GB by the end of the year.

ghshephard 14y ago
Right - but they had to buy those hard drives anyways, so it didn't cost them anything more than the transportation cost of the drives to buy them at the source, load them up, and then ship them.
- jamesaguilar 14y ago
  It would be hell to get the data off before imaging the drives. What do you transfer it to? The shuffling would require a lot of technical work, which also costs money. Also, we don't know if they were on homogenous hardware by then -- they may have still been using budget pcs and refurb'd parts. So it's hard to know if your plan could work. The investment of inventing a process for mass data transfer versus using the normal process but only during a certain time period is also significantly higher. I have to think someone ran the numbers. It's not like the folks in 2000-google were dummies.
  - ghshephard 14y ago
    Admittedly, it would have been a lot of drives. Back in circa 2000, according to http://www.mkomo.com/cost-per-gigabyte http://www.mkomo.com/cost-per-gigabyte, 20 Gigabyte Drives were in vogue @$200/pop. So, a Terabyte would have been 50 drives, 10 Terabytes @500 Drives. 10s of terabytes at a minimum 1000 drives. I'm beginning to question how "simple" an operation this might have been after all....

deleted 14y ago

[deleted]

harshreality 14y ago

Your 16.2TB estimate is for one month. The article says this continued for multiple months.

fallous 14y ago

I can assure you that there were drives larger than 20G in 2000. I had a server with 4 23G Seagate Elites at that time, and the Elite was an _old_ 5 1/4 FH SCSI drive. 18G Seagate Barracudas were certainly alive and kicking since 1997 at minimum (the year I started building real production servers).