5 ms·

When I was hired into AOL as the Sr. Internet Mail Adminstrator in 1995, much of their hardware was still on the Stratus mainframes. That was some seriously po

When I was hired into AOL as the Sr. Internet Mail Adminstrator in 1995, much of their hardware was still on the Stratus mainframes. That was some seriously powerful hardware with an equally powerful but arcane mainframe OS, but it was limited to just 32 nodes -- and even getting to that took some real magic. So, everything was slowly moving off Stratus, and primarily to HP-UX systems. It was a long, slow death. Then the web team brought in DEC Alpha 4100 hardware, and that stuff just absolutely screamed. They were getting performance that many in the company simply couldn't believe. I ended up taking four 4100s, each with four DEC Alpha processors, and ran a copy of BIND-9 on each processor. I suddenly had the biggest and fastest nameserver cluster in the world. I did the benchmarks at the time to prove it. We never did give back that rack of DEC Alphas to the web farm guys. That was around the time that certain leaders in AOL realized that the Internet Operations group under George Boyce was effectively taking over the entire company, and so of course that had to be stopped. George was given a title without portfolio, and all his people were scattered to the four winds of the company. That's also around the time I started seeing the brick walls going up between the operations side of the house and the developers. Those would be the same walls made famous by some of the best known books on DevOps, including "The Phoenix Project" by Gene Kim. Every time I've met Gene at a conference, he and I reminisce and continue our years long discussions over our different experiences at AOL and the different decades when we worked there, and how we bookended that process. I really liked working for George. It's a real shame what they did to him. But boy howdy, was that DEC Alpha hardware fast.

nikau 3y ago
Fun story from a place I worked which got one of the first alpha 4100 sold in our country - it was having intermittent crashes that we couldn't figure out. DEC sent out a very senior engineer to investigate, he eventually discovered some shielding had somehow been bent behind where a bus slot was soldered and with thermal expansion would short against some of the pins causing errors. Aside from that they were rock solid. DEC also sold us a weird windows cluster using where 2 boxes connected to the same scsi array, it was not so solid.
- quercusa 3y ago
  Was that doing a primary/secondary model where the primary wrote a timestamp to the shared disk and if the secondary didn't see that after X seconds, it would take over?
  - nikau 3y ago
    I think it was a Microsoft feature but you had to have certified hardware for support. There were 2 windows servers both plugged into the same scsi disk pack, and it used a couple of crossover cables I believe for the heartbeat. I remember the unix guys were ribbing the windows guy a bit who confessed he regretted pushing for the cluster as it was so unreliable. It was the typical windows experience of the time with a management console that just hung when doing operations.
  - Sylamore 3y ago
    Old versions of NT4 and 2000 supported SCSI disks in the cluster. We had one for a "high availability" file server from Compaq and always regretted it - I'm pretty sure a single box would have been more reliable. There was a cross-over LAN cable between the boxes and there were SCSI heartbeats (using persistent reservations IIRC) over the SCSI bus, but we would almost always have an issue with the SCSI drives deadlocking and the cluster failing BOTH nodes due to a fight over disk ownership. We replaced it with a netapp NAS eventually.
- gjvc 3y ago
  DEC also sold us a weird windows cluster using where 2 boxes connected to the same scsi array, it was not so solid. Was that some witchcraft with a Y-shaped SCSI cable, perchance?
  - nikau 3y ago
    Yep that's the one. We ran exchange and some databases on it.
  - bradknowles 3y ago
    For the GNN mail system, that was built on a pair of DEC Alpha 2100s, that were both connected to a pair of drive arrays. We had inherited that from ANS, and they had inherited that from O'Reilly. Global Network Navigator was the first nation-wide ISP in the US, and was a partnership between O'Reilly and AOL. But then it got too big for O'Reilly to run, and AOL wasn't ready to take it over, so we made ANS take it -- at the time, we had recently acquired ANS as our Tier-1 network provider, and they knew a lot more about DEC Alpha hardware than we did. But then it got too big for them to deal with, and AOL finally had to take it over. The mail system part was given to me and my co-worker. We learned a lot about DEC Alpha hardware from GNN, and we loved it. That system worked great, up until the time that DEC sent out a Field Engineer to upgrade the firmware on the drive arrays. Then we got weird errors when we tried to come back up. So, we switched to the backup systems that were connected to the same drives. And it was also hosed. Fortunately, I had started taking user mailbox backups to our Auspex NFS file server that week, so we had backups. We soon realized that over 50% of the POP3/leave-on-server mailboxes will change radically over the span of an day. And some of our mailbox backups were over eighteen hours old. So, most of those backups were actually useless. We did finally get back up and running by mounting the backup mailboxes via NFS, but we lost pretty much all user confidence in the system by that time. The drive arrays had been set up to mirror at the lowest level, then stripe across the mirrors in hardware. We then striped again across those volumes in software, which was a high performance disk striping technique we called "plaiding". We ended up breaking each of the lowest level mirrors and sending the whole bunch to the guy who wrote the AdvFS filesystem for DEC, but he never got better than 80% recovery of the data, which meant that on average every fifth character was wrong. So, pretty much none of those recovered mailboxes were useful. That was the death knell for GNN. If you can't trust the mail system, then what else can you trust?
AtlasBarfed 3y ago
"That was around the time that certain leaders in AOL realized that the Internet Operations group under George Boyce was effectively taking over the entire company, and so of course that had to be stopped. George was given a title without portfolio, and all his people were scattered to the four winds of the company." So I'm no stranger to middle and upper management Machiavellianism and the costs it inflicts on the organization as a whole, but was that a case of a guy getting too big for his britches and starting to order around / dictate to other bigwigs with the "wrong title", a lack of experience in defending budgetary turf, or the classic tech guy solving problems while the business peers simply scheme? On a sort of related note, it is kind of saddens me that the historical record of hierarchical organizational decisions aren't preserved / document well in critical areas. There's documents and records out there likely showing where global warming research was suppressed by oil companies, cancer research by tobacco companies, or whatever it is that Halliburton gets away with (apparently funneling aid to Russia currently, but that may be a partisan hit job). There are usually well documented hierarchies of management and their ostensible areas of responsibility. There are records of decisions and results by the companies as a whole. There's never a kind of historical record showing X group of people in the company were part of the "graph" of decision making that resulted in it. There's usually lots of fall guys and the like taking the sword for the uppers, but at least you can do some tracing. Anyone that setup stuff like that would probably get suicided by the CIA, or worse, trumped up and sent to Supermax.
- projektfu 3y ago
  A lot of companies used to have in-house historians and they did a surprisingly good job in some cases. However, their efforts are always susceptible to a back room deal followed by some sort of public performance to attempt to make the public record look good. These days I can only think of a handful of companies that do that, usually the old industrial giants like IBM, GE, etc. And who knows at what level the historians are involved or if they have access to internal communications.
  - AtlasBarfed 3y ago
    Companies are limited liability. FOR THE INVESTORS. What is interesting to me is that corporations currently pursue some means for having as short a lifetime of retention as legally allowed. But if corporations keeping detailed decision records led to management being the recipient of liability rather than the corporation/profits (and therefore the actual owners of the corp) .... I'm surprised corporations don't actually track internal comms better. Probably because tort law likes big fat corporations to be the target of class action lawsuits, rather than some far more limited pocket middle manager, and law has aligned to that abstraction (for the good of the lawyer profession).
lanstin 3y ago
I too worked at AOL back then, but on the AIM team. When we bought ICQ the presence side of ICQ ran on the largest machine Dec would sell you. The vertical scaling worked very well, tho. Also porting the host code to Digital Unix made it ready for 64 bit in general. We eventually ported it to the cheaper clusters of Sun then Linux hosts that normal AIM used.
- yjftsjthsd-h 3y ago
  > Also porting the host code to Digital Unix made it ready for 64 bit in general. IIRC, porting Linux to Alpha was credited for its 64-bit support, which made the AMD64 port much easier.
Sylamore 3y ago
I was under the impression that AOL was a big Tandem shop at one point, was that not the case?
- bradknowles 3y ago
  We did have some Tandem hardware, but at that time Tandem couldn't really deliver on the SMP needs that we had, whereas Stratus could. I have heard at least three of the "core four" founders talk about how they had used Tandem at a previous job where they had worked together and got seriously burned by the vendor, and that's why they went with Stratus at what later became known as AOL. Tandem was finally allowed to come into the company later, but had a relatively small part to play. At least, that's the way it was when I left in 1997. I can't speak for what happened after I left.
  - Sylamore 3y ago
    Thanks for the insight, when I was with DTV our HP/Tandem rep always made a big deal about their footprint in AOL in what would have been the same time era you were there (mid 90s) and how they "created the internet you know today" but they always sounded more like sales guy tall tales than anything else. We kept their footprint constrained to database caches (effectively) because our Superdome couldn't handle the dual workload of Siebel and lots of read requests from APIs, which was a role they excelled at for us.
    - bradknowles 3y ago
      Putting SQL engines right on the drive arrays was a pretty cool idea. That let you offload a lot of I/O to the lowest level hardware that you could. That was a real mainframe-class trick. Sadly, Tandem didn't seem to be able to deliver much else for us. Database-wise, AOL decided to go with Sybase as the official company-wide solution. Don't know whose palms got greased on that one, but I could have lived with Sybase if that was the only problem. No, AOL then had to decide who was going to be their sole database hardware provider, and they selected SGI for that. Now, SGI had some great hardware at that time, but databases is not something they did well. And SGI was not a principal hardware platform for Sybase, so everything that was developed had to wait another six months or so to be ported from the main hardware platforms over to SGI. And that's assuming it got ported at all. And that's what was chosen as the infrastructure for implementing the new version of the AOL mail system that lived outside of Stratus -- Sybase on SGI. I could have lived with a database-backed mailbox store for the entire service, but only if it wasn't done using the combined shitshow of Sybase on SGI.

nikau 3y ago

Fun story from a place I worked which got one of the first alpha 4100 sold in our country - it was having intermittent crashes that we couldn't figure out. DEC sent out a very senior engineer to investigate, he eventually discovered some shielding had somehow been bent behind where a bus slot was soldered and with thermal expansion would short against some of the pins causing errors. Aside from that they were rock solid. DEC also sold us a weird windows cluster using where 2 boxes connected to the same scsi array, it was not so solid.

quercusa 3y ago
Was that doing a primary/secondary model where the primary wrote a timestamp to the shared disk and if the secondary didn't see that after X seconds, it would take over?
- nikau 3y ago
  I think it was a Microsoft feature but you had to have certified hardware for support. There were 2 windows servers both plugged into the same scsi disk pack, and it used a couple of crossover cables I believe for the heartbeat. I remember the unix guys were ribbing the windows guy a bit who confessed he regretted pushing for the cluster as it was so unreliable. It was the typical windows experience of the time with a management console that just hung when doing operations.
- Sylamore 3y ago
  Old versions of NT4 and 2000 supported SCSI disks in the cluster. We had one for a "high availability" file server from Compaq and always regretted it - I'm pretty sure a single box would have been more reliable. There was a cross-over LAN cable between the boxes and there were SCSI heartbeats (using persistent reservations IIRC) over the SCSI bus, but we would almost always have an issue with the SCSI drives deadlocking and the cluster failing BOTH nodes due to a fight over disk ownership. We replaced it with a netapp NAS eventually.
gjvc 3y ago
DEC also sold us a weird windows cluster using where 2 boxes connected to the same scsi array, it was not so solid. Was that some witchcraft with a Y-shaped SCSI cable, perchance?
- nikau 3y ago
  Yep that's the one. We ran exchange and some databases on it.
- bradknowles 3y ago
  For the GNN mail system, that was built on a pair of DEC Alpha 2100s, that were both connected to a pair of drive arrays. We had inherited that from ANS, and they had inherited that from O'Reilly. Global Network Navigator was the first nation-wide ISP in the US, and was a partnership between O'Reilly and AOL. But then it got too big for O'Reilly to run, and AOL wasn't ready to take it over, so we made ANS take it -- at the time, we had recently acquired ANS as our Tier-1 network provider, and they knew a lot more about DEC Alpha hardware than we did. But then it got too big for them to deal with, and AOL finally had to take it over. The mail system part was given to me and my co-worker. We learned a lot about DEC Alpha hardware from GNN, and we loved it. That system worked great, up until the time that DEC sent out a Field Engineer to upgrade the firmware on the drive arrays. Then we got weird errors when we tried to come back up. So, we switched to the backup systems that were connected to the same drives. And it was also hosed. Fortunately, I had started taking user mailbox backups to our Auspex NFS file server that week, so we had backups. We soon realized that over 50% of the POP3/leave-on-server mailboxes will change radically over the span of an day. And some of our mailbox backups were over eighteen hours old. So, most of those backups were actually useless. We did finally get back up and running by mounting the backup mailboxes via NFS, but we lost pretty much all user confidence in the system by that time. The drive arrays had been set up to mirror at the lowest level, then stripe across the mirrors in hardware. We then striped again across those volumes in software, which was a high performance disk striping technique we called "plaiding". We ended up breaking each of the lowest level mirrors and sending the whole bunch to the guy who wrote the AdvFS filesystem for DEC, but he never got better than 80% recovery of the data, which meant that on average every fifth character was wrong. So, pretty much none of those recovered mailboxes were useful. That was the death knell for GNN. If you can't trust the mail system, then what else can you trust?

AtlasBarfed 3y ago

"That was around the time that certain leaders in AOL realized that the Internet Operations group under George Boyce was effectively taking over the entire company, and so of course that had to be stopped. George was given a title without portfolio, and all his people were scattered to the four winds of the company." So I'm no stranger to middle and upper management Machiavellianism and the costs it inflicts on the organization as a whole, but was that a case of a guy getting too big for his britches and starting to order around / dictate to other bigwigs with the "wrong title", a lack of experience in defending budgetary turf, or the classic tech guy solving problems while the business peers simply scheme? On a sort of related note, it is kind of saddens me that the historical record of hierarchical organizational decisions aren't preserved / document well in critical areas. There's documents and records out there likely showing where global warming research was suppressed by oil companies, cancer research by tobacco companies, or whatever it is that Halliburton gets away with (apparently funneling aid to Russia currently, but that may be a partisan hit job). There are usually well documented hierarchies of management and their ostensible areas of responsibility. There are records of decisions and results by the companies as a whole. There's never a kind of historical record showing X group of people in the company were part of the "graph" of decision making that resulted in it. There's usually lots of fall guys and the like taking the sword for the uppers, but at least you can do some tracing. Anyone that setup stuff like that would probably get suicided by the CIA, or worse, trumped up and sent to Supermax.

projektfu 3y ago
A lot of companies used to have in-house historians and they did a surprisingly good job in some cases. However, their efforts are always susceptible to a back room deal followed by some sort of public performance to attempt to make the public record look good. These days I can only think of a handful of companies that do that, usually the old industrial giants like IBM, GE, etc. And who knows at what level the historians are involved or if they have access to internal communications.
- AtlasBarfed 3y ago
  Companies are limited liability. FOR THE INVESTORS. What is interesting to me is that corporations currently pursue some means for having as short a lifetime of retention as legally allowed. But if corporations keeping detailed decision records led to management being the recipient of liability rather than the corporation/profits (and therefore the actual owners of the corp) .... I'm surprised corporations don't actually track internal comms better. Probably because tort law likes big fat corporations to be the target of class action lawsuits, rather than some far more limited pocket middle manager, and law has aligned to that abstraction (for the good of the lawyer profession).

lanstin 3y ago

I too worked at AOL back then, but on the AIM team. When we bought ICQ the presence side of ICQ ran on the largest machine Dec would sell you. The vertical scaling worked very well, tho. Also porting the host code to Digital Unix made it ready for 64 bit in general. We eventually ported it to the cheaper clusters of Sun then Linux hosts that normal AIM used.

yjftsjthsd-h 3y ago
> Also porting the host code to Digital Unix made it ready for 64 bit in general. IIRC, porting Linux to Alpha was credited for its 64-bit support, which made the AMD64 port much easier.

Sylamore 3y ago

I was under the impression that AOL was a big Tandem shop at one point, was that not the case?

bradknowles 3y ago
We did have some Tandem hardware, but at that time Tandem couldn't really deliver on the SMP needs that we had, whereas Stratus could. I have heard at least three of the "core four" founders talk about how they had used Tandem at a previous job where they had worked together and got seriously burned by the vendor, and that's why they went with Stratus at what later became known as AOL. Tandem was finally allowed to come into the company later, but had a relatively small part to play. At least, that's the way it was when I left in 1997. I can't speak for what happened after I left.
- Sylamore 3y ago
  Thanks for the insight, when I was with DTV our HP/Tandem rep always made a big deal about their footprint in AOL in what would have been the same time era you were there (mid 90s) and how they "created the internet you know today" but they always sounded more like sales guy tall tales than anything else. We kept their footprint constrained to database caches (effectively) because our Superdome couldn't handle the dual workload of Siebel and lots of read requests from APIs, which was a role they excelled at for us.
  - bradknowles 3y ago
    Putting SQL engines right on the drive arrays was a pretty cool idea. That let you offload a lot of I/O to the lowest level hardware that you could. That was a real mainframe-class trick. Sadly, Tandem didn't seem to be able to deliver much else for us. Database-wise, AOL decided to go with Sybase as the official company-wide solution. Don't know whose palms got greased on that one, but I could have lived with Sybase if that was the only problem. No, AOL then had to decide who was going to be their sole database hardware provider, and they selected SGI for that. Now, SGI had some great hardware at that time, but databases is not something they did well. And SGI was not a principal hardware platform for Sybase, so everything that was developed had to wait another six months or so to be ported from the main hardware platforms over to SGI. And that's assuming it got ported at all. And that's what was chosen as the infrastructure for implementing the new version of the AOL mail system that lived outside of Stratus -- Sybase on SGI. I could have lived with a database-backed mailbox store for the entire service, but only if it wasn't done using the combined shitshow of Sybase on SGI.