8 ms·

“Let's talk about a hypothetical public-facing service”

439 points by kgm 11y ago

zatkin 11y ago
I was hoping they would stay offline. It's really disappointing to see a good service turn into a money wringing desperation.
- pjc50 11y ago
  I was thinking similar thoughts from the other direction - that they have turned to money wringing desperation might mean there is no money to get the thing back up. If the situation turns out to be "it's dead Jim, we need to spend several million on a replacement EMC filer or shut down forever", it might be gone along with all the valuable projects hosted there. In some ways the bundleware served like a SMART warning, causing people to back up and migrate their projects.
- morganvachon 11y ago
  The way I discovered SF was down was when I was trying to install some Slackbuilds and kept getting MD5 mismatches on the source downloads. I noticed I was getting the same MD5 for several different packages, so I tried to manually grab a package and that's when I hit SF's error page (which sbopkg was happily downloading and trying to pass off as somesourcepkg.tar.gz). I found myself in the odd position of hoping SF would come back up so I could finish what I was doing, when I'd normally welcome the news that they had shut down.
  - pascalmemories 11y ago
    Sadly I too found it was screwed a few days ago when trying to pull some source code hosted there (ironically so I would have my own copy in case things went wrong there...). I just got their 'Disaster Recovery Mode' notice and a subset of binary packages available. Lots of google searching for what's wrong with Sourceforge revealed nothing (other than complaints about their packaged installers/crapware). Now I have the answer to the question I was really asking. I just need to wait and see if the developers of the packages I need will somehow migrate to github so I can get the source...
- stuaxo 11y ago
  It does still seem to be offline, for a whole bunch of projects - I like to check in on dosemu from time to time and it is fucked ! http://www.dosemu.org/ http://www.dosemu.org/ What's happened to the mailing list archives for all these projects, are they just down the toilet now? And of course the source, this is bloody terrible !
intrasight 11y ago
Got a good chuckle. Such is life in IT. Gotta admit I missed the "Extremely Massive Corporation" hint.
- microcolonel 11y ago
  I laughed so hard while reading that.
  - tekromancr 11y ago
    Care to elaborate?
    - akamaka 11y ago
      It's a thinly diguised reference to EMC, the storage company.
    - windsurfer 11y ago
      I'm guessing EMC
    - microcolonel 11y ago
      Come on, read Extremely Massive Corporation again, and try to think of an Extremely Massive Corporation which happens to deal in computer storage systems.
      - aidos 11y ago
        Some of us aren't in the environment where we'd know who EMC are :-)
        pjc50 11y ago
        There was a time when they had adverts on seemingly every page of every computing magazine. Although that may have been back when magazines were a thing and Sun were a hardware company.
smacktoward 11y ago
> You have a moment where you envision the future of virtualized storage and think about how great it will be when storage is nearly free and outsourceable and you can stop buying disks from Amazon every few months The future is now! Instead of you sending money to Amazon and them sending you disks, they keep the disks and you send them the money anyway. Progress! :-D
- windsurfer 11y ago
  If the disks only last so long, is it really so terrible that Amazon gets to keep the broken outdated junk?
  - Sanddancer 11y ago
    Yes. One can build a system that holds a petabyte of data, fits in less than one rack, has multiple redundancy, for about a hundred grand these days. Multiplied by three locations, that'll get you to $300000. Add in another $35000 for three years of hosting at three different datacenters, and you get to $335000. Sounds like a lot, right? Amazon's even more expensive. Storing one petabyte on Amazon, at the slowest, cheapest, glacier level, will cost you $10000/month, so at about 10 months, you've paid a hundred grand. At thirty six months, you've paid three hundred sixty grand. Plus, you have no hardware to show for it, while amazon sells that broken outdated junk to a reseller to regain some of their investment. Had you hosted it yourself, you could have sold all that equipment off to refurbishers and resellers and regained at least a portion of your investment back. So yeah, it is pretty terrible to choose Amazon.
    - khuey 11y ago
      Commodity cloud services (and pricing) are meant for the 99%, not the outliers. If you have a petabyte of data to store you should either be doing enough other business with Amazon that you can negotiate rates down or you should be rolling your own.
    - sneak 11y ago
      It costs about a hundred and fifty grand annually (fully loaded, base salary 80-90k) to have the lowest end person who could conceivably keep one of these things running well for three years. n+1 means you need two of them in case one gets the flu. Tell me again about how Amazon's a ripoff, please.
simonebrunozzi 11y ago
"Let's talk about a hypothetical public-facing service that offers tools for collaboration, revision control, and software publishing." - it was hard not to notice that he was referring to SourceForge. :)
- skrause 11y ago
  Especially since it was posted in a thread about SourceForge.
- Spivak 11y ago
  You know the title of the Reddit post was "Sourceforge is down due to storage problems, no ETA", right?
eCa 11y ago
> made by an Extremely Massive Corporation who until now you've generally respected. As usual, one's respect for BigCompany is inversely correlated to one's use of their products.
rosser 11y ago
...Extremely Massive Corporation... I knew EMC storage was utter shit when, upon attempting to create a new RAID group, I realized that the configuration tool's default was to stripe across drives within a shelf, not to create stripes that span shelves. Worse, to create the more fault-tolerant, shelf-spanning RAID volumes, one must manually add drives, one by one to the array, in a process that involves about 44 (slight hyperbole) clicks per disk. And then there was the fact that the configuration tool was Windows-only. Yeah, screw those guys.
- kragen 11y ago
  I guess a "shelf" shares some piece of hardware, such as a controller card or communication backplane, which could itself fail and thus disable access to all the drives on that shelf, which a RAID 4 or 5 stripe that spans shelves could survive?
- angry_octet 11y ago
  Actually it sounds like the EMC software was trying to save you from something which is a bad practice. If you want raid 1+0 it is better to have two disks on the same controller/chassis in raid 0, then raid 1 (or raid 5/6) across chassis. Otherwise a chassis failure will take fail both leaves of your raid mirror. Having said that, EMC is stupidly overpriced bloat.
  - rosser 11y ago
    Did I really need to spell out the geometry of the RAID I was creating to avoid this kind of nitpicky follow-up? It was a 20-something disk RAID 10 [1], arranged so that every mirrored pair of disks spanned different enclosures, in order to mitigate the failure of any one shelf — that is, interleaving mirrors across controllers and shelves, exactly as you suggest I should have done — and further, such that any one shelf failing only affects the mirrors that had disks on that shelf. EMC's software wanted to allocate the drives from two shelves, with an unequal number of drives per shelf. It was just grabbing the next however many disks, linearly. So, no, they weren't trying to balance the mirror across enclosures or controllers. They just weren't thinking. [1] By "RAID 10", I mean "striped mirrors" — that is, create a bunch of mirrors that span shelves and then stripe across them — not "mirrored stripes" which is what you appear to be suggesting, with "it is better to have two disks on the same controller/chassis in raid 0, then raid 1". A striped mirror is recommended in everything I've ever read on the subject, because it puts the redundancy at the lowest level of the array's geometry. Using a mirrored stripe, on the other hand, means that when one disk fails, any other disks striped with it, still presumably perfectly functional, can't be used; the controller must instead read from and write to the mirror. If a disk in that mirror subsequently fails, you've lost data — and remember that when striping, the chance of failure is multiplied by the number of disks in the stripe. EDIT: Footnote.
bootload 11y ago
'all the knowledge about how to handle this moody piece of hardware is lost, like tears in the rain' Offworld, Roy Batty reference.
nitrogen 11y ago
One can hope that if this hypothetical public-facing service never returns, they will ship the backup tapes to the Internet Archive instead of Honest Bob's Social Data Mining and Market Manipulation.
amelius 11y ago
> It writes a full 32 bits of numeric user ID to its filesystem, but to save a few bytes it only stores 16 bits of group IDs. Some engineer probably thought that'd be enough for anybody. I'm having the same issue with the number of hardlinks, which, for linux ext4 systems, is limited to 65000.
galoppini 11y ago
SourceForge has posted info about current infrastructure and service restoration activity at http://sourceforge.net/blog/sourceforge-infrastructure-and-service-restoration/ http://sourceforge.net/blog/sourceforge-infrastructure-and-s... [Disclosure: I work for SourceForge]
- M2Ys4U 11y ago
  >[Disclosure: I work for SourceForge] My condolences.
chris_wot 11y ago
Yeah, they dealt with EMC. That's never going to go well. (continues reading)

439 points by kgm 11y ago

I was hoping they would stay offline. It's really disappointing to see a good service turn into a money wringing desperation.

pjc50 11y ago
I was thinking similar thoughts from the other direction - that they have turned to money wringing desperation might mean there is no money to get the thing back up. If the situation turns out to be "it's dead Jim, we need to spend several million on a replacement EMC filer or shut down forever", it might be gone along with all the valuable projects hosted there. In some ways the bundleware served like a SMART warning, causing people to back up and migrate their projects.
morganvachon 11y ago
The way I discovered SF was down was when I was trying to install some Slackbuilds and kept getting MD5 mismatches on the source downloads. I noticed I was getting the same MD5 for several different packages, so I tried to manually grab a package and that's when I hit SF's error page (which sbopkg was happily downloading and trying to pass off as somesourcepkg.tar.gz). I found myself in the odd position of hoping SF would come back up so I could finish what I was doing, when I'd normally welcome the news that they had shut down.
- pascalmemories 11y ago
  Sadly I too found it was screwed a few days ago when trying to pull some source code hosted there (ironically so I would have my own copy in case things went wrong there...). I just got their 'Disaster Recovery Mode' notice and a subset of binary packages available. Lots of google searching for what's wrong with Sourceforge revealed nothing (other than complaints about their packaged installers/crapware). Now I have the answer to the question I was really asking. I just need to wait and see if the developers of the packages I need will somehow migrate to github so I can get the source...
stuaxo 11y ago
It does still seem to be offline, for a whole bunch of projects - I like to check in on dosemu from time to time and it is fucked ! http://www.dosemu.org/ http://www.dosemu.org/ What's happened to the mailing list archives for all these projects, are they just down the toilet now? And of course the source, this is bloody terrible !

intrasight 11y ago

Got a good chuckle. Such is life in IT. Gotta admit I missed the "Extremely Massive Corporation" hint.

microcolonel 11y ago
I laughed so hard while reading that.
- tekromancr 11y ago
  Care to elaborate?
  - akamaka 11y ago
    It's a thinly diguised reference to EMC, the storage company.
  - windsurfer 11y ago
    I'm guessing EMC
  - microcolonel 11y ago
    Come on, read Extremely Massive Corporation again, and try to think of an Extremely Massive Corporation which happens to deal in computer storage systems.
    - aidos 11y ago
      Some of us aren't in the environment where we'd know who EMC are :-)
      - pjc50 11y ago
        There was a time when they had adverts on seemingly every page of every computing magazine. Although that may have been back when magazines were a thing and Sun were a hardware company.

smacktoward 11y ago

> You have a moment where you envision the future of virtualized storage and think about how great it will be when storage is nearly free and outsourceable and you can stop buying disks from Amazon every few months The future is now! Instead of you sending money to Amazon and them sending you disks, they keep the disks and you send them the money anyway. Progress! :-D

windsurfer 11y ago
If the disks only last so long, is it really so terrible that Amazon gets to keep the broken outdated junk?
- Sanddancer 11y ago
  Yes. One can build a system that holds a petabyte of data, fits in less than one rack, has multiple redundancy, for about a hundred grand these days. Multiplied by three locations, that'll get you to $300000. Add in another $35000 for three years of hosting at three different datacenters, and you get to $335000. Sounds like a lot, right? Amazon's even more expensive. Storing one petabyte on Amazon, at the slowest, cheapest, glacier level, will cost you $10000/month, so at about 10 months, you've paid a hundred grand. At thirty six months, you've paid three hundred sixty grand. Plus, you have no hardware to show for it, while amazon sells that broken outdated junk to a reseller to regain some of their investment. Had you hosted it yourself, you could have sold all that equipment off to refurbishers and resellers and regained at least a portion of your investment back. So yeah, it is pretty terrible to choose Amazon.
  - khuey 11y ago
    Commodity cloud services (and pricing) are meant for the 99%, not the outliers. If you have a petabyte of data to store you should either be doing enough other business with Amazon that you can negotiate rates down or you should be rolling your own.
  - sneak 11y ago
    It costs about a hundred and fifty grand annually (fully loaded, base salary 80-90k) to have the lowest end person who could conceivably keep one of these things running well for three years. n+1 means you need two of them in case one gets the flu. Tell me again about how Amazon's a ripoff, please.

simonebrunozzi 11y ago

"Let's talk about a hypothetical public-facing service that offers tools for collaboration, revision control, and software publishing." - it was hard not to notice that he was referring to SourceForge. :)

skrause 11y ago
Especially since it was posted in a thread about SourceForge.
Spivak 11y ago
You know the title of the Reddit post was "Sourceforge is down due to storage problems, no ETA", right?

eCa 11y ago

> made by an Extremely Massive Corporation who until now you've generally respected. As usual, one's respect for BigCompany is inversely correlated to one's use of their products.

rosser 11y ago

...Extremely Massive Corporation... I knew EMC storage was utter shit when, upon attempting to create a new RAID group, I realized that the configuration tool's default was to stripe across drives within a shelf, not to create stripes that span shelves. Worse, to create the more fault-tolerant, shelf-spanning RAID volumes, one must manually add drives, one by one to the array, in a process that involves about 44 (slight hyperbole) clicks per disk. And then there was the fact that the configuration tool was Windows-only. Yeah, screw those guys.

kragen 11y ago
I guess a "shelf" shares some piece of hardware, such as a controller card or communication backplane, which could itself fail and thus disable access to all the drives on that shelf, which a RAID 4 or 5 stripe that spans shelves could survive?
angry_octet 11y ago
Actually it sounds like the EMC software was trying to save you from something which is a bad practice. If you want raid 1+0 it is better to have two disks on the same controller/chassis in raid 0, then raid 1 (or raid 5/6) across chassis. Otherwise a chassis failure will take fail both leaves of your raid mirror. Having said that, EMC is stupidly overpriced bloat.
- rosser 11y ago
  Did I really need to spell out the geometry of the RAID I was creating to avoid this kind of nitpicky follow-up? It was a 20-something disk RAID 10 [1], arranged so that every mirrored pair of disks spanned different enclosures, in order to mitigate the failure of any one shelf — that is, interleaving mirrors across controllers and shelves, exactly as you suggest I should have done — and further, such that any one shelf failing only affects the mirrors that had disks on that shelf. EMC's software wanted to allocate the drives from two shelves, with an unequal number of drives per shelf. It was just grabbing the next however many disks, linearly. So, no, they weren't trying to balance the mirror across enclosures or controllers. They just weren't thinking. [1] By "RAID 10", I mean "striped mirrors" — that is, create a bunch of mirrors that span shelves and then stripe across them — not "mirrored stripes" which is what you appear to be suggesting, with "it is better to have two disks on the same controller/chassis in raid 0, then raid 1". A striped mirror is recommended in everything I've ever read on the subject, because it puts the redundancy at the lowest level of the array's geometry. Using a mirrored stripe, on the other hand, means that when one disk fails, any other disks striped with it, still presumably perfectly functional, can't be used; the controller must instead read from and write to the mirror. If a disk in that mirror subsequently fails, you've lost data — and remember that when striping, the chance of failure is multiplied by the number of disks in the stripe. EDIT: Footnote.

bootload 11y ago

'all the knowledge about how to handle this moody piece of hardware is lost, like tears in the rain' Offworld, Roy Batty reference.

nitrogen 11y ago

One can hope that if this hypothetical public-facing service never returns, they will ship the backup tapes to the Internet Archive instead of Honest Bob's Social Data Mining and Market Manipulation.

amelius 11y ago

> It writes a full 32 bits of numeric user ID to its filesystem, but to save a few bytes it only stores 16 bits of group IDs. Some engineer probably thought that'd be enough for anybody. I'm having the same issue with the number of hardlinks, which, for linux ext4 systems, is limited to 65000.

galoppini 11y ago

SourceForge has posted info about current infrastructure and service restoration activity at http://sourceforge.net/blog/sourceforge-infrastructure-and-service-restoration/ http://sourceforge.net/blog/sourceforge-infrastructure-and-s... [Disclosure: I work for SourceForge]

M2Ys4U 11y ago
>[Disclosure: I work for SourceForge] My condolences.

chris_wot 11y ago

Yeah, they dealt with EMC. That's never going to go well. (continues reading)