3 ms·

> Just that .NET is way more popular in the enterprise than on the web It is a fallacy to think that the web is the only large network that requires scale. Go

by nikcub 16y ago

> Just that .NET is way more popular in the enterprise than on the web It is a fallacy to think that the web is the only large network that requires scale. Goldman Sachs alone stores more data than the entire web The Visa network has had 4 seconds of downtime in decades Most airline systems took decades to engineer

danudey 16y ago
According to this link (after a few seconds of googling), Visa's network was down for 8 minutes in the five years ending 2001. Do you have citations for any of your claims? Edit: Oops, here's the link: http://www.forbes.com/global/2002/0916/038.html http://www.forbes.com/global/2002/0916/038.html
- nikcub 16y ago
  The Visa figure I heard at the Computer History Museum years ago, it was first decades so that may have changed. Goldman Sachs told me when I was writing a proposal for them years ago that they have over a petabyte of data stored. The web is 80-200TB depending on who you ask. A single department there responsible for the program based trading would alone have an entire copy of the web (and parts of the deep web, and all of twitter, etc.) since they construct those whack trading apps that suck everything up and analyze it for signals. If there are any quants on here they could tell you about this more. The airline system I was referring to is SABRE. Early IBM was built on their rollouts and we are talking about the 50s. V interesting story, lots of references from this page: http://en.wikipedia.org/wiki/Sabre_(computer_system)#History http://en.wikipedia.org/wiki/Sabre_(computer_system)#History
  - rms 16y ago
    The web is only 200TB? That seems wrong by orders of magnitude to me. Maybe you mean Google's publicly indexed web?
    - nikcub 16y ago
      well yes, indexed web - otherwise it would be all the data everywhere bar a few nuclear facilities.
  - simonw 16y ago
    I think you're grossly underestimating the size of the Web. Flickr alone were storing 15TB of new photos a month back in 2006: http://blog.forret.com/2006/10/a-picture-a-day-flickrs-storage-growth/ http://blog.forret.com/2006/10/a-picture-a-day-flickrs-stora...
    - nikcub 16y ago
      it was 8 years ago - the numbers for both have undoubtably increased (it was ~10x the web at the time)
    - nikcub 16y ago
      sorry I just re-read my post, should have been past tense, i have been up for too long. It would be interesting to get more recent figures and compare again, because I do know that the investment banks hoard a lot of data. In terms of just documents, they would easily store more than the web
  - hucker 16y ago
    One of my professors told me YouTube is hosting approximately 100TB of new videos every month, I think 200 TB is off by very, very much. If 200TB was the case I could afford to buy disks to store the entire web with my yearly salary...
famousactress 16y ago
Really great points. Do they add features to those systems rapidly? Cause I think the original premise hinged on the combination of those two desires.
- nikcub 16y ago
  They don't have to add features quickly because they spend years designing and building them based on a spec that somebody else has spent years designing all so that when it launches, you can go online and book an airline ticket or make a transaction or whatever other essential daily activity and not have to think about what is taking place, let alone see an error screen or 404
  - famousactress 16y ago
    Gotcha. Sounds like an environment not likely to breed folks who are constitutionally constructed to create large scalable websites at a startup pace.
  - nikcub 16y ago
    no what I am saying is that it is unusual for them to get themselves in a situation where they have to suddenly code out of a hole at 'startup pace' (whatever that means) startups don't have a monopoly on working hard or working fast, and it is arrogant to generalize about both .NET and enterprise developers in that way since we are all in one way or anther standing on the shoulders of earlier enterprise work (where do you think what we call 'nosql' and think is new and grovvy, was first used?)
    - famousactress 16y ago
      Totally fair. First, apologies if it sounded like I was suggesting startups have a monopoly on working hard and fast. Of course I never meant anything like that at all. Second, arrogant really isn't fair because I certainly never gave an assessment of my own abilities or value. Third, totally.. .NET stands on the shoulders of the same stuff the infrastructure that runs most of the internet stands on. It's just that .NET doesn't run most (or even lots and lots of) the internet.. so to suggest that the .NET development community is less likely on average to be ready to build Facebook doesn't seem like blasphemy. That's very different from suggesting the .NET camp isn't full of awesome, hard-working developers... but really, I apologize if it comes off that way. Definitely don't mean to suggest it in the least.
IgorPartola 16y ago
True, but different networks have different constraints and require different solutions. Data warehousing, which is likely what most of GS's data is doing, is very different than trying to scaling a near-real time data access system. Payment processing only deals with one type of data: money. This gives you as many shortcuts as the number of constraints it imposes. Airline control and departure systems are probably as close to a typical modern web app as you'd get from your list. My point is that while there certainly are engineers that have worked with high scalability issues without ever touching the web, they have also likely been solving slightly different problems. P.S.: Other systems that require high availability but are not the web: telephony and cell communications, broadcasting and doomsday devices.
Kafka 16y ago
> Goldman Sachs alone stores more data than the entire web Why would you believe that Goldman Sachs stores more data than can be reached by HTTP?! Only YouTube amounts to petabytes of data. http://beerpla.net/2008/08/14/how-to-find-out-the-number-of-videos-on-youtube/ http://beerpla.net/2008/08/14/how-to-find-out-the-number-of-...
- nikcub 16y ago
  I completely disagree with his calculation, and GS were storing over 1PB 8 years ago when the web was 80-150TB the point is that it is frikkin big - and most investment firms now scan the entire web for signals

by nikcub 16y ago

danudey 16y ago

According to this link (after a few seconds of googling), Visa's network was down for 8 minutes in the five years ending 2001. Do you have citations for any of your claims? Edit: Oops, here's the link: http://www.forbes.com/global/2002/0916/038.html http://www.forbes.com/global/2002/0916/038.html

nikcub 16y ago
The Visa figure I heard at the Computer History Museum years ago, it was first decades so that may have changed. Goldman Sachs told me when I was writing a proposal for them years ago that they have over a petabyte of data stored. The web is 80-200TB depending on who you ask. A single department there responsible for the program based trading would alone have an entire copy of the web (and parts of the deep web, and all of twitter, etc.) since they construct those whack trading apps that suck everything up and analyze it for signals. If there are any quants on here they could tell you about this more. The airline system I was referring to is SABRE. Early IBM was built on their rollouts and we are talking about the 50s. V interesting story, lots of references from this page: http://en.wikipedia.org/wiki/Sabre_(computer_system)#History http://en.wikipedia.org/wiki/Sabre_(computer_system)#History
- rms 16y ago
  The web is only 200TB? That seems wrong by orders of magnitude to me. Maybe you mean Google's publicly indexed web?
  - nikcub 16y ago
    well yes, indexed web - otherwise it would be all the data everywhere bar a few nuclear facilities.
- simonw 16y ago
  I think you're grossly underestimating the size of the Web. Flickr alone were storing 15TB of new photos a month back in 2006: http://blog.forret.com/2006/10/a-picture-a-day-flickrs-storage-growth/ http://blog.forret.com/2006/10/a-picture-a-day-flickrs-stora...
  - nikcub 16y ago
    it was 8 years ago - the numbers for both have undoubtably increased (it was ~10x the web at the time)
  - nikcub 16y ago
    sorry I just re-read my post, should have been past tense, i have been up for too long. It would be interesting to get more recent figures and compare again, because I do know that the investment banks hoard a lot of data. In terms of just documents, they would easily store more than the web
- hucker 16y ago
  One of my professors told me YouTube is hosting approximately 100TB of new videos every month, I think 200 TB is off by very, very much. If 200TB was the case I could afford to buy disks to store the entire web with my yearly salary...

famousactress 16y ago

Really great points. Do they add features to those systems rapidly? Cause I think the original premise hinged on the combination of those two desires.

nikcub 16y ago
They don't have to add features quickly because they spend years designing and building them based on a spec that somebody else has spent years designing all so that when it launches, you can go online and book an airline ticket or make a transaction or whatever other essential daily activity and not have to think about what is taking place, let alone see an error screen or 404
- famousactress 16y ago
  Gotcha. Sounds like an environment not likely to breed folks who are constitutionally constructed to create large scalable websites at a startup pace.
- nikcub 16y ago
  no what I am saying is that it is unusual for them to get themselves in a situation where they have to suddenly code out of a hole at 'startup pace' (whatever that means) startups don't have a monopoly on working hard or working fast, and it is arrogant to generalize about both .NET and enterprise developers in that way since we are all in one way or anther standing on the shoulders of earlier enterprise work (where do you think what we call 'nosql' and think is new and grovvy, was first used?)
  - famousactress 16y ago
    Totally fair. First, apologies if it sounded like I was suggesting startups have a monopoly on working hard and fast. Of course I never meant anything like that at all. Second, arrogant really isn't fair because I certainly never gave an assessment of my own abilities or value. Third, totally.. .NET stands on the shoulders of the same stuff the infrastructure that runs most of the internet stands on. It's just that .NET doesn't run most (or even lots and lots of) the internet.. so to suggest that the .NET development community is less likely on average to be ready to build Facebook doesn't seem like blasphemy. That's very different from suggesting the .NET camp isn't full of awesome, hard-working developers... but really, I apologize if it comes off that way. Definitely don't mean to suggest it in the least.

IgorPartola 16y ago

True, but different networks have different constraints and require different solutions. Data warehousing, which is likely what most of GS's data is doing, is very different than trying to scaling a near-real time data access system. Payment processing only deals with one type of data: money. This gives you as many shortcuts as the number of constraints it imposes. Airline control and departure systems are probably as close to a typical modern web app as you'd get from your list. My point is that while there certainly are engineers that have worked with high scalability issues without ever touching the web, they have also likely been solving slightly different problems. P.S.: Other systems that require high availability but are not the web: telephony and cell communications, broadcasting and doomsday devices.

Kafka 16y ago

> Goldman Sachs alone stores more data than the entire web Why would you believe that Goldman Sachs stores more data than can be reached by HTTP?! Only YouTube amounts to petabytes of data. http://beerpla.net/2008/08/14/how-to-find-out-the-number-of-videos-on-youtube/ http://beerpla.net/2008/08/14/how-to-find-out-the-number-of-...

nikcub 16y ago
I completely disagree with his calculation, and GS were storing over 1PB 8 years ago when the web was 80-150TB the point is that it is frikkin big - and most investment firms now scan the entire web for signals