9 ms·

Netflix's Open Connect Appliances

86 points by elecboy 12y ago

ubercore 12y ago
Interesting to see some more details on the appliances. Isn't the argument primarily about peering, not open connect appliances, though?
- ceejayoz 12y ago
  Installing these reduces the need for peering dramatically.
  - tonyb 12y ago
    And the two are also not mutually exclusive. You can have an OpenConnect appliance that is populated via peering. Someone like Verizon/Comcast/etc could add OpenConnect appliances and then peer with a single 10Gbps link vs the multiple 10Gpbs transit links they are using now.
- wmf 12y ago
  It's all one discussion in my mind. There are three ways to get Netflix: boxes, peering, or transit (from cheapest to most expensive). If peering saves the ISP money and the boxes save even more money, why are some ISPs refusing to use them?
  - josho 12y ago
    Because some ISPs are trying to maximize their revenue by charging Netflix directly. There are a few other legitimate reasons as well. Some ISPs dislike hosting hardware that they don't own in their data centre. Or it may be that an ISP has contracts in place that commit the ISP to certain bandwidth costs, so removing Netflix traffic may not actually save them money.
    - qq66 12y ago
      Why shouldn't an ISP charge Netflix to host a box in their datacenter? I'd charge you a fee if you wanted to store your DVDs in my house, even if you let me watch them.
      - wmf 12y ago
        Because the box saves the ISP more money than its hosting costs.
        warfangle 12y ago
        On the other hand, if the ISP also runs a video service, they've got some very compelling reasons to do everything they can to not increase the quality of service for Netflix. They're green, and rectangular, and...
deleted 12y ago
[deleted]
Thaxll 12y ago
Lot of mistakes in this article, afair it can't hold all the movies but just the most popular. http://oc.nflxvideo.net/docs/OpenConnect-Deployment-Guide.pdf http://oc.nflxvideo.net/docs/OpenConnect-Deployment-Guide.pd... " Each appliance stores a portion of the catalog, which in general is less than the complete content library for a given region. Popularity changes, new titles being added to the service, and re- encoded movies are all part of the up to 7.5TB of nightly updates each cache must download to remain current. We recommend setting up a download window during off-peak hours when there is sufficient free capacity."
- pessimizer 12y ago
  "But wait, you say. That'd be all fine and dandy if Netflix was etched in stone, but it's not! They add and remove stuff all the time! That's right, which is why most OCAs take a 7.5TB update. Every. Freaking. Day. "If you had to take a 7.5TB update on your home internet, you would be screwed. According to Ookla's Net Index, the average download speed in the United States is 18.6Mbps. So with an average connection, that 7.5TB would take you about 40 days to chew through. "Fortunately, data centers tend to have a little more speed to work with. A firehose, compared to your bendy straw. So Netflix recommends a much shorter 10-hour span (from 2AM to noon, local time) where these boxes can soak up their updates with 1.2Gbps connections. That is to say, 'Google Fiber speeds.'" Source: TFA.
  - ssmoot 12y ago
    I'm sure thought has been put into it. I just wonder why the OCAs aren't caches? I'd imagine a cache you be a lot more (mostly bandwidth) efficient for the ISPs and scale much better. I request a stream. It's proxied through the device. It's not present on the device. So the device requests the 1080P from Netflix. It transcodes the stream it receives on-the-fly and sends me whatever quality I require while writing both the original and my lower-quality version to disk. I mean maybe the limitation in that plan is the number of streams that could be transcoded simultaneously. So maybe instead the on-the-fly version isn't cached. Because I may need to switch quality on-the-fly. So maybe instead jobs are just queued up for the 5 or so formats required based on the 1080P download and they can process on spare transcoders opportunistically. Just interesting to think through it as a programming problem. I'm sure the people working on this are very smart. It just seems really heavy on the bandwidth utilization as is.
JTon 12y ago
Can anyone explain why an ISP would push back Netflix installing one of these boxes? Does the ISP want Netflix service congestion to better sell its own media assets/distribution (i.e. cable TV)?
- osi 12y ago
  From the operator of a smaller ISP: http://www.listbox.com/member/archive/247/2014/07/search/bmV0ZmxpeA/sort/time_rev/page/1/entry/0:7/20140711175317:C1A2027E-0945-11E4-9E99-FF13B5412E71/ http://www.listbox.com/member/archive/247/2014/07/search/bmV...
  - acdha 12y ago
    That's Brett Glass, though, who is … opinionated. It's a good idea to read the entire thread for responses from a number of other network operators as well as some Netflix technical staff: http://seclists.org/nanog/2014/Jul/index.html#265 http://seclists.org/nanog/2014/Jul/index.html#265
    - osi 12y ago
      thanks for the nanog tip. been awhile since i've read that firehose :)
- tobz 12y ago
  Based on the blog posts by Level 3 -- how much internal capacity Comcast / Verizon say they have versus what Level 3 sees for utilization at their peering points with both ISPs -- I would have to assume "yes".
  - dragontamer 12y ago
    Can you please post a link? This sounds very interesting...
    - k3oni 12y ago
      Here's a link to one of the blog posts: http://blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/ http://blog.level3.com/global-connectivity/verizons-accident...
nrzuk 12y ago
Could someone explain to me (it's been a long day, and I'm no storage expert!) how these are useful, wouldn't you need a whole bunch of these at each location they are to be installed? Just basing this on my own experience I have 12 drives in RAID which have a fairly substantial sequential throughput. Start multiple streams of high bandwidth videos and the maximum throughput from the drives drops sharply due to the random reads. I would have assumed that having even 100 people streaming from a single box would be an interesting challenge.
- weaksauce 12y ago
  well, if it means that they need to install n boxes in an isp to avoid peering at a high cost then it might be worth it. > Sometimes it'll even have multiple copies of the same show, if something is in crazy high demand. that statement from the article lends one to believe that the raid arrays are in parallel. These boxes are meant to reduce the number of packets going all the way across the internet not necessarily remove the need. Also, in addition to lots of storage, these boxes probably have loads of ram that they can buffer popular streams into and then serve it.
- twoodfin 12y ago
  These things can pump out 12Gbit during peak viewing, which is enough for about 4,000 avg. 3 Mbit streams. If they can cram in 256GB of DRAM, that's enough room for a 64MB buffer per stream, or about 170 seconds worth of streaming. Now you only need to be able to fill those buffers at a rate of about 24/second. I'm assuming whatever file system is on the disks uses a massive block size, so the number of seeks you'd have to perform to pull 64MB off is probably pretty low. Eight? Sixteen? Even if it's the latter, that's only 384 seeks/second, which you could very plausibly do striped over only a half dozen disks, and the device presumably has many more.
  - runlevel1 12y ago
    > I'm assuming whatever file system is on the disks... Netflix uses UFS+J on FreeBSD 10. Here are some notes from their talk at NYCBSDCon 2014: - 400,000 stream files per appliance. - 5,000 - 25,000 client streams per appliance. - 300 - 500 streams coming off each disk all the time. - Attempt to buffer 1MB ahead, but caching is futile. - Result is completely random disk workload. - System becomes limited by disk latency and CPU load. Video here: https://www.youtube.com/watch?v=FL5U4wr86L4 https://www.youtube.com/watch?v=FL5U4wr86L4
bashinator 12y ago
I'd be interested to find out more of hardware details of these devices. 100+ TB of storage in a 4U is pretty respectable. From the images in the article, it looks like the drives are not hot-swappable, so I'd guess Netflix is able to remotely track loss of redundancy and will just send out an entire replacement unit when needed. For comparison, Supermicro makes one of the highest-density storage servers that I'm aware of[1]. 72 3.5" drives in 36 drive bays, so up to 288tB of raw storage, if you're brave enough to use 4tB drives. [1] http://www.supermicro.com/products/system/4U/6047/SSG-6047R-E1R72L2K.cfm http://www.supermicro.com/products/system/4U/6047/SSG-6047R-...
- pfg 12y ago
  There's some info on the hardware on Netflix Open Connect site [1]. They mention it's heavily influenced by Backblazes Storage Pod[2]. [1]: https://www.netflix.com/openconnect/hardware https://www.netflix.com/openconnect/hardware [2]: http://blog.backblaze.com/2014/03/19/backblaze-storage-pod-4 http://blog.backblaze.com/2014/03/19/backblaze-storage-pod-4
- jedberg 12y ago
  > It looks like the drives are not hot-swappable, That's correct. > so I'd guess Netflix is able to remotely track loss of redundancy and will just send out an entire replacement unit when needed. Exactly. If a drive dies, the capacity of that box is just reduced. Once enough drives die, the box gets an RMA.
  - bashinator 12y ago
    > Exactly. If a drive dies, the capacity of that box is just reduced. So not even redundancy? Just cope with losing whatever media happened to be on the dead drive?
    - wmf 12y ago
      They can always download another copy onto a working drive.
    - 12y ago
frio 12y ago
Isn't this just a CDN? ISPs quite happily host Akamai and Google nodes -- are they being paid for maintenance of those? Empirically, from an old job, the answer is no (in fact, we had to petition for quite a while to get some Google boxen), but that's in a country at the tail end of a long, long pipe. Is it different in the US?
- wmf 12y ago
  People don't know how CDNs work so it looks amazing. Comcast and Verizon now claim that Akamai is indeed paying them. Perhaps the outcry over Netflix being extorted explains why these deals have remained secret for years.
- porpoisemonkey 12y ago
  What this article is describing sounds like a colocation agreement which is a quite common service offered by ISPs and is typically paid for by a monthly rental fee calculated by storage size, dedicated bandwidth use and power consumption.
  - rahimnathwani 12y ago
    It's not the same. If I want Verizon to host my server, they have nothing to gain, but I do. So, yes, they should charge me. If instead they host a CDN server for a big content provider, they save money on routing content within their network.
    - jsz0 12y ago
      That's not always the case though. Sometimes rack space is more valuable to ISPs than improving bandwidth efficiency via CDN co-location deals. It really depends how much space you've got to spare and who else wants it.
      - liveoneggs 12y ago
        a friendly ISP always has rack space.. When you own a few data centers for yourself space is easy to come by.
tzs 12y ago
If the ISPs are reclassified as common carriers, will they still be allowed to let Netflix install those boxes? Presumably there is limited space in the ISP data center, so they cannot let everyone who wants to stick a box in there do so, so I'm wondering what methods a common carrier can use to decide which content providers can have access, and how this fundamentally differs from the "fast lanes" that common carrier status is supposed to prevent.
- wmf 12y ago
  Maybe they can come up with a rule like the box must save 2 Gbps of peak hour bandwidth per U. From the ISP's perspective they are hosting these boxes to save money, not improve performance. If performance happens to improve (ahem) that's a convenient side effect.
deleted 12y ago
[deleted]
tux1968 12y ago
What stops an unscrupulous ISP from making a timely copy of all his favorite movies?
- michaelneale 12y ago
  Same applies to all other CDNs that ISPs host? Although this makes a fairly rich target of concentrated content ;)
mirkules 12y ago
From my understanding, these boxes are basically local copies of the entire Netflix online catalog - a catalog that frequently changes. How do these ~15,000 machines pick up new content? I assume they synchronize with the Netflix mothership every once in a while. My question (without any intent to trivialize what Netflix is doing) is, isn't that just a good-old caching system? But it makes me wonder if it would be possible to create a distributed, localized caching system? For example, if I watch The Avengers, my machine could be configured to cache a part (or all) of the movie. The more people watch it, the more it is distributed. Since Netflix can correlate location data with what movies are watched, they can target certain areas to cache certain movies more aggressively. When the next person in my neighborhood watches The Avengers, they would pull part of the movie from me (maybe the beginning until the rest buffers) and cache another part of it themselves for the next person in line. That way all the people in my neighborhood or adjacent neighborhoods don't have to pull the entire movie across the internet every time - instead, we help each other watch movies with higher quality. On second thought, this may not be cheap considering the amount of development that may be required (and, at the end of the day, someone has to pay for this storage - although Netflix could, for example, offer personal mini-caching boxes in exchange for a monthly discount), but it's fun to dream about a sanctioned, decentralized, peer-to-peer movie service.
- thefreeman 12y ago
  They mention in the article that typically each box downloads a 7.5TB update each day.
- OvidNaso 12y ago
  This seems to be pretty close to what Popcorn Time does, and apparently does it very well. Their site says they already select individual files, so I imagine a 'sanctioned' Netflix style offering would be technically trivial to implement. This whole thing is just a reflection of the open internet/what went wrong debacle where all of this money that could be going to the content producers is instead spent on several layers of network, hardware and technical service providers that a small group from Buenos Aries has demonstrated is unnecessary if we could only, actually use, the network. But we can't.
- 12y ago

86 points by elecboy 12y ago

Interesting to see some more details on the appliances. Isn't the argument primarily about peering, not open connect appliances, though?

ceejayoz 12y ago
Installing these reduces the need for peering dramatically.
- tonyb 12y ago
  And the two are also not mutually exclusive. You can have an OpenConnect appliance that is populated via peering. Someone like Verizon/Comcast/etc could add OpenConnect appliances and then peer with a single 10Gbps link vs the multiple 10Gpbs transit links they are using now.
wmf 12y ago
It's all one discussion in my mind. There are three ways to get Netflix: boxes, peering, or transit (from cheapest to most expensive). If peering saves the ISP money and the boxes save even more money, why are some ISPs refusing to use them?
- josho 12y ago
  Because some ISPs are trying to maximize their revenue by charging Netflix directly. There are a few other legitimate reasons as well. Some ISPs dislike hosting hardware that they don't own in their data centre. Or it may be that an ISP has contracts in place that commit the ISP to certain bandwidth costs, so removing Netflix traffic may not actually save them money.
  - qq66 12y ago
    Why shouldn't an ISP charge Netflix to host a box in their datacenter? I'd charge you a fee if you wanted to store your DVDs in my house, even if you let me watch them.
    - wmf 12y ago
      Because the box saves the ISP more money than its hosting costs.
      - warfangle 12y ago
        On the other hand, if the ISP also runs a video service, they've got some very compelling reasons to do everything they can to not increase the quality of service for Netflix. They're green, and rectangular, and...

deleted 12y ago

[deleted]

Thaxll 12y ago

Lot of mistakes in this article, afair it can't hold all the movies but just the most popular. http://oc.nflxvideo.net/docs/OpenConnect-Deployment-Guide.pdf http://oc.nflxvideo.net/docs/OpenConnect-Deployment-Guide.pd... " Each appliance stores a portion of the catalog, which in general is less than the complete content library for a given region. Popularity changes, new titles being added to the service, and re- encoded movies are all part of the up to 7.5TB of nightly updates each cache must download to remain current. We recommend setting up a download window during off-peak hours when there is sufficient free capacity."

pessimizer 12y ago
"But wait, you say. That'd be all fine and dandy if Netflix was etched in stone, but it's not! They add and remove stuff all the time! That's right, which is why most OCAs take a 7.5TB update. Every. Freaking. Day. "If you had to take a 7.5TB update on your home internet, you would be screwed. According to Ookla's Net Index, the average download speed in the United States is 18.6Mbps. So with an average connection, that 7.5TB would take you about 40 days to chew through. "Fortunately, data centers tend to have a little more speed to work with. A firehose, compared to your bendy straw. So Netflix recommends a much shorter 10-hour span (from 2AM to noon, local time) where these boxes can soak up their updates with 1.2Gbps connections. That is to say, 'Google Fiber speeds.'" Source: TFA.
- ssmoot 12y ago
  I'm sure thought has been put into it. I just wonder why the OCAs aren't caches? I'd imagine a cache you be a lot more (mostly bandwidth) efficient for the ISPs and scale much better. I request a stream. It's proxied through the device. It's not present on the device. So the device requests the 1080P from Netflix. It transcodes the stream it receives on-the-fly and sends me whatever quality I require while writing both the original and my lower-quality version to disk. I mean maybe the limitation in that plan is the number of streams that could be transcoded simultaneously. So maybe instead the on-the-fly version isn't cached. Because I may need to switch quality on-the-fly. So maybe instead jobs are just queued up for the 5 or so formats required based on the 1080P download and they can process on spare transcoders opportunistically. Just interesting to think through it as a programming problem. I'm sure the people working on this are very smart. It just seems really heavy on the bandwidth utilization as is.

JTon 12y ago

Can anyone explain why an ISP would push back Netflix installing one of these boxes? Does the ISP want Netflix service congestion to better sell its own media assets/distribution (i.e. cable TV)?

osi 12y ago
From the operator of a smaller ISP: http://www.listbox.com/member/archive/247/2014/07/search/bmV0ZmxpeA/sort/time_rev/page/1/entry/0:7/20140711175317:C1A2027E-0945-11E4-9E99-FF13B5412E71/ http://www.listbox.com/member/archive/247/2014/07/search/bmV...
- acdha 12y ago
  That's Brett Glass, though, who is … opinionated. It's a good idea to read the entire thread for responses from a number of other network operators as well as some Netflix technical staff: http://seclists.org/nanog/2014/Jul/index.html#265 http://seclists.org/nanog/2014/Jul/index.html#265
  - osi 12y ago
    thanks for the nanog tip. been awhile since i've read that firehose :)
tobz 12y ago
Based on the blog posts by Level 3 -- how much internal capacity Comcast / Verizon say they have versus what Level 3 sees for utilization at their peering points with both ISPs -- I would have to assume "yes".
- dragontamer 12y ago
  Can you please post a link? This sounds very interesting...
  - k3oni 12y ago
    Here's a link to one of the blog posts: http://blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/ http://blog.level3.com/global-connectivity/verizons-accident...

nrzuk 12y ago

Could someone explain to me (it's been a long day, and I'm no storage expert!) how these are useful, wouldn't you need a whole bunch of these at each location they are to be installed? Just basing this on my own experience I have 12 drives in RAID which have a fairly substantial sequential throughput. Start multiple streams of high bandwidth videos and the maximum throughput from the drives drops sharply due to the random reads. I would have assumed that having even 100 people streaming from a single box would be an interesting challenge.

weaksauce 12y ago
well, if it means that they need to install n boxes in an isp to avoid peering at a high cost then it might be worth it. > Sometimes it'll even have multiple copies of the same show, if something is in crazy high demand. that statement from the article lends one to believe that the raid arrays are in parallel. These boxes are meant to reduce the number of packets going all the way across the internet not necessarily remove the need. Also, in addition to lots of storage, these boxes probably have loads of ram that they can buffer popular streams into and then serve it.
twoodfin 12y ago
These things can pump out 12Gbit during peak viewing, which is enough for about 4,000 avg. 3 Mbit streams. If they can cram in 256GB of DRAM, that's enough room for a 64MB buffer per stream, or about 170 seconds worth of streaming. Now you only need to be able to fill those buffers at a rate of about 24/second. I'm assuming whatever file system is on the disks uses a massive block size, so the number of seeks you'd have to perform to pull 64MB off is probably pretty low. Eight? Sixteen? Even if it's the latter, that's only 384 seeks/second, which you could very plausibly do striped over only a half dozen disks, and the device presumably has many more.
- runlevel1 12y ago
  > I'm assuming whatever file system is on the disks... Netflix uses UFS+J on FreeBSD 10. Here are some notes from their talk at NYCBSDCon 2014: - 400,000 stream files per appliance. - 5,000 - 25,000 client streams per appliance. - 300 - 500 streams coming off each disk all the time. - Attempt to buffer 1MB ahead, but caching is futile. - Result is completely random disk workload. - System becomes limited by disk latency and CPU load. Video here: https://www.youtube.com/watch?v=FL5U4wr86L4 https://www.youtube.com/watch?v=FL5U4wr86L4

bashinator 12y ago

I'd be interested to find out more of hardware details of these devices. 100+ TB of storage in a 4U is pretty respectable. From the images in the article, it looks like the drives are not hot-swappable, so I'd guess Netflix is able to remotely track loss of redundancy and will just send out an entire replacement unit when needed. For comparison, Supermicro makes one of the highest-density storage servers that I'm aware of[1]. 72 3.5" drives in 36 drive bays, so up to 288tB of raw storage, if you're brave enough to use 4tB drives. [1] http://www.supermicro.com/products/system/4U/6047/SSG-6047R-E1R72L2K.cfm http://www.supermicro.com/products/system/4U/6047/SSG-6047R-...

pfg 12y ago
There's some info on the hardware on Netflix Open Connect site [1]. They mention it's heavily influenced by Backblazes Storage Pod[2]. [1]: https://www.netflix.com/openconnect/hardware https://www.netflix.com/openconnect/hardware [2]: http://blog.backblaze.com/2014/03/19/backblaze-storage-pod-4 http://blog.backblaze.com/2014/03/19/backblaze-storage-pod-4
jedberg 12y ago
> It looks like the drives are not hot-swappable, That's correct. > so I'd guess Netflix is able to remotely track loss of redundancy and will just send out an entire replacement unit when needed. Exactly. If a drive dies, the capacity of that box is just reduced. Once enough drives die, the box gets an RMA.
- bashinator 12y ago
  > Exactly. If a drive dies, the capacity of that box is just reduced. So not even redundancy? Just cope with losing whatever media happened to be on the dead drive?
  - wmf 12y ago
    They can always download another copy onto a working drive.
  - 12y ago

frio 12y ago

Isn't this just a CDN? ISPs quite happily host Akamai and Google nodes -- are they being paid for maintenance of those? Empirically, from an old job, the answer is no (in fact, we had to petition for quite a while to get some Google boxen), but that's in a country at the tail end of a long, long pipe. Is it different in the US?

wmf 12y ago
People don't know how CDNs work so it looks amazing. Comcast and Verizon now claim that Akamai is indeed paying them. Perhaps the outcry over Netflix being extorted explains why these deals have remained secret for years.
porpoisemonkey 12y ago
What this article is describing sounds like a colocation agreement which is a quite common service offered by ISPs and is typically paid for by a monthly rental fee calculated by storage size, dedicated bandwidth use and power consumption.
- rahimnathwani 12y ago
  It's not the same. If I want Verizon to host my server, they have nothing to gain, but I do. So, yes, they should charge me. If instead they host a CDN server for a big content provider, they save money on routing content within their network.
  - jsz0 12y ago
    That's not always the case though. Sometimes rack space is more valuable to ISPs than improving bandwidth efficiency via CDN co-location deals. It really depends how much space you've got to spare and who else wants it.
    - liveoneggs 12y ago
      a friendly ISP always has rack space.. When you own a few data centers for yourself space is easy to come by.

tzs 12y ago

If the ISPs are reclassified as common carriers, will they still be allowed to let Netflix install those boxes? Presumably there is limited space in the ISP data center, so they cannot let everyone who wants to stick a box in there do so, so I'm wondering what methods a common carrier can use to decide which content providers can have access, and how this fundamentally differs from the "fast lanes" that common carrier status is supposed to prevent.

wmf 12y ago
Maybe they can come up with a rule like the box must save 2 Gbps of peak hour bandwidth per U. From the ISP's perspective they are hosting these boxes to save money, not improve performance. If performance happens to improve (ahem) that's a convenient side effect.

deleted 12y ago

[deleted]

tux1968 12y ago

What stops an unscrupulous ISP from making a timely copy of all his favorite movies?

michaelneale 12y ago
Same applies to all other CDNs that ISPs host? Although this makes a fairly rich target of concentrated content ;)

mirkules 12y ago

From my understanding, these boxes are basically local copies of the entire Netflix online catalog - a catalog that frequently changes. How do these ~15,000 machines pick up new content? I assume they synchronize with the Netflix mothership every once in a while. My question (without any intent to trivialize what Netflix is doing) is, isn't that just a good-old caching system? But it makes me wonder if it would be possible to create a distributed, localized caching system? For example, if I watch The Avengers, my machine could be configured to cache a part (or all) of the movie. The more people watch it, the more it is distributed. Since Netflix can correlate location data with what movies are watched, they can target certain areas to cache certain movies more aggressively. When the next person in my neighborhood watches The Avengers, they would pull part of the movie from me (maybe the beginning until the rest buffers) and cache another part of it themselves for the next person in line. That way all the people in my neighborhood or adjacent neighborhoods don't have to pull the entire movie across the internet every time - instead, we help each other watch movies with higher quality. On second thought, this may not be cheap considering the amount of development that may be required (and, at the end of the day, someone has to pay for this storage - although Netflix could, for example, offer personal mini-caching boxes in exchange for a monthly discount), but it's fun to dream about a sanctioned, decentralized, peer-to-peer movie service.

thefreeman 12y ago
They mention in the article that typically each box downloads a 7.5TB update each day.
OvidNaso 12y ago
This seems to be pretty close to what Popcorn Time does, and apparently does it very well. Their site says they already select individual files, so I imagine a 'sanctioned' Netflix style offering would be technically trivial to implement. This whole thing is just a reflection of the open internet/what went wrong debacle where all of this money that could be going to the content producers is instead spent on several layers of network, hardware and technical service providers that a small group from Buenos Aries has demonstrated is unnecessary if we could only, actually use, the network. But we can't.
12y ago