7 ms·
Actual ISP network engineer here: I'm sure these people are having fun and all, but it's insignificant in the larger scheme of things. (correction to original
by walrus01 9y ago
Actual ISP network engineer here: I'm sure these people are having fun and all, but it's insignificant in the larger scheme of things.
(correction to original post: I did not at first see an ASN for them, or announcements of their own IP space. They have an ASN and announce two /24s, and peer at one physical location in metro NYC).
Real five nines network infrastructure takes significant capital investment.
This part is so much unadulerated bullshit:
""The internet doesn't really cost you anything, it's just the connection [that has a fee]. So however you can get plugged in — then you're on the internet. Nobody owns the internet, there's no one to pay.""
Actual, reliable internet costs real money, both in the salaries of people to engineer and architect it, salaries for 24x7x365 clued-in NOC staff, equipment, salaries and expenses for field technicians to build it. And that's before you get into things like establishing colo at major IX points, serious core routers that cost $15k each (do you really want to deploy something in the year 2018 to take several full BGP tables that doesn't have a 4 million FIB capacity?), etc. The Internet is a significant construction project at OSI layer 1, whether you're putting PTP radios on rooftops, running aerial fiber, or underground fiber. Otherwise you're just piggybacking on something that another, larger entity has already built.
I give them an A for enthusiasm and effort. It just needs to be channeled the right way so that they can figure out what it actually costs to run a reliable ISP. I'm all in favor of new startup ISPs.
- dboreham 9y agoAgree the article is BS. Mesh networking is incompatible with QoS. So they can surely deliver crappy Internet cheaply. More than that? Nope. That'd need serious infrastructure such as licensed P2P, 60GHz short haul links, leased fiber. $$$$$ Oh, and they'd need peering: more $$$$$
- nycmesh 9y agoDid you read any of our comments?
- deleted 9y ago[deleted]
- rdiddly 9y agoI think maybe the whole point is that not everybody wants all that shit, or they want to experiment with doing without some of it. That last 5% of reliability is on the order of 50% of the cost, or more, isn't it? Is it worth paying double for the small 5% improvement? Not for everyone, obviously. And that's true even assuming just bare costs without markups. Obviously when you take into account how much profit & graft is attached to every single level of that stack, it really starts to add up. Throw in the absolutely, inexcusably bad, awful experience nearly everyone has with the big telecom companies and their employees, and that's enough ill will to propel many an alternative DIY network. Metaphorically speaking, you just told a bicycle owner what a terrible car he has. Or rather, warned him how much the car is going to cost that he doesn't want in the first place. Is it possible the bicyclist can actually tell the difference between a bike and a car? Maybe he even sees some advantage you don't? (As opposed to the implausible but apparently favored hypothesis by car-owners that no, he just likes punishing himself while unsuccessfully pretending the bike is a car. When we don't know a stranger's rationale we glibly tend to assume there is none, and they're just cray-cray.) To make yet another metaphor, when all you know is hammers, the whole world looks like it's full of 1) nails, 2) hammers, and 3) things that suck at being hammers.
- mentos 9y agoYou have any opinion on if internet should be treated as a utility?
- bogomipz 9y agoI have to wonder at the level of condescension in your comments. I'm wondering if it's because you actually feel threatened by this. >"Actual ISP network engineer here: I'm sure these people are having fun and all, but it's insignificant in the larger scheme of things" No it is actually very significant. In a past life I was also an "Actual ISP network engineer" at a large ISP. And what I can say is that large ISPs aren't exactly wellsprings of innovation. However these folks are hacking on things, solving an actual problem and seem to be having fun in the process. It's exactly this ethos that usually ends up moving the needle. Sometimes by accident. It's curious that you are so keen to poo-pooh them. >"Real five nines network infrastructure takes significant capital investment" There is no ISP that provides customers 5 9's of uptime. Not even for business customers. With a Time Warner or Comcast you are lucky if you get 3 9's from their network. This is what you get from "the duopoly." >"And that's before you get into things like establishing colo at major IX points, serious core routers that cost $15k each (do you really want to deploy something in the year 2018 to take several full BGP tables that doesn't have a 4 million FIB capacity?), etc." There a clue in the article about which IXP they're - 32 Ave of the Americas(DE-CIX.) It's one of the 3 of the most "wired" buildings in NYC. You can peer with everyone there. These people seem to have done their homework. And you don't need to buy $15K Cisco ASR any more. You can take full tables on commodity hardware running Linux and ExaBGP or BIRD. There are many CDNs and IXP's doing just that.
- chriswarbo 9y agoThis comment sounds very much like the AT&T engineers mentioned in http://www.shirky.com/weblog/2010/04/the-collapse-of-complex-business-models http://www.shirky.com/weblog/2010/04/the-collapse-of-complex... (a post which appears on HN quite regularly). --- > In the mid-90s, I got a call from some friends at ATT, asking me to help them research the nascent web-hosting business. They thought ATT’s famous “five 9’s” reliability (services that work 99.999% of the time) would be valuable, but they couldn’t figure out how $20 a month, then the going rate, could cover the costs for good web hosting, much less leave a profit. ... > The ATT guys had correctly understood that the income from $20-a-month customers wouldn’t pay for good web hosting. What they hadn’t understood, were in fact professionally incapable of understanding, was that the industry solution, circa 1996, was to offer hosting that wasn’t very good. ... > For a century, ATT’s culture had prized—insisted on—quality of service; they ran their own power grid to keep the dial-tone humming during blackouts. ATT, like most organizations, could not be good at the thing it was good at and good at the opposite thing at the same time. The web hosting business, because it followed the “Simplicity first, quality later” model, didn’t just present a new market, it required new cultural imperatives.
- rsync 9y agoGod, this reply couldn't be worse: "Real five nines network infrastructure takes significant capital investment." No, it doesn't. Overly complex, corporatized infrastructures designed by committees in meetings - that takes significant capital investment. Simple infrastructure designed by hackers can achieve bullshit metrics like "five nines" without breaking the bank. "And that's before you get into things like establishing colo at major IX points, serious core routers that cost $15k each" Establishing colo at a "major IX point" is a few emails and phonecalls away. Your salesperson will fall all over him or herself and waive the NRCs to get you in there. It doesn't cost much, even for a full rack with 20 or 30 amps. If you're just doing networking, you won't even need that. Then you get a fiber cross connect to he.net for 10 or 20 or 40 gbps and you're in business. You can buy very, very nice used routers and since you're a hacker and not Mr. Enterprisey IT guy, you can run it and support it yourself. We had many iterations of this exact critique in our early days - explaining how we couldn't possibly deploy this volume of storage without fancy bullshit enterprise solutions from EMC or Compaq and how we couldn't possibly run a global storage network without fancy routers (we currently own zero routers) and fantastically complex clustering and failover ... and 17 years later we're still making it all work just fine. Fuck those people. Fuck those school trained, passionless, non-hacker, no-talent-pussies. Good luck with your mesh network!
- niroze 9y agoI wish I could upvote this many times. Great reply. Agree 100% and thanks for speaking for us that work behind the scenes to make things happen.
- nycmesh 9y agoyou are upvoting [things that are not true about us]
- mediocrejoker 9y ago> you are upvoting nonsense I don't think this is a helpful reply. Since it sounds like you're associated with the project, can you address some of the specific concerns mentioned, like reliability? Do you disagree that hundreds of thousands of dollars are required for five nines connectivity? Do you disagree that that level of reliability is required for your customers etc.
- bigiain 9y agoFWIW, I have many many companies as customers for whom I've offered pricing for three, four, and five nines of reliability (for mobile app backends, not residential ISP service, but lets run with it any way). Not a single one of them has ever signed up for anything more expensive than three nines. Only a few have even discussed the differences between 3 and 4 nines solutions. I'm reasonably sure I'd choose the same for my home internet connection - if offered representative pricing based on costs of providing 3, 4, or 5 nines, or possibly even 2 nines - I'd choose the least expensive because losing a few minutes or even occasionally 15mins a day of connectivity at home (or maybe more realistically an ~8 hour outage per month) really isn't going to bug me greatly - not if it's two or more orders of magnitude cheaper than a four of five nines connection.
- nycmesh 9y agoNo you don't need hundreds of thousands. You do need tens of thousands. We've had no downtime this year at supernode1. Yes reliability is a big priority, but it is not a big priority for the big ISPs in NYC. Time Warner Cable (now Spectrum) has barely two nines. In Manhattan it would go down in the East Village after every large storm. We are already providing faster more reliable connections than this.
- nycmesh 9y agoOur ASN is AS395853. Our peering info is here- https://nycmesh.net/peering/ https://nycmesh.net/peering/ We have v4 and v6 space. Our uptime when connecting to supernode1 is significantly better than Time Warner Cable as many of our members have reported. We are already established. This is something we have already done! It isn't impossible. Our IXP is DE-CIX, the worlds biggest IXP. We are currently in one data-center/IXP and talking to a few more colocs.
- bogomipz 9y ago>"Our IXP is DE-CIX, the worlds biggest IXP." Yep, DE-CIX is great and 32 Ave of the Americas is one of the most wired buildings in NYC. Great choice. Kudos on you folks, this is a great initiative. Reading this article made me happy. And I'm glad you are getting some press!
- walrus01 9y agoDigi Desert LLC, which existed in 2010? Looks like that's the AS of some entity that predates the existence of your project and current public relations effort. You announce a whopping total of two v4 /24. Kudos for having enough clue to know that you needed to establish a presence at a major IX, and actually doing it, because it looks like you're adding peers. But your actual network presence is minuscule. How do you intend to compete with the six NYC based companies I can think of off the top of my head that are putting fiber fed, $9,000 to $20,000 5 to 10Gbps 71-86 GHz PTP links on rooftops to build their own backbones, when you're playing around with 1000BaseT to the roof and AF24s? You have high uptime? Do you have any of the following, because some of your much larger competitors sure do: sites with parallel A and B side power systems -48VDC rectifier + battery systems sized for 24 hour runtime at load diesel generators propane generators generator resupply contracts chassis-based routers with hotswap fan, N+1 power supplies, dual redundant routing engines pair of identical core routers singlemode fiber to the roof -48vdc power to the roof ironclad rooftop lease agreements with building owners, drawn up by professional telecom/real estate lawyers that run for 5+ year terms a 24x7x365 NOC staffed by live humans
- nycmesh 9y ago
- koolba 9y ago> Actual, reliable internet costs real money, both in the salaries of people to engineer and architect it, salaries for 24x7x365 clued-in NOC staff, equipment, salaries and expenses for field technicians to build it. A phone network that objectively sucked compared to AT&T and Verizon but was boatloads cheaper seems to have worked out just fine for T-Mobile. I can see the same thing working for this. $20/mo for an internet connection that works most of the time would have plenty of customers. I'm not familiar with the wireless tech involved so no clue how well this scales outside of high density cities but within some place like NYC there's plenty of people that would pay for this over $50+/mo (min!) to the local cable companies.
- jimktrains2 9y ago> Actual ISP network engineer her So then you must intimately understand the depth of the lies and anti-consumer business practices that spur such networks. > Real five nines network infrastructure takes significant capital investment. 5-9s is 5 minutes of downtime a year. I've never had a commercial, let alone residential ISP hit that mark. > Actual, reliable internet costs real money, both in the salaries of people to engineer and architect it, salaries for 24x7x365 clued-in NOC staff, equipment, salaries and expenses for field technicians to build it. ... Going back to my first point, almost no one is unwilling to pay for their internet connection. They're unwilling to pay less to get TV service they have no use for and internet than they are internet alone. They're unwilling to look at the cost of network access in other countries and be happy when comparing that to the cost of internet in a major, densely populated area in the US. They're unwilling to accept transfer caps and accept the lies they they help the network by making people share when they know that transfer is irrelevant and bandwidth is the limited commodity. They're unwilling to accept lies of bandwidth speed that isn't attainable even under the best of conditions. They're unwilling to accept shady and underhanded business practices such as random, "accidental" charges. > It just needs to be channeled the right way so that they can figure out what it actually costs to run a reliable ISP. I'm all in favor of new startup ISPs. However, as an "actual ISP network engineer", you must understand the (very arguably unfair) regulatory issues that the big players have put in place to make this impossible to do. Google failed at starting an ISP, and it's not like they are tight on either cash or lawyers.
- rayiner 9y agoDo you have any experience with either ISP level network engineering or regulation? Because you speak awfully authoritatively.
- adrianN 9y agoI'm no network engineer, but when you have a large network with lots of equipment, you want the individual components to have a couple more nines than the resulting network. Maybe that's what they meant.
- walrus01 9y agoGoogle did not fail at starting an ISP. They decided that the ROI for doing individual residential home FTTH was not worth it, considering the cost per house passed. Then they went and bought Webpass, which has a totally different condo/apartment based business model, which has been wildly successful. Starting a small ISP based out of extreme dissatisfaction with huge, evil incumbents is something that I encourage any plucky upstart to do. If you don't like Comcast, Charter, RCN, etc, if you have sufficient clue, go do it yourself. Just don't try to do it with $20,000 and some good wishes, because something with much deeper pockets like Webpass will come along and obliterate you. The point that I am rather bluntly trying to get across to these folks is that if they're going to invest significant amounts of their personal time/engineering resources into it (unpaid labor?), that they should treat it like a serious business and not a nonprofit. Raise sufficient funds, somewhere in the $500,000+ range to start, to build a network that actually does meet five nines through sufficient diversity in topology and routing. Compared to random YC companies that seem to raise $1m to $5m to do some vaguely defined web2.0 SAAS thing, the amount of money that a group of people need to raise to build the core of a serious ISP is not huge. Elsewhere in this thread I threw out a figure like $750,000 in beginning CapEx if they're serious. For some good reasons, however, it's easier to get $5m of series A funding from a VC for a pure software development business than a wholly-owned-facilities-based startup ISP. Even if you completely ignore things like colocation costs, IP transit costs, IX/peering costs, office rental, office utilities and overhead, running a serious ISP with live, clued in humans watching the network takes a lot of money per year for salaries. 3 to 5 NOC persons for 24x7x365 NOC phone coverage. At a certain point if you peer with the big boy ASNs they will expect a live human to answer the phone at 3am when something unusual happens in traffic between your mutual networks. That live human needs to be somebody who has the equivalent of 'enable' on your peering router and sufficient clue to wield it. 2 network engineering/architecture positions, multiply by expected fully loaded yearly salary and benefits package for each person. at least 2 field technicians vehicles, vehicle insurance, tools for technicians At NYC salaries? In a greenfield scenario, I could easily spend $900,000 a year in just payroll and benefits, just to begin to attempt to duplicate what some other serious ISPs already have in place for manpower. And again that is before you spend one dollar on equipment.
- Roritharr 9y agoThe whole 5 nines aspect is interesting. I wonder why there isn't more cheaper more unreliable service available for customers built on cheaper hardware using less reliable infrastructure. Laser Links to cross distances etc. I imagine consumers would be fine with 95% uptime of a gigabit connection compared to 99,99999% uptime of 16mbit.
- deleted 9y ago[deleted]
- juliangoldsmith 9y ago>I imagine consumers would be fine with 95% uptime of a gigabit connection compared to 99,99999% uptime of 16mbit. I can guarantee that nobody would be okay with 95% uptime. That means that your Internet access is down for over an hour every day. Given that people go berserk when their Internet is down for even a few minutes, 95% uptime will be unacceptable to consumers.
- ForHackernews 9y agoDepends which hour it's down. Plenty of people would be happy to have their internet down from 3-4 am every morning if it meant they paid $10/month instead of $70/month.
- bigiain 9y agoFWIW, many of the people selling five nines would just claim those 3-4am windows as "scheduled maintenance" and go down for an hour every night anyway whether they need to or not... (Actually, if I were an evil ISP lawyer/marketer, I'd try to claim a "Maintenance window of 1hr downtime sometime between 4am and 3am", then ignore any downtime of less than an hour during the day and if you complained about not getting yur expensive five nines I'd laugh and say "published schedule maintenance not included!"...)
- adamsea 9y agoFirst guess is it would be most likely to be down when traffic is highest.