17 ms·

'Anti-authority' tech rebels take on ISPs, connect NYC with cheap Wi-Fi

417 points by standeven 8y ago

nickthemagicman 8y ago
Hate to sound stupid but how does DNS work on a mesh?
- nycmesh 8y ago
  We originally used the qMp/OpenWRT package that included mdns, which is a version of zeroconf for the internal network. It isn't scaling so we now use our own DNS server to resolve internal addresses on our 10/ network.
  - nickthemagicman 8y ago
    Mesh networks are going to be incredible when they're widespread! It's truly superior to centralized control by ISPs/Governments and the world will be grateful for this after this whole net neutrality debacle recently. You guys are brilliant and doing an amazing job and I'm grateful people like you guys exist! Thank you for the response!
tlrobinson 8y ago
There's an impressive amount of negativity in these comments. A community-run mesh network? HN should be all over it, but it appears to have turned into a bunch of network engineers telling us why they are very smart and it won't work, or is "insignificant". More power to you, NYC Mesh.
- pavs 8y ago
  It's OK to be optimistic about something different. It's also ok to discuss the reality, that's not necessarily negativity towards a project. Are we only supposed to discuss one side of the argument and ignore reality just because it makes us feel nice inside? There are bazillion reasons that this is nothing more than a cute side project that will fall on its face when its time to scale in any meaningful way. Unless they have discovered a new technology they are not taking on ISPs by any meaningful way. Especially in NYC. There is a difference between, just "connecting to the internet" and having a "reliable connectivity to the internet", the difference is a massive investment in infrastructure. Wifi p2p mesh networks don't work on magic. They are very unreliable and hard/impossible to scale to 100,000 users, let alone millions of users. In some places, wifi p2p connection hardware actually costs more money than a traditional wired network. Mesh networks have been around for a while, technical and reliability aspect of this is well understood within the networking industry. There is a reason why it's not widely used, especially in cities. In remote areas, this kind of network makes sense, if you don't have a choice or an alternative. Don't brush of educated criticisms from people within the industry. I own an ISP for 5+ years (mostly wired). I am not in US and I am not at all worried if this kind of network comes in my country/area. We are not worried because we have tried and deployed this kind of network in cities and they are extremely unreliable and costs real money. I also don't understand the "Anti-authority" angle of the argument, unless I live in a different universe, they still have to get license/permit from the local government to do this business (non-profit or not), they still have to follow the same rules enforced by local and federal government when it comes to log/customer information sharing to LEO. So not sure how they are Anti-authority? Unless they mean anti-establishment?
  - hux_ 8y ago
    Once upon a time the network, storage and data center guys thought Facebook/YouTubes one to many broadcasting of content would never scale too. Now they fall over themselves to work there.
    - js1k374 8y ago
      I think pavs is referring to physical limitations of the technology that prevent it from scaling.
    - pavs 8y ago
      They scaled by making a huge investment in their network. They didn't make a fundamental new technical discovery that didn't exist before. They had money, they threw money at the problem, as expected it worked. The problem was an investment (which takes time), not technical. https://peering.google.com/#/ https://peering.google.com/#/ https://code.facebook.com/posts/565767133547005/steering-oceans-of-content-to-the-world/ https://code.facebook.com/posts/565767133547005/steering-oce... Any ISP (if big enough), can host their own Google, Akamai, Facebook, Netflix nodes on their premise.
- 8bitsrule 8y ago
  It has been a few years since I learned about Guifi.net (non-English site; English portal:guifi.net/en/node/38392) in Catalonia/Valencia - a 2004 startup. WP says that (as of a year ago) it has 'over 33,000 active nodes and about 46,000 km of wireless links.' It's described (in English) in this 2016 paper http://dsg.ac.upc.edu/sites/default/files/dsg/acm-sigcomm-gaia-guifi-econ.pdf http://dsg.ac.upc.edu/sites/default/files/dsg/acm-sigcomm-ga... as 'known to be the largest community network in the world ... and is still growing while successfully evading the collapse of the tragedy of the commons. Undoubtedly, this is due to the governance model that the foundation has developed....' Since it's been around for a while, and still viable, that says something about the idea worth attending to. More info: http://www.dtic.upf.edu/~jbarcelo/tmp/guifi.net.pdf http://www.dtic.upf.edu/~jbarcelo/tmp/guifi.net.pdf
- rayiner 8y ago
  The weirdest part of HN for me is the optimism. Engineers are pessimistic because engineering is hard and physics is unforgiving. Engineers like to point out things won't work because it probably won't and engineers like to be right. Mesh networking is incredibly hard to get right to the point where I wouldn't even say anyone really has ever gotten it right at scale.
- BLKNSLVR 8y ago
  What's so ironically funny, is that this article is also on HN's front page: https://www.newyorker.com/culture/cultural-comment/brilliant-1895-novel-on-the-emptiness-of-literary-fame https://www.newyorker.com/culture/cultural-comment/brilliant... HN Comments: https://news.ycombinator.com/item?id=16980503 https://news.ycombinator.com/item?id=16980503 The jealousy amongst peers for non-recognition. "More power to you, NYC Mesh." is the correct answer. Thumbs up tlrobinson.
- John_KZ 8y ago
  It would get a lot more positive comments if it was called what it is: A low-bandwidth mesh network. It's nice, but not going to replace ISPs anytime soon. It consumes more energy, it's slower, less reliable and higher latency than regular wired/fiber networks. These things don't scale well and all those radios can be universally jammed by a single drone over the city. Not a great backup plan if you're anti-establishment. I'd love to partake in such a project but keeping your expectations grounded is very important.
EGreg 8y ago
I love seeing the growth of mesh network. “We don’t have any use cases that show how nice it can be if you have enough users.” They are going to need software. The supernodes are still a symptom of the thinking that you need the signal to go to another stage in order to communicate locally. I think there should be more software that works on a local network, like things used to work before broadband. This is what I’m talking about: https://www.youtube.com/watch?v=pZ1O_gmPneI https://www.youtube.com/watch?v=pZ1O_gmPneI
Zigurd 8y ago
When you compare comments here a few themes emerge: 1. The internet has been captured by the telcos, who operate the internet like telcos have operated the circuit switched voice network. 2. We are all overlooking the fact that smart radios make spectrum ownership obsolete. That means that the potential of mesh wireless connectivity is orders of magnitude larger than can be realized in current unlicensed spectrum. 3. The assumption that anyone is owed a business model in internet service is going unquestioned. Of course it costs a lot of money to run the internet the way the Bell System ran phones. But there are plausible alternatives, not all of which are compatible with investor-owned near-monopoly "markets." This is an economic tragedy and a case of local optimization holding a global optimum hostage. ISPs are a sweet business, in a way that costs the rest of the economy dearly. Kill the incumbents and reap an across the board boost in GDP.
0xBA5ED 8y ago
Well, it seems this idea has touched a few nerves. In this case, it may indicate you're on the right track. Keep it up!
10065 8y ago
NYC public housing has had rock bottom cost, high speed Internet for years. Have to now do something with it.
MessageLife1122 8y ago
Just by reading the comments, I d really don't need to read the article as a whole. haha
mapmap 8y ago
I'm curious about the technology they are using. Is it an actual mesh network or point-to-point wireless?
- tlrobinson 8y ago
  Here's a map of their network: https://nycmesh.net/map/ https://nycmesh.net/map/ (uncheck everything except "Link" to see the network)
  - ybloviator 8y ago
    "The network connects directly to the internet backbone, so we do not rely on an ISP." That makes no sense. I doubt they're a tier-1 backbone themselves, they have to be purchasing transit from an ISP. It feels like lots of the docs are written by marketing-type folks. Also there's a weird diagram that insinuates NAT happens before bgp peering with the not-ISPs they buy internet from. Weird shit.
  - dawiddutoit 8y ago
    Down here in South Africa we have a similar network. We have had quite shite connections down here for so long that a mesh network was one of the few ways to get a good connection. Link for our map of Cape Town. https://wind.ctwug.za.net/index.php/nodes https://wind.ctwug.za.net/index.php/nodes
- Zekio 8y ago
  Looks like point to point in the video
- crishoj 8y ago
  Seems to be an actual mesh network, based on the BMX6 mesh protocol [0]. [0] http://nycmeshnet.github.io/faq/ http://nycmeshnet.github.io/faq/
- nycmesh 8y ago
  We are a hybrid of mesh, p2p and point to multi-point (sector antennas). Whatever is most appropriate to give the best connection for reasonable price.
stcredzero 8y ago
Is there an inexpensive point to point technology that isn't hampered by rain?
- notyourday 8y ago
  Yes. Whatever TowerStream uses. But it is definitely line of sight and you have to be on Empire State or Trump Tower or a few other buildings to have any reasonable visibility
- jeffreyrogers 8y ago
  2.4 and 5 GHz aren't too affected by rain fade. And they're cheap.
  - mholt 8y ago
    I thought wifi couldn't go through water? (Or is rain just so sparse/diluted that it doesn't really affect it?)
    - jeffreyrogers 8y ago
      I'm not sure about water, but rain doesn't have much effect at those frequencies. At higher frequencies (e.g. 24GHz) rain can make the link useless.
notyourday 8y ago
I'm in NYC. I know a few people who played with it. It barely works. I'm all for this kind of tech solutions, but for crying out loud pretending that this is a replacement for ISPs in NYC really discredits techies.
- isostatic 8y ago
  Why doesn't a startup run their own fibre? Are building owners that protective? Actually scratch that, I tried getting a 1 gig cross manhattan link. Cost more than 1g to London.
  - cimmanom 8y ago
    Building owners? Before you get anywhere near the buildings, you have to dig up the streets to run cabling around the city. Any idea how much that costs?
    - ybloviator 8y ago
      Pilot is doing this cheaper than others, look into how they get their fiber in the ground...
    - notyourday 8y ago
      That's what you outsource to the latest incarnation of Metromedia Fiber Network - Zayo.
    - Symbiote 8y ago
      Speaking of London, talking to building owners is exactly what Hyperoptic did to build out a fibre network in London. Unfortunately, the building owner where I used to live wasn't interested -- it turned out most of the building was empty, people keeping apartments for occasional visits or as an investment :-( https://www.hyperoptic.com/how-hyperoptic-works/ https://www.hyperoptic.com/how-hyperoptic-works/ or https://www.hyperoptic.com/map/ https://www.hyperoptic.com/map/
    - isostatic 8y ago
tlrobinson 8y ago
This sounds neat. Is anyone working on something like it in the SF bay area? I know there's Webpass and Monkeybrains, which are great, but I'd love to be part of a community-owned, actual mesh network.
- dannyobrien 8y ago
  There's the People's Open Network: https://peoplesopen.net/ https://peoplesopen.net/ co-ordinated by the SudoRoom folks and the wider community.
- ziton 8y ago
  Not "community owned" per say, but Facebook has deployed a Mesh network in DT San Jose called Terragraph.
jeffreyrogers 8y ago
This stuff is cool and maybe useful in far-fetched scenarios, but it's not going to replace ISPs anytime soon. Mesh networks don't make economic sense in most circumstances. Also, I'm guessing that users share the same IP address, which can cause problems. Plus, if one user is doing something illegal and causes the supernode to get shutdown then no one on the mesh network will have access to the internet. Addition: quotes like "The internet doesn't really cost you anything, it's just the connection [that has a fee]... Nobody owns the internet, there's no one to pay." Are shockingly naive. Who does he expects pays for the fiber that runs across the country and along the ocean floor? The marginal cost of a new user on the internet is effectively zero, but that doesn't say anything important. The infrastructure that makes the internet work is enormously expensive.
- bogomipz 8y ago
  >"This stuff is cool and maybe useful in far-fetched scenarios, but it's not going to replace ISPs anytime soon. Mesh networks don't make economic sense in most circumstances" Far fetched you mean like getting instantly connected to the internet for a fraction of the cost and with better bandwidth in major city? You realize that mesh is just for the access network correct? There's still fiber between them and their transit providers. Mesh makes a lot economic sense compared to the costs of trenching and laying fiber miles.
- nycmesh 8y ago
  We are replacing ISPs in NYC for our members. This is now, not the future. The main thing holding us back is scaling up the number of installs. All networks, including us, add to this expensive infrastructure. The difference with our infrastructure is it is owned and shared by members.
  - Paul-ish 8y ago
    Very cool. Is this a customer coop or a non-profit (or something else)?
    - nycmesh 8y ago
lbotos 8y ago
Who's responsible when someone torrents copyrighted material through that Internet exchange point? "NYC Mesh will comply with all federal laws in the countries it operates, however, as policy, no data is collected and therefore no data exists to provide requestors." This is cute, but if you aren't hunting down people abusing the mesh, the whole connection to the internet will go away for a ToS violation.
- nickthemagicman 8y ago
  The beauty of it is as it becomes popular people will start torrenting WITHIN the mesh network.
  - kalleboo 8y ago
    Brings back memories of using DC++ hubs on the university network
- ybloviator 8y ago
  Didn't one of the major ISPs get their ass handed to them in court for trying to say that they were somehow exempt from keeping exactly this sort of information? I mean, you can SAY it, but it would be sort of insane to make a statement like that without a lawyer signing off on it. The law doesn't have a "this is too hard for us" or "we didn't build our network for this" exemption, does it?
- jlgaddis 8y ago
  I've worked for an ISP for about six and a half years. In that time, I have received I-don't-know-how-many e-mails about people torrenting and so on. I've had one conversation with a Special Agent from DHS investigating credit card fraud. Over this same period, I have given out a customer's identifying information zero times (if more than 24 hours have passed, there is a very small chance that I am able to identify the customer). In every instance, it was "sorry, I am not able to identify the customer". Zero is also the number of times I have heard anything from any of my upstreams regarding any ToS violations or similar.
jessaustin 8y ago
The group also knows there will be growing pains as they challenge the status quo and that it's only a matter of time before the big ISPs take notice, which could bring new challenges. I'd like to imagine this would be competition on price and service. Sadly, that won't be the case. They can expect everything from spurious CALEA requirements to rooftop lease shenanigans to unwarranted radio interference claims. Eventually, though, this mesh model will take over the world, from both a network perspective and a commercial one. We just have to squelch our authoritarian impulse. Of course, technical challenges may still exist, but those certainly don't discredit the model.
- jeffreyrogers 8y ago
  I don't think mesh makes sense from an economics perspective. The number of edges scales O(n^2), where n is the number of people in the network. That's not very efficient. Even if you think wireless internet is feasible in dense urban areas (I think this is still an open question) it seems like a star topology connected by wireless backhaul to fiber makes more sense. This scales O(n).
  - teraflop 8y ago
    > The number of edges scales O(n^2), where n is the number of people in the network. I don't see how this is true for any reasonable definition of "edge". It's certainly not the case that every node directly communicates with every other node; if they did, you wouldn't need a mesh.
    - jeffreyrogers 8y ago
      A mesh network is defined as a network where the nodes connect to as many other nodes as possible. In the limit this is a complete graph with O(n^2) edges. It's true that in practice you won't have a complete graph but it's still less efficient than a more hierarchical, tree-like network structure.
      - mpfundstein 8y ago
deleted 8y ago
[deleted]
peterwwillis 8y ago
Mesh networks are too complicated to be reliable over a long term. They are useful in disaster situations or places with no infrastructure. Project Byzantium (http://project-byzantium.org/faqs/ http://project-byzantium.org/faqs/) is a great way to quickly stand up a mesh network as a temporary thing.
- jimktrains2 8y ago
  > Mesh networks are too complicated to be reliable over a long term I mean, this isn't true. I'm not even sure to begin. Perhaps specific points instead of nebulous nonsense would help us better understand you.
  - peterwwillis 8y ago
    Sure. The protocols, the routing, the interference, backend and gear maintenance, regulatory requirements, expense and complexity of multiple WAN links, and ad hoc setup of said links in an uneven physical environment. To say nothing of managing uneven customer load. Running fiber is expensive but less complicated and less error prone. And unless I missed something, there aren't any initiatives by big companies (that obviously don't want extra expense and headache) to go mesh. Big ISPs use their customer nodes as wifi hotspots but that is obviously not mesh.
    - woah 8y ago
      What are you talking about? Most wireless ISPs running OSPF could be classified as “meshnets”, plus the whole internet backbone is routed by BGP which works in the same way.
    - jimktrains2 8y ago
      > Sure. The protocols, the routing, the interference, backend and gear maintenance, regulatory requirements, expense and complexity of multiple WAN links, and ad hoc setup of said links in an uneven physical environment. To say nothing of managing uneven customer load. I think you have an overly narrow view of what meshnets can be.
tlrobinson 8y ago
It looks like they use a lot of Ubiquiti gear, which I'm a fan of (at least as a "prosumer", haven't used it professionally)
inetsee 8y ago
Can anyone comment on the utility of mesh networks in rural areas, where the big ISPs say "Sorry, not interested", and the distances between potential nodes is high? I have read articles in the past about hardware hackers setting up systems to broadcast WiFi signals for exceptional distances. Could a system of mesh networks provide rural America (or other countries) with low cost / high speed internet service?
- Rjevski 8y ago
  Point to point wireless links can definitely be used as long as you’ve got line of sight between the endpoints.
- gl00pp 8y ago
  I have set up wireless bridges that could supposedly under optimum conditions, connect to one another over 11 miles. So with line of sight I know you could 'beam internet' from either a plain old ISP connection OR from an internet exchange. This was consumer grade equipment too, I am sure there is much much stronger out there. As far as the potential of mesh in rural areas, I think one of those two options should be workable. Of course there are costs and geography that may hinder.
- johnnyforeigner 8y ago
  In Catalunya we’ve had such a network for many years and it is specifically used for providing connectivity to rural parts of the country. It’s called Guifi.net and has over 30,000 nodes. I was surprised it wasn’t mentioned in the article actually.
  - deleted 8y ago
    [deleted]
  - g8oz 8y ago
    Guifi.net is actually mentioned halfway through the article.
  - Paul-ish 8y ago
    It was mentioned. > While there are mesh networks dotting the U.S., she says the best working example of what mesh technology can do is in Spain. Guifi.net has more than 34,000 nodes covering an area of roughly 50,000 square kilometres across the Catalonia region.
walrus01 8y ago
Actual ISP network engineer here: I'm sure these people are having fun and all, but it's insignificant in the larger scheme of things. (correction to original post: I did not at first see an ASN for them, or announcements of their own IP space. They have an ASN and announce two /24s, and peer at one physical location in metro NYC). Real five nines network infrastructure takes significant capital investment. This part is so much unadulerated bullshit: ""The internet doesn't really cost you anything, it's just the connection [that has a fee]. So however you can get plugged in — then you're on the internet. Nobody owns the internet, there's no one to pay."" Actual, reliable internet costs real money, both in the salaries of people to engineer and architect it, salaries for 24x7x365 clued-in NOC staff, equipment, salaries and expenses for field technicians to build it. And that's before you get into things like establishing colo at major IX points, serious core routers that cost $15k each (do you really want to deploy something in the year 2018 to take several full BGP tables that doesn't have a 4 million FIB capacity?), etc. The Internet is a significant construction project at OSI layer 1, whether you're putting PTP radios on rooftops, running aerial fiber, or underground fiber. Otherwise you're just piggybacking on something that another, larger entity has already built. I give them an A for enthusiasm and effort. It just needs to be channeled the right way so that they can figure out what it actually costs to run a reliable ISP. I'm all in favor of new startup ISPs.
- dboreham 8y ago
  Agree the article is BS. Mesh networking is incompatible with QoS. So they can surely deliver crappy Internet cheaply. More than that? Nope. That'd need serious infrastructure such as licensed P2P, 60GHz short haul links, leased fiber. $$$$$ Oh, and they'd need peering: more $$$$$
  - nycmesh 8y ago
    Did you read any of our comments?
- deleted 8y ago
  [deleted]
- rdiddly 8y ago
badrabbit 8y ago
This can work for small scale networks but it needs to be a paid service model to scale it(at least for usable performance). A spine/leaf arch would be great for this imo. The leaves can be local cells of wifi networks. The spine can be operated by payment collecting individuals that maintain a high speed(10g+) and low latency uplink to the internet and network gear that can handle the traffic from the leaves. In spine leaf, leaves connect to multiple spine nodes as well (don't have to worry about spine operator reliability too much,could potentially load balance over spine) That being said,I hope any modern wifi network assumes wifi encryption is not reliable and instead implements a layer2 or 3 tunnel (macsec/.11ae and wireguard respectively)
chiefalchemist 8y ago
This is a great ideal. Sticking it to The Man is never a bad thing :) The question seems to be one of level of service and reliability. Collectives (if you will) can certainly work. But at this scale? And necessity? The ultimate benefit, I would think, would be to cause downward price pressure on the establishment ISPs. But wasn't that also Google Fiber's quest? Which we know didn't last long. Profits or not, there is still a certain level of cost to maintain sustainability of the model. I hope I don't sound cynical. I honestly just curious and will to share my thoughts out loud.
grahamburger 8y ago
Shameless plug: I run a website that helps people do this kind of thing: https://startyourownisp.com https://startyourownisp.com (although it's free and ad free, so not much of a plug really.) Also have a matrix chatroom on this topic: https://riot.im/app/#/room/#startyourownisp:matrix.org https://riot.im/app/#/room/#startyourownisp:matrix.org
alexandercrohde 8y ago
I really don't understand the HN reaction here. Why all the presumed negativity? This is tech, the best encyclopedia in the world is built on volunteer work and donations. A genius builds the de-facto version control system on his own in 3 months for fun. 2 guys in a garage build revolutionize search. Certainly there are logistical differences between these things, but I don't understand why one would assert with confidence that such a project (that already seems to be working) couldn't work.
- GCU-Empiricist 8y ago
  Because this requires brick and mortar industrial hardware to be logistically salable, lacking that it becomes vaporware. I think an unconscious bias may be that a bad but free connection that can negatively impact the perception of web solutions from casual users, which in the end is the revenue base for everyone here involved in B2C business models.
  - bigiain 8y ago
    I wonder? Unreliable but redundant commodity grade hardware pretty much allowed "cloud computing" to eat 99.9% of "big iron" sales. I strongly suspect what Google does internally with software defined networking looks a lot closer to what these people are doing than the traditional "buy a small number of powerful-but-expensive entrenched vendor network hardware" approach. This mesh is kinda the network equivalent of autoscaling or autorepairing cloud computing strategies that are so common these days. Once this grows to 2 or 3 IXP connections it won't surprise me much to find they stomp the traditional ISP model reliability-wise, and if they can build that "virtually", where they don't need much more than a bunch of somewhat transient mesh participants with a few medium reliability "super nodes" which are mostly completely remotely configurable...
- bigiain 8y ago
  "It is difficult to get a man to understand something, when his salary depends upon his not understanding it!" -- Upton Sinclair I wonder how much it's because so many Silicon Valley salaries are dependant on people assuming that replicating what they do is way harder than it really is? What happened to Britannica or BitKeeper or AltaVista might just happen to the ISP industry - or any other Start-up-de-jour...
hop 8y ago
Nice. Our 200+ unit condo building in Portland just got gigabit internet via line of sight radio on the roof from another building with fiber. This is a trend that will definitely take off and removes the huge barriers of entry that laying cable requires. 1ms latency too.
- menacers 8y ago
  I understand you must be in USA, but I'm from Canada, and this trend is certainly very popular in most parts of Ontario. In the USA it's also catching on, have a look at Webpass. The Canadian equivalent would be Cloudwifi who offers gigabit service most condos and apartments built after 2005. http://cloudwifi.ca http://cloudwifi.ca Oh, on average these next generation networks have much lower latency than traditional cable and DSL lines.
- tvchurch 8y ago
  I've got to know more about this. Who set it up? Local start up? Bored engineers?
  - hop 8y ago
    Wave G. I think its a spin off a small isp.
freeflight 8y ago
Förderverein Freie Netzwerke e. V. has been doing this, rather successfully, in Germany since 2003 [0] Similar projects exist in Austria (FunkFeuer) and Switzerland (Openwireless). [0] https://en.wikipedia.org/wiki/Freifunk https://en.wikipedia.org/wiki/Freifunk
- nisa 8y ago
  There are several distributions for mesh-networks: - Gluon is the most popular: https://gluon.readthedocs.io https://gluon.readthedocs.io - Libremesh is more flexible: https://libremesh.org/ https://libremesh.org/ Like NYC Mesh both are based on OpenWRT.org a wonderful distribtion for wireless devices from 4mb flash / 32mb memory up to 64bit x86 support.
_bxg1 8y ago
[Ron Paul voice] It's happeniiiiinnnnng
blhack 8y ago
"Anti-authority" seems like a pretty good marketing campaign that WISPs have figured out. I mean: good for them.
throwaway932 8y ago
'Anti-authority', even with scare quotes around it. God forbids somebody uses the word "anarchist"...
- jessaustin 8y ago
  Synonyms are fine. Those with knowledge know what's going on. Those with prejudice can remain in the dark for a time.

417 points by standeven 8y ago

nickthemagicman 8y ago

Hate to sound stupid but how does DNS work on a mesh?

nycmesh 8y ago
We originally used the qMp/OpenWRT package that included mdns, which is a version of zeroconf for the internal network. It isn't scaling so we now use our own DNS server to resolve internal addresses on our 10/ network.
- nickthemagicman 8y ago
  Mesh networks are going to be incredible when they're widespread! It's truly superior to centralized control by ISPs/Governments and the world will be grateful for this after this whole net neutrality debacle recently. You guys are brilliant and doing an amazing job and I'm grateful people like you guys exist! Thank you for the response!

tlrobinson 8y ago

There's an impressive amount of negativity in these comments. A community-run mesh network? HN should be all over it, but it appears to have turned into a bunch of network engineers telling us why they are very smart and it won't work, or is "insignificant". More power to you, NYC Mesh.

pavs 8y ago
It's OK to be optimistic about something different. It's also ok to discuss the reality, that's not necessarily negativity towards a project. Are we only supposed to discuss one side of the argument and ignore reality just because it makes us feel nice inside? There are bazillion reasons that this is nothing more than a cute side project that will fall on its face when its time to scale in any meaningful way. Unless they have discovered a new technology they are not taking on ISPs by any meaningful way. Especially in NYC. There is a difference between, just "connecting to the internet" and having a "reliable connectivity to the internet", the difference is a massive investment in infrastructure. Wifi p2p mesh networks don't work on magic. They are very unreliable and hard/impossible to scale to 100,000 users, let alone millions of users. In some places, wifi p2p connection hardware actually costs more money than a traditional wired network. Mesh networks have been around for a while, technical and reliability aspect of this is well understood within the networking industry. There is a reason why it's not widely used, especially in cities. In remote areas, this kind of network makes sense, if you don't have a choice or an alternative. Don't brush of educated criticisms from people within the industry. I own an ISP for 5+ years (mostly wired). I am not in US and I am not at all worried if this kind of network comes in my country/area. We are not worried because we have tried and deployed this kind of network in cities and they are extremely unreliable and costs real money. I also don't understand the "Anti-authority" angle of the argument, unless I live in a different universe, they still have to get license/permit from the local government to do this business (non-profit or not), they still have to follow the same rules enforced by local and federal government when it comes to log/customer information sharing to LEO. So not sure how they are Anti-authority? Unless they mean anti-establishment?
- hux_ 8y ago
  Once upon a time the network, storage and data center guys thought Facebook/YouTubes one to many broadcasting of content would never scale too. Now they fall over themselves to work there.
  - js1k374 8y ago
    I think pavs is referring to physical limitations of the technology that prevent it from scaling.
  - pavs 8y ago
    They scaled by making a huge investment in their network. They didn't make a fundamental new technical discovery that didn't exist before. They had money, they threw money at the problem, as expected it worked. The problem was an investment (which takes time), not technical. https://peering.google.com/#/ https://peering.google.com/#/ https://code.facebook.com/posts/565767133547005/steering-oceans-of-content-to-the-world/ https://code.facebook.com/posts/565767133547005/steering-oce... Any ISP (if big enough), can host their own Google, Akamai, Facebook, Netflix nodes on their premise.
8bitsrule 8y ago
It has been a few years since I learned about Guifi.net (non-English site; English portal:guifi.net/en/node/38392) in Catalonia/Valencia - a 2004 startup. WP says that (as of a year ago) it has 'over 33,000 active nodes and about 46,000 km of wireless links.' It's described (in English) in this 2016 paper http://dsg.ac.upc.edu/sites/default/files/dsg/acm-sigcomm-gaia-guifi-econ.pdf http://dsg.ac.upc.edu/sites/default/files/dsg/acm-sigcomm-ga... as 'known to be the largest community network in the world ... and is still growing while successfully evading the collapse of the tragedy of the commons. Undoubtedly, this is due to the governance model that the foundation has developed....' Since it's been around for a while, and still viable, that says something about the idea worth attending to. More info: http://www.dtic.upf.edu/~jbarcelo/tmp/guifi.net.pdf http://www.dtic.upf.edu/~jbarcelo/tmp/guifi.net.pdf
rayiner 8y ago
The weirdest part of HN for me is the optimism. Engineers are pessimistic because engineering is hard and physics is unforgiving. Engineers like to point out things won't work because it probably won't and engineers like to be right. Mesh networking is incredibly hard to get right to the point where I wouldn't even say anyone really has ever gotten it right at scale.
BLKNSLVR 8y ago
What's so ironically funny, is that this article is also on HN's front page: https://www.newyorker.com/culture/cultural-comment/brilliant-1895-novel-on-the-emptiness-of-literary-fame https://www.newyorker.com/culture/cultural-comment/brilliant... HN Comments: https://news.ycombinator.com/item?id=16980503 https://news.ycombinator.com/item?id=16980503 The jealousy amongst peers for non-recognition. "More power to you, NYC Mesh." is the correct answer. Thumbs up tlrobinson.
John_KZ 8y ago
It would get a lot more positive comments if it was called what it is: A low-bandwidth mesh network. It's nice, but not going to replace ISPs anytime soon. It consumes more energy, it's slower, less reliable and higher latency than regular wired/fiber networks. These things don't scale well and all those radios can be universally jammed by a single drone over the city. Not a great backup plan if you're anti-establishment. I'd love to partake in such a project but keeping your expectations grounded is very important.

EGreg 8y ago

I love seeing the growth of mesh network. “We don’t have any use cases that show how nice it can be if you have enough users.” They are going to need software. The supernodes are still a symptom of the thinking that you need the signal to go to another stage in order to communicate locally. I think there should be more software that works on a local network, like things used to work before broadband. This is what I’m talking about: https://www.youtube.com/watch?v=pZ1O_gmPneI https://www.youtube.com/watch?v=pZ1O_gmPneI

Zigurd 8y ago

When you compare comments here a few themes emerge: 1. The internet has been captured by the telcos, who operate the internet like telcos have operated the circuit switched voice network. 2. We are all overlooking the fact that smart radios make spectrum ownership obsolete. That means that the potential of mesh wireless connectivity is orders of magnitude larger than can be realized in current unlicensed spectrum. 3. The assumption that anyone is owed a business model in internet service is going unquestioned. Of course it costs a lot of money to run the internet the way the Bell System ran phones. But there are plausible alternatives, not all of which are compatible with investor-owned near-monopoly "markets." This is an economic tragedy and a case of local optimization holding a global optimum hostage. ISPs are a sweet business, in a way that costs the rest of the economy dearly. Kill the incumbents and reap an across the board boost in GDP.

0xBA5ED 8y ago

Well, it seems this idea has touched a few nerves. In this case, it may indicate you're on the right track. Keep it up!

10065 8y ago

NYC public housing has had rock bottom cost, high speed Internet for years. Have to now do something with it.

MessageLife1122 8y ago

Just by reading the comments, I d really don't need to read the article as a whole. haha

mapmap 8y ago

I'm curious about the technology they are using. Is it an actual mesh network or point-to-point wireless?

tlrobinson 8y ago
Here's a map of their network: https://nycmesh.net/map/ https://nycmesh.net/map/ (uncheck everything except "Link" to see the network)
- ybloviator 8y ago
  "The network connects directly to the internet backbone, so we do not rely on an ISP." That makes no sense. I doubt they're a tier-1 backbone themselves, they have to be purchasing transit from an ISP. It feels like lots of the docs are written by marketing-type folks. Also there's a weird diagram that insinuates NAT happens before bgp peering with the not-ISPs they buy internet from. Weird shit.
- dawiddutoit 8y ago
  Down here in South Africa we have a similar network. We have had quite shite connections down here for so long that a mesh network was one of the few ways to get a good connection. Link for our map of Cape Town. https://wind.ctwug.za.net/index.php/nodes https://wind.ctwug.za.net/index.php/nodes
Zekio 8y ago
Looks like point to point in the video
crishoj 8y ago
Seems to be an actual mesh network, based on the BMX6 mesh protocol [0]. [0] http://nycmeshnet.github.io/faq/ http://nycmeshnet.github.io/faq/
nycmesh 8y ago
We are a hybrid of mesh, p2p and point to multi-point (sector antennas). Whatever is most appropriate to give the best connection for reasonable price.

stcredzero 8y ago

Is there an inexpensive point to point technology that isn't hampered by rain?

notyourday 8y ago
Yes. Whatever TowerStream uses. But it is definitely line of sight and you have to be on Empire State or Trump Tower or a few other buildings to have any reasonable visibility
jeffreyrogers 8y ago
2.4 and 5 GHz aren't too affected by rain fade. And they're cheap.
- mholt 8y ago
  I thought wifi couldn't go through water? (Or is rain just so sparse/diluted that it doesn't really affect it?)
  - jeffreyrogers 8y ago
    I'm not sure about water, but rain doesn't have much effect at those frequencies. At higher frequencies (e.g. 24GHz) rain can make the link useless.

notyourday 8y ago

I'm in NYC. I know a few people who played with it. It barely works. I'm all for this kind of tech solutions, but for crying out loud pretending that this is a replacement for ISPs in NYC really discredits techies.

isostatic 8y ago
Why doesn't a startup run their own fibre? Are building owners that protective? Actually scratch that, I tried getting a 1 gig cross manhattan link. Cost more than 1g to London.
- cimmanom 8y ago
  Building owners? Before you get anywhere near the buildings, you have to dig up the streets to run cabling around the city. Any idea how much that costs?
  - ybloviator 8y ago
    Pilot is doing this cheaper than others, look into how they get their fiber in the ground...
  - notyourday 8y ago
    That's what you outsource to the latest incarnation of Metromedia Fiber Network - Zayo.
  - Symbiote 8y ago
    Speaking of London, talking to building owners is exactly what Hyperoptic did to build out a fibre network in London. Unfortunately, the building owner where I used to live wasn't interested -- it turned out most of the building was empty, people keeping apartments for occasional visits or as an investment :-( https://www.hyperoptic.com/how-hyperoptic-works/ https://www.hyperoptic.com/how-hyperoptic-works/ or https://www.hyperoptic.com/map/ https://www.hyperoptic.com/map/
  - isostatic 8y ago

tlrobinson 8y ago

This sounds neat. Is anyone working on something like it in the SF bay area? I know there's Webpass and Monkeybrains, which are great, but I'd love to be part of a community-owned, actual mesh network.

dannyobrien 8y ago
There's the People's Open Network: https://peoplesopen.net/ https://peoplesopen.net/ co-ordinated by the SudoRoom folks and the wider community.
ziton 8y ago
Not "community owned" per say, but Facebook has deployed a Mesh network in DT San Jose called Terragraph.

jeffreyrogers 8y ago

This stuff is cool and maybe useful in far-fetched scenarios, but it's not going to replace ISPs anytime soon. Mesh networks don't make economic sense in most circumstances. Also, I'm guessing that users share the same IP address, which can cause problems. Plus, if one user is doing something illegal and causes the supernode to get shutdown then no one on the mesh network will have access to the internet. Addition: quotes like "The internet doesn't really cost you anything, it's just the connection [that has a fee]... Nobody owns the internet, there's no one to pay." Are shockingly naive. Who does he expects pays for the fiber that runs across the country and along the ocean floor? The marginal cost of a new user on the internet is effectively zero, but that doesn't say anything important. The infrastructure that makes the internet work is enormously expensive.

bogomipz 8y ago
>"This stuff is cool and maybe useful in far-fetched scenarios, but it's not going to replace ISPs anytime soon. Mesh networks don't make economic sense in most circumstances" Far fetched you mean like getting instantly connected to the internet for a fraction of the cost and with better bandwidth in major city? You realize that mesh is just for the access network correct? There's still fiber between them and their transit providers. Mesh makes a lot economic sense compared to the costs of trenching and laying fiber miles.
nycmesh 8y ago
We are replacing ISPs in NYC for our members. This is now, not the future. The main thing holding us back is scaling up the number of installs. All networks, including us, add to this expensive infrastructure. The difference with our infrastructure is it is owned and shared by members.
- Paul-ish 8y ago
  Very cool. Is this a customer coop or a non-profit (or something else)?
  - nycmesh 8y ago

lbotos 8y ago

Who's responsible when someone torrents copyrighted material through that Internet exchange point? "NYC Mesh will comply with all federal laws in the countries it operates, however, as policy, no data is collected and therefore no data exists to provide requestors." This is cute, but if you aren't hunting down people abusing the mesh, the whole connection to the internet will go away for a ToS violation.

nickthemagicman 8y ago
The beauty of it is as it becomes popular people will start torrenting WITHIN the mesh network.
- kalleboo 8y ago
  Brings back memories of using DC++ hubs on the university network
ybloviator 8y ago
Didn't one of the major ISPs get their ass handed to them in court for trying to say that they were somehow exempt from keeping exactly this sort of information? I mean, you can SAY it, but it would be sort of insane to make a statement like that without a lawyer signing off on it. The law doesn't have a "this is too hard for us" or "we didn't build our network for this" exemption, does it?
jlgaddis 8y ago
I've worked for an ISP for about six and a half years. In that time, I have received I-don't-know-how-many e-mails about people torrenting and so on. I've had one conversation with a Special Agent from DHS investigating credit card fraud. Over this same period, I have given out a customer's identifying information zero times (if more than 24 hours have passed, there is a very small chance that I am able to identify the customer). In every instance, it was "sorry, I am not able to identify the customer". Zero is also the number of times I have heard anything from any of my upstreams regarding any ToS violations or similar.

jessaustin 8y ago

The group also knows there will be growing pains as they challenge the status quo and that it's only a matter of time before the big ISPs take notice, which could bring new challenges. I'd like to imagine this would be competition on price and service. Sadly, that won't be the case. They can expect everything from spurious CALEA requirements to rooftop lease shenanigans to unwarranted radio interference claims. Eventually, though, this mesh model will take over the world, from both a network perspective and a commercial one. We just have to squelch our authoritarian impulse. Of course, technical challenges may still exist, but those certainly don't discredit the model.

jeffreyrogers 8y ago
I don't think mesh makes sense from an economics perspective. The number of edges scales O(n^2), where n is the number of people in the network. That's not very efficient. Even if you think wireless internet is feasible in dense urban areas (I think this is still an open question) it seems like a star topology connected by wireless backhaul to fiber makes more sense. This scales O(n).
- teraflop 8y ago
  > The number of edges scales O(n^2), where n is the number of people in the network. I don't see how this is true for any reasonable definition of "edge". It's certainly not the case that every node directly communicates with every other node; if they did, you wouldn't need a mesh.
  - jeffreyrogers 8y ago
    A mesh network is defined as a network where the nodes connect to as many other nodes as possible. In the limit this is a complete graph with O(n^2) edges. It's true that in practice you won't have a complete graph but it's still less efficient than a more hierarchical, tree-like network structure.
    - mpfundstein 8y ago

deleted 8y ago

[deleted]

peterwwillis 8y ago

Mesh networks are too complicated to be reliable over a long term. They are useful in disaster situations or places with no infrastructure. Project Byzantium (http://project-byzantium.org/faqs/ http://project-byzantium.org/faqs/) is a great way to quickly stand up a mesh network as a temporary thing.

jimktrains2 8y ago
> Mesh networks are too complicated to be reliable over a long term I mean, this isn't true. I'm not even sure to begin. Perhaps specific points instead of nebulous nonsense would help us better understand you.
- peterwwillis 8y ago
  Sure. The protocols, the routing, the interference, backend and gear maintenance, regulatory requirements, expense and complexity of multiple WAN links, and ad hoc setup of said links in an uneven physical environment. To say nothing of managing uneven customer load. Running fiber is expensive but less complicated and less error prone. And unless I missed something, there aren't any initiatives by big companies (that obviously don't want extra expense and headache) to go mesh. Big ISPs use their customer nodes as wifi hotspots but that is obviously not mesh.
  - woah 8y ago
    What are you talking about? Most wireless ISPs running OSPF could be classified as “meshnets”, plus the whole internet backbone is routed by BGP which works in the same way.
  - jimktrains2 8y ago
    > Sure. The protocols, the routing, the interference, backend and gear maintenance, regulatory requirements, expense and complexity of multiple WAN links, and ad hoc setup of said links in an uneven physical environment. To say nothing of managing uneven customer load. I think you have an overly narrow view of what meshnets can be.

tlrobinson 8y ago

It looks like they use a lot of Ubiquiti gear, which I'm a fan of (at least as a "prosumer", haven't used it professionally)

inetsee 8y ago

Can anyone comment on the utility of mesh networks in rural areas, where the big ISPs say "Sorry, not interested", and the distances between potential nodes is high? I have read articles in the past about hardware hackers setting up systems to broadcast WiFi signals for exceptional distances. Could a system of mesh networks provide rural America (or other countries) with low cost / high speed internet service?

Rjevski 8y ago
Point to point wireless links can definitely be used as long as you’ve got line of sight between the endpoints.
gl00pp 8y ago
I have set up wireless bridges that could supposedly under optimum conditions, connect to one another over 11 miles. So with line of sight I know you could 'beam internet' from either a plain old ISP connection OR from an internet exchange. This was consumer grade equipment too, I am sure there is much much stronger out there. As far as the potential of mesh in rural areas, I think one of those two options should be workable. Of course there are costs and geography that may hinder.
johnnyforeigner 8y ago
In Catalunya we’ve had such a network for many years and it is specifically used for providing connectivity to rural parts of the country. It’s called Guifi.net and has over 30,000 nodes. I was surprised it wasn’t mentioned in the article actually.
- deleted 8y ago
  [deleted]
- g8oz 8y ago
  Guifi.net is actually mentioned halfway through the article.
- Paul-ish 8y ago
  It was mentioned. > While there are mesh networks dotting the U.S., she says the best working example of what mesh technology can do is in Spain. Guifi.net has more than 34,000 nodes covering an area of roughly 50,000 square kilometres across the Catalonia region.

walrus01 8y ago

Actual ISP network engineer here: I'm sure these people are having fun and all, but it's insignificant in the larger scheme of things. (correction to original post: I did not at first see an ASN for them, or announcements of their own IP space. They have an ASN and announce two /24s, and peer at one physical location in metro NYC). Real five nines network infrastructure takes significant capital investment. This part is so much unadulerated bullshit: ""The internet doesn't really cost you anything, it's just the connection [that has a fee]. So however you can get plugged in — then you're on the internet. Nobody owns the internet, there's no one to pay."" Actual, reliable internet costs real money, both in the salaries of people to engineer and architect it, salaries for 24x7x365 clued-in NOC staff, equipment, salaries and expenses for field technicians to build it. And that's before you get into things like establishing colo at major IX points, serious core routers that cost $15k each (do you really want to deploy something in the year 2018 to take several full BGP tables that doesn't have a 4 million FIB capacity?), etc. The Internet is a significant construction project at OSI layer 1, whether you're putting PTP radios on rooftops, running aerial fiber, or underground fiber. Otherwise you're just piggybacking on something that another, larger entity has already built. I give them an A for enthusiasm and effort. It just needs to be channeled the right way so that they can figure out what it actually costs to run a reliable ISP. I'm all in favor of new startup ISPs.

dboreham 8y ago
Agree the article is BS. Mesh networking is incompatible with QoS. So they can surely deliver crappy Internet cheaply. More than that? Nope. That'd need serious infrastructure such as licensed P2P, 60GHz short haul links, leased fiber. $$$$$ Oh, and they'd need peering: more $$$$$
- nycmesh 8y ago
  Did you read any of our comments?
deleted 8y ago
[deleted]
rdiddly 8y ago

badrabbit 8y ago

This can work for small scale networks but it needs to be a paid service model to scale it(at least for usable performance). A spine/leaf arch would be great for this imo. The leaves can be local cells of wifi networks. The spine can be operated by payment collecting individuals that maintain a high speed(10g+) and low latency uplink to the internet and network gear that can handle the traffic from the leaves. In spine leaf, leaves connect to multiple spine nodes as well (don't have to worry about spine operator reliability too much,could potentially load balance over spine) That being said,I hope any modern wifi network assumes wifi encryption is not reliable and instead implements a layer2 or 3 tunnel (macsec/.11ae and wireguard respectively)

chiefalchemist 8y ago

This is a great ideal. Sticking it to The Man is never a bad thing :) The question seems to be one of level of service and reliability. Collectives (if you will) can certainly work. But at this scale? And necessity? The ultimate benefit, I would think, would be to cause downward price pressure on the establishment ISPs. But wasn't that also Google Fiber's quest? Which we know didn't last long. Profits or not, there is still a certain level of cost to maintain sustainability of the model. I hope I don't sound cynical. I honestly just curious and will to share my thoughts out loud.

grahamburger 8y ago

Shameless plug: I run a website that helps people do this kind of thing: https://startyourownisp.com https://startyourownisp.com (although it's free and ad free, so not much of a plug really.) Also have a matrix chatroom on this topic: https://riot.im/app/#/room/#startyourownisp:matrix.org https://riot.im/app/#/room/#startyourownisp:matrix.org

alexandercrohde 8y ago

I really don't understand the HN reaction here. Why all the presumed negativity? This is tech, the best encyclopedia in the world is built on volunteer work and donations. A genius builds the de-facto version control system on his own in 3 months for fun. 2 guys in a garage build revolutionize search. Certainly there are logistical differences between these things, but I don't understand why one would assert with confidence that such a project (that already seems to be working) couldn't work.

GCU-Empiricist 8y ago
Because this requires brick and mortar industrial hardware to be logistically salable, lacking that it becomes vaporware. I think an unconscious bias may be that a bad but free connection that can negatively impact the perception of web solutions from casual users, which in the end is the revenue base for everyone here involved in B2C business models.
- bigiain 8y ago
  I wonder? Unreliable but redundant commodity grade hardware pretty much allowed "cloud computing" to eat 99.9% of "big iron" sales. I strongly suspect what Google does internally with software defined networking looks a lot closer to what these people are doing than the traditional "buy a small number of powerful-but-expensive entrenched vendor network hardware" approach. This mesh is kinda the network equivalent of autoscaling or autorepairing cloud computing strategies that are so common these days. Once this grows to 2 or 3 IXP connections it won't surprise me much to find they stomp the traditional ISP model reliability-wise, and if they can build that "virtually", where they don't need much more than a bunch of somewhat transient mesh participants with a few medium reliability "super nodes" which are mostly completely remotely configurable...
bigiain 8y ago
"It is difficult to get a man to understand something, when his salary depends upon his not understanding it!" -- Upton Sinclair I wonder how much it's because so many Silicon Valley salaries are dependant on people assuming that replicating what they do is way harder than it really is? What happened to Britannica or BitKeeper or AltaVista might just happen to the ISP industry - or any other Start-up-de-jour...

hop 8y ago

Nice. Our 200+ unit condo building in Portland just got gigabit internet via line of sight radio on the roof from another building with fiber. This is a trend that will definitely take off and removes the huge barriers of entry that laying cable requires. 1ms latency too.

menacers 8y ago
I understand you must be in USA, but I'm from Canada, and this trend is certainly very popular in most parts of Ontario. In the USA it's also catching on, have a look at Webpass. The Canadian equivalent would be Cloudwifi who offers gigabit service most condos and apartments built after 2005. http://cloudwifi.ca http://cloudwifi.ca Oh, on average these next generation networks have much lower latency than traditional cable and DSL lines.
tvchurch 8y ago
I've got to know more about this. Who set it up? Local start up? Bored engineers?
- hop 8y ago
  Wave G. I think its a spin off a small isp.

freeflight 8y ago

Förderverein Freie Netzwerke e. V. has been doing this, rather successfully, in Germany since 2003 [0] Similar projects exist in Austria (FunkFeuer) and Switzerland (Openwireless). [0] https://en.wikipedia.org/wiki/Freifunk https://en.wikipedia.org/wiki/Freifunk

nisa 8y ago
There are several distributions for mesh-networks: - Gluon is the most popular: https://gluon.readthedocs.io https://gluon.readthedocs.io - Libremesh is more flexible: https://libremesh.org/ https://libremesh.org/ Like NYC Mesh both are based on OpenWRT.org a wonderful distribtion for wireless devices from 4mb flash / 32mb memory up to 64bit x86 support.

_bxg1 8y ago

[Ron Paul voice] It's happeniiiiinnnnng

blhack 8y ago

"Anti-authority" seems like a pretty good marketing campaign that WISPs have figured out. I mean: good for them.

throwaway932 8y ago

'Anti-authority', even with scare quotes around it. God forbids somebody uses the word "anarchist"...

jessaustin 8y ago
Synonyms are fine. Those with knowledge know what's going on. Those with prejudice can remain in the dark for a time.