8 ms·

This article is great overall, but there is some stuff you should do differently, if you're wanting to follow in the author's footsteps: 1. Just use UPC connec

This article is great overall, but there is some stuff you should do differently, if you're wanting to follow in the author's footsteps: 1. Just use UPC connectors everywhere unless you have really good reasons not to. 2. Use single mode fiber (SMF) everywhere, including internal wiring. SMF is infinitely scalable for future needs, multimode (MMF) is not. These days the cost differences for cabling and optics is negligible. In fact, SMF cabling is often cheaper than MMF. And no, you generally won’t have to worry about attenuating signal. Also, in my extensive experience, FS.com optics modules are typically better manufactured and more reliable than OEM modules. I resisted the "cheap chinese" modules for a long, long time, but then discovered that most of my uplink carriers used them. It's been years now. I've had many Cisco OEM, Dell OEM, and a couple from another 3rd party manufacturer go bad, but I have yet for one FS.com SFP to go bad. I've got several hundred deployed in a wide variety of environments, from great (cool, clean areas) to bad (hot, dirty areas).

sponaugle 4y ago
Fiber is, as runjake mentioned, very very cheap. The raw cost is less than copper, and that is part of why I ran so much of it in my build. (I am not the OP). I did however include some multimode in that equation only because some AV equipment uses is, and not all of them use replaceable optics modules. I would strongly agree that if you are running fiber, run single mode everywhere. It is amazing how far singlemode fiber can go compared to copper if you are willing to spend money on the endpoints.
- yakak 4y ago
  What kind of equipment outlay was necessary if you don't mind my asking? I.e. the splicer the OP bought seems a bit expensive and impractical to own compared to an ethernet crimper, etc.
  - conk 4y ago
    I just ordered the length I needed from fs.com. Granted I only have a single run between 2 switches home<->garage. I ran the fiber in a PVC conduit. Put a 4”x4” pvc junction box in the run with a couple feet excess coiled inside because my measurements were a little off. At the termination of the runs I have fiber keystone jacks and I can extend/replace the last bit if I need to move equipment.
sschueller 4y ago
I had a hard time finding UPC wall outlets since most that are available in the Swiss format are for FTTH use, so they all have APC LC connectors on them. The keystone versions are available as UPC but they lack any fiber routing in the back.
- runjake 4y ago
  This is what I suspected. Great write-up. Thanks for putting all that effort into it.
- jagrsw 4y ago
  I use the Feller one - https://www.feller.ch/de/Produktangebot/Kommunikations-und-Netzwerktechnik/FTTH-DSL-Telefon-Aschlussdosen/Highlight-Produkte https://www.feller.ch/de/Produktangebot/Kommunikations-und-N... - and it's indeed LC/APC. My setup is much simpler - I get 25Gbit/s from init7 to Mikrotik CCR2004 https://mikrotik.com/product/ccr2004_1g_12s_2xs https://mikrotik.com/product/ccr2004_1g_12s_2xs and then distribute it to 2x1Gbit access points, and use the other SFP28 port to power my PC (in another room) via the Intel E810 card. I use the Feller fiber extension cable (20m) and bought some cable puller to pull the fiber via the plastic piping, and then finished it with the Feller outlet. Looks rather clean :) Photo: https://pbs.twimg.com/media/FWhHwD0WAAIb3Dt?format=jpg&name=4096x4096 https://pbs.twimg.com/media/FWhHwD0WAAIb3Dt?format=jpg&name=... PS: ccr2004_1g_12s_2xs is barely coping with 25Gb/s routing. It can do it, but for multiple TCP sessions, a single one chokes the router at something like 20Gbps (down), 15 Gbps (up).
  - sschueller 4y ago
    Very nice, how much does one of those plates cost? I can't find any prices easily.
    - jagrsw 4y ago
      IIRC ~70CHF for the outlet (which only works with the Feller electrical wall outlets AFAIK) + 10CHF (RJ45 module) + 20 CHF (single-port LC/APC module). The cable extension is expensive, 150CHF or so for 20m, but you'd probably need it for this outlet because it's specially designed so it can be attached to the LC/APC module manually (no splicing). Something like Swisscom's ClickLC, just a different system.
jbverschoor 4y ago
What about bi-directional vs running two fibers? Will this increase latency, even if negligible?
- ilyt 4y ago
  It will increase cost and you need to remember which side uses which module. Bi-directional just uses different wavelength in each direction. Generally not worth it unless you have existent infrastructure and want some more density
- sponaugle 4y ago
  And given some of the inexpensive single mode fiber spools are two fibers, it is probably easier to just have two fibers instead of trying to do 2 wavelengths.
voidwtf 4y ago
Is there a best practice for which bidi direction to choose for which device type? i.e. I assume you don't want to just wing it for every link, so do people typically choose to use all A/B on the switch side, and all B/A on the desktop/server side? If I were walking into a business, not having setup their network, what direction would I expect to need on the desktop/server side?
- hackmiester 4y ago
  I can't speak for anyone else, but we wing it for every link. When we light up a new link, we grab a matched pair, randomly select one to put in the switch, and then put the other one downstream.
- coder543 4y ago
  Aren’t BiDi transceivers usually a good bit more expensive, especially compared to how low the material cost of fiber is? What’s your use case for BiDi? I don’t have a lot of first hand experience with this stuff, though. I can easily imagine BiDi being useful to retrofit/repurpose some existing fiber, since labor costs can be a lot.
  - fifteenforty 4y ago
    I can see it being useful for running ultra thin bend insensitive fibre around base boards; just like the FTTR thing Huawei is trying to sell. I wish the concept was more clearly legal in Australia. I’m pretty sure once you glue the fiber in place it becomes ‘permanent’ and thus has to be done by a licensed cabler with a fiber endorsement.
Hikikomori 4y ago
3rd party and maybe 1-2 of each type from the vendor (used to show their support that the 3rd party transceiver isn't the problem) is what we usually buy, basically an open secret. Though I've seen 97% off the price for vendor optics, still more expensive than 3rd party
dijit 4y ago
Had a really good experience with FS.com. To the point that we even printed out a picture of their warehouse from google maps and wrote “thank you” on it before stapling it to the wall. They were the best support we received.
- dmd 4y ago
  Which is why I'm so annoyed that my employer (which happens to be the largest employer in my entire state) mandates a vendor that costs 6x as much and whose products fail much more often.
  - kevin_nisbet 4y ago
    I'm really out of the game these days so don't know the state of different options, but to be fair, a decade ago I had a team member try and cheap out on modules and it caused a fairly decent mess. Lots of failures, but not just outright failures, but problems where the link would mostly work, but through a bunch of fiber taps and bypass modules produced a high error rates when installed. So I'm not saying it's right, but I can at least understand where a mandate like that might come from, if perhaps outdated compared to the current state of the industry.
  - Spooky23 4y ago
    The details matter. Depending on how you contract for support, etc. 6x on optics may be the logical choice as compared to being in some situation where the vendor won’t fix stuff until you verify that it’s broken with their OEM parts. If you’re the largest employer in the state, there may be some personal implications to a CIO telling a major OEM to f-off.
- secabeen 4y ago
  The new FS-Box is out, allowing reprogramming of most FS transceivers for a one-time $500 cost.
anonymousiam 4y ago
I agree there's a lot of useful information in the article. Two things: 1) I question his choice of OpnSense vs. OpenWRT. I've found OpenWRT to be less demanding on the hypervisor/host, and quite scalable. 2) Not about the article, but about your comment on Chinese modules. I've tried Chinese 10Gb SFP+ modules and I've returned all that I tried. They would overheat, and they would not operate up to the specified data rates. The name brands cost twice as much, but they work.
fire 4y ago
what's the big difference between single and multi mode fiber? Why is it scalable but multi-mode isn't?
- runjake 4y ago
  It comes down to multiple factors but this article covers them well. https://www.tlnetworx.com/blogs/news/single-mode-vs-multimode-fiber-whats-the-difference https://www.tlnetworx.com/blogs/news/single-mode-vs-multimod... And to be clear, there will be hacks that scale MMF a bit, but it’s not really “infinitely” scalable due to relatively poor attenuation and light mode. Keep in mind, there isn’t one MMF spec, there’s many: eg. OM1 through OM4
deleted 4y ago
[deleted]
stephen_g 4y ago
Good advice. I’ve had the same policy with regard to SMF vs MMF for quite a while now, when I saw each new speed dropping in distance with multimode. You just know SMF will do what you need. We’ve been using the FS modules too for years, they are very reliable. It does totally make a mockery of the vendors who claim they need to lock out other brand’s modules (from their 100% standardised port) for reliability etc.!
kotaKat 4y ago
> I resisted the "cheap chinese" modules for a long, long time, but then discovered that most of my uplink carriers used them. MSP ISP here, we just rock the FS optics combined with an FSbox to code them as required. ;) Just a warning for the weary: don't put a non-Fiberstore optic in their FS coding box -- you'll lock your account for a week. Ask our field team how they know this.

sponaugle 4y ago

Fiber is, as runjake mentioned, very very cheap. The raw cost is less than copper, and that is part of why I ran so much of it in my build. (I am not the OP). I did however include some multimode in that equation only because some AV equipment uses is, and not all of them use replaceable optics modules. I would strongly agree that if you are running fiber, run single mode everywhere. It is amazing how far singlemode fiber can go compared to copper if you are willing to spend money on the endpoints.

yakak 4y ago
What kind of equipment outlay was necessary if you don't mind my asking? I.e. the splicer the OP bought seems a bit expensive and impractical to own compared to an ethernet crimper, etc.
- conk 4y ago
  I just ordered the length I needed from fs.com. Granted I only have a single run between 2 switches home<->garage. I ran the fiber in a PVC conduit. Put a 4”x4” pvc junction box in the run with a couple feet excess coiled inside because my measurements were a little off. At the termination of the runs I have fiber keystone jacks and I can extend/replace the last bit if I need to move equipment.

sschueller 4y ago

I had a hard time finding UPC wall outlets since most that are available in the Swiss format are for FTTH use, so they all have APC LC connectors on them. The keystone versions are available as UPC but they lack any fiber routing in the back.

runjake 4y ago
This is what I suspected. Great write-up. Thanks for putting all that effort into it.
jagrsw 4y ago
I use the Feller one - https://www.feller.ch/de/Produktangebot/Kommunikations-und-Netzwerktechnik/FTTH-DSL-Telefon-Aschlussdosen/Highlight-Produkte https://www.feller.ch/de/Produktangebot/Kommunikations-und-N... - and it's indeed LC/APC. My setup is much simpler - I get 25Gbit/s from init7 to Mikrotik CCR2004 https://mikrotik.com/product/ccr2004_1g_12s_2xs https://mikrotik.com/product/ccr2004_1g_12s_2xs and then distribute it to 2x1Gbit access points, and use the other SFP28 port to power my PC (in another room) via the Intel E810 card. I use the Feller fiber extension cable (20m) and bought some cable puller to pull the fiber via the plastic piping, and then finished it with the Feller outlet. Looks rather clean :) Photo: https://pbs.twimg.com/media/FWhHwD0WAAIb3Dt?format=jpg&name=4096x4096 https://pbs.twimg.com/media/FWhHwD0WAAIb3Dt?format=jpg&name=... PS: ccr2004_1g_12s_2xs is barely coping with 25Gb/s routing. It can do it, but for multiple TCP sessions, a single one chokes the router at something like 20Gbps (down), 15 Gbps (up).
- sschueller 4y ago
  Very nice, how much does one of those plates cost? I can't find any prices easily.
  - jagrsw 4y ago
    IIRC ~70CHF for the outlet (which only works with the Feller electrical wall outlets AFAIK) + 10CHF (RJ45 module) + 20 CHF (single-port LC/APC module). The cable extension is expensive, 150CHF or so for 20m, but you'd probably need it for this outlet because it's specially designed so it can be attached to the LC/APC module manually (no splicing). Something like Swisscom's ClickLC, just a different system.

jbverschoor 4y ago

What about bi-directional vs running two fibers? Will this increase latency, even if negligible?

ilyt 4y ago
It will increase cost and you need to remember which side uses which module. Bi-directional just uses different wavelength in each direction. Generally not worth it unless you have existent infrastructure and want some more density
sponaugle 4y ago
And given some of the inexpensive single mode fiber spools are two fibers, it is probably easier to just have two fibers instead of trying to do 2 wavelengths.

voidwtf 4y ago

Is there a best practice for which bidi direction to choose for which device type? i.e. I assume you don't want to just wing it for every link, so do people typically choose to use all A/B on the switch side, and all B/A on the desktop/server side? If I were walking into a business, not having setup their network, what direction would I expect to need on the desktop/server side?

hackmiester 4y ago
I can't speak for anyone else, but we wing it for every link. When we light up a new link, we grab a matched pair, randomly select one to put in the switch, and then put the other one downstream.
coder543 4y ago
Aren’t BiDi transceivers usually a good bit more expensive, especially compared to how low the material cost of fiber is? What’s your use case for BiDi? I don’t have a lot of first hand experience with this stuff, though. I can easily imagine BiDi being useful to retrofit/repurpose some existing fiber, since labor costs can be a lot.
- fifteenforty 4y ago
  I can see it being useful for running ultra thin bend insensitive fibre around base boards; just like the FTTR thing Huawei is trying to sell. I wish the concept was more clearly legal in Australia. I’m pretty sure once you glue the fiber in place it becomes ‘permanent’ and thus has to be done by a licensed cabler with a fiber endorsement.

Hikikomori 4y ago

3rd party and maybe 1-2 of each type from the vendor (used to show their support that the 3rd party transceiver isn't the problem) is what we usually buy, basically an open secret. Though I've seen 97% off the price for vendor optics, still more expensive than 3rd party

dijit 4y ago

Had a really good experience with FS.com. To the point that we even printed out a picture of their warehouse from google maps and wrote “thank you” on it before stapling it to the wall. They were the best support we received.

dmd 4y ago
Which is why I'm so annoyed that my employer (which happens to be the largest employer in my entire state) mandates a vendor that costs 6x as much and whose products fail much more often.
- kevin_nisbet 4y ago
  I'm really out of the game these days so don't know the state of different options, but to be fair, a decade ago I had a team member try and cheap out on modules and it caused a fairly decent mess. Lots of failures, but not just outright failures, but problems where the link would mostly work, but through a bunch of fiber taps and bypass modules produced a high error rates when installed. So I'm not saying it's right, but I can at least understand where a mandate like that might come from, if perhaps outdated compared to the current state of the industry.
- Spooky23 4y ago
  The details matter. Depending on how you contract for support, etc. 6x on optics may be the logical choice as compared to being in some situation where the vendor won’t fix stuff until you verify that it’s broken with their OEM parts. If you’re the largest employer in the state, there may be some personal implications to a CIO telling a major OEM to f-off.
secabeen 4y ago
The new FS-Box is out, allowing reprogramming of most FS transceivers for a one-time $500 cost.

anonymousiam 4y ago

I agree there's a lot of useful information in the article. Two things: 1) I question his choice of OpnSense vs. OpenWRT. I've found OpenWRT to be less demanding on the hypervisor/host, and quite scalable. 2) Not about the article, but about your comment on Chinese modules. I've tried Chinese 10Gb SFP+ modules and I've returned all that I tried. They would overheat, and they would not operate up to the specified data rates. The name brands cost twice as much, but they work.

fire 4y ago

what's the big difference between single and multi mode fiber? Why is it scalable but multi-mode isn't?

runjake 4y ago
It comes down to multiple factors but this article covers them well. https://www.tlnetworx.com/blogs/news/single-mode-vs-multimode-fiber-whats-the-difference https://www.tlnetworx.com/blogs/news/single-mode-vs-multimod... And to be clear, there will be hacks that scale MMF a bit, but it’s not really “infinitely” scalable due to relatively poor attenuation and light mode. Keep in mind, there isn’t one MMF spec, there’s many: eg. OM1 through OM4

deleted 4y ago

[deleted]

stephen_g 4y ago

Good advice. I’ve had the same policy with regard to SMF vs MMF for quite a while now, when I saw each new speed dropping in distance with multimode. You just know SMF will do what you need. We’ve been using the FS modules too for years, they are very reliable. It does totally make a mockery of the vendors who claim they need to lock out other brand’s modules (from their 100% standardised port) for reliability etc.!

kotaKat 4y ago

> I resisted the "cheap chinese" modules for a long, long time, but then discovered that most of my uplink carriers used them. MSP ISP here, we just rock the FS optics combined with an FSbox to code them as required. ;) Just a warning for the weary: don't put a non-Fiberstore optic in their FS coding box -- you'll lock your account for a week. Ask our field team how they know this.