4 ms·

> Can you share, at least vaguely, what was your work? We work with gis/satellite data. Nothing groundbreaking technically, but we aggregate it into products t

by zhdc1 4y ago

> Can you share, at least vaguely, what was your work? We work with gis/satellite data. Nothing groundbreaking technically, but we aggregate it into products that are useful for social science researchers who aren’t comfortable with or don’t have the computing resources to use this data by themselves.

antihero 4y ago
What exactly about gis/satellite data required such an insane uplink?
- arcade79 4y ago
  What exactly is insane about it? I was surprised when my parents had 1Gbit in 2009. I was delighted when I could get 500Mbit in 2013. I'm slightly miffed that I don't even get 1Gbit in 2022. I literally laugh at providers attempting to convince me to get whatever with 10 or 20Mbit uplink speeds, in 2022l Yes, they exist. Anyhow. What's insane about a 10Gbit uplink? I wired up my apartment for 1Gbit in 2001. I've been frustrated about home network speeds for 21 years. I do not understand why you consider 10Gbit insane.
  - nine_k 4y ago
    At home you are free to make everything 10G-capable, have a NAS that can saturate such a link, etc. On the wider internet, few resources are happy to dedicate such bandwidth to you, even if you can consume it. Not even every popular torrent can muster so much incoming bandwidth. I can see how it may make sense if you run a popular public server from your home; maybe more convenient than colocation, but still is not a typical use case.
    - lmz 4y ago
      Also, even at colocation providers, 10 Gigabits committed rate is going to be a bit pricy.
    - throwaway2037 4y ago
      I feel the same. I am an übergeek, but I see no reason to pay for more than 100Mbit/sec, but 1Gbit and (soon'ish) 10Gbit will be available. Where I live, 100Mbit/sec is laughably cheap. I get a sales call once per month: "100Mit/sec is so slow! Would you like a upgrade?" I have no use for it. That said, I view myself as the exception to the rule. Please don't read this post as "1Gbit+ is useless" -- only that I don't have a use. Real question: If you are an Internet gamer, I assume that latency per packet is lower for 1Gbit+ connections, so it is useful for many customers. More: I feel the same about monitor resolutions above 4K. Anything above 4K on a 27inch monitor makes no sense for my body size / sitting height. I cannot use a vertical monitor taller than 27inch, and above 4K is too small for my aging eyesight! Again: I am sure there are many interesting applications for 4K+ and 27inch+. I hereby wish good luck to all the whippersnappers who will prove me wrong with interesting applications of 4K+ and 27inch+!
    - fragmede 4y ago
      > Not even every popular torrent can muster so much incoming bandwidth. So? The process of wringing out the bottlenecks is to eliminate the ones you can, and be mad about the ones you can't.
    - the_biot 4y ago
      You can look at it as too much for anyone's needs, or you can see it as enabling tomorrow's applications. Looking back, or looking forward. Back in the dialup days, nobody in their right mind would run a torrent, or play online shooters, or watch streaming HD video -- such things were utterly impractical over ultra slow links with very high latency. Once people started getting broadband, these applications started appearing. Of course we need faster links into people's homes. Of course under-1Gbit links into homes in 2022 are ridiculous.
  - s3233323 4y ago
  - beAbU 4y ago
    I have 100up/50down (mbps, not giga) at home, and its the fastest I can afford. If money is not a concern then it can go up to 1gb, but not in my area yet. I can barely saturate my line unless I'm donwloading something big. 10gig is insane.
  - antihero 4y ago
    I have 1Gbit Virgin in the U.K. and that speed us still very rare here. Any faster and I’m going to have to be searching for a better USB NIC in order to actually utilise anything. Most consumer hardware seems to still be 1000baseT so how would you utilise anything more than a gigabit connection unless you have 10+devices all working at maximum capacity? Also I rarely max it downloading anything at all, is there some sort of special internet somewhere that you can download stuff at 10Gbps? Finally, can your SSD even keep up with speeds like this? **Oh and I thought I’d note, it is the positive use of “insane” like “mad good”.
    - consp 4y ago
      > Finally, can your SSD even keep up with speeds like this? COTS nvme SSD can easily do 1GB/s which easily maxes out non-optimized 10Gbit due to overhead alone. Burst that is for write, but you get the idea.
  - jjav 4y ago
    > I literally laugh at providers attempting to convince me to get whatever with 10 or 20Mbit uplink speeds, in 2022l Yes, they exist. Meanwhile here in Silicon Valley, the fastest connection I can get at home (sonic.net) is 22Mbps/1.7Mbps. In the office I have Comcast Business which is a bit faster (don't recall exactly) but certainly well below 100Mbps. So yes 1Gbps sounds quite insane. 10Gbps is out of this universe.
- merlyn 4y ago
  25Gbps is not insane if you are using commodity hardware. If you are buying any sort of datacenter switch, most likely it will be 25G/100G already. 10G/40G is dead tech. Now, 400G uplinks are still pretty reserved, but 100G (and its' breakout, 4 x 25G) is common as dirt now.
  - namibj 4y ago
    Are there any small (affordable, energy-efficient) switches with those speeds? Say, around 8-24 of the 25G lanes? You can buy a dual-port SFP28 card for 250 EUR by using the QNAP "accessory"[0] which is just a ConnectX-6 Lx EN with a fan to work in the NAS systems they market it for. This is quite efficient due to the PCIe4 support not hogging many lanes (half of the 8 lanes can be enough depending on usage), but to use the speed one needs a 25G switch. Is the cheapest way just an Ryzen with 5 of these directly attached to the CPU, and running routing on the CPU via DPDK? [0]: https://geizhals.eu/qnap-lan-adapter-qxg-25g2sf-cx6-a2609459.html https://geizhals.eu/qnap-lan-adapter-qxg-25g2sf-cx6-a2609459...
    - throwaway2037 4y ago
      MikroTik is a market leader in barebones 10Gbit/sec routers. However, multi-port 25Gbit/sec is very expensive! This model is 2800 USD and has 12x 25Gbit/sec and 2x 100Gbit/sec ports: https://mikrotik.com/product/ccr2216_1g_12xs_2xq https://mikrotik.com/product/ccr2216_1g_12xs_2xq
  - antihero 4y ago
    I struggle to find a 10Gbps USB NIC for my laptop.
    - zhdc1 4y ago
      Go with surplus commercial hardware and be judicious about when you turn your equipment on.
- OJFord 4y ago
  I'm sure there's >10Gbits of 'gis/satellite data' available every second if you want it? Or what if you want a bunch of historical data for some sort of analysis, and you'd like to start that now, not next week when it finishes downloading? If it's historical (i.e. sitting there available to download) then faster is always better right? If your connection is the bottleneck then all else equal why wouldn't you want it? (I imagine 10Gbps is about the point currently where it's not necessarily going to help going faster on your end, unless you you have multiple (concurrent) sources, or know that even at that you're still still the bottleneck.)
- brailsafe 4y ago
  GIS and remote sensing has you working with gargantuan TIFF files and massive amounts of other kinds of data. It's a whole thing. If you're doing data aggregation, it's possible that without an absolute unit of a pipe just to do a download of imagery, the prototype wouldn't have been feasible. https://www.esri.com/arcgis-blog/products/product/imagery/making-landsat-on-aws-accessible/ https://www.esri.com/arcgis-blog/products/product/imagery/ma...
- randomluck040 4y ago
  I can think of two things. Large time series data for large areas with a lot of attributes like temperature or other atmospheric data or a huge amount of point clouds or comparable 3D Data.
  - zhdc1 4y ago
    SAR and multispectral imagery. It’s not difficult to work with when you’re only dealing a small and well defined regions, but the bandwidth and storage requirement definitely increase once you start doing daily global composites. We’ve actually moved everything into a data center with a 1gb connection. The trade of being that we have several orders of magnitude more storage and computing capacity.
    - randomluck040 4y ago
      It’s crazy because among friends I’m the only one working with geodata (remote sensing, too) and the last few days I keep stumbling upon people who already do some work in the same area and are probably much more advanced than I am. Good luck with your endeavour!

by zhdc1 4y ago

antihero 4y ago

What exactly about gis/satellite data required such an insane uplink?

arcade79 4y ago
What exactly is insane about it? I was surprised when my parents had 1Gbit in 2009. I was delighted when I could get 500Mbit in 2013. I'm slightly miffed that I don't even get 1Gbit in 2022. I literally laugh at providers attempting to convince me to get whatever with 10 or 20Mbit uplink speeds, in 2022l Yes, they exist. Anyhow. What's insane about a 10Gbit uplink? I wired up my apartment for 1Gbit in 2001. I've been frustrated about home network speeds for 21 years. I do not understand why you consider 10Gbit insane.
- nine_k 4y ago
  At home you are free to make everything 10G-capable, have a NAS that can saturate such a link, etc. On the wider internet, few resources are happy to dedicate such bandwidth to you, even if you can consume it. Not even every popular torrent can muster so much incoming bandwidth. I can see how it may make sense if you run a popular public server from your home; maybe more convenient than colocation, but still is not a typical use case.
  - lmz 4y ago
    Also, even at colocation providers, 10 Gigabits committed rate is going to be a bit pricy.
  - throwaway2037 4y ago
    I feel the same. I am an übergeek, but I see no reason to pay for more than 100Mbit/sec, but 1Gbit and (soon'ish) 10Gbit will be available. Where I live, 100Mbit/sec is laughably cheap. I get a sales call once per month: "100Mit/sec is so slow! Would you like a upgrade?" I have no use for it. That said, I view myself as the exception to the rule. Please don't read this post as "1Gbit+ is useless" -- only that I don't have a use. Real question: If you are an Internet gamer, I assume that latency per packet is lower for 1Gbit+ connections, so it is useful for many customers. More: I feel the same about monitor resolutions above 4K. Anything above 4K on a 27inch monitor makes no sense for my body size / sitting height. I cannot use a vertical monitor taller than 27inch, and above 4K is too small for my aging eyesight! Again: I am sure there are many interesting applications for 4K+ and 27inch+. I hereby wish good luck to all the whippersnappers who will prove me wrong with interesting applications of 4K+ and 27inch+!
  - fragmede 4y ago
    > Not even every popular torrent can muster so much incoming bandwidth. So? The process of wringing out the bottlenecks is to eliminate the ones you can, and be mad about the ones you can't.
  - the_biot 4y ago
    You can look at it as too much for anyone's needs, or you can see it as enabling tomorrow's applications. Looking back, or looking forward. Back in the dialup days, nobody in their right mind would run a torrent, or play online shooters, or watch streaming HD video -- such things were utterly impractical over ultra slow links with very high latency. Once people started getting broadband, these applications started appearing. Of course we need faster links into people's homes. Of course under-1Gbit links into homes in 2022 are ridiculous.
- s3233323 4y ago
- beAbU 4y ago
  I have 100up/50down (mbps, not giga) at home, and its the fastest I can afford. If money is not a concern then it can go up to 1gb, but not in my area yet. I can barely saturate my line unless I'm donwloading something big. 10gig is insane.
- antihero 4y ago
  I have 1Gbit Virgin in the U.K. and that speed us still very rare here. Any faster and I’m going to have to be searching for a better USB NIC in order to actually utilise anything. Most consumer hardware seems to still be 1000baseT so how would you utilise anything more than a gigabit connection unless you have 10+devices all working at maximum capacity? Also I rarely max it downloading anything at all, is there some sort of special internet somewhere that you can download stuff at 10Gbps? Finally, can your SSD even keep up with speeds like this? **Oh and I thought I’d note, it is the positive use of “insane” like “mad good”.
  - consp 4y ago
    > Finally, can your SSD even keep up with speeds like this? COTS nvme SSD can easily do 1GB/s which easily maxes out non-optimized 10Gbit due to overhead alone. Burst that is for write, but you get the idea.
- jjav 4y ago
  > I literally laugh at providers attempting to convince me to get whatever with 10 or 20Mbit uplink speeds, in 2022l Yes, they exist. Meanwhile here in Silicon Valley, the fastest connection I can get at home (sonic.net) is 22Mbps/1.7Mbps. In the office I have Comcast Business which is a bit faster (don't recall exactly) but certainly well below 100Mbps. So yes 1Gbps sounds quite insane. 10Gbps is out of this universe.
merlyn 4y ago
25Gbps is not insane if you are using commodity hardware. If you are buying any sort of datacenter switch, most likely it will be 25G/100G already. 10G/40G is dead tech. Now, 400G uplinks are still pretty reserved, but 100G (and its' breakout, 4 x 25G) is common as dirt now.
- namibj 4y ago
  Are there any small (affordable, energy-efficient) switches with those speeds? Say, around 8-24 of the 25G lanes? You can buy a dual-port SFP28 card for 250 EUR by using the QNAP "accessory"[0] which is just a ConnectX-6 Lx EN with a fan to work in the NAS systems they market it for. This is quite efficient due to the PCIe4 support not hogging many lanes (half of the 8 lanes can be enough depending on usage), but to use the speed one needs a 25G switch. Is the cheapest way just an Ryzen with 5 of these directly attached to the CPU, and running routing on the CPU via DPDK? [0]: https://geizhals.eu/qnap-lan-adapter-qxg-25g2sf-cx6-a2609459.html https://geizhals.eu/qnap-lan-adapter-qxg-25g2sf-cx6-a2609459...
  - throwaway2037 4y ago
    MikroTik is a market leader in barebones 10Gbit/sec routers. However, multi-port 25Gbit/sec is very expensive! This model is 2800 USD and has 12x 25Gbit/sec and 2x 100Gbit/sec ports: https://mikrotik.com/product/ccr2216_1g_12xs_2xq https://mikrotik.com/product/ccr2216_1g_12xs_2xq
- antihero 4y ago
  I struggle to find a 10Gbps USB NIC for my laptop.
  - zhdc1 4y ago
    Go with surplus commercial hardware and be judicious about when you turn your equipment on.
OJFord 4y ago
I'm sure there's >10Gbits of 'gis/satellite data' available every second if you want it? Or what if you want a bunch of historical data for some sort of analysis, and you'd like to start that now, not next week when it finishes downloading? If it's historical (i.e. sitting there available to download) then faster is always better right? If your connection is the bottleneck then all else equal why wouldn't you want it? (I imagine 10Gbps is about the point currently where it's not necessarily going to help going faster on your end, unless you you have multiple (concurrent) sources, or know that even at that you're still still the bottleneck.)
brailsafe 4y ago
GIS and remote sensing has you working with gargantuan TIFF files and massive amounts of other kinds of data. It's a whole thing. If you're doing data aggregation, it's possible that without an absolute unit of a pipe just to do a download of imagery, the prototype wouldn't have been feasible. https://www.esri.com/arcgis-blog/products/product/imagery/making-landsat-on-aws-accessible/ https://www.esri.com/arcgis-blog/products/product/imagery/ma...
randomluck040 4y ago
I can think of two things. Large time series data for large areas with a lot of attributes like temperature or other atmospheric data or a huge amount of point clouds or comparable 3D Data.
- zhdc1 4y ago
  SAR and multispectral imagery. It’s not difficult to work with when you’re only dealing a small and well defined regions, but the bandwidth and storage requirement definitely increase once you start doing daily global composites. We’ve actually moved everything into a data center with a 1gb connection. The trade of being that we have several orders of magnitude more storage and computing capacity.
  - randomluck040 4y ago
    It’s crazy because among friends I’m the only one working with geodata (remote sensing, too) and the last few days I keep stumbling upon people who already do some work in the same area and are probably much more advanced than I am. Good luck with your endeavour!