5 ms·

The hardest thing for me to understand is how every telco can complain of congestion, but they're perfectly willing to introduce unnecessary overhead for DPI.

The hardest thing for me to understand is how every telco can complain of congestion, but they're perfectly willing to introduce unnecessary overhead for DPI. This is not a good day, not a good day at all.

MichaelGG 14y ago
How is DPI related to needing more bandwidth needed per base station or residential area?
- kcbanner 14y ago
  I think he is referring to the fact that significant CPU resources are required for DPI and if the DPI can't keep up, things will slow down.
  - Jach 14y ago
    I think you're underestimating the limit to how fast DPI is or can get. The computations aren't complicated and like many kinds of algorithms you can trade space with time.
    - jessaustin 14y ago
      This statement might be correct (although I've never seen any evidence to support it), but it's still misleading because "forward the packet" is always going to require fewer resources than "read the packet, parse it using this set of algos, use the parse results to search your DB of shit you want to fuck with, optionally fuck with the packet, optionally forward the packet". An exception to this would be if you have big iron on the edge that protects resource-poor interior nodes. This situation is unavoidable sometimes (DDOS), but it's not what anyone should aim for.
      - gknoy 14y ago
        They don't necessarily need to do anything with the packet then -- they could always pass a copy to cold storage, and then crunch the bits at their leisure in a massive data center.
        jessaustin 14y ago
        I have several responses: A) this is probably true to an extent. B) slurping everything to disk is impractical, and no one has that much storage, so you're back to parsing and deciding at the boundary. This "leisure" time doesn't ever happen when you save everything all the time. C) this point seems to contradict the various claims of reasonable network maintenance I've seen; if it's trash you want to drop, you want to drop it on the floor not on your disk. If you're keeping actual traffic rather than just summaries of traffic, you're not doing reasonable network maintenance.
      - mfukar 14y ago
        That's how DPI was done in the 00's, yes.
    - jholman 14y ago
      I think you're underestimating how much traffic can be out there. DPI requires decoding application-layer protocols. This requires many layers of decoding. That's all well and good for one client (feature phones happily load web pages, after all), but to do it inline at the router means the router needs to be as powerful as the sum of all of the networking hardware of the hosts it routes to. Source: at my last job, we were working on fixes for security holes that could cost billions of dollars of damages per incident, and the fixes were solvable in software, but we couldn't persuade customers of the value of buying beefy enough network equipment to actual block the attacks. Mind you, I didn't run any numbers myself on the cost of the hardware; that's just what my colleagues said.
      - signa11 14y ago
        narus-networks has solved this 'problem' long time ago...
        jholman 14y ago
        Oh, it's obviously possible. Reading a little about Narus does not give me the impression that it's cost-effective, for any non-NSA sense of the word "cost". In any case, in my opinion our customers were not taking their problems very seriously, and their price sensitivity reflected that. If you have any trustworthy source of ballpark pricing for a 10Gb/s DPI solution, Narus or otherwise, I'm curious. I'm talking BALLPARK here. I looked up some of Narus's whitepapers, and what I'm seeing looks like Grade A Bullshit. That doesn't mean they ARE bullshit -- their whitepapers would look like this in either case. But I would like to point out that it is absolutely possible to build a company like this without actually delivering the goods, because many of your potential customers are largely incapable of understanding whether or not the goods have really been delivered. ALSO, I guess I should remember that DPI for stopping malware, and DPI for spying on citizens, are really different things. When you're spying on citizens, 90% effectiveness is good enough. When you're trying to block malware (versus e.g. commited cyberterrorism), 90% is a little better than 0%, but not much, because your adversary can try many times and they only have to succeed once. So in that sense, the DPI being discussed here is a much easier problem than the DPI that we were trying to tackle.
- politician 14y ago
  Bandwidth is the measure of data over time. DPI is a process which inspects each piece of data and that takes time. Therefore, the bandwidth will decrease.
  - swalberg 14y ago
    Latency != bandwidth. Also, one goal of DPI is to enable granular QoS policy. So the total bandwidth available would stay the same, but the bandwidth available to each app would vary depending on network conditions and the will of the network operator.
    - 65b 14y ago
      "In TCP connections, the large bandwidth-delay product of high latency connections, combined with relatively small TCP window sizes on many devices, effectively causes the throughput of a high latency connection to drop sharply with latency" --Wikipedia It does equal bandwidth until everything is jumbo frames UDP
      - swalberg 14y ago
        If you would have pasted a larger part of the quote, you would have included the part of increasing the window size (e.g. window scaling, selective ACKs, like almost everything out there supports) and the mention of satellite links to refer to high latency conditions. And that notwithstanding that, we are talking about additional latency of microseconds against a typical delay of at least 30 milliseconds. "drop sharply" just doesn't apply here. If you want to argue against DPI that's fine but "it'll make the Internet slower" comes off as whining plus you're fighting against the exponential effects of Moore's law. A quick Google search tells me there are several products that'll do line rate DPI at 10Gbps. There are much better arguments to be made against DPI such as privacy, a slippery slope to a walled garden, or just plain unfairness.
    - politician 14y ago
      Wikipedia says, "In computer networking and computer science, the words bandwidth, network bandwidth, data bandwidth, or digital bandwidth are terms used to refer to various bit-rate measures, representing the available or consumed data communication resources expressed in bits per second or multiples of it (bit/s, kbit/s, Mbit/s, Gbit/s, etc.)." All else being equal, increasing the denominator reduces the rate. It's just math.
- ffk 14y ago
  Other restrictions besides bandwidth exist. For example, DPI requires more CPU and memory, which are some of the major constraints of networking equipment, especially when the equipment resides at the edge (borders) of a Tier 1 ISP.
  - signa11 14y ago
    from (http://en.wikipedia.org/wiki/Narus_(company) http://en.wikipedia.org/wiki/Narus_(company)) we have the following "A single NarusInsight machine can monitor traffic equal to the maximum capacity (10 Gbit/s) of around 39,000 DSL lines or 195,000 telephone modems. But, in practical terms, since individual internet connections are not continually filled to capacity, the 10 Gbit/s capacity of one NarusInsight installation enables it to monitor the combined traffic of several million broadband users." also if you have not already read this (http://www.wired.com/science/discoveries/news/2006/05/70908 http://www.wired.com/science/discoveries/news/2006/05/70908) earlier, doesn't hurt to read once again.
    - abecedarius 14y ago
      How much does it cost? (With operating expenses.)
- aidenn0 14y ago
  The main limit to bandwith is speed of routing the packets. This is why QoS typically fails for IP; it is usually cheaper to just use a faster dumb router than to use a slower smart router.
sliverstorm 14y ago
Perhaps they are of the opinion that DPI will better-enable them to apply QoS rules, resulting in a net improvement in perceived performance?
noselasd 14y ago
Monitoring systems are usually extra boxes that you place as either pass-through on the cables, or you tap the signals with attenuators on electric cables, or you bend the fiber enough to tap about 10% light off them, enoufg to regenerate the signal. Other solutions such as just using mirror ports on an ethernet switch exists as well. This only costs money, not bandwidth.

MichaelGG 14y ago

How is DPI related to needing more bandwidth needed per base station or residential area?

kcbanner 14y ago
I think he is referring to the fact that significant CPU resources are required for DPI and if the DPI can't keep up, things will slow down.
- Jach 14y ago
  I think you're underestimating the limit to how fast DPI is or can get. The computations aren't complicated and like many kinds of algorithms you can trade space with time.
  - jessaustin 14y ago
    This statement might be correct (although I've never seen any evidence to support it), but it's still misleading because "forward the packet" is always going to require fewer resources than "read the packet, parse it using this set of algos, use the parse results to search your DB of shit you want to fuck with, optionally fuck with the packet, optionally forward the packet". An exception to this would be if you have big iron on the edge that protects resource-poor interior nodes. This situation is unavoidable sometimes (DDOS), but it's not what anyone should aim for.
    - gknoy 14y ago
      They don't necessarily need to do anything with the packet then -- they could always pass a copy to cold storage, and then crunch the bits at their leisure in a massive data center.
      - jessaustin 14y ago
        I have several responses: A) this is probably true to an extent. B) slurping everything to disk is impractical, and no one has that much storage, so you're back to parsing and deciding at the boundary. This "leisure" time doesn't ever happen when you save everything all the time. C) this point seems to contradict the various claims of reasonable network maintenance I've seen; if it's trash you want to drop, you want to drop it on the floor not on your disk. If you're keeping actual traffic rather than just summaries of traffic, you're not doing reasonable network maintenance.
    - mfukar 14y ago
      That's how DPI was done in the 00's, yes.
  - jholman 14y ago
    I think you're underestimating how much traffic can be out there. DPI requires decoding application-layer protocols. This requires many layers of decoding. That's all well and good for one client (feature phones happily load web pages, after all), but to do it inline at the router means the router needs to be as powerful as the sum of all of the networking hardware of the hosts it routes to. Source: at my last job, we were working on fixes for security holes that could cost billions of dollars of damages per incident, and the fixes were solvable in software, but we couldn't persuade customers of the value of buying beefy enough network equipment to actual block the attacks. Mind you, I didn't run any numbers myself on the cost of the hardware; that's just what my colleagues said.
    - signa11 14y ago
      narus-networks has solved this 'problem' long time ago...
      - jholman 14y ago
        Oh, it's obviously possible. Reading a little about Narus does not give me the impression that it's cost-effective, for any non-NSA sense of the word "cost". In any case, in my opinion our customers were not taking their problems very seriously, and their price sensitivity reflected that. If you have any trustworthy source of ballpark pricing for a 10Gb/s DPI solution, Narus or otherwise, I'm curious. I'm talking BALLPARK here. I looked up some of Narus's whitepapers, and what I'm seeing looks like Grade A Bullshit. That doesn't mean they ARE bullshit -- their whitepapers would look like this in either case. But I would like to point out that it is absolutely possible to build a company like this without actually delivering the goods, because many of your potential customers are largely incapable of understanding whether or not the goods have really been delivered. ALSO, I guess I should remember that DPI for stopping malware, and DPI for spying on citizens, are really different things. When you're spying on citizens, 90% effectiveness is good enough. When you're trying to block malware (versus e.g. commited cyberterrorism), 90% is a little better than 0%, but not much, because your adversary can try many times and they only have to succeed once. So in that sense, the DPI being discussed here is a much easier problem than the DPI that we were trying to tackle.
politician 14y ago
Bandwidth is the measure of data over time. DPI is a process which inspects each piece of data and that takes time. Therefore, the bandwidth will decrease.
- swalberg 14y ago
  Latency != bandwidth. Also, one goal of DPI is to enable granular QoS policy. So the total bandwidth available would stay the same, but the bandwidth available to each app would vary depending on network conditions and the will of the network operator.
  - 65b 14y ago
    "In TCP connections, the large bandwidth-delay product of high latency connections, combined with relatively small TCP window sizes on many devices, effectively causes the throughput of a high latency connection to drop sharply with latency" --Wikipedia It does equal bandwidth until everything is jumbo frames UDP
    - swalberg 14y ago
      If you would have pasted a larger part of the quote, you would have included the part of increasing the window size (e.g. window scaling, selective ACKs, like almost everything out there supports) and the mention of satellite links to refer to high latency conditions. And that notwithstanding that, we are talking about additional latency of microseconds against a typical delay of at least 30 milliseconds. "drop sharply" just doesn't apply here. If you want to argue against DPI that's fine but "it'll make the Internet slower" comes off as whining plus you're fighting against the exponential effects of Moore's law. A quick Google search tells me there are several products that'll do line rate DPI at 10Gbps. There are much better arguments to be made against DPI such as privacy, a slippery slope to a walled garden, or just plain unfairness.
  - politician 14y ago
    Wikipedia says, "In computer networking and computer science, the words bandwidth, network bandwidth, data bandwidth, or digital bandwidth are terms used to refer to various bit-rate measures, representing the available or consumed data communication resources expressed in bits per second or multiples of it (bit/s, kbit/s, Mbit/s, Gbit/s, etc.)." All else being equal, increasing the denominator reduces the rate. It's just math.
ffk 14y ago
Other restrictions besides bandwidth exist. For example, DPI requires more CPU and memory, which are some of the major constraints of networking equipment, especially when the equipment resides at the edge (borders) of a Tier 1 ISP.
- signa11 14y ago
  from (http://en.wikipedia.org/wiki/Narus_(company) http://en.wikipedia.org/wiki/Narus_(company)) we have the following "A single NarusInsight machine can monitor traffic equal to the maximum capacity (10 Gbit/s) of around 39,000 DSL lines or 195,000 telephone modems. But, in practical terms, since individual internet connections are not continually filled to capacity, the 10 Gbit/s capacity of one NarusInsight installation enables it to monitor the combined traffic of several million broadband users." also if you have not already read this (http://www.wired.com/science/discoveries/news/2006/05/70908 http://www.wired.com/science/discoveries/news/2006/05/70908) earlier, doesn't hurt to read once again.
  - abecedarius 14y ago
    How much does it cost? (With operating expenses.)
aidenn0 14y ago
The main limit to bandwith is speed of routing the packets. This is why QoS typically fails for IP; it is usually cheaper to just use a faster dumb router than to use a slower smart router.

sliverstorm 14y ago

Perhaps they are of the opinion that DPI will better-enable them to apply QoS rules, resulting in a net improvement in perceived performance?

noselasd 14y ago

Monitoring systems are usually extra boxes that you place as either pass-through on the cables, or you tap the signals with attenuators on electric cables, or you bend the fiber enough to tap about 10% light off them, enoufg to regenerate the signal. Other solutions such as just using mirror ports on an ethernet switch exists as well. This only costs money, not bandwidth.