4 ms·

It's easy to underestimate just how slow digital systems are in many real time scenarios compared to analog systems. Consider it in the context of camera based

by yesco 1y ago

It's easy to underestimate just how slow digital systems are in many real time scenarios compared to analog systems. Consider it in the context of camera based self-driving cars, it's tangential to this discussion but it's an easy to visualize metaphor: - A car traveling 60mph is traveling at 88 feet per second - Assuming a 60hz camera, there would be a 16.67 ms gap between each frame - The car is traveling 1.5 feet between each frame interval - A certain amount of exposure time is necessary for the camera to generate even 1 frame or it will be blurry - High framerate cameras often work around this by staggering/interlacing multiple sensors, but doing this implicitly increases the latency of each frame - A 120hz camera might deliver double the frames per second, but each frame could be arriving 4 frames late in exchange - 4 frames would be imperceptible to humans, it would be 3 feet for the car - You haven't even processed any of these frames yet. - Your off the shelf library introduces a random 1 second delay for some reason and costs you 88 miles in processing time - The car can drive as fast as 120mph All digital sensors implicitly have a sampling frequency, and the fundamental disconnect is always high sampling frequency =/= low latency. People constantly make this mistake over and over, and by the time you notice you are already too deep into development to make a change. Decreasing latency is expensive, and requires specialized knowledge. Often you get expert software engineers who end up bottlenecked by the hardware limitations they can't even comprehend or the reverse, hardware guys bottlenecked by the software they can't introspect. The latency is only truly understood when you get to integration testing. Nearly every step of the way you discover you need specialized hardware, software, operating systems, sensors. Every part of the chain each costing you more latency. It's like it's own ecosystem where almost everyone writes everything from scratch and doesn't share anything. It's gotten better in recent years though. Full disclosure: I work in medtech and don't actually deal with cars, but it's a very similar problem space. We often use the same hardware/software cars use for this reason.

neom 1y ago
88ft not 88 miles. I don't understand your 120hz camera line, I also have a degree in imaging tech and that part I do not understand, would you be so kind as to expand upon it??
- yesco 1y ago
  Wew I'm getting a lot of pedantic replies here. Yeah I got the units wrong there my bad, you get the point I was making though right? In regards to the 120hz camera line, I would be happy to expand on that for you. To be clear, I'm specifically talking about how when you try to increase sampling rate by interlacing multiple concurrent digital sensors you need to deal with the following potential problems (this is a property of all digital sensors): - Real time synchronization between concurrent sensors requires additional processing time to ensure proper ordering. The samples need to be processed together in small batches. This adds latency and the more things you are trying to synchronize the more latency is introduced. - Inter-sensor calibration to account for variations between individual sensors can be used to reduce the latency introduced by synchronization but lacking this, you are bottlenecked by the slowest sensor. There are a lot of different ways to handle this though so I'm speaking very generally. - Broadly speaking most signals need to go through some kind of filter to remove noise, most digital filters have a certain amount of algorithmic latency built-in that is physically unavoidable. When you are interlacing multiple different sensors, you are getting more noise, so the likelihood of a filter being required starts to increase. I want to stress here that these are not impossible challenges. In fact they are largely solved problems. But they are not universally solved in the same way, you need to balance between precision manufacturing, signal quality and signal latency. In practice most people are not prioritizing latency, so a 120hz camera might be optimizing for video recordings and not live processing scenerios. So long as you know what you are looking for you can avoid this when choosing which camera to use. Computers can be fast, but fast can mean different things when dealing with real-time situations in high speeds. Bottlenecks need to be considered from all levels. The clock speed of the computers CPU often gives product managers weird ideas about what is possible. This was the main point I was trying to make here.
  - potato3732842 1y ago
    > I'm getting a lot of pedantic replies here. Yeah I got the units wrong there my bad, you get the point I was making though right? You get the most pedantic replies when you're right and people don't like it. I work on a latency sensitive product. You're pretty correct about most of it.
krisoft 1y ago
I can’t quite understand your comment. > Assuming a 60hz camera, there would be a 16.67 ms gap between each frame. The car is traveling 1.5 feet between each frame interval. Ok? So? You are just stating this as if we should understand the implications. I do in fact work with self-driving cars. What you say is true, but it is not a big deal. Why do you feel this maters? Or what is your point? > A certain amount of exposure time is necessary for the camera to generate even 1 frame or it will be blurry This is a confused statement. A certain amount of exposure is necessary for the camera to collect light. If you don’t have long enough exposure the picture will be dark, not blurry. Your statement makes it sound as if avoiding blur is why we need exposure time, which is a complete nonsense. In reality none of this is a problem. There are automotive grade cameras which can collect enough light fast enough that the images are not blurry in practice. Yes, these cameras have a non-zero exposure time. Yes, this adds latency. No, this is not a problem. > Your off the shelf library introduces a random 1 second delay for some reason and costs you 88 miles in processing time You mean 88 feet. If my off the shelf library introduces a random 1 second delay i will chuck it in the bin post haste. Use stuff whose performance characteristics are well understood by you and is not terrible. > Nearly every step of the way you discover you need specialized hardware, software, operating systems, sensors. I do not recognise the world you describe.
- yesco 1y ago
  > I do in fact work with self-driving cars. What you say is true, but it is not a big deal. Why do you feel this maters? Or what is your point? Sorry not trying to dunk on you here, but this reads like something a junior engineer would complain to me about. These are not trivial problems, and I'm sure your co-workers who resolved them already so you didn't need to worry about them would agree with me. > In reality none of this is a problem. There are automotive grade cameras which can collect enough light fast enough that the images are not blurry in practice. Yes, these cameras have a non-zero exposure time. Yes, this adds latency. No, this is not a problem. You are contradicting yourself here, yes there are automotive grade cameras, but if this wasn't a problem, why would automotive grade cameras need to exist? My post wasn't saying these were impossible problems but hard ones. > You mean 88 feet. If my off the shelf library introduces a random 1 second delay i will chuck it in the bin post haste. Use stuff whose performance characteristics are well understood by you and is not terrible. Look I might have typed the wrong unit but it's a bit ironic you gave me this word salad right after complaining about this... Yeah you obviously don't use the library that adds a 1 second delay, but often you don't have the luxury of knowing that until after you learn about it through integration testing. Libraries don't usually come with latency stats calibrated to your desired hardware right on the tin, would be pretty sweet if they did though. > I do not recognise the world you describe. I don't find this surprising :)
chrisweekly 1y ago
Awesome post, illuminating metaphor, +1. Tangent: why do so many people (even smart articulate ones like you) in our field (which involves precision with syntax and grammar) get apostrophes wrong? "It's [IT IS] like its [ITS/HIS/HER/MY/YOUR/THEIR] own ecosystem..." I anticipate downvotes for picking nits (let alone mentioning karma points), but FTR my intent is to help non-native English speakers (and mostly-literate English language-natives). Getting this "it's : its" distinction wrong is increasingly common and sometimes leads to signal loss. Yesco, rereading your comment makes me slightly ashamed of this apostrophe rant, I hope others comment on the substance here. / end tangent As someone who spent years doing web performance optimization for a living, your observations resonate. Beyond obvious low-hanging fruit, latency gains are rarely simple to achieve in practice; tradeoffs abound.
- yesco 1y ago
  Ha yeah it reads a bit silly, to be honest I lazily typed my original post on my phone while sitting on a hammock outside, I didn't really review it much before posting. Autocorrect can make it challenging for me to do apostrophes correctly since it often overassumes my intent.
  - chrisweekly 1y ago
    IME autocorrect costs me more time and keystrokes than doing without, so I always disable it (shrug) different strokes...
- saratogacx 1y ago
  To your tangent. When I was taught its vs it's, I was never given the association to his/hers/etc. It was just arbitrary and something you just had to memorize. Reading your post I am shocked I never made the connection earlier but it was just never knowledge I was provided with to work from.
foobarian 1y ago
Honestly I'm just a dumb web dev and even though many of our performance problems can be solved by horizontal scaling, there are a minority of cases where latency is important and that's when things get really fun :-) No more downloading random halfbaked libraries off the web, architecture starts to matter, vm settings start to matter, gateways/load balancers in the middle start to matter, even code optimization starts to matter again just like in good old days.

by yesco 1y ago

neom 1y ago

88ft not 88 miles. I don't understand your 120hz camera line, I also have a degree in imaging tech and that part I do not understand, would you be so kind as to expand upon it??

yesco 1y ago
Wew I'm getting a lot of pedantic replies here. Yeah I got the units wrong there my bad, you get the point I was making though right? In regards to the 120hz camera line, I would be happy to expand on that for you. To be clear, I'm specifically talking about how when you try to increase sampling rate by interlacing multiple concurrent digital sensors you need to deal with the following potential problems (this is a property of all digital sensors): - Real time synchronization between concurrent sensors requires additional processing time to ensure proper ordering. The samples need to be processed together in small batches. This adds latency and the more things you are trying to synchronize the more latency is introduced. - Inter-sensor calibration to account for variations between individual sensors can be used to reduce the latency introduced by synchronization but lacking this, you are bottlenecked by the slowest sensor. There are a lot of different ways to handle this though so I'm speaking very generally. - Broadly speaking most signals need to go through some kind of filter to remove noise, most digital filters have a certain amount of algorithmic latency built-in that is physically unavoidable. When you are interlacing multiple different sensors, you are getting more noise, so the likelihood of a filter being required starts to increase. I want to stress here that these are not impossible challenges. In fact they are largely solved problems. But they are not universally solved in the same way, you need to balance between precision manufacturing, signal quality and signal latency. In practice most people are not prioritizing latency, so a 120hz camera might be optimizing for video recordings and not live processing scenerios. So long as you know what you are looking for you can avoid this when choosing which camera to use. Computers can be fast, but fast can mean different things when dealing with real-time situations in high speeds. Bottlenecks need to be considered from all levels. The clock speed of the computers CPU often gives product managers weird ideas about what is possible. This was the main point I was trying to make here.
- potato3732842 1y ago
  > I'm getting a lot of pedantic replies here. Yeah I got the units wrong there my bad, you get the point I was making though right? You get the most pedantic replies when you're right and people don't like it. I work on a latency sensitive product. You're pretty correct about most of it.

krisoft 1y ago

I can’t quite understand your comment. > Assuming a 60hz camera, there would be a 16.67 ms gap between each frame. The car is traveling 1.5 feet between each frame interval. Ok? So? You are just stating this as if we should understand the implications. I do in fact work with self-driving cars. What you say is true, but it is not a big deal. Why do you feel this maters? Or what is your point? > A certain amount of exposure time is necessary for the camera to generate even 1 frame or it will be blurry This is a confused statement. A certain amount of exposure is necessary for the camera to collect light. If you don’t have long enough exposure the picture will be dark, not blurry. Your statement makes it sound as if avoiding blur is why we need exposure time, which is a complete nonsense. In reality none of this is a problem. There are automotive grade cameras which can collect enough light fast enough that the images are not blurry in practice. Yes, these cameras have a non-zero exposure time. Yes, this adds latency. No, this is not a problem. > Your off the shelf library introduces a random 1 second delay for some reason and costs you 88 miles in processing time You mean 88 feet. If my off the shelf library introduces a random 1 second delay i will chuck it in the bin post haste. Use stuff whose performance characteristics are well understood by you and is not terrible. > Nearly every step of the way you discover you need specialized hardware, software, operating systems, sensors. I do not recognise the world you describe.

yesco 1y ago
> I do in fact work with self-driving cars. What you say is true, but it is not a big deal. Why do you feel this maters? Or what is your point? Sorry not trying to dunk on you here, but this reads like something a junior engineer would complain to me about. These are not trivial problems, and I'm sure your co-workers who resolved them already so you didn't need to worry about them would agree with me. > In reality none of this is a problem. There are automotive grade cameras which can collect enough light fast enough that the images are not blurry in practice. Yes, these cameras have a non-zero exposure time. Yes, this adds latency. No, this is not a problem. You are contradicting yourself here, yes there are automotive grade cameras, but if this wasn't a problem, why would automotive grade cameras need to exist? My post wasn't saying these were impossible problems but hard ones. > You mean 88 feet. If my off the shelf library introduces a random 1 second delay i will chuck it in the bin post haste. Use stuff whose performance characteristics are well understood by you and is not terrible. Look I might have typed the wrong unit but it's a bit ironic you gave me this word salad right after complaining about this... Yeah you obviously don't use the library that adds a 1 second delay, but often you don't have the luxury of knowing that until after you learn about it through integration testing. Libraries don't usually come with latency stats calibrated to your desired hardware right on the tin, would be pretty sweet if they did though. > I do not recognise the world you describe. I don't find this surprising :)

chrisweekly 1y ago

Awesome post, illuminating metaphor, +1. Tangent: why do so many people (even smart articulate ones like you) in our field (which involves precision with syntax and grammar) get apostrophes wrong? "It's [IT IS] like its [ITS/HIS/HER/MY/YOUR/THEIR] own ecosystem..." I anticipate downvotes for picking nits (let alone mentioning karma points), but FTR my intent is to help non-native English speakers (and mostly-literate English language-natives). Getting this "it's : its" distinction wrong is increasingly common and sometimes leads to signal loss. Yesco, rereading your comment makes me slightly ashamed of this apostrophe rant, I hope others comment on the substance here. / end tangent As someone who spent years doing web performance optimization for a living, your observations resonate. Beyond obvious low-hanging fruit, latency gains are rarely simple to achieve in practice; tradeoffs abound.

yesco 1y ago
Ha yeah it reads a bit silly, to be honest I lazily typed my original post on my phone while sitting on a hammock outside, I didn't really review it much before posting. Autocorrect can make it challenging for me to do apostrophes correctly since it often overassumes my intent.
- chrisweekly 1y ago
  IME autocorrect costs me more time and keystrokes than doing without, so I always disable it (shrug) different strokes...
saratogacx 1y ago
To your tangent. When I was taught its vs it's, I was never given the association to his/hers/etc. It was just arbitrary and something you just had to memorize. Reading your post I am shocked I never made the connection earlier but it was just never knowledge I was provided with to work from.

foobarian 1y ago

Honestly I'm just a dumb web dev and even though many of our performance problems can be solved by horizontal scaling, there are a minority of cases where latency is important and that's when things get really fun :-) No more downloading random halfbaked libraries off the web, architecture starts to matter, vm settings start to matter, gateways/load balancers in the middle start to matter, even code optimization starts to matter again just like in good old days.