4 ms·

Espargos: ESP32-based WiFi sensing array

199 points by leoedin 2y ago

leoedin 2y ago
This project uses an array of ESP32s and some clever calibration techniques to do pretty accurate localisation of WiFi signals. There's an excellent companion video here: https://www.youtube.com/watch?v=sXwDrcd1t-E https://www.youtube.com/watch?v=sXwDrcd1t-E
- transpute 2y ago
  This educational visualization will be helpful as WiFi 7 Sensing comes to consumer routers and NPU/AI laptops, rendering home and business walls transparent to human activity like typing, breathing & more. "Inside a $1 radar motion sensor", 100 comments, https://news.ycombinator.com/item?id=40834349 https://news.ycombinator.com/item?id=40834349 https://www.cnx-software.com/2022/08/08/esp-wifi-csi-detects-humans-with-wifi-signals-only-no-sensor-needed/ https://www.cnx-software.com/2022/08/08/esp-wifi-csi-detects... > Espressif claims it can also capture subtle movements caused by small movements such as breathing and chewing of people or animals in a static environment.. works with all ESP32 series microcontrollers including ESP32, ESP32-S2, and ESP32-C3, and does not require any changes to the hardware 24Ghz/60Ghz mmWave sensors for finer resolution, https://www.dfrobot.com/product-2795.html https://www.dfrobot.com/product-2795.html
  - zimpenfish 2y ago
    I've got a 60Ghz mmWave sensor mounted above my bed (at the suggested distance and orientation) and it has been terrible at detecting sleep / heart rate / breathing. Reasonably good at detecting presence though. Maybe there's some tuning/calibration can be done but I've not seen anything about that.
    - transpute 2y ago
      RF sensor has to be coupled with activity-specific ML models for detection. Google Scholar has hundreds of academic papers on wireless sensing with machine learning, targeting specific activities. Commercially, Google Nest Hub 2nd gen includes mmWave sensor for sleep tracking.
Ecco 2y ago
Very cool! Website bug report (hoping somebody sees that): on my phone (iPhone latest iOS) the videos would automatically play in full screen when scrolling the page. Really annoying :)
- saaaaaam 2y ago
  Same for me. Though I just assumed it was some sort of annoying “feature”.
- Jeija 2y ago
  Oops, never tested the website on Safari / iOS. I will look into this at some point, opened a GitHub issue for myself so that I will remember: https://github.com/ESPARGOS/espargos.github.io/issues/1 https://github.com/ESPARGOS/espargos.github.io/issues/1
tonetegeatinst 2y ago
Is this locked to WiFi spectrum, or could the antennas be swappen for whatever frequency band you want to target? Say starlink frequency for tracking or The NOAA satellites?
- gfkclzhzo 2y ago
  ESP32 will be just 2.4ghz, and the bandwidth of the antennas will also limit. Totally doable to make a wideband version with different hardware. This is called a phased array, which is common in radar applications (e.g. Aegis cruisers).
  - leptons 2y ago
    Espressif has a new dual-band chip that does 2.4ghz and 5ghz wifi. That isn't the subject of this article, but ESP32 is gaining some new wifi capabilities.
    - Jeija 2y ago
      I'm currently experimenting with ESP32-C5 engineering samples provided by Espressif, so hopefully some future ESPARGOS revision will be 5GHz-capable :) But yeah, as far as I can tell, the chip can only extract CSI from WiFi (unless there is some hidden raw I/Q sample debug mode or something).
      - ipdashc 2y ago
        This project is amazing! I know there's some prior art, but I feel like I've never seen anything wifi-related moving so... smoothly? (With that high of an update/sample rate, I guess?) as your visualizations do. I do have one question that wasn't quite clear to me, in case you happen to see this. There's a bunch of mentions of Channel State Information, but Google doesn't give a ton of info on it. So I'm just wondering, is CSI a "client-side" thing provided by the ESP32 for any Wi-Fi connection, or is it some special protocol feature that the AP or client have to enable?
SEJeff 2y ago
I’d note that Xandem while a total flop commercially, does own patents on tomographic motion detection. It’s a shame honestly, it was a really cool product.
deleted 2y ago
[deleted]
ChrisArchitect 2y ago
Earlier video: https://www.youtube.com/watch?v=sXwDrcd1t-E https://www.youtube.com/watch?v=sXwDrcd1t-E (https://news.ycombinator.com/item?id=43062982 https://news.ycombinator.com/item?id=43062982)
slicktux 2y ago
Hmm, seems like I’ll be able to replace my cantenna with this to find malicious IoT devices spamming the airwaves with DeAuth attacks…! Imagine being able to catch crooks using Flipper Zeros to defeat wireless cameras…can track them in real time ;)
- MisterTea 2y ago
  > Imagine being able to catch crooks using Flipper Zeros to defeat wireless cameras…can track them in real time ;) One of the reasons I run Ethernet to everything. Also, I assume you work in security if you deal deal with malicious WiFi devices and sophisticated crooks?
  - slicktux 2y ago
    Wired is definitely the best! But sometimes wireless is the most feasible and quickest way.
- adfm 2y ago
  +1 for cantenna reference.
newman314 2y ago
I wonder if this can be used to aid in per room presence detection.
- Havoc 2y ago
  24ghz radar sensors are pretty damn accurate too (way better than the infra red based ones). About 10 bucks off aliexpress
- transpute 2y ago
  e.g. count people watching TV for pay-per-view billing, https://news.ycombinator.com/item?id=43076335 https://news.ycombinator.com/item?id=43076335
jbl0ndie 2y ago
That is very nice! It reminds me of Usman Haque's WiFi Camera project from 2006. That wasn't anywhere near realtime but it did use wasabi-pea cans https://haque.co.uk/work/wifi-camera/ https://haque.co.uk/work/wifi-camera/ I miss tech art that doesn't take itself too seriously.
pfyy 2y ago
The hackster article has a bit more technical infos: https://www.hackster.io/news/exploring-the-invisible-world-with-the-esp32-b0a72365a967 https://www.hackster.io/news/exploring-the-invisible-world-w...
janmo 2y ago
I am very interested in knowing how they are able to get such precise measurements with an ESP32. In their chart there is a time of arrival difference that has nano second accuracy. The ESP32 runs at 240MHZ max, and usually it takes several cycles to read certain values, so I really wonder how they were able to pull it off. Please enlighten me.
- sigmaprimus 2y ago
  Just a gas but maybe it's because they're using multiple esp32's in an array? But also I seem to recall something similar in the Snowden papers? But I get your point, given that the esp32 and all its design files are available from SPRESIF maybe they built a board that's running out of much faster clock rate? PS: sorry about the typos and miss takes but sadly I have to use Voice typing due to a recent traumatic injury.
- bobmcnamara 2y ago
  > The ESP32 runs at 240MHZ max Luckily for 802.11mc, host CPU speed is not used in the distance estimation.
- rollulus 2y ago
  As shared elsewhere in this thread: https://www.youtube.com/watch?v=sXwDrcd1t-E https://www.youtube.com/watch?v=sXwDrcd1t-E TL;DW: the ESP’s WiFi driver provides per incoming packet the carrier’s phase and amplitude. All ESPs are calibrated relatively to each other.
- Jeija 2y ago
  Every WiFi chip needs to be able to synchronize to the received signal down to a few I/Q samples (the remaining synchronization error is accounted for by the cyclic prefix of OFDM). For example, with 40MHz of bandwidth, there is an I/Q sample every 25 nanoseconds (the wave travels 7.5m during that time), so the level of synchronization will be on that order. The ESP32 reports the time of arrival, the code for extracting a nanosecond-precision timestamp is here: https://github.com/ESPARGOS/pyespargos/blob/main/espargos/pool.py#L255 https://github.com/ESPARGOS/pyespargos/blob/main/espargos/po... Additionally, you can use the phase information in the estimated channel coefficients (CSI) to determine the more precise time of arrival. Very briefly, a time delay in time domain will be visible as a frequency(=subcarrier)-dependent phase shift in frequency(=subcarrier)-domain. Now synchronization is only limited by impairments like thermal noise and phase noise. In practice in nice lab conditions, the time of arrival accuracy is on the level of <<1m. On top of phase differences between subcarriers (--> timing information), we also measure phase differences between antennas, which provides angle of arrival and, with multiple arrays, phase of arrival information. You can go even further by measuring phase differences between subsequent packets, which provides frequency offset and Doppler (--> velocity) information. I have some nice results on that, but that's a topic for the future ;)
gonzagued 2y ago
[dead]

199 points by leoedin 2y ago

This project uses an array of ESP32s and some clever calibration techniques to do pretty accurate localisation of WiFi signals. There's an excellent companion video here: https://www.youtube.com/watch?v=sXwDrcd1t-E https://www.youtube.com/watch?v=sXwDrcd1t-E

transpute 2y ago
This educational visualization will be helpful as WiFi 7 Sensing comes to consumer routers and NPU/AI laptops, rendering home and business walls transparent to human activity like typing, breathing & more. "Inside a $1 radar motion sensor", 100 comments, https://news.ycombinator.com/item?id=40834349 https://news.ycombinator.com/item?id=40834349 https://www.cnx-software.com/2022/08/08/esp-wifi-csi-detects-humans-with-wifi-signals-only-no-sensor-needed/ https://www.cnx-software.com/2022/08/08/esp-wifi-csi-detects... > Espressif claims it can also capture subtle movements caused by small movements such as breathing and chewing of people or animals in a static environment.. works with all ESP32 series microcontrollers including ESP32, ESP32-S2, and ESP32-C3, and does not require any changes to the hardware 24Ghz/60Ghz mmWave sensors for finer resolution, https://www.dfrobot.com/product-2795.html https://www.dfrobot.com/product-2795.html
- zimpenfish 2y ago
  I've got a 60Ghz mmWave sensor mounted above my bed (at the suggested distance and orientation) and it has been terrible at detecting sleep / heart rate / breathing. Reasonably good at detecting presence though. Maybe there's some tuning/calibration can be done but I've not seen anything about that.
  - transpute 2y ago
    RF sensor has to be coupled with activity-specific ML models for detection. Google Scholar has hundreds of academic papers on wireless sensing with machine learning, targeting specific activities. Commercially, Google Nest Hub 2nd gen includes mmWave sensor for sleep tracking.

Ecco 2y ago

Very cool! Website bug report (hoping somebody sees that): on my phone (iPhone latest iOS) the videos would automatically play in full screen when scrolling the page. Really annoying :)

saaaaaam 2y ago
Same for me. Though I just assumed it was some sort of annoying “feature”.
Jeija 2y ago
Oops, never tested the website on Safari / iOS. I will look into this at some point, opened a GitHub issue for myself so that I will remember: https://github.com/ESPARGOS/espargos.github.io/issues/1 https://github.com/ESPARGOS/espargos.github.io/issues/1

tonetegeatinst 2y ago

Is this locked to WiFi spectrum, or could the antennas be swappen for whatever frequency band you want to target? Say starlink frequency for tracking or The NOAA satellites?

gfkclzhzo 2y ago
ESP32 will be just 2.4ghz, and the bandwidth of the antennas will also limit. Totally doable to make a wideband version with different hardware. This is called a phased array, which is common in radar applications (e.g. Aegis cruisers).
- leptons 2y ago
  Espressif has a new dual-band chip that does 2.4ghz and 5ghz wifi. That isn't the subject of this article, but ESP32 is gaining some new wifi capabilities.
  - Jeija 2y ago
    I'm currently experimenting with ESP32-C5 engineering samples provided by Espressif, so hopefully some future ESPARGOS revision will be 5GHz-capable :) But yeah, as far as I can tell, the chip can only extract CSI from WiFi (unless there is some hidden raw I/Q sample debug mode or something).
    - ipdashc 2y ago
      This project is amazing! I know there's some prior art, but I feel like I've never seen anything wifi-related moving so... smoothly? (With that high of an update/sample rate, I guess?) as your visualizations do. I do have one question that wasn't quite clear to me, in case you happen to see this. There's a bunch of mentions of Channel State Information, but Google doesn't give a ton of info on it. So I'm just wondering, is CSI a "client-side" thing provided by the ESP32 for any Wi-Fi connection, or is it some special protocol feature that the AP or client have to enable?

SEJeff 2y ago

I’d note that Xandem while a total flop commercially, does own patents on tomographic motion detection. It’s a shame honestly, it was a really cool product.

deleted 2y ago

[deleted]

ChrisArchitect 2y ago

Earlier video: https://www.youtube.com/watch?v=sXwDrcd1t-E https://www.youtube.com/watch?v=sXwDrcd1t-E (https://news.ycombinator.com/item?id=43062982 https://news.ycombinator.com/item?id=43062982)

slicktux 2y ago

Hmm, seems like I’ll be able to replace my cantenna with this to find malicious IoT devices spamming the airwaves with DeAuth attacks…! Imagine being able to catch crooks using Flipper Zeros to defeat wireless cameras…can track them in real time ;)

MisterTea 2y ago
> Imagine being able to catch crooks using Flipper Zeros to defeat wireless cameras…can track them in real time ;) One of the reasons I run Ethernet to everything. Also, I assume you work in security if you deal deal with malicious WiFi devices and sophisticated crooks?
- slicktux 2y ago
  Wired is definitely the best! But sometimes wireless is the most feasible and quickest way.
adfm 2y ago
+1 for cantenna reference.

newman314 2y ago

I wonder if this can be used to aid in per room presence detection.

Havoc 2y ago
24ghz radar sensors are pretty damn accurate too (way better than the infra red based ones). About 10 bucks off aliexpress
transpute 2y ago
e.g. count people watching TV for pay-per-view billing, https://news.ycombinator.com/item?id=43076335 https://news.ycombinator.com/item?id=43076335

jbl0ndie 2y ago

That is very nice! It reminds me of Usman Haque's WiFi Camera project from 2006. That wasn't anywhere near realtime but it did use wasabi-pea cans https://haque.co.uk/work/wifi-camera/ https://haque.co.uk/work/wifi-camera/ I miss tech art that doesn't take itself too seriously.

pfyy 2y ago

The hackster article has a bit more technical infos: https://www.hackster.io/news/exploring-the-invisible-world-with-the-esp32-b0a72365a967 https://www.hackster.io/news/exploring-the-invisible-world-w...

janmo 2y ago

I am very interested in knowing how they are able to get such precise measurements with an ESP32. In their chart there is a time of arrival difference that has nano second accuracy. The ESP32 runs at 240MHZ max, and usually it takes several cycles to read certain values, so I really wonder how they were able to pull it off. Please enlighten me.

sigmaprimus 2y ago
Just a gas but maybe it's because they're using multiple esp32's in an array? But also I seem to recall something similar in the Snowden papers? But I get your point, given that the esp32 and all its design files are available from SPRESIF maybe they built a board that's running out of much faster clock rate? PS: sorry about the typos and miss takes but sadly I have to use Voice typing due to a recent traumatic injury.
bobmcnamara 2y ago
> The ESP32 runs at 240MHZ max Luckily for 802.11mc, host CPU speed is not used in the distance estimation.
rollulus 2y ago
As shared elsewhere in this thread: https://www.youtube.com/watch?v=sXwDrcd1t-E https://www.youtube.com/watch?v=sXwDrcd1t-E TL;DW: the ESP’s WiFi driver provides per incoming packet the carrier’s phase and amplitude. All ESPs are calibrated relatively to each other.
Jeija 2y ago
Every WiFi chip needs to be able to synchronize to the received signal down to a few I/Q samples (the remaining synchronization error is accounted for by the cyclic prefix of OFDM). For example, with 40MHz of bandwidth, there is an I/Q sample every 25 nanoseconds (the wave travels 7.5m during that time), so the level of synchronization will be on that order. The ESP32 reports the time of arrival, the code for extracting a nanosecond-precision timestamp is here: https://github.com/ESPARGOS/pyespargos/blob/main/espargos/pool.py#L255 https://github.com/ESPARGOS/pyespargos/blob/main/espargos/po... Additionally, you can use the phase information in the estimated channel coefficients (CSI) to determine the more precise time of arrival. Very briefly, a time delay in time domain will be visible as a frequency(=subcarrier)-dependent phase shift in frequency(=subcarrier)-domain. Now synchronization is only limited by impairments like thermal noise and phase noise. In practice in nice lab conditions, the time of arrival accuracy is on the level of <<1m. On top of phase differences between subcarriers (--> timing information), we also measure phase differences between antennas, which provides angle of arrival and, with multiple arrays, phase of arrival information. You can go even further by measuring phase differences between subsequent packets, which provides frequency offset and Doppler (--> velocity) information. I have some nice results on that, but that's a topic for the future ;)

gonzagued 2y ago

[dead]