6 ms·

Ask HN: Are there any projects to do historical archiving of RF spectrum?

32 points by adamgamble 10y ago

Something like archive.org but for rf spectrum. I'm not even sure if its legal, but I was playing with my SDR the other day and thinking of all the historical information that passes over the airwaves that is ephemeral. Being able to view this data after a major event would be amazing. Imagine listening to all the radio information as 9/11 transpired or during the coup attempt in turkey. Obviously this couldn't be done everywhere all the time, but it would be interesting to be able to archive radio spectrum during major events. A quick google search didn't turn up anything.

deleted 10y ago
[deleted]
wyldfire 10y ago
No but that sounds like a good idea!
rfw1z 10y ago
there are commercial solutions that do this
- shawn81 10y ago
  Such as... ?
pigeons 10y ago
Hard enough to get good coverage, and very hard to get the storage resources. There was a CCC talk about setting up coverage, but that was ambitious enough, archiving was mostly out of scope: http://www.cw-complex.com/rfarray/ http://www.cw-complex.com/rfarray/ http://www.rtl-sdr.com/talk-monitoring-spectrum-building-distributed-rf-scanner-array/ http://www.rtl-sdr.com/talk-monitoring-spectrum-building-dis...
vitovito 10y ago
From my notes on a radio archiving project I'm doing: > Could you record the entire radio spectrum and extract stations and broadcasts later? > In the USA, the AM radio band is 540-1710 kHz, a spread of 1170 kHZ, > http://rtl-sdr.better-than.tv/?page_id=237 http://rtl-sdr.better-than.tv/?page_id=237 states that spectrum recordings are also a function of how many samples per second you choose to record: > 2.8msps - 44.8mbps = 5.6 MB/sec. > 2msps - 32mbps = 4 MB/sec. > 1msps - 16mbps 2 MB/sec. > http://www.myradiobase.de/perseus/ http://www.myradiobase.de/perseus/ has sample files. 3.5 minutes is ~360MB. 60 seconds of 1500 kHZ is a gig. All too much. 24 hours at 1MB/min is 1.5GB, but 17MB/sec. is 1.4TB. Spectrum recordings are out.
- x1798DE 10y ago
  Assuming most of this is dead air, don't you think this would be a highly compressible data set (even lossless)?
  - niftich 10y ago
    Highly is perhaps an overstatement, but compressible like most clear signal data, behaving similar to PCM audio. I compressed '20090922_950_GLFS_outdoor.wav' (150 kHz - 1750 kHz) from the Perseus site [1] with some lossless compressors. In each case, I independently round-tripped the de/compression and the results were confirmed to be lossless. The compression ratios are in line or slightly (but not overwhelmingly) better than what you'd expect [2] from audio: original: 1.000 7zip-lzma2-xz-ultra: 0.820 (not a signal compressor) wavpack-normal: 0.724 wavpack-high: 0.691 wavpack-hhx6: 0.593 I would've liked to test additional compressors. In addition to WavPack, FLAC, TAK, ALAC, OptimFROG, TTA, WMA Lossless all purport to have some level of high-resolution support, but I'm not sure how many kHz they go up to. FLAC wouldn't take the file, and the rest I don't have access to right this moment. [1] http://www.myradiobase.de/perseus/ http://www.myradiobase.de/perseus/ [2] http://wiki.hydrogenaud.io/index.php?title=Lossless_comparison http://wiki.hydrogenaud.io/index.php?title=Lossless_comparis...
  - kastnerkyle 10y ago
    At least in HF, this "dead air" is anything but dead (lightning, channels with lots of fading, things you only catch due to ionospheric effects, etc). Noise is basically any signal you don't care about in lower frequency bands. Also many transmissions look like wideband noise at higher frequency due to direct sequence spreadspectrum type techniques.
derptacious 10y ago
This is super interesting! And if anyone else is curious about creating a project that monitors the range of RF amplitude and frequency at many data points around the world, then send me your contact info! I want to see a record of this spectrum (human and non-human made) in frequency and amplitude to see how much it is (and hopefully "has") changed in physical space. I think interesting things would come out of this data.
- adamgamble 10y ago
  Any idea if its legal? I assume it is but I have no idea.
  - gravypod 10y ago
    Listening is 100% fine in the USA. Transmission without a ham license, and in certain bands, will net you some fines and maybe jail time.
    - vitovito 10y ago
      Any spectrum recordings would almost be guaranteed to include copyrighted content, and in the US distributing those recordings would fall afoul of the DMCA.
      - gravypod 10y ago
        That's not how RF works. The courts have heard cases like this. Look up the talk on CreepyDOL from defcon.
jf 10y ago
The Internet Archive is archiving some of the broadcast TV part of the RF spectrum, which is where this comes from: https://archive.org/details/tv https://archive.org/details/tv
superuser2 10y ago
>all the radio information as 9/11 transpired I know it's not everything, but if you are interested in this the New York Times has an excellent feature using the ATC tapes and some recorded phone lines in the military/air defense system. http://www.nytimes.com/interactive/2011/09/08/nyregion/911-tapes.html?_r=0 http://www.nytimes.com/interactive/2011/09/08/nyregion/911-t...
- adamgamble 10y ago
  Also here is beeper messages that somehow wikileaks got a hold of. https://911.wikileaks.org/ https://911.wikileaks.org/
vesche 10y ago
As vitovito and pigeons mentioned storage and collection locations are the main issues. Things like this exist, but they don't archive data and are designed for military use- Google: Wideband Recording System . They typically are mobile units that focus on fast capture rate and decently large storage (a few terabytes). sigidwiki is worth looking at, but it's more for classification: http://www.sigidwiki.com/wiki/Signal_Identification_Guide http://www.sigidwiki.com/wiki/Signal_Identification_Guide
niftich 10y ago
I find this fascinating because the context of transmissions occurring simultaneously is indeed ephemeral. This data gets lost with traditional archiving, which is squarely considers context (including temporal simultaneity across multiple objects) to be metadata. These sorts of efforts can be partially retrofitted/approximated with timecodes on existing archived material, which may also be a separate, worthwhile endeavor to pursue.
kastnerkyle 10y ago
As vitovito mentions, wideband data in high resolution is incredibly expensive from a storage perspective. There are entities that do it, but this also has the same privacy concerns as saving every packet that comes over a wire - in fact if you record 2.4 MHz and 5 MHz you will be picking up someone's (hopefully encrypted) private data. I don't think it is illegal, but the technical aspects of this are pretty daunting. You would be better off decoding and storing that, but that gets into v& territory pretty quickly, depending on who and what you decode (definitely don't decrypt/crack).
- elbrownos 10y ago
  Google was deemed to have broken wiretapping laws when it scooped up unsecured wifi data while collecting images for Streetview. http://www.bbc.co.uk/news/technology-24047235 http://www.bbc.co.uk/news/technology-24047235
  - throwaway7767 10y ago
    In my country back in the eighties, the TV networks broadcasting scrambled content over the airwaves sued some people who were making a profit modifying receivers to unscramble the broadcasts without entering the numeric key (it was a very weak scheme). The court ruled that no law was broken as the signals were being sent over the air, and therefore had no expectation of privacy. And since at that time the descramblers were sold to the user, there was no problem with modifying the hardware. This is why they switched to leasing the descrambler boxes when they upgraded their system (and chose a system for which descramblers were not easily publicly available). It always seemed a sensible ruling to me. Though I don't think anything has changed legally, I suspect the ruling might be different if such a case were tried today. :/
gravypod 10y ago
Archiving the entire spectrum can range anywhere from infeasible to impossible. First you'd need to specify the frequency ranges you want to listen to. Then you've got a data storage problem which has been covered. After that you've got the propagation problem. Not everything is seen from everywhere. In any event you might want to check out http://www.reversebeacon.net/ http://www.reversebeacon.net/ It's a project that's got a different "simpler" archiving goal. It is just to track spots of people calling CQ in the CW bands.
barrystaes 10y ago
Likely there are top-secret satellites doing exactly that for specific geographic regions. These likely wont retain all gathered data indefinitely however, and even if data is shared its tainted politically.
theknarf 10y ago
The closest thing I know about is websdr (http://websdr.org http://websdr.org)
privong 10y ago
Most of the major radio astronomy observatories maintain archives (e.g., [0]). Of course, they mostly use highly directional dishes pointed at the sky, but local radio signals can create interference[1]. So, those archives provide some probe/record of terrestrial RF spectrum. Of course, those telescopes are generally placed in radio-quiet locations. [0] https://archive.nrao.edu/archive/advquery.jsp https://archive.nrao.edu/archive/advquery.jsp [1] http://www.gb.nrao.edu/IPG/ http://www.gb.nrao.edu/IPG/