4 ms·
I'm always amused when I see the criterion for a "very dense" time series as data being collected more than once per second. In my business (telemetry), we ofte
by engi_nerd 11y ago
I'm always amused when I see the criterion for a "very dense" time series as data being collected more than once per second. In my business (telemetry), we often record parameters thousands of times per second, depending on what we are trying to measure.
- tsdbase 11y agoCan you be more specific? Is it car engine telemetry?
- engi_nerd 11y agoAircraft telemetry.
- maaku 11y agoYou really need to know the position of an aircraft to a thousandth of a second?
- engi_nerd 11y agoI said nothing about position. But a lot of things measured have to do with how the structure of the aircraft responds to turbulence, rough air, and aeroacoustic vibration (aka flutter). So there might be modes where structural components have harmonics that are pretty high (several hundred to over 1000 Hz). Therefore you must use a transducer that has a frequency response that can cover that range, and sample the output of the transducer at least twice that rate (at an absolute theoretical minimum, but rule of thumb is 5x oversampling).
- akavel 11y agoPossibly he might measure some other thing, but still after your question I started to wonder how much a passenger plane moves per thousandth of a second. So: some quick googling suggests that "economical cruising speed" of an Airbus A320 is 840km/h [1]. After quick back-of-envelope calculations, this gives ~230m/s, so 0.2m per 0.001s. Given some possible uncertainty of a single measurement, I'd imagine that's not unreasonable level of precision when e.g. your Airbus is landing on an airport. [1]: http://www.airliners.net/aircraft-data/stats.main?id=23 http://www.airliners.net/aircraft-data/stats.main?id=23
- endymi0n 11y agoNot that funny anymore if you try to record 1MM time series at once. :)
- engi_nerd 11y agoI'm not anywhere in that realm because I don't have a data system that could support such a large amount of data. But then, I also have a physical system that does not need observations of 1 million distinct things (at least, not yet).
- nether 11y agoDo you use a database for your data? We store our flight test data as individual DATAQ files with a naming scheme. This is manageable for our very simple civil aircraft so far.
- engi_nerd 11y agoAh, a fellow flight test person. I don't handle the warehousing of data, so my knowledge is limited. I know a database is involved and used but I could not say what database or provide any specifics. Sorry. EDIT: I would like to know how other flight test people visualize their test data. Seems like you can only make so many envelope charts and strip charts (they work and are effective, but man, sometimes I'd like something different)
- nether 11y agoAll we use are strip charts. It's all I saw at Boeing as well (through some proprietary charting software IIRC). What I'd like is some "rich" time series data format that stores a history of modifications (removing noise, filtering etc.), and an overall better interface for scrolling through the data (Windaq is pretty barebones).
- engi_nerd 11y agoWow, that is pretty rough. That's so rough that I would consider figuring out some way to export your data from that software and then visualize it with python/matplotlib or similar.
- nether 11y agoI've been working on a Python module to read the files directly: https://github.com/choj/cldumpread/tree/master/WindaqReader/src https://github.com/choj/cldumpread/tree/master/WindaqReader/.... Scipy/numpy/matplotlib totally opens up what we can do with the data. The hard part is convincing a mechanical engineering manager who's extremely averse to new tools.
- kbenson 11y agoThe collection time is only a single dimension, to see the scope you also need to know how many different pieces of data are being collected and for how many discrete groupings. Compare 100 inputs for a single device 1000 times a second to 100 inputs times 1000 devices once a second.
- engi_nerd 11y agoFair enough. In the telemetry world the second case you mention is rarely seen. I'm sure someone, somewhere is doing something like that, so I won't say never. In flight testing you have anywhere from a few measurements to thousands of measurements gathered from physical transducers, and then digital data bus information that can vary dramatically from platform to platform, from just a few kilobytes/sec to gigabytes/sec.
- kbenson 11y agoThere's many, many cases where you have lots of discrete inputs but a lower time threshold. In/out bytes per port on switches in a datacenter. CPU/memory/disk stats on virts you spin up for on-demand load, or just your normal datacenter deployment. Employees/vehicle locations in the field. Often you don't even need per-second time granularity, but that can be made up for with a larger number of discrete inputs (how many trucks does UPS have in service?)
- engi_nerd 11y agoIndeed, those cases are normal in other industries, but not in mine.
- elektronaut 11y agoI've got a system here capable of collecting data close to a hundred thousand times per second from tens of sources, with perceptual real time (< 10ms) processing and aggregate monitoring. It's called multitrack audio recording, and commonly runs on your run of the mill laptop. HD video would probably be a few orders of magnitude more data than that, with even more processing. Computers are really, really good at these kinds of things if only the software is efficient enough.
- engi_nerd 11y agoYour last sentence is well taken, but the rest leaves me a bit puzzled about what you meant. Your run of the mill laptop is a wonderful machine capable of amazing things. But that laptop is not rated to withstand extreme vibration and shock and huge temperature ranges. Nor does it have the external interfaces to deal with large numbers of custom, high speed digital data buses, nor does it contain the signal conditioning hardware to deal with a huge array of transducers working with differing physical phenomena, each with its own unique power and signal processing requirements. The system I support has to deal with all of these things, plus more constraints than are relevant to the conversation here. Much (but not all!) of this doesn't even require a general purpose computer. It can be done with state machines implemented in FPGAs. Sometimes you need something different than a run of the mill laptop.
- elektronaut 11y agoI apologize if it came across as a snarky comment or one-upmanship, I merely intended to support your original comment.
- engi_nerd 11y agoNo problem. Like I said, you do have a very good point. I am often frustrated by the realization that my laptop is much more powerful than this specialized data collection equipment. But we have a very large fixed base of equipment, with no plans to upgrade anytime soon.