4 ms·

Measuring a sensor is so complicated...Influx, Grafana and Docker is not enough. You may want to throw in AWS, some Google API and not to forget: Tensorflow. A

Measuring a sensor is so complicated...Influx, Grafana and Docker is not enough. You may want to throw in AWS, some Google API and not to forget: Tensorflow. A pinch of opencv with GPU support might make it real software. Cross compile all this for windows that runs on RPi so corporate clients can trust it -- because buying a license always reflects seriousness.

Ysx 8y ago
I'd have solved it same way as the article, as they're tools I'm familiar with - feels like overkill though. How would you have managed it?
- IanSanders 8y ago
  Arduino nano, a few sensors, display, and a few buttons
- mschaef 8y ago
  Not OP, but here's how I did solve it: https://github.com/mschaef/metlog https://github.com/mschaef/metlog This was mainly an exploratory thing to learn some new technologies, but the gist is that it's a Clojure/Clojurescript application with HSQLDB on the back end. It runs reliably and has for years, but the more 'roll your own' experience means continually finding out that there's some other bit of missing functionality you either need to implement or live without. A couple specific observations on various elements of the tech stack: * HSQLDB - Nice little database, well documented, and has served its purpose well. Just now getting to the point where query lengths on long running time series are a performance issue. I think my mitigation strategy will be to do more in memory. * core.async - I use this in the front end to manage sourcing data from the server and getting it into the graphing components. Three or four different approaches later, I'm still not sure I fully get how or why this should be used. * Reagent - Definitely the right choice. About the perfect level of abstraction. Wish I'd spent a bit more time reading the docs ahead of time, though. * Clojure - This has been a great choice, but to be honest, there's not much code on the server side, so it's not at all heavily used. * ClojureScript - Also a good choice (Figwheelr helps), but it's fundamentally a different language with different core data structures sitting in a land that's solidly entrenched in JavaScript. That impedence mismatch is a continual low grade annoyance. * HTML Canvas - Easy to use, but lots of tricky little edge cases to worry about. (The latest being how to find a way to get non-blurry pixel accurate display on a Retina laptop.)
  - dmix 8y ago
    So you'd design your own UI + graphs?
    - mschaef 8y ago
      I did. It's simple, but intentionally so - essentially just a linear list of stripcharts and some basic formatting. The general idea was that in steady-state operation, the only thing that should really be seen is the data of interest. Honestly, the thing that's been the most time-consuming is getting data transported efficiently and quickly to the front end for rendering. The rendering itself hasn't been a huge deal, and is essentially the one part of the system where I haven't thought seriously about replacing it with something 'real'. (The stripchart plot code itself is here, and I haven't touched it in three years. https://github.com/mschaef/metlog/blob/master/metlog-vault/src-client/metlog_client/tsplot.cljs https://github.com/mschaef/metlog/blob/master/metlog-vault/s... )
scrollaway 8y ago
What is this, off-brand sarcasm? > Measuring a sensor is so complicated Well, you need something to ingest & query the data. InfluxDB or Prometheus is a good choice. You need something compatible to display it. Grafana is perfect for that. Docker is a pretty simple and decently-lightweight choice to standardize how this installs and runs. And letting it run on a Pi makes it cheap and accessible. What exactly is the problem with this setup? Not enough NIH?
- moosingin3space 8y ago
  Also, one very nice benefit this has is that balenaOS has safe, automatic updating of the base system.
mschaef 8y ago
I've built one of these types of systems without a lot of that technology stack. I have my own dashboard, use in-process HSQLDB for the back end, and have a deployment process build around uberjars, shell scripts, and init scripts. https://github.com/mschaef/metlog https://github.com/mschaef/metlog There are reasons I made the choices I did, and I think they still make sense, but there's been enough of various sorts of implementation pain that I am absolutely sympathetic to the idea that this person used some off the shelf software. You should probably be more aware of your target's constraints and goals before you start questioning the means they used to achieve them.
crispyambulance 8y ago
> Measuring a sensor is so complicated... I get the sarcasm, but sometimes it is useful to exercise "big boy" tools for projects that don't need it. What better way is there to learn/explore these things than with a very simple project that does something remotely interesting or useful?
- MisterTea 8y ago
  > "big boy" tools I read that as "big boy stools".
- tarsinge 8y ago
  I think on the contrary we need to seriously reconsider these tools that don't make sense at the scale of the vast majority of projects. It's sad we've gone full circle to the 00's, didn't miss those Java EE days of real big boy tools. Edit: current real-life example: I am currently developing a core tool for helping business users of a big industrial group. Client is amazed I was able to quickly develop a custom tool to answer their business needs. Code repo: 3 files: one HTML page, one vanilla JS file (less than a thousand lines), one lib to open XLSX files in JS.
  - crispyambulance 8y ago
    Yeah, I remember the bad old days. The most spectacular P.O.S. from those days, for me, was SOAP. It makes me clench just thinking about it and it's wildly ironic name "SIMPLE object access protocol." What, like there are even more complicated protocols to transfer data? FML! That said, sometimes you need/want to pick up some skills. I think it's good to do these on tractable problems that one can imagine scaling up smoothly, like this one. It's better than overly simple "hello world" and also better than trying to learn from a project where the technology is appropriately scaled and it would take weeks just to figure out what's going on.
rlonn 8y ago
If you don't feel like setting up and maintaining your own data storage & visualisation solution for your time series data, you could use adafruit.io or blynk.io. And if you're really lazy or impatient you can try my ridiculously-low-friction service pushdata.io :)
nakkaya 8y ago
At work one of the projects I work on is the telemetry system for a solar race car. I have pretty much the exact setup. We grab the data from the CAN bus transmit it over the radio to the chase vehicle. Data needs to be logged so you can create/test algorithms for predicting optimum speed for the car and also run the algorithms during the race. We don't use PI but a headless linux box will receive all data. During the race both the electrical and mechanical teams needs to see this data. Grafana pools the DB and everyone with a PC, Smart phone or tablet has access to the telemetry. Computers break if the server on the chase vehicle (centos box) breaks I will have reasonable confidance that I can run the telemetry system on my laptop (ubuntu box) with a couple of docker commands. So on top of OP I have a pinch of Clojure, Matlab, Postgres.

Ysx 8y ago

I'd have solved it same way as the article, as they're tools I'm familiar with - feels like overkill though. How would you have managed it?

IanSanders 8y ago
Arduino nano, a few sensors, display, and a few buttons
mschaef 8y ago
Not OP, but here's how I did solve it: https://github.com/mschaef/metlog https://github.com/mschaef/metlog This was mainly an exploratory thing to learn some new technologies, but the gist is that it's a Clojure/Clojurescript application with HSQLDB on the back end. It runs reliably and has for years, but the more 'roll your own' experience means continually finding out that there's some other bit of missing functionality you either need to implement or live without. A couple specific observations on various elements of the tech stack: * HSQLDB - Nice little database, well documented, and has served its purpose well. Just now getting to the point where query lengths on long running time series are a performance issue. I think my mitigation strategy will be to do more in memory. * core.async - I use this in the front end to manage sourcing data from the server and getting it into the graphing components. Three or four different approaches later, I'm still not sure I fully get how or why this should be used. * Reagent - Definitely the right choice. About the perfect level of abstraction. Wish I'd spent a bit more time reading the docs ahead of time, though. * Clojure - This has been a great choice, but to be honest, there's not much code on the server side, so it's not at all heavily used. * ClojureScript - Also a good choice (Figwheelr helps), but it's fundamentally a different language with different core data structures sitting in a land that's solidly entrenched in JavaScript. That impedence mismatch is a continual low grade annoyance. * HTML Canvas - Easy to use, but lots of tricky little edge cases to worry about. (The latest being how to find a way to get non-blurry pixel accurate display on a Retina laptop.)
- dmix 8y ago
  So you'd design your own UI + graphs?
  - mschaef 8y ago
    I did. It's simple, but intentionally so - essentially just a linear list of stripcharts and some basic formatting. The general idea was that in steady-state operation, the only thing that should really be seen is the data of interest. Honestly, the thing that's been the most time-consuming is getting data transported efficiently and quickly to the front end for rendering. The rendering itself hasn't been a huge deal, and is essentially the one part of the system where I haven't thought seriously about replacing it with something 'real'. (The stripchart plot code itself is here, and I haven't touched it in three years. https://github.com/mschaef/metlog/blob/master/metlog-vault/src-client/metlog_client/tsplot.cljs https://github.com/mschaef/metlog/blob/master/metlog-vault/s... )

scrollaway 8y ago

What is this, off-brand sarcasm? > Measuring a sensor is so complicated Well, you need something to ingest & query the data. InfluxDB or Prometheus is a good choice. You need something compatible to display it. Grafana is perfect for that. Docker is a pretty simple and decently-lightweight choice to standardize how this installs and runs. And letting it run on a Pi makes it cheap and accessible. What exactly is the problem with this setup? Not enough NIH?

moosingin3space 8y ago
Also, one very nice benefit this has is that balenaOS has safe, automatic updating of the base system.

mschaef 8y ago

I've built one of these types of systems without a lot of that technology stack. I have my own dashboard, use in-process HSQLDB for the back end, and have a deployment process build around uberjars, shell scripts, and init scripts. https://github.com/mschaef/metlog https://github.com/mschaef/metlog There are reasons I made the choices I did, and I think they still make sense, but there's been enough of various sorts of implementation pain that I am absolutely sympathetic to the idea that this person used some off the shelf software. You should probably be more aware of your target's constraints and goals before you start questioning the means they used to achieve them.

crispyambulance 8y ago

> Measuring a sensor is so complicated... I get the sarcasm, but sometimes it is useful to exercise "big boy" tools for projects that don't need it. What better way is there to learn/explore these things than with a very simple project that does something remotely interesting or useful?

MisterTea 8y ago
> "big boy" tools I read that as "big boy stools".
tarsinge 8y ago
I think on the contrary we need to seriously reconsider these tools that don't make sense at the scale of the vast majority of projects. It's sad we've gone full circle to the 00's, didn't miss those Java EE days of real big boy tools. Edit: current real-life example: I am currently developing a core tool for helping business users of a big industrial group. Client is amazed I was able to quickly develop a custom tool to answer their business needs. Code repo: 3 files: one HTML page, one vanilla JS file (less than a thousand lines), one lib to open XLSX files in JS.
- crispyambulance 8y ago
  Yeah, I remember the bad old days. The most spectacular P.O.S. from those days, for me, was SOAP. It makes me clench just thinking about it and it's wildly ironic name "SIMPLE object access protocol." What, like there are even more complicated protocols to transfer data? FML! That said, sometimes you need/want to pick up some skills. I think it's good to do these on tractable problems that one can imagine scaling up smoothly, like this one. It's better than overly simple "hello world" and also better than trying to learn from a project where the technology is appropriately scaled and it would take weeks just to figure out what's going on.

rlonn 8y ago

If you don't feel like setting up and maintaining your own data storage & visualisation solution for your time series data, you could use adafruit.io or blynk.io. And if you're really lazy or impatient you can try my ridiculously-low-friction service pushdata.io :)

nakkaya 8y ago

At work one of the projects I work on is the telemetry system for a solar race car. I have pretty much the exact setup. We grab the data from the CAN bus transmit it over the radio to the chase vehicle. Data needs to be logged so you can create/test algorithms for predicting optimum speed for the car and also run the algorithms during the race. We don't use PI but a headless linux box will receive all data. During the race both the electrical and mechanical teams needs to see this data. Grafana pools the DB and everyone with a PC, Smart phone or tablet has access to the telemetry. Computers break if the server on the chase vehicle (centos box) breaks I will have reasonable confidance that I can run the telemetry system on my laptop (ubuntu box) with a couple of docker commands. So on top of OP I have a pinch of Clojure, Matlab, Postgres.