We never needed to track who was using the web sites since it was only our group and related researchers, so there was just one password and everyone had it. T

We never needed to track who was using the web sites since it was only our group and related researchers, so there was just one password and everyone had it. The data displayed was not very sensitive. If you are concerned about ownership transfer and long term maintenance then using proprietary products is not a good idea, you never know when the company might fold or be bought or decide to do a total revamp of their products. Open source means you at least have a copy of the code. Writing your own analysis code makes it easier to change. If you are using commercial analysis code you never know exactly what it is it does. I've never found a good live analysis product or open source solution, maybe due to the fact that we were a research group and our needs were not typical. For example, we collected a huge amount of data from a campus wireless network over a decade or more. The format sometimes changed as the campus network changed, so multiple custom processors were needed and the analysis we wanted was nothing like what a network manager would want for day to day operations (e.g., tracking the kinds of devices the student population used by semester.) We wrote parsers to extract and clean up the data, automatic feature extractors, we automated the generation of graphs and status data using gnuplot with the output automatically posted as a way to make sure the data collection was functioning ok. We used an open source principal components analysis tool and ran extraction and analysis scripts in batches to automate analysis. You can find some of the details here: https://cs.dartmouth.edu/~dfk/research/project/wifi-measure/index.html https://cs.dartmouth.edu/~dfk/research/project/wifi-measure/... Amazon Kinesis looks interesting, it didn't exist when I was doing this kind of work. We had to process a lot of video at times which was used as ground truth for training ML algorithms (e.g., to detect when a person was eating). The automated tools at the time for processing video were not capable of that kind of determination so we used Mechanical Turk (or a Chinese equivalent). Kinesis does not seem capable of doing something like that, you'd still have to write your own ML algorithm to extract features (eating, not eating) and if you trust that well enough to be ground truth for training a ML algorithm why even use it to generate training data, just use it directly. For data streams Kinesis looks like it might make ingestion easier, but it is still up to you to create the data analysis tools. If you can use their analytics, then great, but if you are doing something new you'll still probably end up writing your own analysis code. And then you are tied to Amazon and are paying them for CPU time, network traffic, storage, etc. AWS and related services are probably going to be stable long term to some extent. Last time (6 years ago?) I considered using AWS for dealing with a lot of streaming data the network data transfer costs were too much for us. We would have had to move data in and out of AWS often and it would have cost us more than our entire grant funding.