5 ms·

Not trying to be too negative here, but why wasn't a broad association analysis done on this dataset first? The number of outages correlated with train activit

Not trying to be too negative here, but why wasn't a broad association analysis done on this dataset first? The number of outages correlated with train activity should have popped up like a sore thumb, no? Hindsight is always twenty-twenty, but I feel the author just wanted to show of his cool visualization knowledge, while the same could have been achieved with a much more boring association table. EDIT: I am corrected by "Rifu". Activity data of all traincars was not available in the first instance, so a broad association analysis would probably have yielded nothing. The type of analysis the authors use seems warranted. Although I'd still recommend doing broad association analysis on any dataset, before moving into complicated visualization techniques.

folli 10y ago
I'm not sure if I understand correctly. I assume that the trains are in service the majority of the day. Wouldn't that make it very hard to correlate activity of a single train to the outages?
- deugtniet 10y ago
  Not really, I believe the article states that 95% of the outages could be attributed to the single faulty train. That's a really big effect. Correlating traincar activity to outages will only NOT work when the traincar is active on the same track all the time, which it clearly isn't
deleted 10y ago
[deleted]
Rifu 10y ago
As per TFA, they did say that the only data they had on hand were from the trains that broke down, so it would have been impossible to even detect the "rogue train" from a simple analysis of the data they had. Though it also said SMRT was slowly extracting the train logs from the incidents so I'm sure they would have gone with that if their preliminary analysis yielded nothing.
- deugtniet 10y ago
  Right I overlooked this. That makes it more meaningful to do the analysis as in the article. I stand corrected.
jaclaz 10y ago
I believe you are actualy "right", the article states "we were given ..." but WHY didn't they ASK for single train activities? These data is normally available in any railway, as it is used for several things, including programmed maintenance. In other words the Authors did IMHO a very interesting analysis of the data they were given, but - had they actually asked for the "right" kind of data - it would have been much simpler and faster. If you prefer they approached the problem from the viewpoint of data analysts (which is what they are, and evidently very good at it) but not from the viewpoint of an accident investigator.
- kalleboo 10y ago
  It's in the article. This all happened in one single day, while the train company was still compiling the extra data.
  - jaclaz 10y ago
    Yes and no. It was done in a single day, OK, but the train usage data is not "extra data" it is something that is "ordinary data" that you have typically for two to four weeks in advance (as a planning) and daily or at the most weekly afterwards, let alone - in November - the actual usage data for August-October.
    - nicky0 10y ago
      You seem very knowledgeable about the data recording system of the Singapore train system. Perhaps they should have called you.
      - jaclaz 10y ago
        Actually I know nothing about the specific Singapore train system, but the basics are "trans-national" and "well established", a metro is an extremely complex structure to manage, but the principles are well established on the experiences of early railways. You have to manage a time table, the personnel and the maintenance of both the rails (and station and power lines, etc. i.e. the "static" parts of the infrastructure) and of the "moving" parts (the trains). In order to do so you have "allocation" tables, usually made in advance two or four weeks at least, stating "who" (personnel) and "what" (train) are "where". These "programs" are monitored and changes to it (think of people not showing up at work, a train having a malfunction, etc.) modified to allow for these impredictable events. So at the end of the day (each day) you had a printed piece of paper with modifications - if any - scribbled on it. Nowadays the same thing is done on computers, possibly in a much more detailed way, but the basic data "which train was in service on which line at what time" has been available on paper since day one (or two) of any railway in the world. While I can (barely) understand how these basic data for the 5th of November was not available on the 5th night, there is no real reason why data since August and until - say - the 4th of november was not available. As said, the data analysis carried on the data provided by SMRT has been carried on in a clever way, as finding the culprit from that set of data would have not been at all easy through other simpler analysis methods, but somehow SMRT (and LTA) provided the "wrong" set of data.
Hnrobert42 10y ago
Is there another name for broad association analysis? I have been unsuccessful searching for more information on it.

folli 10y ago

I'm not sure if I understand correctly. I assume that the trains are in service the majority of the day. Wouldn't that make it very hard to correlate activity of a single train to the outages?

deugtniet 10y ago
Not really, I believe the article states that 95% of the outages could be attributed to the single faulty train. That's a really big effect. Correlating traincar activity to outages will only NOT work when the traincar is active on the same track all the time, which it clearly isn't

deleted 10y ago

[deleted]

Rifu 10y ago

As per TFA, they did say that the only data they had on hand were from the trains that broke down, so it would have been impossible to even detect the "rogue train" from a simple analysis of the data they had. Though it also said SMRT was slowly extracting the train logs from the incidents so I'm sure they would have gone with that if their preliminary analysis yielded nothing.

deugtniet 10y ago
Right I overlooked this. That makes it more meaningful to do the analysis as in the article. I stand corrected.

jaclaz 10y ago

I believe you are actualy "right", the article states "we were given ..." but WHY didn't they ASK for single train activities? These data is normally available in any railway, as it is used for several things, including programmed maintenance. In other words the Authors did IMHO a very interesting analysis of the data they were given, but - had they actually asked for the "right" kind of data - it would have been much simpler and faster. If you prefer they approached the problem from the viewpoint of data analysts (which is what they are, and evidently very good at it) but not from the viewpoint of an accident investigator.

kalleboo 10y ago
It's in the article. This all happened in one single day, while the train company was still compiling the extra data.
- jaclaz 10y ago
  Yes and no. It was done in a single day, OK, but the train usage data is not "extra data" it is something that is "ordinary data" that you have typically for two to four weeks in advance (as a planning) and daily or at the most weekly afterwards, let alone - in November - the actual usage data for August-October.
  - nicky0 10y ago
    You seem very knowledgeable about the data recording system of the Singapore train system. Perhaps they should have called you.
    - jaclaz 10y ago
      Actually I know nothing about the specific Singapore train system, but the basics are "trans-national" and "well established", a metro is an extremely complex structure to manage, but the principles are well established on the experiences of early railways. You have to manage a time table, the personnel and the maintenance of both the rails (and station and power lines, etc. i.e. the "static" parts of the infrastructure) and of the "moving" parts (the trains). In order to do so you have "allocation" tables, usually made in advance two or four weeks at least, stating "who" (personnel) and "what" (train) are "where". These "programs" are monitored and changes to it (think of people not showing up at work, a train having a malfunction, etc.) modified to allow for these impredictable events. So at the end of the day (each day) you had a printed piece of paper with modifications - if any - scribbled on it. Nowadays the same thing is done on computers, possibly in a much more detailed way, but the basic data "which train was in service on which line at what time" has been available on paper since day one (or two) of any railway in the world. While I can (barely) understand how these basic data for the 5th of November was not available on the 5th night, there is no real reason why data since August and until - say - the 4th of november was not available. As said, the data analysis carried on the data provided by SMRT has been carried on in a clever way, as finding the culprit from that set of data would have not been at all easy through other simpler analysis methods, but somehow SMRT (and LTA) provided the "wrong" set of data.

Hnrobert42 10y ago

Is there another name for broad association analysis? I have been unsuccessful searching for more information on it.