5 ms·
I used to work at Standard and Poors. They do media monitoring for lots of financial companies. When I joined, I thought they using state of the art tools to m
by AznHisoka 9y ago
I used to work at Standard and Poors. They do media monitoring for lots of financial companies.
When I joined, I thought they using state of the art tools to monitor millions of sources.
Turns out they just had a team of thousands in India that manually visited certain websites to check for press releases, transcripts, earnings reports, etc and push them to a database if they found them.
- sharemywin 9y agowonder if that could be crawled and save money.
- bm1362 9y agoWoah there cowboy
- sjg007 9y agoor just set google alerts.
- dismantlethesun 9y agoAccuracy. Every solution I've seen that relies on automatic crawling will eventually have a parsing error when someone changes their sentence structure of a press release. It's not so obvious when you're looking at the breaking releases for a few stocks or companies, but historical records have at least 1 error per stock per year.
- platz 9y agoright, it depends on what the reward for high precision vs high recall is
- greggyb 9y agoSo split your stream: 1. Data matching expectations (you do have a definition of correct, right?) 2. Log for manual review -> manual inserts or correction and placed into queue for (1) Monitor (2). When inserts start trending up, it may be time to update your processing logic.
- flukus 9y agoI came up with a similar idea for a company several years ago where we had a team of people doing data entry from faxed documents. I wanted to build something that would do all the OCR it could and then display it to users to verify, which should have been a 10 times efficiency increase, not to mention speed and accuracy. The idea was rejected, they wanted either a perfect solution or nothing. I don't know why, but for some reason the idea computers removing humans is acceptable to management, but computers augmenting humans wasn't.
- thehardsphere 9y agoWould writing and maintaining the crawler be less expensive than paying the small army of third world employees to manually check pages all day?
- DigDugDoug 9y agoThe short answer is that it would depend on the value of any precision loss that occurred and whether or not the shift would disrupt any other systems, be they social or technical, within the company. But with that in mind, there's also the issue of whether or not the money saved would be worth it. It's very possible that S&P simply has enough money to play with that any savings from swapping to an automated solution for media monitoring aren't worth the potential disruption to the company workflow.
- tomascot 9y agofinancial companies usually use the worst formats to extract info from.
- steve371 9y agothis somehow made me laugh so hard. made my day~
- kobeya 9y agolaugh, and then cry
- westoque 9y agoRelated. I know a company that does analysis for soccer games. I thought it was state of the art too until they told me they just hired a bunch of people from China that logged every move (distance ran, kicks, etc) from eveyone and also put it in the database.
- tomascot 9y agogo on...
- JBlue42 9y agoDon't know about the China angle but the life of a soccer analyst isn't all it's made out to be: - https://www.pastemagazine.com/articles/2017/01/the-secret-soccer-analyst-no-pay-no-gain.html https://www.pastemagazine.com/articles/2017/01/the-secret-so... - https://www.pastemagazine.com/articles/2017/01/the-secret-soccer-analyst-mo-moneyball-mo-problems.html https://www.pastemagazine.com/articles/2017/01/the-secret-so... There's plenty of other blogs / articles out there now that people are trying to "Moneyball" soccer a bit more. Like anything that has a tinge of glamour and a lot of money around it, you have to take poor earnings to "pay your dues" in the trenches. Same as other aspects of pro sports, entertainment, etc. Hopefully something motivates you other than money.
- tomascot 9y agoI was curious about the logging system for that kind of data. Now that I know the name of the position I made a search and found that most systems use people to input the data. Thank you.
- ci5er 9y agoWell, that makes sense, doesn't it? Back in "the day" when certain US companies were trying to convert all the Whitepages and Yellowpages nation-wide into CD-ROMs, they'd ship phonebooks from each zip-code to China to have them triple entered (error correction), and then that's where the nation-wide CD-ROM phone-books came from. This is the same deal, no?
- rkwasny 9y ago@AznHisoka I would like to hear more about this! This is exactly a problem we are trying to solve (http://evolution.ai http://evolution.ai) One cannot even imagine how many people in the world have a boring job of looking at word document and C&P some of the data into excel spreadsheet. Other half have a job of looking at some document and effectively making Yes/No decisions all day. Disclaimer, I am the CTO
- s_kilk 9y agoHumans are the cheapest machines.