6 ms·
Wouldn't some sophisticated shops potentially use crawlers, etc. hosted from cloud providers (i.e., third-party IPs) that might significantly skew this data?
by piker 9y ago
Wouldn't some sophisticated shops potentially use crawlers, etc. hosted from cloud providers (i.e., third-party IPs) that might significantly skew this data?
- scarletham 9y agoYes, exactly. Wouldn't surprise me if most of these firms were rescraping the data every night and overwriting.
- fwdpropaganda 9y agoWhy would they do that?
- scarletham 9y agoTake this with a grain of salt as I don't work at any of the funds mentioned (but still in the industry), but why not? Filings are subject to change due to errors, updates, etc. It's cheaper, easier, and less error prone to just scrape and overwrite data than search for any updates. I may have been heavyhanded in saying that the firms are rescraping the entire universe of filings on a daily basis, but I guess my point here is that these numbers can very easily be skewed and are generally a pretty poor indicator of actual data access.
- calabin 9y agoSEC EDGAR is terrible for a variety of reasons. It breaks, few things are truly standardized/normalized, etc. If you have the time and an invested interest there's no reason not to. Filings are subject to change but they are restated, not modified in place. That said, anything goes on the site. The only real reason not to scrape everything on a continuous loop is that there is a rate limiter in place, so you may end up impeding your intake of new data by loading yourself down with old jobs to handle.
- fwdpropaganda 9y agoI assumed the sophisticated shops ran their own datacenters. Can you imagine RenTech running their code on other people's computers? I can't.
- jeffreyrogers 9y agoSome do use AWS and other cloud providers for collecting and processing data.
- dsacco 9y agoYes, Renaissance runs everything on its own infrastructure (as do most firms of that caliber, and even a tier below). That said, many of these firms also ingest data from aggregated sources, so their research would activity be very difficult to accurately track via IP presence.
- sseveran 9y agoWhy? The reasons for having their own datacenters are either legacy or needing some sort of specialized hardware that is not available at a cloud provider. Mana which is a new pure quant fund uses AWS extensively.
- MagnumOpus 9y agoThe reason for having your own computers is assured secrecy of your algorithms, and complete control over redundancy, latency and network linkages. The reason for not having your own computers is because you can't afford them. A consideration for upstart funds, not a problem for RenTec.
- sseveran 9y agoIf it is discovered that Amazon is stealing data from AWS customers than AWS will go out of business fairly quickly. Also even though Mana is new they do have ~$1B AUM.
- fwdpropaganda 9y ago> Why? Because you don't want anyone to have access to your code and data. If you found out that Facebook is running on AWS, wouldn't you find that strange?