3 ms·
For the search database, I did have to manually download each and every company, all 856,000 of them. To cut down on size though, I included all data points exc
by anonyonoor 3y ago
For the search database, I did have to manually download each and every company, all 856,000 of them. To cut down on size though, I included all data points except for filings. This is because they're large, but also because they're updated often. By excluding them the database stays more up-to-date.
Other than the search database which needs to be available on demand, the rest of the filers are queried and analyzed on demand. This is required because some filers get really, really big. Blackrock Inc alone is about a 30MB file.
- sroussey 3y agoI get the list of filings, and for now only download the ones I want, which are related to S-1 or 8k (there are lot that are related, including withdrawals of a filing, updates to a filing, etc., and some companies use an S-4 etc). Right now I scan the list of daily filings, and for every cik I mark them as dirty. Another pass looks at ciks marked dirty, and downloads the xml or json of the filings. I then scan for anything new, create filing rows and mark them as dirty so to speak. Annual reports can be big, and for my purposes I don't need them so I skip them. The HTML of these things is garbage! But a lot of financial data is in the company facts which is nice and clean.