7 ms·

Show HN: A tool to scrape senators' stock transactions for your own analysis

350 points by nsomani 6y ago

nsomani 6y ago
Hi everyone, with the recent interest in U.S. senator insider trading, I open-sourced this repo for you to scrape and analyze the senators' stock transactions yourself: https://github.com/neelsomani/senator-filings https://github.com/neelsomani/senator-filings I provided a Jupyter notebook with my own simple analysis (limitations included in the README), but of course it can be modified to suit your needs. Let me know if you have any questions - happy to help!
- rasengan 6y ago
  This is amazing transparency that we needed. With this tool, you may have really changed the future of America to that of one that’s less corrupt. Bravo!
  - matthewaveryusa 6y ago
    It will force senators to need a partner in crime. Communication increases the likelihood of getting solid evidence of insider trading, and also gives another leverage point if the partner in crime can be pressured to produce evidence. The smart ones will still get away with it.
    - dpau 6y ago
      I was surprised to hear of blatant Covid-related insider trading by individuals in government, either some really are too stupid to know better or they feel impunity. I fear it's the latter.
      - craftinator 6y ago
        At least this will filter out some of the ones too stupid to be allowed in leadership roles.
  - cosmodisk 6y ago
    I don't think so. Hardly anyone is that stupid nowadays that they would run their own assets against privileged information. Corruption, especially in western countries,tend to be a bit more sophisticated.A simplified example: Senator A figures out that stock value of company X will drop because of upcoming legislation. He contacts his college years friend,who happens to own a lot of this company's stocks. The friend sell on time just before stock price crashes amd nets millions. Now senator A wants something im return. So it happens that his daughter is finishing marketing studies and will be in need for a job. The friend places senator's daughter in one of his firms and now she's newly baked VP of Marketing.
    - bredren 6y ago
      I agree, although I think currency of favors is usually less than direct stock tips. That is too easy to prove.
    - ra7 6y ago
      > The friend sell on time just before stock price crashes amd nets millions. Doesn’t this get looked at by the SEC? Not always, but they monitor these kinds of transactions especially when millions are involved.
      - jchook 6y ago
        Since the "upcoming legislation" is public information, wouldn't this count as simple market speculation rather than insider trading?
      - cosmodisk 6y ago
        This was a simplified example. In real life people usually have very complex social networks and it can very easily become Senator A - Senator B- Friend's friend- Neighbors daughter and etc.
    - deleted 6y ago
      [deleted]
- amykyta 6y ago
  Great idea. Have you done any analysis on the data? I’d be curious to see if a passive index to track members of the Congress would beat the S&P. Although it may not account for other sweet deals like this option expiry extension and accelerated vesting https://theintercept.com/2020/05/12/david-purdue-senate-cardlytics-stock/ https://theintercept.com/2020/05/12/david-purdue-senate-card...
  - creaghpatr 6y ago
    That's neat idea, but I think the reporting lag is too long to action it. Would still be good for retroactive benchmarketing. Also consider that extremely wealthy senators (many are) might only have a small part of their assets in stocks and could be part of a bigger financial strategy, as a hedge against other asset classes rather than an indicator of where they or their agent thinks the market will move.
  - bko 6y ago
    I didn't use this tool, I instead used Senate Stock Watcher [0]. But a cursory review [1] of the trades made by senators showed the amounts to be very small (66% of trades were < 15k) and the typical Senator is a millionaire. In the 4 month period I analyzed, only 3 trades were between 250-500k. I looked at performance as well, but its tough to benchmark these things. There were net outflows in the market during that period and if you sold you did relatively well in a falling market, so the typical trade did well. But the relatively small size of the trades makes me skeptical that insider information was used. Matt Levine said it best: > Of course they could be lying, but in context the defense seems pretty plausible. (Kelly Loeffler, for instance, controversially dumped about 0.6% of her portfolio at around the same time, which sure seems like the sort of thing an investment adviser would do without any input from her? You could call your adviser and say “a disaster is coming, sell everything!,” but calling them to say “a disaster is coming, sell a tiny bit!” seems pointless.) But this doesn't include William Barr as he is not a Senator. [0] https://senatestockwatcher.com/ https://senatestockwatcher.com/ [1] https://medium.com/ml-everything/analyzing-us-senators-stock-picks-e8106bbbe8e https://medium.com/ml-everything/analyzing-us-senators-stock... [2] https://www.bloomberg.com/opinion/articles/2020-05-14/senator-s-stock-trades-make-trouble https://www.bloomberg.com/opinion/articles/2020-05-14/senato...
- christiansakai 6y ago
  Seems to add the PDF analyzing feature is not that hard or I'm missing something? Have you tried to incorporate this feature?
  - pc86 6y ago
    The good news is that it's open source, so instead of asking OP why they didn't add a feature, you can add it.
  - nsomani 6y ago
    If someone could help out and make a PR to do the OCR for PDFs, that'd be awesome! Would love to add it in.
- k3oni 6y ago
  I would move away from the Yahoo Finance API, there's a few data sources that you can use that have free access to their API, see [1] for an example. - [1] https://www.alphavantage.co/ https://www.alphavantage.co/
  - nsomani 6y ago
    I took a look at Alphavantage, but I wanted other data like daily transaction volume + market caps to do things like weighted regressions. The rate limiting was also a little restrictive.
inglor 6y ago
Hey probably email the guys at tipranks (support@tipranks.com) with this. They have a tool that takes returns and back tests then and finds correlations they run on hedge funds and can probably find patterns quickly.
- inglor 6y ago
  Plus finding from Elliot Spitzer to do things that bring transparency to the stock market. Source: I worked there ~4 years ago.
fratimo66 6y ago
Feinstein selling $ALLO on Feb 18th. Not a good call, I'd say.
- rasengan 6y ago
  Agreed - and she says it was her husband and not her. There needs to be a law that politicians are no longer single people but in fact come as a package - and politicians should have 0 expectation of privacy as they serve the public.
  - mosselman 6y ago
    What nonsense. Why would a politician have to disclose/not be allowed to keep secret when they masturbate, for example?
  - deleted 6y ago
    [deleted]
  - fratimo66 6y ago
    I still believe in the value of individuality. Politiciants should have same expectation of privacy as regular citizens. Transactions should not.
    - save_ferris 6y ago
      The argument that politicians should have the same expectation of privacy as regular citizens while also making their stock transparent is inherently contradictory. Regular citizens do not have to make their stock transactions public, but politicians should be required to. I think it’s entirely reasonable for politicians to assume a life of less privacy since every aspect of their lives is scrutinized by the press anyway.
    - pc86 6y ago
      This whole debate stems from the fact that we have career politicians, which (at the federal level, at least) is not what the founders envisioned. A citizen legislature cannot, by definition, exist when it's filled by people who have never had a real job. Term limits fixes this from both ends. Politicians are allowed their privacy, because they're just regular people and nobody is going to get rid from a single Senate term or 3-4 House terms. It eliminates the incentive to associate one's self more with DC than your home district. And it would probably result in people having a better opinion of Congress as a whole because it's not filled with people who have never done anything else.
pjc50 6y ago
While the impact of this on corruption is likely to be small, it would be interesting to see how much of an edge on the market it would give you...
- awinder 6y ago
  Probably not a very good one, somewhat surprisingly. I haven't run any analyses / haven't looked at this dataset in depth but from past forays into congressional datasets I don't really get what a lot of these people are doing. Usually you can throw a dart, check the financial portfolio, and see massive glaring problems immediately. Huge tilts into single / a handful of companies, totally outsized amounts of money in some random local small cap / mid cap, weird biases and tilts into commercial real estate (for example), you name it and you can see it. If anything I might investigate a counter-congress trading plan if you're looking for edge...
  - odonnellryan 6y ago
    Actually, there have been studies done showing their trades beat the market! It is not possible to know their exact return but the studies I've read suggest they have alpha.
    - awinder 6y ago
      I think I’m gonna have to read up a little more, but I’ve seen this in support of the excess returns view: https://www.nytimes.com/2011/07/10/business/mutfund/congressional-portfolios-outpacing-the-market-essay.html https://www.nytimes.com/2011/07/10/business/mutfund/congress... And this in support that avg congressional portfolio is not performing well: https://www.mit.edu/~jhainm/Paper/Eggmueller_CapitolLosses.pdf https://www.mit.edu/~jhainm/Paper/Eggmueller_CapitolLosses.p... https://www.morningstar.com/news/marketwatch/2020041854/senators-privileged-information-does-not-often-result-in-market-beating-returns-dartmouth-research-team-concludes https://www.morningstar.com/news/marketwatch/2020041854/sena... Looks like there might be a historical component around the STOCK act that different studies play with for their data window. In either case — I still say for the poor level of diversification within & around asset classes that I don’t see a lot of shining examples of how to achieve success.
stuaxo 6y ago
Could this be adapted to work for British MPs ?
- alibaba_x 6y ago
  Is their data public? The US publishes this data about their senators but I’m not sure if the UK does as well
TekMol 6y ago
It says it uses the "Yahoo Finance API" but it seems no such API is offered by Yahoo anymore. There are other sites that offer access to what they call the "Yahoo Finance API". I wonder how that works and if those are legit offers. Does anybody know?
- nsomani 6y ago
  Here's the specific package that I'm using: https://github.com/ranaroussi/yfinance https://github.com/ranaroussi/yfinance
  - TekMol 6y ago
    That seems to be a scraper. I wonder if it is legal to scrape stock market data and then republish it.
    - Ill_ban_myself 6y ago
      Not only is it legal it’s insanely profitable! See: https://www.spglobal.com/marketintelligence/en/campaigns/snl-financial https://www.spglobal.com/marketintelligence/en/campaigns/snl...
      - nl 6y ago
        Firstly, this doesn't scrape prices - they pay for a live feed direct from the market. Secondly, it isn't legal. They are protected by license agreement. But stock prices is not what the value of SNL Financial and/or Capital IQ is for most people. I know a bunch of people who pay the $30K license fee for these (as well as Bloomberg etc) who barely use the stock price at all.
- snazz 6y ago
  There definitely is still a Yahoo Finance API for large corporate customers. That's where Apple's Stocks app gets its data, for example. I'm not sure about the third party services---they probably just scrape Yahoo's website and repackage the data.
beager 6y ago
It would be fun to compare the timeline of stock transactions by legislators with the timeline of public statements by those legislators and their colleagues pertaining to the companies’ stocks they’re selling, or those companies’ industries in general. Not that I expect such nakedly corrupt pumping and dumping or hyping... but I’ve been more surprised in the past.
- olah_1 6y ago
  You'd get more meaningful results if you checked _their_ statements with the transactions of their _family members_. But that's too creepy for me (which is probably what is being counted on).
- wdn 6y ago
  "Throw Them All Out" by Peter Schweizer documented a lot of this. https://www.amazon.com/Throw-Them-All-Out-Politicians-ebook/dp/B0062N35X8 https://www.amazon.com/Throw-Them-All-Out-Politicians-ebook/...
blargmaster33 6y ago
Waste of time! They are exempt from the insider trading laws.
JoshuaEddy 6y ago
Sounds like "Senate Stock Watcher": https://news.ycombinator.com/item?id=22834524 https://news.ycombinator.com/item?id=22834524 https://senatestockwatcher.com/ https://senatestockwatcher.com/
- nsomani 6y ago
  Senate Stock Watcher is really cool. I chatted a bit with Tim Carambat (who built the site), and right now he's basically crowdsourcing the labeling for PDFs. Hoping that we can develop on this package and eliminate the manpower required for projects like Senate Stock Watcher.
zeeone 6y ago
Great effort, though these kind of tools are useless unless you can include senators’ spouses and family </sarcasm>
filtercoffee37 6y ago
Nice idea! This should be an easily navigable web app that every common person should be able to use. Thanks for building.
nathell 6y ago
I've created a Clojure library to facilitate scraping scenarios like this one: https://github.com/nathell/skyscraper https://github.com/nathell/skyscraper Writeup of a sample data acquisition + analysis usecase: http://blog.danieljanus.pl/2020/05/08/making-of-clojure-dependency/ http://blog.danieljanus.pl/2020/05/08/making-of-clojure-depe...
dhruvkar 6y ago
Nice! Any particular reason you picked using selenium+chromedriver instead of an http lib (like requests) and beautifulsoup?
- nsomani 6y ago
  It's a dynamic page, so I have to do things like mark checkboxes, click buttons, etc. But if there's a better way to do it, I'm down to incorporate it!
  - dhruvkar 6y ago
    Usually for dynamic pages, it's still working off of some internal API. So when I search for first name starts with "a", it's sending a POST request to https://efdsearch.senate.gov/search/report/data/ https://efdsearch.senate.gov/search/report/data/ with the following form data, which returns a nice JSON response. Obviously you'd have to handle with correct headers and CSRF tokens, but it'll be easier than Selenium for sure. draw: 1 columns[0][data]: 0 columns[0][name]: columns[0][searchable]: true columns[0][orderable]: true columns[0][search][value]: columns[0][search][regex]: false columns[1][data]: 1 columns[1][name]: columns[1][searchable]: true columns[1][orderable]: true columns[1][search][value]: columns[1][search][regex]: false columns[2][data]: 2 columns[2][name]: columns[2][searchable]: true columns[2][orderable]: true columns[2][search][value]: columns[2][search][regex]: false columns[3][data]: 3 columns[3][name]: columns[3][searchable]: true columns[3][orderable]: true columns[3][search][value]: columns[3][search][regex]: false columns[4][data]: 4 columns[4][name]: columns[4][searchable]: true columns[4][orderable]: true columns[4][search][value]: columns[4][search][regex]: false order[0][column]: 1 order[0][dir]: asc order[1][column]: 0 order[1][dir]: asc start: 0 length: 25 search[value]: search[regex]: false report_types: [] filer_types: [] submitted_start_date: 01/01/2012 00:00:00 submitted_end_date: candidate_state: senator_state: office_id: first_name: a last_name:
    - nsomani 6y ago
      That works for the search results, but the pages to display the periodic transaction reports embed the data directly into tables. Here's an example: https://efdsearch.senate.gov/search/view/ptr/8db16cde-8a14-4ea2-92ed-71d7f73ad131/ https://efdsearch.senate.gov/search/view/ptr/8db16cde-8a14-4... So I think we'll still need to parse from the HTML. But it might still be cleaner than using Selenium, so I'm open to any code changes.
at_a_remove 6y ago
Too bad they do not have to report their activity real-time, then the next step would be to create something like an "index fund" that simply mimics their choices.
- buckhx 6y ago
  I've always wondered if some hedge fund uses senator's transactions as input in their model
dznodes 6y ago
Is it legal for companies to pay politicians in stock? ("donate" stock to them for favorable policies)
davidkuennen 6y ago
This is nice. I think I will integrate it in my new stock events app. It's exactly the kind of content that belongs there. Thanks!
kumarski 6y ago
You could actually implement the trades with https://alpaca.markets https://alpaca.markets if you wanted to.
- jchook 6y ago
  Unfortunately the senators get 45 days to disclose their trades.
feep 6y ago
Takes about 2.5 hours to run on a slow machine (doesn’t look like CPU is an issue) with a slow internet connection. There is more processing to be done. There is a Jupyter cell that he says takes 3 hours to run. I will make a processed data file available, hopefully this weekend. If anybody wants it, let me know.
- malshe 6y ago
  That would be great! I am interested in the data. Thanks
  - feep 6y ago
    I hope you have a notifier running, because I cannot find an email for you. Published at https://qri.cloud/feep/senate_financial_disclosures https://qri.cloud/feep/senate_financial_disclosures I may add more fields, so check for updates.
mvkel 6y ago
What’s the lag time on this? 30 days?

350 points by nsomani 6y ago

Hi everyone, with the recent interest in U.S. senator insider trading, I open-sourced this repo for you to scrape and analyze the senators' stock transactions yourself: https://github.com/neelsomani/senator-filings https://github.com/neelsomani/senator-filings I provided a Jupyter notebook with my own simple analysis (limitations included in the README), but of course it can be modified to suit your needs. Let me know if you have any questions - happy to help!

rasengan 6y ago
This is amazing transparency that we needed. With this tool, you may have really changed the future of America to that of one that’s less corrupt. Bravo!
- matthewaveryusa 6y ago
  It will force senators to need a partner in crime. Communication increases the likelihood of getting solid evidence of insider trading, and also gives another leverage point if the partner in crime can be pressured to produce evidence. The smart ones will still get away with it.
  - dpau 6y ago
    I was surprised to hear of blatant Covid-related insider trading by individuals in government, either some really are too stupid to know better or they feel impunity. I fear it's the latter.
    - craftinator 6y ago
      At least this will filter out some of the ones too stupid to be allowed in leadership roles.
- cosmodisk 6y ago
  I don't think so. Hardly anyone is that stupid nowadays that they would run their own assets against privileged information. Corruption, especially in western countries,tend to be a bit more sophisticated.A simplified example: Senator A figures out that stock value of company X will drop because of upcoming legislation. He contacts his college years friend,who happens to own a lot of this company's stocks. The friend sell on time just before stock price crashes amd nets millions. Now senator A wants something im return. So it happens that his daughter is finishing marketing studies and will be in need for a job. The friend places senator's daughter in one of his firms and now she's newly baked VP of Marketing.
  - bredren 6y ago
    I agree, although I think currency of favors is usually less than direct stock tips. That is too easy to prove.
  - ra7 6y ago
    > The friend sell on time just before stock price crashes amd nets millions. Doesn’t this get looked at by the SEC? Not always, but they monitor these kinds of transactions especially when millions are involved.
    - jchook 6y ago
      Since the "upcoming legislation" is public information, wouldn't this count as simple market speculation rather than insider trading?
    - cosmodisk 6y ago
      This was a simplified example. In real life people usually have very complex social networks and it can very easily become Senator A - Senator B- Friend's friend- Neighbors daughter and etc.
  - deleted 6y ago
    [deleted]
amykyta 6y ago
Great idea. Have you done any analysis on the data? I’d be curious to see if a passive index to track members of the Congress would beat the S&P. Although it may not account for other sweet deals like this option expiry extension and accelerated vesting https://theintercept.com/2020/05/12/david-purdue-senate-cardlytics-stock/ https://theintercept.com/2020/05/12/david-purdue-senate-card...
- creaghpatr 6y ago
  That's neat idea, but I think the reporting lag is too long to action it. Would still be good for retroactive benchmarketing. Also consider that extremely wealthy senators (many are) might only have a small part of their assets in stocks and could be part of a bigger financial strategy, as a hedge against other asset classes rather than an indicator of where they or their agent thinks the market will move.
- bko 6y ago
  I didn't use this tool, I instead used Senate Stock Watcher [0]. But a cursory review [1] of the trades made by senators showed the amounts to be very small (66% of trades were < 15k) and the typical Senator is a millionaire. In the 4 month period I analyzed, only 3 trades were between 250-500k. I looked at performance as well, but its tough to benchmark these things. There were net outflows in the market during that period and if you sold you did relatively well in a falling market, so the typical trade did well. But the relatively small size of the trades makes me skeptical that insider information was used. Matt Levine said it best: > Of course they could be lying, but in context the defense seems pretty plausible. (Kelly Loeffler, for instance, controversially dumped about 0.6% of her portfolio at around the same time, which sure seems like the sort of thing an investment adviser would do without any input from her? You could call your adviser and say “a disaster is coming, sell everything!,” but calling them to say “a disaster is coming, sell a tiny bit!” seems pointless.) But this doesn't include William Barr as he is not a Senator. [0] https://senatestockwatcher.com/ https://senatestockwatcher.com/ [1] https://medium.com/ml-everything/analyzing-us-senators-stock-picks-e8106bbbe8e https://medium.com/ml-everything/analyzing-us-senators-stock... [2] https://www.bloomberg.com/opinion/articles/2020-05-14/senator-s-stock-trades-make-trouble https://www.bloomberg.com/opinion/articles/2020-05-14/senato...
christiansakai 6y ago
Seems to add the PDF analyzing feature is not that hard or I'm missing something? Have you tried to incorporate this feature?
- pc86 6y ago
  The good news is that it's open source, so instead of asking OP why they didn't add a feature, you can add it.
- nsomani 6y ago
  If someone could help out and make a PR to do the OCR for PDFs, that'd be awesome! Would love to add it in.
k3oni 6y ago
I would move away from the Yahoo Finance API, there's a few data sources that you can use that have free access to their API, see [1] for an example. - [1] https://www.alphavantage.co/ https://www.alphavantage.co/
- nsomani 6y ago
  I took a look at Alphavantage, but I wanted other data like daily transaction volume + market caps to do things like weighted regressions. The rate limiting was also a little restrictive.

inglor 6y ago

Hey probably email the guys at tipranks (support@tipranks.com) with this. They have a tool that takes returns and back tests then and finds correlations they run on hedge funds and can probably find patterns quickly.

inglor 6y ago
Plus finding from Elliot Spitzer to do things that bring transparency to the stock market. Source: I worked there ~4 years ago.

fratimo66 6y ago

Feinstein selling $ALLO on Feb 18th. Not a good call, I'd say.

rasengan 6y ago
Agreed - and she says it was her husband and not her. There needs to be a law that politicians are no longer single people but in fact come as a package - and politicians should have 0 expectation of privacy as they serve the public.
- mosselman 6y ago
  What nonsense. Why would a politician have to disclose/not be allowed to keep secret when they masturbate, for example?
- deleted 6y ago
  [deleted]
- fratimo66 6y ago
  I still believe in the value of individuality. Politiciants should have same expectation of privacy as regular citizens. Transactions should not.
  - save_ferris 6y ago
    The argument that politicians should have the same expectation of privacy as regular citizens while also making their stock transparent is inherently contradictory. Regular citizens do not have to make their stock transactions public, but politicians should be required to. I think it’s entirely reasonable for politicians to assume a life of less privacy since every aspect of their lives is scrutinized by the press anyway.
  - pc86 6y ago
    This whole debate stems from the fact that we have career politicians, which (at the federal level, at least) is not what the founders envisioned. A citizen legislature cannot, by definition, exist when it's filled by people who have never had a real job. Term limits fixes this from both ends. Politicians are allowed their privacy, because they're just regular people and nobody is going to get rid from a single Senate term or 3-4 House terms. It eliminates the incentive to associate one's self more with DC than your home district. And it would probably result in people having a better opinion of Congress as a whole because it's not filled with people who have never done anything else.

pjc50 6y ago

While the impact of this on corruption is likely to be small, it would be interesting to see how much of an edge on the market it would give you...

awinder 6y ago
Probably not a very good one, somewhat surprisingly. I haven't run any analyses / haven't looked at this dataset in depth but from past forays into congressional datasets I don't really get what a lot of these people are doing. Usually you can throw a dart, check the financial portfolio, and see massive glaring problems immediately. Huge tilts into single / a handful of companies, totally outsized amounts of money in some random local small cap / mid cap, weird biases and tilts into commercial real estate (for example), you name it and you can see it. If anything I might investigate a counter-congress trading plan if you're looking for edge...
- odonnellryan 6y ago
  Actually, there have been studies done showing their trades beat the market! It is not possible to know their exact return but the studies I've read suggest they have alpha.
  - awinder 6y ago
    I think I’m gonna have to read up a little more, but I’ve seen this in support of the excess returns view: https://www.nytimes.com/2011/07/10/business/mutfund/congressional-portfolios-outpacing-the-market-essay.html https://www.nytimes.com/2011/07/10/business/mutfund/congress... And this in support that avg congressional portfolio is not performing well: https://www.mit.edu/~jhainm/Paper/Eggmueller_CapitolLosses.pdf https://www.mit.edu/~jhainm/Paper/Eggmueller_CapitolLosses.p... https://www.morningstar.com/news/marketwatch/2020041854/senators-privileged-information-does-not-often-result-in-market-beating-returns-dartmouth-research-team-concludes https://www.morningstar.com/news/marketwatch/2020041854/sena... Looks like there might be a historical component around the STOCK act that different studies play with for their data window. In either case — I still say for the poor level of diversification within & around asset classes that I don’t see a lot of shining examples of how to achieve success.

stuaxo 6y ago

Could this be adapted to work for British MPs ?

alibaba_x 6y ago
Is their data public? The US publishes this data about their senators but I’m not sure if the UK does as well

TekMol 6y ago

It says it uses the "Yahoo Finance API" but it seems no such API is offered by Yahoo anymore. There are other sites that offer access to what they call the "Yahoo Finance API". I wonder how that works and if those are legit offers. Does anybody know?

nsomani 6y ago
Here's the specific package that I'm using: https://github.com/ranaroussi/yfinance https://github.com/ranaroussi/yfinance
- TekMol 6y ago
  That seems to be a scraper. I wonder if it is legal to scrape stock market data and then republish it.
  - Ill_ban_myself 6y ago
    Not only is it legal it’s insanely profitable! See: https://www.spglobal.com/marketintelligence/en/campaigns/snl-financial https://www.spglobal.com/marketintelligence/en/campaigns/snl...
    - nl 6y ago
      Firstly, this doesn't scrape prices - they pay for a live feed direct from the market. Secondly, it isn't legal. They are protected by license agreement. But stock prices is not what the value of SNL Financial and/or Capital IQ is for most people. I know a bunch of people who pay the $30K license fee for these (as well as Bloomberg etc) who barely use the stock price at all.
snazz 6y ago
There definitely is still a Yahoo Finance API for large corporate customers. That's where Apple's Stocks app gets its data, for example. I'm not sure about the third party services---they probably just scrape Yahoo's website and repackage the data.

beager 6y ago

It would be fun to compare the timeline of stock transactions by legislators with the timeline of public statements by those legislators and their colleagues pertaining to the companies’ stocks they’re selling, or those companies’ industries in general. Not that I expect such nakedly corrupt pumping and dumping or hyping... but I’ve been more surprised in the past.

olah_1 6y ago
You'd get more meaningful results if you checked _their_ statements with the transactions of their _family members_. But that's too creepy for me (which is probably what is being counted on).
wdn 6y ago
"Throw Them All Out" by Peter Schweizer documented a lot of this. https://www.amazon.com/Throw-Them-All-Out-Politicians-ebook/dp/B0062N35X8 https://www.amazon.com/Throw-Them-All-Out-Politicians-ebook/...

blargmaster33 6y ago

Waste of time! They are exempt from the insider trading laws.

JoshuaEddy 6y ago

Sounds like "Senate Stock Watcher": https://news.ycombinator.com/item?id=22834524 https://news.ycombinator.com/item?id=22834524 https://senatestockwatcher.com/ https://senatestockwatcher.com/

nsomani 6y ago
Senate Stock Watcher is really cool. I chatted a bit with Tim Carambat (who built the site), and right now he's basically crowdsourcing the labeling for PDFs. Hoping that we can develop on this package and eliminate the manpower required for projects like Senate Stock Watcher.

zeeone 6y ago

Great effort, though these kind of tools are useless unless you can include senators’ spouses and family </sarcasm>

filtercoffee37 6y ago

Nice idea! This should be an easily navigable web app that every common person should be able to use. Thanks for building.

nathell 6y ago

I've created a Clojure library to facilitate scraping scenarios like this one: https://github.com/nathell/skyscraper https://github.com/nathell/skyscraper Writeup of a sample data acquisition + analysis usecase: http://blog.danieljanus.pl/2020/05/08/making-of-clojure-dependency/ http://blog.danieljanus.pl/2020/05/08/making-of-clojure-depe...

dhruvkar 6y ago

Nice! Any particular reason you picked using selenium+chromedriver instead of an http lib (like requests) and beautifulsoup?

nsomani 6y ago
It's a dynamic page, so I have to do things like mark checkboxes, click buttons, etc. But if there's a better way to do it, I'm down to incorporate it!
- dhruvkar 6y ago
  Usually for dynamic pages, it's still working off of some internal API. So when I search for first name starts with "a", it's sending a POST request to https://efdsearch.senate.gov/search/report/data/ https://efdsearch.senate.gov/search/report/data/ with the following form data, which returns a nice JSON response. Obviously you'd have to handle with correct headers and CSRF tokens, but it'll be easier than Selenium for sure. draw: 1 columns[0][data]: 0 columns[0][name]: columns[0][searchable]: true columns[0][orderable]: true columns[0][search][value]: columns[0][search][regex]: false columns[1][data]: 1 columns[1][name]: columns[1][searchable]: true columns[1][orderable]: true columns[1][search][value]: columns[1][search][regex]: false columns[2][data]: 2 columns[2][name]: columns[2][searchable]: true columns[2][orderable]: true columns[2][search][value]: columns[2][search][regex]: false columns[3][data]: 3 columns[3][name]: columns[3][searchable]: true columns[3][orderable]: true columns[3][search][value]: columns[3][search][regex]: false columns[4][data]: 4 columns[4][name]: columns[4][searchable]: true columns[4][orderable]: true columns[4][search][value]: columns[4][search][regex]: false order[0][column]: 1 order[0][dir]: asc order[1][column]: 0 order[1][dir]: asc start: 0 length: 25 search[value]: search[regex]: false report_types: [] filer_types: [] submitted_start_date: 01/01/2012 00:00:00 submitted_end_date: candidate_state: senator_state: office_id: first_name: a last_name:
  - nsomani 6y ago
    That works for the search results, but the pages to display the periodic transaction reports embed the data directly into tables. Here's an example: https://efdsearch.senate.gov/search/view/ptr/8db16cde-8a14-4ea2-92ed-71d7f73ad131/ https://efdsearch.senate.gov/search/view/ptr/8db16cde-8a14-4... So I think we'll still need to parse from the HTML. But it might still be cleaner than using Selenium, so I'm open to any code changes.

at_a_remove 6y ago

Too bad they do not have to report their activity real-time, then the next step would be to create something like an "index fund" that simply mimics their choices.

buckhx 6y ago
I've always wondered if some hedge fund uses senator's transactions as input in their model

dznodes 6y ago

Is it legal for companies to pay politicians in stock? ("donate" stock to them for favorable policies)

davidkuennen 6y ago

This is nice. I think I will integrate it in my new stock events app. It's exactly the kind of content that belongs there. Thanks!

kumarski 6y ago

You could actually implement the trades with https://alpaca.markets https://alpaca.markets if you wanted to.

jchook 6y ago
Unfortunately the senators get 45 days to disclose their trades.

feep 6y ago

Takes about 2.5 hours to run on a slow machine (doesn’t look like CPU is an issue) with a slow internet connection. There is more processing to be done. There is a Jupyter cell that he says takes 3 hours to run. I will make a processed data file available, hopefully this weekend. If anybody wants it, let me know.

malshe 6y ago
That would be great! I am interested in the data. Thanks
- feep 6y ago
  I hope you have a notifier running, because I cannot find an email for you. Published at https://qri.cloud/feep/senate_financial_disclosures https://qri.cloud/feep/senate_financial_disclosures I may add more fields, so check for updates.

mvkel 6y ago

What’s the lag time on this? 30 days?