3 ms·

It's a dynamic page, so I have to do things like mark checkboxes, click buttons, etc. But if there's a better way to do it, I'm down to incorporate it!

dhruvkar 6y ago
Usually for dynamic pages, it's still working off of some internal API. So when I search for first name starts with "a", it's sending a POST request to https://efdsearch.senate.gov/search/report/data/ https://efdsearch.senate.gov/search/report/data/ with the following form data, which returns a nice JSON response. Obviously you'd have to handle with correct headers and CSRF tokens, but it'll be easier than Selenium for sure. draw: 1 columns[0][data]: 0 columns[0][name]: columns[0][searchable]: true columns[0][orderable]: true columns[0][search][value]: columns[0][search][regex]: false columns[1][data]: 1 columns[1][name]: columns[1][searchable]: true columns[1][orderable]: true columns[1][search][value]: columns[1][search][regex]: false columns[2][data]: 2 columns[2][name]: columns[2][searchable]: true columns[2][orderable]: true columns[2][search][value]: columns[2][search][regex]: false columns[3][data]: 3 columns[3][name]: columns[3][searchable]: true columns[3][orderable]: true columns[3][search][value]: columns[3][search][regex]: false columns[4][data]: 4 columns[4][name]: columns[4][searchable]: true columns[4][orderable]: true columns[4][search][value]: columns[4][search][regex]: false order[0][column]: 1 order[0][dir]: asc order[1][column]: 0 order[1][dir]: asc start: 0 length: 25 search[value]: search[regex]: false report_types: [] filer_types: [] submitted_start_date: 01/01/2012 00:00:00 submitted_end_date: candidate_state: senator_state: office_id: first_name: a last_name:
- nsomani 6y ago
  That works for the search results, but the pages to display the periodic transaction reports embed the data directly into tables. Here's an example: https://efdsearch.senate.gov/search/view/ptr/8db16cde-8a14-4ea2-92ed-71d7f73ad131/ https://efdsearch.senate.gov/search/view/ptr/8db16cde-8a14-4... So I think we'll still need to parse from the HTML. But it might still be cleaner than using Selenium, so I'm open to any code changes.
  - dhruvkar 6y ago
    Ah got it. Yeah seems like that's direct HTML. It would make the script faster and less error prone for sure. I'll check out the repo in depth and see if there are any low hanging fruit.

dhruvkar 6y ago

Usually for dynamic pages, it's still working off of some internal API. So when I search for first name starts with "a", it's sending a POST request to https://efdsearch.senate.gov/search/report/data/ https://efdsearch.senate.gov/search/report/data/ with the following form data, which returns a nice JSON response. Obviously you'd have to handle with correct headers and CSRF tokens, but it'll be easier than Selenium for sure. draw: 1 columns[0][data]: 0 columns[0][name]: columns[0][searchable]: true columns[0][orderable]: true columns[0][search][value]: columns[0][search][regex]: false columns[1][data]: 1 columns[1][name]: columns[1][searchable]: true columns[1][orderable]: true columns[1][search][value]: columns[1][search][regex]: false columns[2][data]: 2 columns[2][name]: columns[2][searchable]: true columns[2][orderable]: true columns[2][search][value]: columns[2][search][regex]: false columns[3][data]: 3 columns[3][name]: columns[3][searchable]: true columns[3][orderable]: true columns[3][search][value]: columns[3][search][regex]: false columns[4][data]: 4 columns[4][name]: columns[4][searchable]: true columns[4][orderable]: true columns[4][search][value]: columns[4][search][regex]: false order[0][column]: 1 order[0][dir]: asc order[1][column]: 0 order[1][dir]: asc start: 0 length: 25 search[value]: search[regex]: false report_types: [] filer_types: [] submitted_start_date: 01/01/2012 00:00:00 submitted_end_date: candidate_state: senator_state: office_id: first_name: a last_name:

nsomani 6y ago
That works for the search results, but the pages to display the periodic transaction reports embed the data directly into tables. Here's an example: https://efdsearch.senate.gov/search/view/ptr/8db16cde-8a14-4ea2-92ed-71d7f73ad131/ https://efdsearch.senate.gov/search/view/ptr/8db16cde-8a14-4... So I think we'll still need to parse from the HTML. But it might still be cleaner than using Selenium, so I'm open to any code changes.
- dhruvkar 6y ago
  Ah got it. Yeah seems like that's direct HTML. It would make the script faster and less error prone for sure. I'll check out the repo in depth and see if there are any low hanging fruit.