4 ms·

- Opening and closing the driver 10000 times Can you be more clear on this? I don't see where this is happening. There's a start_drive function that they call

- Opening and closing the driver 10000 times Can you be more clear on this? I don't see where this is happening. There's a start_drive function that they call once in main and use that reference the entire time. https://github.com/SeanDaBlack/KelloggBot/blob/main/req.py#L54 https://github.com/SeanDaBlack/KelloggBot/blob/main/req.py#L... - Complicated flakey in-line xpaths '//*[@id="content"]/div/div[2]/div/div[1]/div[1]/div/div/button' How else would you get around navigating the DOM which this has to match precisely? - case first_name | last_name both uses a first_name() generator, should just be generate_name() Very subjective? - Time.sleep(x) is flakey, you should wait for the element to appear. Agree with this but maybe it's not possible for some reason in regards to how the webpage is configured?

gruez 5y ago
>Can you be more clear on this? I don't see where this is happening. There's a start_drive function that they call once in main and use that reference the entire time. >https://github.com/SeanDaBlack/KelloggBot/blob/main/req.py#L https://github.com/SeanDaBlack/KelloggBot/blob/main/req.py#L... the call to "start_driver" is inside the "while (i < 10000)" loop
- edmcnulty101 5y ago
  I mean wouldn't that allow 10,000 parallel operations? If you waited for each one to complete, then close each one, it would take forever.
  - gruez 5y ago
    Theoretically it can be rewritten to do the requests in parallel, but currently it isn't.
    - edmcnulty101 5y ago
      Ah good point! Yeah. That is a definite issue then. I should link this thread on the code's github. The author is getting a free code review. haha.
      - exdsq 5y ago
        Hahaha I'll link this entire thread to their github
    - exdsq 5y ago
      If it was you don't want 10k non-headless Chrome browsers on your computer, there isn't enough RAM in the world
exdsq 5y ago
> Can you be more clear on this? I don't see where this is happening. There's a start_drive function that they call once in main and use that reference the entire time. https://github.com/SeanDaBlack/KelloggBot/blob/main/req.py#L179 https://github.com/SeanDaBlack/KelloggBot/blob/main/req.py#L... They run start_driver() inside a while loop with 10000 iterations > How else would you get around navigating the DOM which this has to match precisely? For that specific one I'd probably try "//*[contains(text(), 'Apply now')]", if they add a single element to that page the entire xpath will fail in the code example > Very subjective? Do you have two first names? > Agree with this but maybe it's not possible for some reason in regards to how the webpage is configured? Maybe! Still, I bet you it's possible to wait for an element even if it's your own action taking effect.
- edmcnulty101 5y ago
  > They run start_driver() inside a while loop with 10000 iterations I mean wouldn't that allow 10,000 parallel operations? If you waited for each one to complete, then close each one, it would take forever. EDIT: Gruez(below) made the point that the code isn't written to be async so this is definite issue. > For that specific one I'd probably try "//*[contains(text(), 'Apply now')]", if they add a single element to that page the entire xpath will fail in the code example You're prob right about this to be honest. However, there's two elements with 'Apply now' on the page. A button with 'Apply now' and a dropdown with 'Apply now' that opens when you click that button. Lol https://jobs.kellogg.com/job/Lancaster-Permanent-Production-Associate-Lancaster-PA-17601/817684800/# https://jobs.kellogg.com/job/Lancaster-Permanent-Production-... Welcome to scraping hell! > Do you have two first names? It's generating fake names. It doesn't matter.
  - exdsq 5y ago
    > However, there's two elements with 'Apply now' on the page. A button with 'Apply now' and a dropdown with 'Apply now' that opens when you click that button. Lol You can chain them! It's still better to [0] the text one and then click the pop-up than use div chains the whole way down. Yes it's not perfect, if I was testing the site as part of Kellogg I'd ask for test ids to make them unique, but it's still immune to additional divs. > It's generating fake names. It doesn't matter. This hurts my soul

gruez 5y ago

>Can you be more clear on this? I don't see where this is happening. There's a start_drive function that they call once in main and use that reference the entire time. >https://github.com/SeanDaBlack/KelloggBot/blob/main/req.py#L https://github.com/SeanDaBlack/KelloggBot/blob/main/req.py#L... the call to "start_driver" is inside the "while (i < 10000)" loop

edmcnulty101 5y ago
I mean wouldn't that allow 10,000 parallel operations? If you waited for each one to complete, then close each one, it would take forever.
- gruez 5y ago
  Theoretically it can be rewritten to do the requests in parallel, but currently it isn't.
  - edmcnulty101 5y ago
    Ah good point! Yeah. That is a definite issue then. I should link this thread on the code's github. The author is getting a free code review. haha.
    - exdsq 5y ago
      Hahaha I'll link this entire thread to their github
  - exdsq 5y ago
    If it was you don't want 10k non-headless Chrome browsers on your computer, there isn't enough RAM in the world

exdsq 5y ago

> Can you be more clear on this? I don't see where this is happening. There's a start_drive function that they call once in main and use that reference the entire time. https://github.com/SeanDaBlack/KelloggBot/blob/main/req.py#L179 https://github.com/SeanDaBlack/KelloggBot/blob/main/req.py#L... They run start_driver() inside a while loop with 10000 iterations > How else would you get around navigating the DOM which this has to match precisely? For that specific one I'd probably try "//*[contains(text(), 'Apply now')]", if they add a single element to that page the entire xpath will fail in the code example > Very subjective? Do you have two first names? > Agree with this but maybe it's not possible for some reason in regards to how the webpage is configured? Maybe! Still, I bet you it's possible to wait for an element even if it's your own action taking effect.

edmcnulty101 5y ago
> They run start_driver() inside a while loop with 10000 iterations I mean wouldn't that allow 10,000 parallel operations? If you waited for each one to complete, then close each one, it would take forever. EDIT: Gruez(below) made the point that the code isn't written to be async so this is definite issue. > For that specific one I'd probably try "//*[contains(text(), 'Apply now')]", if they add a single element to that page the entire xpath will fail in the code example You're prob right about this to be honest. However, there's two elements with 'Apply now' on the page. A button with 'Apply now' and a dropdown with 'Apply now' that opens when you click that button. Lol https://jobs.kellogg.com/job/Lancaster-Permanent-Production-Associate-Lancaster-PA-17601/817684800/# https://jobs.kellogg.com/job/Lancaster-Permanent-Production-... Welcome to scraping hell! > Do you have two first names? It's generating fake names. It doesn't matter.
- exdsq 5y ago
  > However, there's two elements with 'Apply now' on the page. A button with 'Apply now' and a dropdown with 'Apply now' that opens when you click that button. Lol You can chain them! It's still better to [0] the text one and then click the pop-up than use div chains the whole way down. Yes it's not perfect, if I was testing the site as part of Kellogg I'd ask for test ids to make them unique, but it's still immune to additional divs. > It's generating fake names. It doesn't matter. This hurts my soul