5 ms·

Thank you. This data should be available in all states, but each state has a proprietary site I have to write a new scraper for and write a new data pipeline fo

Thank you. This data should be available in all states, but each state has a proprietary site I have to write a new scraper for and write a new data pipeline for. I'm thinking about open-sourcing the project and gathering community support for the rest now that I have a good pattern set. Yes, I was harmed by a doctor and the medical board dismissed my complaint. This made me curious about what the board actually does take action for, but this wasn't searchable because all that was available were PDF scans that weren't digitized/OCR'd. I also noticed some documents that were referenced in doctors' profiles had disappeared. As part of this, I archived every PDF available on the wayback machine to protect against that going forward. I also want to give credit to a similar site that I took inspiration from called 4patientsafety.org, which did some of the same stuff I have here, but completely manually. I chatted with the creator of that and he agreed making an automated version would help because they fell behind updating it due to how intensive the manual process was.

cobertos 3y ago
Is there a link to the community where this is being developed? I was unable to find one on the site itself. I'd be interested in writing a scraper for my state and the digitization pipeline.
- the_sleaze9 3y ago
  I'd be interested in helping out as well, looking forward to it
  - jamesfly 3y ago
    Thank you. I'm going to refine the site and data a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B https://discord.gg/bXdk4CuR7B for anyone interested in contributing.
- jamesfly 3y ago
  Thank you for offering to help. I'm going to refine the site and data a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B https://discord.gg/bXdk4CuR7B for anyone interested in contributing.
heliostatic 3y ago
You may want to look at courtlistener.org as inspiration--similar set of problems, but in a different space obviously.
- jamesfly 3y ago
  Thanks but the site seems to be down. Do you have another link?
  - levelhorizon 3y ago
    It's www.courtlistener.com
    - jamesfly 3y ago
      Thanks! I'm looking at the Contribute page and seeing some cool ideas.
yorwba 3y ago
> I archived every PDF available on the wayback machine How did you tell archive.org to store the PDFs? It doesn't seem to have worked in all cases, since when I went to check out the single "order vacating automatic cancellation of certificate", there was no snapshot yet. https://web.archive.org/web/20231009140931/https://www2.mbc.ca.gov/PDL/document.aspx?path=%5CDIDOCS%5C20111130%5CDMRAAADE3%5C&did=AAADE111130232525437.DID https://web.archive.org/web/20231009140931/https://www2.mbc....
- jamesfly 3y ago
  I have to admit I missed a few hundred out of about 18k PDFs. I didn't keep track of which ones I archived on my initial pass, so I need to rerun the whole script which takes about a week with Web Archive's rate limit, so I've just been putting it off. I was also thinking people who visit them can click the "archive" button in cases it's not yet archived to crowdsource the effort, like you seem to have done.
codeTired 3y ago
I’m interested in helping out. I have done a lot of research in medicine due to my poor experiences with even top specialists. Ignoring symptoms, denying side effects of meds I took, even though those were present on the label. Do you have a discord? Any plans on allowing people to comment on physicians? Any plans of adding how much doctors have received from corporate interests?(as this data is available)
- jamesfly 3y ago
  Sorry to hear you've also had poor experiences. I too had to delve into Pubmed to figure out what was going on with me after the doctor harmed me. Thank you for offering to help out. I do want to refine the site and data a little more over the next few days and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo. I've now made a discord here https://discord.gg/bXdk4CuR7B https://discord.gg/bXdk4CuR7B for anyone interested in contributing. Comments on doctors should already be enabled via Disqus if you scroll to the bottom of a profile page. Yes, I definitely want to add that data!
radlad 3y ago
I'm also interested in writing a scraper for my site (Michigan) if time allows.
- jamesfly 3y ago
  Thank you. I'm going to refine it a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B https://discord.gg/bXdk4CuR7B for anyone interested in contributing.
noman-land 3y ago
Please open source this. I would love to help.
- jamesfly 3y ago
  Thank you. I'm going to refine the site and data a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B https://discord.gg/bXdk4CuR7B for anyone interested in contributing.
mywrathacademia 3y ago
Open sourcing this would be a great idea and a lot of people would happily contribute, it already looks great. The biggest secret is that most doctors do more harm than good when a patient has anything worse than a cold. The psychopaths who couldn't make it as CEOs or serial killers end up becoming doctors, in charge of people's health.
- jamesfly 3y ago
  Thank you. I'm going to refine the site a bit more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B https://discord.gg/bXdk4CuR7B for anyone interested in contributing.
deleted 3y ago
[deleted]

cobertos 3y ago

Is there a link to the community where this is being developed? I was unable to find one on the site itself. I'd be interested in writing a scraper for my state and the digitization pipeline.

the_sleaze9 3y ago
I'd be interested in helping out as well, looking forward to it
- jamesfly 3y ago
  Thank you. I'm going to refine the site and data a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B https://discord.gg/bXdk4CuR7B for anyone interested in contributing.
jamesfly 3y ago
Thank you for offering to help. I'm going to refine the site and data a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B https://discord.gg/bXdk4CuR7B for anyone interested in contributing.

heliostatic 3y ago

You may want to look at courtlistener.org as inspiration--similar set of problems, but in a different space obviously.

jamesfly 3y ago
Thanks but the site seems to be down. Do you have another link?
- levelhorizon 3y ago
  It's www.courtlistener.com
  - jamesfly 3y ago
    Thanks! I'm looking at the Contribute page and seeing some cool ideas.

yorwba 3y ago

> I archived every PDF available on the wayback machine How did you tell archive.org to store the PDFs? It doesn't seem to have worked in all cases, since when I went to check out the single "order vacating automatic cancellation of certificate", there was no snapshot yet. https://web.archive.org/web/20231009140931/https://www2.mbc.ca.gov/PDL/document.aspx?path=%5CDIDOCS%5C20111130%5CDMRAAADE3%5C&did=AAADE111130232525437.DID https://web.archive.org/web/20231009140931/https://www2.mbc....

jamesfly 3y ago
I have to admit I missed a few hundred out of about 18k PDFs. I didn't keep track of which ones I archived on my initial pass, so I need to rerun the whole script which takes about a week with Web Archive's rate limit, so I've just been putting it off. I was also thinking people who visit them can click the "archive" button in cases it's not yet archived to crowdsource the effort, like you seem to have done.

codeTired 3y ago

I’m interested in helping out. I have done a lot of research in medicine due to my poor experiences with even top specialists. Ignoring symptoms, denying side effects of meds I took, even though those were present on the label. Do you have a discord? Any plans on allowing people to comment on physicians? Any plans of adding how much doctors have received from corporate interests?(as this data is available)

jamesfly 3y ago
Sorry to hear you've also had poor experiences. I too had to delve into Pubmed to figure out what was going on with me after the doctor harmed me. Thank you for offering to help out. I do want to refine the site and data a little more over the next few days and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo. I've now made a discord here https://discord.gg/bXdk4CuR7B https://discord.gg/bXdk4CuR7B for anyone interested in contributing. Comments on doctors should already be enabled via Disqus if you scroll to the bottom of a profile page. Yes, I definitely want to add that data!

radlad 3y ago

I'm also interested in writing a scraper for my site (Michigan) if time allows.

jamesfly 3y ago
Thank you. I'm going to refine it a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B https://discord.gg/bXdk4CuR7B for anyone interested in contributing.

noman-land 3y ago

Please open source this. I would love to help.

jamesfly 3y ago
Thank you. I'm going to refine the site and data a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B https://discord.gg/bXdk4CuR7B for anyone interested in contributing.

mywrathacademia 3y ago

Open sourcing this would be a great idea and a lot of people would happily contribute, it already looks great. The biggest secret is that most doctors do more harm than good when a patient has anything worse than a cold. The psychopaths who couldn't make it as CEOs or serial killers end up becoming doctors, in charge of people's health.

jamesfly 3y ago
Thank you. I'm going to refine the site a bit more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B https://discord.gg/bXdk4CuR7B for anyone interested in contributing.

deleted 3y ago

[deleted]