5 ms·

Getting my models dunked on by people who can't open MS Outlook more than 3 tries out of 5, however, have a remarkable depth and insight into their chosen domai

Getting my models dunked on by people who can't open MS Outlook more than 3 tries out of 5, however, have a remarkable depth and insight into their chosen domain of expertise. It's rather humbling. Collaborating with nontechnical people is oddly my favorite part of doing MLE work right now. It wasn't the case when I did basic web/db stuff. They see me as a magician. I see them as voodoo priests and priestesses. When we get something trained up and forecasting that we both like, it's super fulfilling. I think for both sides. Most of my modeling is healthcare related. I tease insights out of a monstrous data lake of claims, Rx, doctor notes, vital signs, diagnostic imagery, etc. What is also monstrous is how accessible this information is. HIPAA my left foot. Since you seemed to be asking about the temporal realities, it's about 3 hours of meetings a week, probably another 3 doing task grooming/preparatory stuff, fixing some ETL problem, or doing a one-off query for the business, the rest is swimming around in the data trying to find a slight edge to forecast something that surprised us for a $million or two using our historical snapshots. It's like playing wheres waldo with math. And the waldo scene ends up being about 50TB or so in size. :D

soared 2y ago
3 hours of meetings a week, that’s incredible. Sounds like your employer understands and values your time!
- visarga 2y ago
  13 meetings/week, at least one full day of work wasted for me
- schmookeeg 2y ago
  They really do. This has been my longest tenure at any position by far and Engineer QoL is a massive part of it. Our CTO came up through the DBA/Data/Engineering Management ranks and the empathy is solidly there. As we grow, I'm ever watchful for our metamorphosis into a big-dumb-company, but no symptoms yet. :)
saulrh 2y ago
> HIPAA my left foot. That was my experience as well - training documentation for fresh college grads (i.e. me) directed new engineers to just... send SQL queries to production to learn. There was a process for gaining permissions, there were audit logs, but the only sign-off you needed was your manager, permission lasted 12 months, and the managers just rubber-stamped everyone. That was ten years ago. Every time I think about it I find myself hoping that things have gotten better and knowing they haven't.
- bick_nyers 2y ago
  ... you didn't have a UAT environment?
  - saulrh 2y ago
    There were a couple "not prod" environments, but they were either replicated directly from prod or so poorly maintained that they were unusable (empty tables, wrong schemas, off by multiple DB major versions, etc), no middle ground. So institutional culture was to just run everything against prod (for bare selects that could be copied and pasted into the textbox in the prod-access web tool) or a prod replica (for anything that needed a db connection). The training docs actually did specify Real Production, and first-week tasks included gaining Real Production access. If I walked in and was handed that training documentation today I'd raise hell and/or quit on the spot, but that was my first job out of college - it was basically everyone's first job out of college, they strongly preferred hiring new graduates - and I'd just had to give up on my PhD so I didn't have the confidence, energy, or pull to do anything about it, even bail out. That was also the company where prod pushes happened once a month, over the weekend, and were all hands on deck in case of hiccups. It was an extraordinarily strong lesson in how not to organize software development. (edit: if what you're really asking is "did every engineer have write access to production", the answer was, I believe, that only managers did, and they were at least not totally careless with it. not, like, actually responsible, no "formal post-mortem for why we had to use break-glass access", but it generally only got brought out to unbreak prod pushes. Still miserable.)
  - roughly 2y ago
constantinum 2y ago
> I tease insights out of a monstrous data lake of claims, Rx, doctor notes, vital signs I'm curious to know the tech stack behind converting unstructured to structured data(for reporting and analysis)
- dax77 2y ago
  Take a look at AWS Healthlake and AWS Comprehend Medical
hamasho 2y ago
I worked on a project to analyze endoscope videos to find diseases. I examined a lot of images and videos annotated with symptoms of various diseases labeled by assistants and doctors. Most of them are really obvious, but others are almost impossible to detect. In rare cases, despite my best efforts, I couldn't see any difference between the spot labeled as a symptom of cancer and the surrounding area. There's no a-ha moment, like finding an insect mimicking its environment. No matter how many times I tried, I just couldn't see any difference.
- aswegs8 2y ago
  Mind sharing how to get a foot into the field? I've got a good amount of domain knowledge from my studies in life science and rather meager experience from learning to code on my own for a few years. It seems like I cant compete with CS majors and gotta find a way to leverage my domain knowledge.
  - hamasho 2y ago
    I'm not an expert in machine learning, but rather a web developer and data engineer helping develop a system to detect diseases from endoscopy images using the model developed by other ML engineers. And it was 5 years ago when I worked on the project, so please take it with a grain of salt. If you want to learn machine learning for healthcare in general, it may help to start problems with tabular data like CSVs instead of images. Image processing is a lot harder, and takes a lot of time and computational power. But it's best to learn what you're interested in the most. Anyway, first you need to be familiar with basic; Python, machine learning, and popular libraries like scikit-learn, matplotlib, numpy, and pandas. Those are tons of articles, textbooks, and videos to help you learn them. If you grasp the basics, I think it's better to learn from actual code to train/evaluate models rather than more theories. Kaggle may be a good starting point. They host a lot of competitions for machine learning problems. There are easy competitions for beginners, and competitions and datasets in the medical field. You can view notebooks (actual code to solve those problems well written by experts) and popular ones are very educational. You can learn a lot by reading those code, understanding concepts and how to use libraries, and modifying some code to see how it changes the result. ChatGPT is also helpful. If you want to learn image classification, the technology used to detect objects from images and videos is called image classification and object detection. It uses CNN, one of the deep neural networks. You also need to learn basic image processing, how to train a deep neural network, how to evaluate, and libraries like OpenCV/Pillow/PyTorch/TorchVision. There are a lot of image classification competitions in the medical field on Kaggle too[0][1]. To run those notebooks, I recommend Google Colab. Image processing often uses a lot of GPUs, and you may not have GPUs, or even if you have it's difficult to set up the right environment. It's easier to use those dedicated cloud services and it doesn't cost much. It's hard to learn, but sometimes fun, so enjoy your journey! [0] https://www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia https://www.kaggle.com/datasets/paultimothymooney/chest-xray... [1] https://www.kaggle.com/code/arkapravagupta/endoscopy-multiclassification-explained https://www.kaggle.com/code/arkapravagupta/endoscopy-multicl...
nomilk 2y ago
The Dead Internet Theory says most activity on the internet is by bots [1]. The Dead Privacy Theory says approximately all private data is not private; but rather is accessible on whim by any data scientist, SWE, analyst, or db admin with access to the database. [1] https://en.wikipedia.org/wiki/Dead_Internet_theory https://en.wikipedia.org/wiki/Dead_Internet_theory
- htrp 2y ago
  > The Dead Privacy Theory says approximately all private data is not private; but rather is accessible on whim by any data scientist, SWE, analyst, or db admin with access to the database. I like this so much I'm definitely stealing it!
- bearjaws 2y ago
  Damn, I've talked about this many times at my last job (startup that went from 100k patients to ~2.5M in 5 years). I love the name Dead Privacy Theory
- mr_toad 2y ago
  At least more of that access is logged now. It used to be only the production databases that were properly logged, now it’s more common for every query to be logged, even in dev environments. The next step will be more monitoring of those logs.
htrp 2y ago
>Getting my models dunked on by people who can't open MS Outlook more than 3 tries out of 5, however, have a remarkable depth and insight into their chosen domain of expertise. It's rather humbling. The people who have lasted in those roles have built up a large degree of intuition on how their domains work (or they would've done something else).
ProjectArcturis 2y ago
What business are you in that predicting health data can make you millions?
- dr_kiszonka 2y ago
  Insurance, health benefits.
- apwheele 2y ago
  The scale of health insurance claims is incredible, my company has a process that simply identifies when car insurance should pay the bill instead of medicaid/medicare post traffic accident (subrogation). Seems a minor thing right? We process right at 1 billion a year in related claims (and I don't know our US market share, maybe like 10-20%). I am guessing every datascientist that works for a BlueCross BlueShield at individual states deals with processes that touch multiple-million dollars of claims. We even now have various dueling systems -- one company has a model to tack on more diagnoses to push up the bill, another has a process to identify that upcoding. One company has models to auto accept/deny claims, another has an automatic prior authorization process to try to usurp that later denial stage, etc.
  - DontchaKnowit 2y ago
    Man, respectfully, sounds like you are doing the dirty immoral work of insurabce companies. AI algos for adding diagnoses to rack up the bill? That is completely disgusting. Wholly unnacceptable and if I ever find out ive been the target of such a scam I will raise holy hell.
    - apwheele 2y ago
      I am speaking more generally of the industry. You can look at my profile and follow the bread-trail to see whom I work for. It is not insurance, we are a client of state systems, and I work to do the opposite of what you suggest (identify fraud/waste/abuse in insurance claims). I do not know the chances that any individual provider uses a system to identify what codes to tack onto your bill.
conkeisterdoor 2y ago
This sounds almost exactly like my day-to-day as a solo senior data engineer — minus building and training ML models, and I don't work in healthcare. My peers are all very non-technical business directors who are very knowledgeable about their domains, and I'm like a wizard who can conjure up time savings/custom reporting/actionable insights for them. Collaborating with them is great, and has been a great exercise in learning how to explain complex ideas to non-technical business people. Which has the side effect of helping me get better at what I do (because you need a good understanding of a topic to be able to explain it both succinctly and accurately to others). It has also taught me to appreciate the business context and reasoning that can drive decisions about how a business uses or develops data/software.
kvakerok 2y ago
HIPAA your left foot because nobody reads fine print anymore and signs their soul away for a one-time $60 discount.

soared 2y ago

3 hours of meetings a week, that’s incredible. Sounds like your employer understands and values your time!

visarga 2y ago
13 meetings/week, at least one full day of work wasted for me
schmookeeg 2y ago
They really do. This has been my longest tenure at any position by far and Engineer QoL is a massive part of it. Our CTO came up through the DBA/Data/Engineering Management ranks and the empathy is solidly there. As we grow, I'm ever watchful for our metamorphosis into a big-dumb-company, but no symptoms yet. :)

saulrh 2y ago

> HIPAA my left foot. That was my experience as well - training documentation for fresh college grads (i.e. me) directed new engineers to just... send SQL queries to production to learn. There was a process for gaining permissions, there were audit logs, but the only sign-off you needed was your manager, permission lasted 12 months, and the managers just rubber-stamped everyone. That was ten years ago. Every time I think about it I find myself hoping that things have gotten better and knowing they haven't.

bick_nyers 2y ago
... you didn't have a UAT environment?
- saulrh 2y ago
  There were a couple "not prod" environments, but they were either replicated directly from prod or so poorly maintained that they were unusable (empty tables, wrong schemas, off by multiple DB major versions, etc), no middle ground. So institutional culture was to just run everything against prod (for bare selects that could be copied and pasted into the textbox in the prod-access web tool) or a prod replica (for anything that needed a db connection). The training docs actually did specify Real Production, and first-week tasks included gaining Real Production access. If I walked in and was handed that training documentation today I'd raise hell and/or quit on the spot, but that was my first job out of college - it was basically everyone's first job out of college, they strongly preferred hiring new graduates - and I'd just had to give up on my PhD so I didn't have the confidence, energy, or pull to do anything about it, even bail out. That was also the company where prod pushes happened once a month, over the weekend, and were all hands on deck in case of hiccups. It was an extraordinarily strong lesson in how not to organize software development. (edit: if what you're really asking is "did every engineer have write access to production", the answer was, I believe, that only managers did, and they were at least not totally careless with it. not, like, actually responsible, no "formal post-mortem for why we had to use break-glass access", but it generally only got brought out to unbreak prod pushes. Still miserable.)
- roughly 2y ago

constantinum 2y ago

> I tease insights out of a monstrous data lake of claims, Rx, doctor notes, vital signs I'm curious to know the tech stack behind converting unstructured to structured data(for reporting and analysis)

dax77 2y ago
Take a look at AWS Healthlake and AWS Comprehend Medical

hamasho 2y ago

I worked on a project to analyze endoscope videos to find diseases. I examined a lot of images and videos annotated with symptoms of various diseases labeled by assistants and doctors. Most of them are really obvious, but others are almost impossible to detect. In rare cases, despite my best efforts, I couldn't see any difference between the spot labeled as a symptom of cancer and the surrounding area. There's no a-ha moment, like finding an insect mimicking its environment. No matter how many times I tried, I just couldn't see any difference.

aswegs8 2y ago
Mind sharing how to get a foot into the field? I've got a good amount of domain knowledge from my studies in life science and rather meager experience from learning to code on my own for a few years. It seems like I cant compete with CS majors and gotta find a way to leverage my domain knowledge.
- hamasho 2y ago
  I'm not an expert in machine learning, but rather a web developer and data engineer helping develop a system to detect diseases from endoscopy images using the model developed by other ML engineers. And it was 5 years ago when I worked on the project, so please take it with a grain of salt. If you want to learn machine learning for healthcare in general, it may help to start problems with tabular data like CSVs instead of images. Image processing is a lot harder, and takes a lot of time and computational power. But it's best to learn what you're interested in the most. Anyway, first you need to be familiar with basic; Python, machine learning, and popular libraries like scikit-learn, matplotlib, numpy, and pandas. Those are tons of articles, textbooks, and videos to help you learn them. If you grasp the basics, I think it's better to learn from actual code to train/evaluate models rather than more theories. Kaggle may be a good starting point. They host a lot of competitions for machine learning problems. There are easy competitions for beginners, and competitions and datasets in the medical field. You can view notebooks (actual code to solve those problems well written by experts) and popular ones are very educational. You can learn a lot by reading those code, understanding concepts and how to use libraries, and modifying some code to see how it changes the result. ChatGPT is also helpful. If you want to learn image classification, the technology used to detect objects from images and videos is called image classification and object detection. It uses CNN, one of the deep neural networks. You also need to learn basic image processing, how to train a deep neural network, how to evaluate, and libraries like OpenCV/Pillow/PyTorch/TorchVision. There are a lot of image classification competitions in the medical field on Kaggle too[0][1]. To run those notebooks, I recommend Google Colab. Image processing often uses a lot of GPUs, and you may not have GPUs, or even if you have it's difficult to set up the right environment. It's easier to use those dedicated cloud services and it doesn't cost much. It's hard to learn, but sometimes fun, so enjoy your journey! [0] https://www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia https://www.kaggle.com/datasets/paultimothymooney/chest-xray... [1] https://www.kaggle.com/code/arkapravagupta/endoscopy-multiclassification-explained https://www.kaggle.com/code/arkapravagupta/endoscopy-multicl...

nomilk 2y ago

The Dead Internet Theory says most activity on the internet is by bots [1]. The Dead Privacy Theory says approximately all private data is not private; but rather is accessible on whim by any data scientist, SWE, analyst, or db admin with access to the database. [1] https://en.wikipedia.org/wiki/Dead_Internet_theory https://en.wikipedia.org/wiki/Dead_Internet_theory

htrp 2y ago
> The Dead Privacy Theory says approximately all private data is not private; but rather is accessible on whim by any data scientist, SWE, analyst, or db admin with access to the database. I like this so much I'm definitely stealing it!
bearjaws 2y ago
Damn, I've talked about this many times at my last job (startup that went from 100k patients to ~2.5M in 5 years). I love the name Dead Privacy Theory
mr_toad 2y ago
At least more of that access is logged now. It used to be only the production databases that were properly logged, now it’s more common for every query to be logged, even in dev environments. The next step will be more monitoring of those logs.

htrp 2y ago

>Getting my models dunked on by people who can't open MS Outlook more than 3 tries out of 5, however, have a remarkable depth and insight into their chosen domain of expertise. It's rather humbling. The people who have lasted in those roles have built up a large degree of intuition on how their domains work (or they would've done something else).

ProjectArcturis 2y ago

What business are you in that predicting health data can make you millions?

dr_kiszonka 2y ago
Insurance, health benefits.
apwheele 2y ago
The scale of health insurance claims is incredible, my company has a process that simply identifies when car insurance should pay the bill instead of medicaid/medicare post traffic accident (subrogation). Seems a minor thing right? We process right at 1 billion a year in related claims (and I don't know our US market share, maybe like 10-20%). I am guessing every datascientist that works for a BlueCross BlueShield at individual states deals with processes that touch multiple-million dollars of claims. We even now have various dueling systems -- one company has a model to tack on more diagnoses to push up the bill, another has a process to identify that upcoding. One company has models to auto accept/deny claims, another has an automatic prior authorization process to try to usurp that later denial stage, etc.
- DontchaKnowit 2y ago
  Man, respectfully, sounds like you are doing the dirty immoral work of insurabce companies. AI algos for adding diagnoses to rack up the bill? That is completely disgusting. Wholly unnacceptable and if I ever find out ive been the target of such a scam I will raise holy hell.
  - apwheele 2y ago
    I am speaking more generally of the industry. You can look at my profile and follow the bread-trail to see whom I work for. It is not insurance, we are a client of state systems, and I work to do the opposite of what you suggest (identify fraud/waste/abuse in insurance claims). I do not know the chances that any individual provider uses a system to identify what codes to tack onto your bill.

conkeisterdoor 2y ago

This sounds almost exactly like my day-to-day as a solo senior data engineer — minus building and training ML models, and I don't work in healthcare. My peers are all very non-technical business directors who are very knowledgeable about their domains, and I'm like a wizard who can conjure up time savings/custom reporting/actionable insights for them. Collaborating with them is great, and has been a great exercise in learning how to explain complex ideas to non-technical business people. Which has the side effect of helping me get better at what I do (because you need a good understanding of a topic to be able to explain it both succinctly and accurately to others). It has also taught me to appreciate the business context and reasoning that can drive decisions about how a business uses or develops data/software.

kvakerok 2y ago

HIPAA your left foot because nobody reads fine print anymore and signs their soul away for a one-time $60 discount.