7 ms·

What's a good way to learn how to do this work, if you have scripting knowledge but not physics? I am working on a set of images to parse with OpenCV (non astro

What's a good way to learn how to do this work, if you have scripting knowledge but not physics? I am working on a set of images to parse with OpenCV (non astronomical) but could apply the same techniques as those in the OP. Often I find lately the barrier to a project is sociological, not technical.

anovikov 4y ago
If you have 4 dots on a photo or different photos and you want to find out if they belong to the same object (shot at different times) and find orbit of that object, it's trivial to do it. You find centroids of them using opencv, then recognise the plate using the like of www.astrometry.net to build WCS and convert (x,y) coordinates into (Ra, Dec) coordinates, and then you can run orbit determination code and check if resulting orbit is sane (that is, is within certain boundaries of "possible" asteroid orbits). If these are just random dots that don't belong to the same object, or some of them are noise/false detections (like hot pixel, meteor, or similar), the orbit will be invalid/impossible. Trick is now, how to put together these possible groups of 4 for detection. Because they say they have 68 billion observations, certainly combining each to each 4 times is not possible. There must be some heuristic and/or trick to it.
- hkgjjgjfjfjfjf 4y ago
- dontbenebby 4y ago
  >Because they say they have 68 billion observations, certainly combining each to each 4 times is not possible. There must be some heuristic and/or trick to it. Maybe define heuristics for what ISN'T useful first to pare down the data set (looking for negatives), then focus on the sort where you're actually looking for positives? When I'm sorting photos for art purposes, I shoot a lot, years at a time, then spend a month or two sorting into albums, I actualy lost hours of work because I had to restore my phone from a backup recently. But the general idea, when doing a completely manual process, is you grab say, 1kish of 20k images, then from there further split them into 3-4 albums. Now if they were astronomy photos then you could go into that set of about 500 and manually look, think what's a heuristic to get me from where I started to here more quickly. (Sorry if articulating myself poorly, this site spikes my anxiety -- if I give useful information people just siphon it up, but if you give an opinion they don't like, folks get real ornery.)
  - anovikov 4y ago
    In fact, it might be that their method is a lot lot simpler: - Detect any protracted images on the photos which shouldn't be protracted - Interpret ends of them as 2 observations and dots at 1/3 and 2/3 as 2 others (exploiting the fact that speed of angular motion at such a short interval is essentially constant). - Run a (very crude) orbit detection using these - Look for any suspicious dots on other images within the margin of error provided by this orbit (that is, where the object with that orbit should've been if it was real, at a time each other shot was made). - If found, use these for orbit refinement, rinse and repeat until on all shots where the object should've been apparent, it is present, or reject it as a false detection if that doesn't happen.
ad404b8a372f2b9 4y ago
The hard part of research is the domain/scientific knowledge, not the scripting. If you look at the original paper, the method is informed by their knowledge of astrophysics, and their knowledge of the current state of the field and existing methods. Many (maybe most) researchers are terrible programmers, it's a non-issue. If you want to learn how to do this kind of work you have to learn the science.
- dontbenebby 4y ago
  I guess my question is what you mean by "know the science". For example I know different types of stars and how their compositions differ, but not detailed mathematical models. If by "the science" you mean the latter, great, I can drop the idea, if the former I might be able to add value. (For example, I've read a lot about black holes, Hawking radiation etc, but not in a detailed mathematical sense, just more general ideas like the concept of an event horizon.)
  - pvg 4y ago
    You don't have to 'know the science' to the degree of detail the people driving the research part of the project are, I think the bigger problem is many of the opportunities for grunt work go to undergrads and graduate students where the hope (in theory at least) is that it also helps advance their educational goals.
    - dontbenebby 4y ago
      No, the goal is cheap labor, they hire more PhD students than there are tenure track positions, this was a common gripe when so many undergrads commited suicide at IU, they threatened to shut down the newspaper if they reported it and told grad students who were basically extorted into finishing under the threat their advisers would forever gatekeep roles that would take them out of precarity. (That's why I dropped out, worked briefly in policy, learned to hack, then sold stock out of my retirement fund to get an open source intelligence certification from Bellingcat -- because THAT MANY sets of people refused to follow basic social norms.)
  - 4y ago
gammarator 4y ago
There are open positions for software engineers working with these teams: https://b612foundation.org/open-positions/ https://b612foundation.org/open-positions/ https://www.lsstcorporation.org/lincc/job_opportunities https://www.lsstcorporation.org/lincc/job_opportunities
- dontbenebby 4y ago
  Sorry, this is a bit much for my skillset -- I'm more suited to the title of "analyst" but struggle to connect with those roles, but I'll point any folks I know who are stronger software engineers at it. (It's a common issue for us security folks: we're good at writing out pretty code that automates things on the command line with bash and a few esoteric CLI tools, but a full stack developer, I am not.)

anovikov 4y ago

If you have 4 dots on a photo or different photos and you want to find out if they belong to the same object (shot at different times) and find orbit of that object, it's trivial to do it. You find centroids of them using opencv, then recognise the plate using the like of www.astrometry.net to build WCS and convert (x,y) coordinates into (Ra, Dec) coordinates, and then you can run orbit determination code and check if resulting orbit is sane (that is, is within certain boundaries of "possible" asteroid orbits). If these are just random dots that don't belong to the same object, or some of them are noise/false detections (like hot pixel, meteor, or similar), the orbit will be invalid/impossible. Trick is now, how to put together these possible groups of 4 for detection. Because they say they have 68 billion observations, certainly combining each to each 4 times is not possible. There must be some heuristic and/or trick to it.

hkgjjgjfjfjfjf 4y ago
dontbenebby 4y ago
>Because they say they have 68 billion observations, certainly combining each to each 4 times is not possible. There must be some heuristic and/or trick to it. Maybe define heuristics for what ISN'T useful first to pare down the data set (looking for negatives), then focus on the sort where you're actually looking for positives? When I'm sorting photos for art purposes, I shoot a lot, years at a time, then spend a month or two sorting into albums, I actualy lost hours of work because I had to restore my phone from a backup recently. But the general idea, when doing a completely manual process, is you grab say, 1kish of 20k images, then from there further split them into 3-4 albums. Now if they were astronomy photos then you could go into that set of about 500 and manually look, think what's a heuristic to get me from where I started to here more quickly. (Sorry if articulating myself poorly, this site spikes my anxiety -- if I give useful information people just siphon it up, but if you give an opinion they don't like, folks get real ornery.)
- anovikov 4y ago
  In fact, it might be that their method is a lot lot simpler: - Detect any protracted images on the photos which shouldn't be protracted - Interpret ends of them as 2 observations and dots at 1/3 and 2/3 as 2 others (exploiting the fact that speed of angular motion at such a short interval is essentially constant). - Run a (very crude) orbit detection using these - Look for any suspicious dots on other images within the margin of error provided by this orbit (that is, where the object with that orbit should've been if it was real, at a time each other shot was made). - If found, use these for orbit refinement, rinse and repeat until on all shots where the object should've been apparent, it is present, or reject it as a false detection if that doesn't happen.

ad404b8a372f2b9 4y ago

The hard part of research is the domain/scientific knowledge, not the scripting. If you look at the original paper, the method is informed by their knowledge of astrophysics, and their knowledge of the current state of the field and existing methods. Many (maybe most) researchers are terrible programmers, it's a non-issue. If you want to learn how to do this kind of work you have to learn the science.

dontbenebby 4y ago
I guess my question is what you mean by "know the science". For example I know different types of stars and how their compositions differ, but not detailed mathematical models. If by "the science" you mean the latter, great, I can drop the idea, if the former I might be able to add value. (For example, I've read a lot about black holes, Hawking radiation etc, but not in a detailed mathematical sense, just more general ideas like the concept of an event horizon.)
- pvg 4y ago
  You don't have to 'know the science' to the degree of detail the people driving the research part of the project are, I think the bigger problem is many of the opportunities for grunt work go to undergrads and graduate students where the hope (in theory at least) is that it also helps advance their educational goals.
  - dontbenebby 4y ago
    No, the goal is cheap labor, they hire more PhD students than there are tenure track positions, this was a common gripe when so many undergrads commited suicide at IU, they threatened to shut down the newspaper if they reported it and told grad students who were basically extorted into finishing under the threat their advisers would forever gatekeep roles that would take them out of precarity. (That's why I dropped out, worked briefly in policy, learned to hack, then sold stock out of my retirement fund to get an open source intelligence certification from Bellingcat -- because THAT MANY sets of people refused to follow basic social norms.)
- 4y ago

gammarator 4y ago

There are open positions for software engineers working with these teams: https://b612foundation.org/open-positions/ https://b612foundation.org/open-positions/ https://www.lsstcorporation.org/lincc/job_opportunities https://www.lsstcorporation.org/lincc/job_opportunities

dontbenebby 4y ago
Sorry, this is a bit much for my skillset -- I'm more suited to the title of "analyst" but struggle to connect with those roles, but I'll point any folks I know who are stronger software engineers at it. (It's a common issue for us security folks: we're good at writing out pretty code that automates things on the command line with bash and a few esoteric CLI tools, but a full stack developer, I am not.)