7 ms·
PyWhat: Identify Anything
- gigatexal 5y agoThere really is a Python module for everything.
- cosmic_quanta 5y agoIn the same vague theme of "I don't know what I'm dealing with" : https://github.com/ajalt/fuckitpy https://github.com/ajalt/fuckitpy
- kilnr 5y agoAnother one sort of related is hachoir, and specifically the hachoir-metadata script: https://github.com/vstinner/hachoir https://github.com/vstinner/hachoir
- antongribok 5y agoI can't decide what I'm more impressed with: The 110% code coverage, the downloads per month, or the license.
- bee_rider 5y agoI'm not sure if it was intentional or not, but I love that the Hebrew characters that they found look visually similar to Nan.
- dangrie158 5y agoRead the test.py then ;)
- 0-_-0 5y agoI like the Versioning section: The web devs tell me that fuckit's versioning scheme is confusing, and that I should use "Semitic Versioning" instead. So starting with fuckit version ה.ג.א, package versions will use Hebrew Numerals.
- doubleunplussed 5y agoI was disappointed that it's actually 4.8.1 in the setup.py
- ehsankia 5y agoDidn't know there was a python version, but as the README says, this is based on the classic fuckitjs: https://github.com/mattdiamond/fuckitjs https://github.com/mattdiamond/fuckitjs
- belval 5y ago> Still getting errors? Chain fuckit calls. This module is like violence: if it doesn't work, you just need more of it. From the README. Jokes aside it seems like something I could actually have a use for.
- cecilpl2 5y agoCool, but it seems like 80% of the results in your example demos are Youtube video IDs.
- Mogzol 5y agoI find it kind of funny that they would choose to show those as demos when it's obvious that most of them really aren't Youtube video IDs. Like "Accept-Lang" is pretty obviously not actually a video ID, even if it matches the [A-Za-z0-9_-]{11} pattern and technically could be a valid ID. On the other hand, I don't know how you would actually verify whether an 11-character string is or isn't a Youtube ID (short of querying Youtube itself), so I suppose it's nice that potential IDs are shown, just seems they have a very high chance of being false positives.
- meowface 5y agoYou can reduce false positives by trying to identify base64-seeming strings that are 11 characters long. Above a certain amount of entropy and uppercase/lowercase/digit distribution, etc. You might risk false negatives, but different flags for different levels of sensitivity could help with that.
- ehsankia 5y agoAt least from the config I see that the rarity for it is set pretty low (0.2), so you can filter our the low rarity stuff. I would probably run it by default with like --rarity 0.5 or something.
- bbno4 5y agoAuthor here, that's because I made the gifs like weeks before the actual program was made and I am too lazy to spend an hour making gifs again
- dec0dedab0de 5y agoAt first I thought this was going to be like google lens. It's instead a way to probabilistically Identify things in strings. I have wished for this to exist, and made my own dumbed down version of it before. This could be very useful for less fragile screen scraping.
- acidbaseextract 5y agoSome more great probabilistic python libraries: https://github.com/datamade/usaddress https://github.com/datamade/usaddress - "usaddress is a Python library for parsing unstructured address strings into address components, using advanced NLP methods." https://github.com/datamade/probablepeople https://github.com/datamade/probablepeople - "probablepeople is a python library for parsing unstructured romanized name or company strings into components, using advanced NLP methods."
- nerdponx 5y agoI have used and benefited tremendously from both of these libraries. While the methods are sound, the training data they used is not that comprehensive. He will probably want to apply some heuristic clean up before and after processing. Or if your organization has a lot of time and money, add additional training data.
- ok123456 5y agohttps://github.com/chardet/chardet https://github.com/chardet/chardet - Detects the most likely encoding of a raw byte string.
- ssivark 5y agoNice! In the same spirit, here’s an interesting talk on using Gen.jl (a probabilistic programming library/framework) for cleaning messy data in tables: https://youtu.be/vUxrtqY84AM https://youtu.be/vUxrtqY84AM
- cge 5y agoNote that for the usaddress library, as I was surprised that it failed spectacularly when I played with it: the 'us' in the name appears to refer to the US, not 'unstructured'. There's no note of this in the readme, though there is a small US flag emoji in the Github about string.
- lettergram 5y agoWe built a similar tool, utilizing a CNN. It works on structured (and unstructured) data and provides additional info. https://github.com/capitalone/DataProfiler https://github.com/capitalone/DataProfiler Cool part, is you can “extend” the intern name-entity recognition model by refitting with the new data. Out if the box, the DataProfiler does something like 18 entities including most of the PII dada.
- deleted 5y ago[deleted]
- lapp0 5y agoWhy would I need this when I already have a full Tome of Identify with 50 charges?
- saas_sam 5y agoPyWhat only uses one inventory slot vs. 2 for Tome. That's one extra SoJ!
- nknealk 5y agoTome of identify only holds 20 charges
- AbraKdabra 5y agoI'm pretty sure he's playing the Project Diablo II mod.
- mgraczyk 5y agoSomewhat odd result for s3.amazonaws.com: ~> python3 -m pywhat s3.amazonaws.com Possible Identification ┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓ ┃ Matched Text ┃ Identified as ┃ Description ┃ ┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩ │ s3.amazonaws.com │ JSON Web Token (JWT) │ None │ └──────────────────┴──────────────────────┴─────────────┘
- vitus 5y agoI'm admittedly not impressed by the pcap processing. It identifies a bunch of fragments of HTTP headers as "YouTube Video ID". Meanwhile, I can get the same info and more by running $ strings FollowTheLeader.pcap *]?> GET / HTTP/1.1 Host: 10.0.2.5 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Connection: keep-alive Upgrade-Insecure-Requests: 1 Pragma: no-cache Cache-Control: no-cache HTTP/1.0 200 OK Server: SimpleHTTP/0.6 Python/3.7.3rc1 Date: Sun, 14 Jul 2019 02:42:13 GMT Content-type: text/html Content-Length: 105 Last-Modified: Sun, 14 Jul 2019 02:41:10 GMT <h1>My Flag Web Page</h1> <p>Hi there! Have a flag!</p> <p>Here is your flag: ctfa{terrific_traffic}</p>
- iab 5y agoHas anyone tried using this on the GIMBAL/GOFAST UAP videos?
- mkl 5y agoWhy are these screenshots animated? The command is still visible in the final frame, and the final frame shows the output we're interested in, but not long enough to read and understand it.
- bbno4 5y agothat's a good point, i'll take it into consideration thanks :)
- rainonmoon 5y agoBee is a really tremendous and generous developer. I use a few of their other projects near-daily (Rustscan especially has changed my life.) Definitely one of those open source devs you follow just to see whatever they come up with next.
- bbno4 5y agoHey! Bee here :))) Thank you so much for the kind words!!! I hope to make things that help people in their lives :)
- MrYellowP 5y agoI am confused and amazed at the same time. What is this sorcery?
- Daneil 5y agoGood program! I think? it's can bi useful in OSINT, or many more things!
- bbno4 5y agohaha yup!! I love OSINT :)
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]