8 ms·
1.5 TB of Dark Net Market scrapes
- b6 11y agogwern, you are an absolute force of nature when it comes to generating and collecting and presenting information in a useful way. Thank you.
- icpmacdo 11y agoI hope a he gets some good donations from HN, Ill send him a few dollars in BTC.
- deleted 11y ago[deleted]
- curiousg 11y agoWhat amazing work! I am very interested in doing research with Tor and a dataset like this could make my job a heck of a lot easier. I have a legal question though: Are your scrapes text only? Before I work with this dataset, I want to make sure that there's no possibility it contains illegal images (child porn).
- mslev 11y agoLower down on the page, he says he did scrape at least one site with such images, although he specifically only took text. Can't verify that this was the case for all scraped sites.
- fineman 11y agoHow about ascii art? Actually, this is an interesting topic. Poisoning a dataset. CP would work for private security investigators, and to poison against government investigators you could use leaked classified secrets. Could you work around this by operating on the files on VPS you don't own, streaming a very low-res ('Basilisk'-proof - https://en.wikipedia.org/wiki/BLIT_(short_story) https://en.wikipedia.org/wiki/BLIT_(short_story) ) remote desktop image.
- curiousg 11y agoPossession laws are pretty strict and hard to decode. I wouldn't want to be the test case in court. The idea of "poisoning" a dataset is an interesting theoretical. But in practice, I just want to judge the likelihood that the dataset is poisoned by the presence of images. If it is then there's not much I can do with it.
- setuptools 11y agoYes, this absolutely needs to be clarified by Gwern. This is a very dangerous thing to link researchers to if it contains any illegal content.
- fineman 11y agoNonsense. Gwern doesn't need to do anything for anyone. It's an interesting issue, and a way investigators may be attacked, but it's their responsibility alone. There exists data. This is that data. The data may bite. Touch the data at your own risk.
- tomcam 11y agoUm... Dark market
- tomcam 11y agoUm.. It says Black Market right on the tin.
- gwern 11y agoThey are generally not text only. I feel that images are useful to allow browsing the markets as they were and may be highly valuable in their own right as research material, so I tried to collects images where applicable. (The forums usually did not support any kind of image upload other than avatars, so this is more relevant to the markets than forum scrapes.) As far as CP goes, there should be essentially zero CP anywhere in the archive. DNM users almost universally loathe CP, and no market has ever dared to permit sales. (You may find this funny: CP is so taboo, on the DNMs like elsewhere, that it's been used in at least one attack - SR2's DoctorClu/Brian Farrell infamously attacked a rival market's forum by posting CP to it.)
- userbinator 11y agoDNM users almost universally loathe CP, and no market has ever dared to permit sales. These users are willing to do so many other illegal things, but the thought of being known as a pedophile or supporting pedophilia in any way, is completely abhorrent to them? Interesting datapoint.
- RickHull 11y ago> These users are willing to do so many other illegal things, but the thought of being known as a pedophile or supporting pedophilia in any way, is completely abhorrent to them? I'm pretty sure the distinction is that voluntary transactions have no victims, and DNM folks care more about morality and ethics than legalities.
- pluma 11y agoMorality is not binary. That someone is fine with some things that are illegal does not mean that they are fine with everything that is illegal.
- davidgerard 11y agoIt's quite usual for career criminals to have a detailed and complicated worked-out ethical system, where that is a victimless crime but that is reprehensible.
- joshmn 11y agoThis is so cool. Thanks gwern. If someone's feeling bored, you're welcome to put the entire archive on a web server for us to look at...... Or maybe I'll just do it.
- fractalcat 11y agoYou mean like this? https://archive.org/download/dnmarchives https://archive.org/download/dnmarchives
- gwern 11y agoI don't think that works. It's not remotely browsable or searchable. It would be quite challenging to put these scrapes up, anyway. They're regular wget crawls with a regular directory/file structure, the problem is that there's so much material and so many files that it can be almost impossible to find what you are looking for... (Plus you need to rewrite links into relative links to make everything render properly.)
- joshmn 11y agoHmm. Now I'm thinking that I might end up using your idea (scraping the dark web) and using something like httrack[0] to do exactly that: structure. [0] https://en.wikipedia.org/wiki/HTTrack https://en.wikipedia.org/wiki/HTTrack
- ryanlol 11y ago>collating and creating these scrapes has absorbed an enormous amount of my time & energy due to the need to solve CAPTCHAs,... Have you considered automating this?
- smeyer 11y agoThe whole point of CAPTCHAs is to be difficult to automate. Or are you suggesting automating farming out the CAPTCHA solutions to cheap workers?
- jMyles 11y agoI assumed it was a joke. But maybe not?
- MichaelGG 11y agoThere are plenty of services that do exactly that. They're pretty cheap.
- ryanlol 11y agoThere's a plenty of service providers that sell APIs to captcha solving services at reasonable prices.
- amenghra 11y agoOr if you don't want to spend money, you can always re-host things on your own site. Let your visitors do the work for you.
- davidgerard 11y agoThe main problem is that most CAPTCHAs are terrible, and do a better job of keeping humans out than robots.
- timboslice 11y agoYes. Quick list of providers I've worked with, in no particular order: http://www.imagetyperz.com/ http://www.imagetyperz.com/ http://expertdecoders.com/ http://expertdecoders.com/ http://www.deathbycaptcha.com/user/login http://www.deathbycaptcha.com/user/login https://de-captcher.com/ https://de-captcher.com/ http://bypasscaptcha.com/ http://bypasscaptcha.com/
- branchless 11y agoThis is why I love the internet. This article has given me a fascinating glimpse into a world I have no idea about. Author: thank you so much for taking the time to document this.
- deleted 11y ago[deleted]
- ethanbond 11y agoYou should take the time to read his other posts. Gwern is one of the most fascinating people/websites I know of.
- branchless 11y agoThanks, I will read at least another by him.
- jmduke 11y agoUnrelated to the darknet, but this Twitter account evoked a very similar feeling in me (random snippets from userboards in the 80's and early 90's): https://twitter.com/wwwtxt https://twitter.com/wwwtxt
- oxide 11y agowhat a great twitter account. thanks for bringing this to my attention.
- WalterGR 11y ago"I heard about the Apple Watch recently and was going to check it out—but not now. It can't even transmit or input data. ☯93JAN" Prescient.
- hfsktr 11y agoSome of them a bit less clairvoyant about the future: "The actual date for the end of the world is July 5, 1998. ☯92NOV" It is interesting how many were accurate or onto something. I just didn't expect it. edit: Found this gem after wasting more time that I cared to. "I think the future of personal communications holds great things in store for us, but privacy won't be one those things. ☯94AUG"
- acosmism 11y agoyou could also use a library to handle captchas. have a look at 'tesseract'
- rcpt 11y agoI've had a very slow-moving hobby project to parse and analyze a subset of this data: https://github.com/rcompton/black-market-recommender-systems https://github.com/rcompton/black-market-recommender-systems So far I've had some ok results along the lines of "91.7% of vendors who sold speed and MDMA also sold ecstasy" http://ryancompton.net/2015/03/24/darknet-market-basket-analysis/ http://ryancompton.net/2015/03/24/darknet-market-basket-anal... I am working on extending this to markets besides evolution now.
- ez123 11y agoJust FYI, MDMA is ecstasy.
- deleted 11y ago[deleted]
- rcpt 11y agoI chose that statistic to be a bit tongue-in-cheek, cf > While 'ecstasy' is the popular name for MDMA, the functional definition of ecstasy is any pill represented as MDMA on the street. Ecstasy pills are notoriously unreliable in content, more so than most other street drugs, and commonly contain either caffeine, ephedrine, amphetamines, MDA, MDE, DXM, or--in rare cases--DOB, and don't necessarily contain MDMA or any psychoactive. https://www.erowid.org/chemicals/mdma/mdma_basics.shtml https://www.erowid.org/chemicals/mdma/mdma_basics.shtml
- escape_goat 11y agoTo follow up on your response and clarify some: in your dataset, not only do 91.7% of the vendors of 'speed and MDMA' also sell 'Ecstasy'; 76.8% of the vendors of 'Stimulants and Ecstasy' also sell 'MDMA'. If I understand correctly, this means that vendors were significantly more likely to mention 'Ecstasy' in a product name (or description) if they claimed to be selling 'MDMA' than they were to mention 'MDMA' in a product name/description if they claimed to be selling 'Ecstasy', reinforcing the point that you have just made.
- 11y ago
- Moshe_Silnorin 11y agoGwern has a Patreon page now for anyone interested in supporting his research: https://www.patreon.com/gwern?ty=h https://www.patreon.com/gwern?ty=h
- mapt 11y agoGwern does more interesting things with lower monetary burn rate than most anyone I've met.
- deleted 11y ago[deleted]
- stevewepay 11y agoWhat is the legality with respect to downloading this file? Could it contain material that would put us at legal risk?
- NhanH 11y agoI really don't know the answer. But I just want to remark that it seems to be a terrible situation that more than one of us have to wonder about the legality of downloading the dataset (there was a comment thread below on the same topic)
- brandon272 11y agoI wouldn't download it unless you want a visit from the feds.
- belorn 11y agoSince when would it be illegal to posses statistics. Are we afraid that dark markets might claiming proprietary ownership of that data?
- SergeyHack 11y ago"HOW TO CRAWL MARKETS" section has good tips for general crawling as well.