7 ms·
Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
- jtwaleson 11y agoOn a side note, I recently discovered https://scans.io/ https://scans.io/ where you can find pretty much all of the data that I collected as well. Might be interesting.
- metafex 11y agoCensys (https://www.censys.io/ https://www.censys.io/) is also from them and it's a search frontend for a quick lookup in their data. It can come in real handy.
- hdmoore 11y agoYou might find the processing tips on the Project Sonar wiki useful: https://github.com/rapid7/sonar/wiki/Analyzing-Datasets https://github.com/rapid7/sonar/wiki/Analyzing-Datasets Project Sonar is one of the primary contributors to scans.io. The DAP utility is handy for parsing raw x509 certificates and generating JSON output.
- yazriel 11y agoCan you comment on how many additional domains you mined - compared (for example) to the 1M domains from alexa top-1M
- jtwaleson 11y ago$ cat alexa myset myset | sort | uniq -u | wc -l 773733 0.77M of Alexa top 1M were not in my list. $ cat alexa alexa myset | sort | uniq -u | wc -l 25842205 I mined 25,842,205 additional domain names.
- yazriel 11y agoman.. the internet really is full of crappy domains... (and yes.. now i see that you mentioned it in the article.. took me time to get there)
- howaboutit 11y agoDid you consider using the gTLD zone files (from the respective registries) and the ccTLD zone files found @ http://viewdns.info/data/ http://viewdns.info/data/? A much bigger initial dataset than 25M domains right there?
- jtwaleson 11y agoNo, getting access will probably take a couple of days (or in case of viewdns, more than 100$) and thereby all the fun out of the project. If you know of any other way to get the list I'd be happy to hear it though!
- adamseabrook 11y agohttp://meanpath.com/freedirectory.html http://meanpath.com/freedirectory.html Feel free to grab a copy of our domain list. The "All domains with NS records" is the one you want. Has 191 million in it.
- breakingcups 11y agoAmazing, thanks!
- jtwaleson 11y agoWow! That's awesome!
- hdmoore 11y agoA shortcut to getting com, net, info, org, us, sk, and biz is to give premiumdrops.com $24.95/mo. You can get these for free from the TLD operators, but it takes a few weeks of snail mail (last I checked). The gTLD access via CZDAP is free, but takes a few days for approvals to process.
- adamseabrook 11y agohttps://czds.icann.org/en https://czds.icann.org/en It has been largely automated now so you can request access to the files with one click vs having to sign and email hundreds of forms. Approval seems to be automatic for most of them.
- hdmoore 11y agoThe Sonar FDNS set contains about 1.4 billion host names (50m+ domains). The FDNS set is seeded from TLD zones, CZDAP, PTR lookups (RDNS), SSL/TLS scans, and HTTP link extraction. It updates every two weeks: https://github.com/rapid7/sonar/wiki/Forward-DNS https://github.com/rapid7/sonar/wiki/Forward-DNS
- magicmu 11y agoGreat write-up; really interesting that the CPU ended up being the bottleneck in this experiment! Regarding the cost of sending this data out of AWS, did you run into any issues there using rsync? IIRC rsync copies the data over TCP, so wouldn't this end up being expensive as well? Generally, though, that was my favorite part of the experiment!
- jtwaleson 11y agoMy use case converted 10TB in only a couple of GB after processing. Downloading that was very cheap.
- visarga 11y agoSort uses only a fixed amount of memory, you can sort files larger than memory, but for such situations where you have only a few tens of millions of distinct values you can just use a python dictionary and it works even faster. While sort would shuffle data around a lot, the memory dictionary would just hold a key and a count as it gobbles the logs. It works because it is a special case of sorting where there are relatively few different values relative to the count of the whole list.
- jtwaleson 11y agoDon't you mean a python set? But yes, for use cases containing many duplicates where the result easily fits in memory, that is probably the fastest.
- bacr 11y agoFun fact: they are nearly the same implementation. See: http://markmail.org/message/ktzomp4uwrmnzao6 http://markmail.org/message/ktzomp4uwrmnzao6
- masklinn 11y agoAs one would generally expect, the backing store of most hashsets is little more than a hashmap with zero-sized/no values. In fact, that's exactly how Rust's standard library hashset is implemented since rust supports zero-sized types "in userland" (and unit `()` is a ZST): pub struct HashSet<T, S = RandomState> { map: HashMap<T, (), S> } http://doc.rust-lang.org/src/std/collections/hash/set.rs.html#112-114 http://doc.rust-lang.org/src/std/collections/hash/set.rs.htm...
- justinsaccount 11y agoa set replaces 'sort -u' or 'sort | uniq'. A dictionary replaces 'sort | uniq -c'
- justinsaccount 11y ago'sort | uniq' is another special case of this, and it is much better to replace that with 'sort -u' the 'sort' in 'sort | uniq' doesn't know you are going to be throwing away all the duplicate data. If anyone is wondering, here is an implementation of the python approach i have lying around: #!/usr/bin/env python2 import sys from collections import defaultdict c = defaultdict(int) for line in sys.stdin: c[line] += 1 top = sorted(c.items(), key=lambda (k,v): v) for k, v in top: print v, k,
- tyingq 11y agoThe key to the low cost seems to be that he needed to process 10TB. You get 10TB "data in" free, per month. Had it been 10TB more, or if he needed to run more than once a month, or if he needed to get that 10TB back out, the bill is around $920. Edit: Inbound might be unlimited free. The calculator did show me an inbound total a few times, but I can't reproduce it now.
- jtwaleson 11y agoPretty sure it's unlimited.
- tyingq 11y agoAppears that way. Still $920 if you had wanted to extract the 10TB back out though.
- mej10 11y agoYeah... that is one of the really unfortunate lock-ins with AWS. Hopefully they will add data export to their "Snowball" product, but they don't really have a lot of incentive to.
- MichaelBurge 11y agoI haven't used it, but I do see that they have both data in and data out on their Snowball pricing[1] page. Data in is free, but data out is $0.03/GB(plus $200 per job). So it would cost a minimum of $500 to use Snowball to transmit 10TB. Still, it does appear to reduce the price of getting data out of Amazon compared to using the internet. [1] https://aws.amazon.com/importexport/pricing https://aws.amazon.com/importexport/pricing
- devonkim 11y agoYou may have used an out of date calculator or it's for specific cases. AWS inbound traffic has been free since 2011 https://aws.amazon.com/blogs/aws/aws-lowers-its-pricing-again-free-inbound-data-transfer-and-lower-outbound-data-transfer-for-all-ser/ https://aws.amazon.com/blogs/aws/aws-lowers-its-pricing-agai...
- workitout 11y agoHow much did storing the data on S3 cost where you said, "However, the data is on S3" or was it there for such a transient time that it didn't cost much? Bandwidth costs in/out of S3 too? Edit: Actually I read the S3 parts again, it sounds like the CommonCrawl project pays the S3 costs, I think, since it looks like you're using their domain data?
- jtwaleson 11y agoThe results of the Common Crawl project are hosted on AWS Public Data Sets, so it's not in my account. https://aws.amazon.com/datasets/ https://aws.amazon.com/datasets/
- fitzwatermellow 11y agoSorry, but is this golang concurrent networking pattern correct: func main() { ch := make(chan string) for i := 0; i < MAX; i++ { go fetchCert(ch) } scanner := bufio.NewScanner(os.Stdin) for Scanner.Scan() { line := scanner.Text() ch <- line } } All goroutines receive on the same channel! Instead a new goroutine should be launched for each net conn. One should be able to spawn 1000s (or 1Ms) of conns and avoid ulimits using buffered chans, waitgroups, timeouts, or counters...
- andmarios 11y agoThis pattern is correct (but has a flaw). It is a simple worker pool. The first available worker will grab the first piece of work from the channel and process it. If you set MAX to 1000, you will have 1000 workers — and simultaneous connections. The flaw is that when the last piece of work gets taken from the channel, the program will end, thus the last pieces of work that at the time are being processed, will get canceled. You could mitigate this by using a second channel, that the workers will access at the end of their work, thus ensuring that it will close only when the last worker finishes its work.
- Liru 11y agoThe in-article version has a time.Sleep(2 * time.Second) after the scan loop. Not exactly reliable (waitgroups or channel signaling would be better) but better than nothing.
- jtwaleson 11y agoAs you can see from almost all commands / snippets in the article, I took the pragmatic approach for this project ;)
- Liru 11y agoWell, that whole file can be improved on (for instance: analyzeDomains and analyzeDomain can be combined into one; the range/close operators can be used on a channel), but the pattern itself (spawning a certain amount of workers instead of one for each job) is decent for certain cases. For this case, you may be right (I haven't tested it), since a) not much data is transferred, and b) He was running this on an AWS instance instead of a low-end machine. However, at a certain point, you may encounter bandwidth issues, timeouts, and the like due to local network congestion; that pattern has its uses there. I've tried writing a downloader that downloads every file it's given at once, and it went about as well as one would expect.
- jnsaff2 11y agoit will not help for data transfer pricing but for cpu/vm time spot instances can be amazing value for these short lived projects. typically 1/8 of the price of the on demand. always take care to not set your bid higher than on demand price as wild fluctuations can happen. also if you are afraid of losing your work, there is an api you can query from within the vm that tells you 2 minutes ahead that its going to get killed. also, price is per ZONE, so there are zones in the same region that people use less.
- jvehent 11y agoI have about the same amount of data in a Postgres database as part of the TLS Observatory project [1]. observatory=> select count(distinct(sha256_fingerprint)) from certificates; count --------- 1239943 observatory=> select count(distinct(target)) from scans; count --------- 6483386 The scanner evaluates both certificate and ciphersuites and stores the results in DB, so we can run complex analysis [2,3]. There is also have a public client [4]. I don't have a good way to provide direct access to the database yet, but if you're a researcher, ping me directly and we can figure something out. [1] https://github.com/mozilla/tls-observatory https://github.com/mozilla/tls-observatory [2] https://twitter.com/jvehent/status/684127067005390848 https://twitter.com/jvehent/status/684127067005390848 [3] https://twitter.com/jvehent/status/686938805413232640 https://twitter.com/jvehent/status/686938805413232640 [4] https://twitter.com/jvehent/status/687429007680376833 https://twitter.com/jvehent/status/687429007680376833
- mtrn 11y ago> I have about the same amount of data in a Postgres database ... I'm curious, how fast can one load data into Postgres? Is it possible to import data directly from CSV files?
- jvehent 11y agoOur dataset is not loaded from an external source, it is generated by scanners. But to answer your question: yes, postgres can load data from csv files: http://stackoverflow.com/questions/2987433/how-to-import-csv-file-data-into-a-postgresql-table http://stackoverflow.com/questions/2987433/how-to-import-csv...
- underyx 11y ago>Is it possible to import data directly from CSV files? Yup! http://www.postgresql.org/docs/current/static/sql-copy.html http://www.postgresql.org/docs/current/static/sql-copy.html
- masklinn 11y ago> I'm curious, how fast can one load data into Postgres? Hard to answer considering the number of variables impacting. pg_bulkload[0] quotes 18MB/s for parallel loading on DBT-2 (221s to load 4GB), and 12MB/s for the built-in COPY (with post-indexing, that is first import all the data then enable and build the indexes) > Is it possible to import data directly from CSV files? Yes, the COPY command[1] can probably be configured to support whatever your *SV format is. There's also pg_bulkload (which should be faster but works offline). [0] http://ossc-db.github.io/pg_bulkload/index.html http://ossc-db.github.io/pg_bulkload/index.html [1] http://www.postgresql.org/docs/current/interactive/sql-copy.html http://www.postgresql.org/docs/current/interactive/sql-copy....