Smerity

31 ms·

211.
by Smerity 11y ago
I cannot recommend FlashGraph strongly enough. FlashGraph was one of the first graph computation engines to enable near trivial analysis of the Web Data Commons Hyperlink Graph - 3.5 billion web pages and 128 billion links. For smaller grap
212.
by Smerity 11y ago
If you're interested in a publicly queryable index of the web, you could try running a search server such as ElasticSearch on the Common Crawl[1] corpus. ElasticSearch runs the search backend of WordPress, 600 million+ documents in tot
213.
by Smerity 11y ago
You're indeed correct as this is built upon bloom filters and standard bloom filters don't support deletion[1]. You could reconstruct the bloom filter minus the deleted element from each of the individual relations but then you&#x
214.
by Smerity 12y ago
I'm on Webpass (100/100) and can hit 11MB/s fairly consistently on torrents and downloads, so it's certainly possible. This was enough of a problem that I had to connect my media center directly via cables instead of wif
215.
Google Ads Take a Step Towards “HTTPS Everywhere”(adwords.blogspot.com)
3 points by Smerity 12y ago|0 comments
216.
by Smerity 12y ago
Agreed that it's malformed. My point was more that I'm curious how many other libraries fail badly when confronted with an unexpectedly large port. It's the edge cases / "no-one would ever be silly enough to do that
217.
by Smerity 12y ago
Tangentially related, I was performing an analysis of the billions of links available in Common Crawl[1] and ran across an interesting error: there was one link where the port number was 18779690999[2]. The library I was using converted the
218.
The lack of HTTPS at Amazon: identifying items purchased via information leakage(smerity.com)
128 points by Smerity 12y ago|35 comments
219.
by Smerity 12y ago
With all due respect, Karpathy is serious -- and funnily enough he implemented deep learning (convolutional neural nets) in Javascript[1]. Admittedly he did it mainly for fun + browser demos (it's on the front page of Stanford's C
220.
by Smerity 12y ago
Whilst an option, I feel it's a bad one if security is actually a priority. Having worked at companies who actually have a high level concern over computer security, telling someone simply isn't enough. Being told is passive. Pass
221.
by Smerity 12y ago
Someone joked that this would be useful to ensure people won't randomly plug USB drives into their computers. Sounds insane, except that... "During a stop-over in Hong Kong, he finds a spare USB key in his hotel room. Curious, he
222.
Analyzing a Web graph with 129B edges using FlashGraph(blog.commoncrawl.org)
1 points by Smerity 12y ago|0 comments
223.
by Smerity 12y ago
The Bitcoin wiki gives full details[1], but tldr, 51% attack breaks most elements of the blockchain (reverse attacker's transactions => double spending, prevent transactions from getting confirmation, prevent valid blocks originatin
224.
by Smerity 12y ago
It really depends on the use case. I mentioned m1.xlarge as they provide a good general cluster setup. The mix of CPU, RAM, and disk space should work well for most experiments one might want to perform. m1.xlarge instances also have 1Gbps
225.
by Smerity 12y ago
There's are two levels of optimizations that come into play: AWS setup and the choice of primary language. I'm always happy to speak about both as we love seeing experiments run over the data! AWS optimizations: For that level of
226.
by Smerity 12y ago
[lightly modified version of a comment I put on the article as I love HN for discussion!] Great article -- we're excited there's so much interest in the web as a dataset! I'm part of the team at Common Crawl and thought I&#x2
227.
by Smerity 12y ago
For my favourite variation of this, see "Practical Linguistic Steganography using Contextual Synonym Substitution and a Novel Vertex Coding Method"[1]. Chang et al. use synonym substitution to encode hidden data into standard text
228.
by Smerity 12y ago
Good link, thanks for pointing it out. Re: not clear cut, that's always the case to varying degrees :) To quote the author's in document response to the Google Doc you just linked to: "Update by Richard Socher (Nov 2014): Thi
229.
by Smerity 12y ago
The paper compares in detail against word2vec, but (spoiler alert) GloVe using 42 billion tokens from Common Crawl beats word2vec using 100 billion tokens from the Google News corpus! They don't actually use the 840 billion token model
230.
by Smerity 12y ago
Python + Numpy: core loop is computationally intensive + benefits from vector ops Cython: repeated invocations overpower any core loop advantage In the case of NLP, which is the author's field, you'll find not only do you have an
231.
by Smerity 12y ago
For those who don't know the story about the Australian internet industry, Telstra plays a pivotal and depressingly stupid role. Telstra was a government monopoly in the past and owned + owns almost all the copper network in Australia.
232.
by Smerity 12y ago
I usually don't react to such things but Robin Williams is such a strong part of my childhood and youth. Jumanji, Aladdin, Mrs. Doubtfire, Flubber, Hook, Bicentennial Man, Good Morning, Vietnam, Good Will Hunting ... RIP mate, you&#x27
233.
by Smerity 12y ago
We're aiming to do monthly crawls from this point on. The main hold was automating the intensive manual steps of our crawl process. Now we have scripts that make running our 100 node EC2 cluster and processing the terabytes of web data
234.
by Smerity 12y ago
Partial plug, but very related to the topic: if you're doing large scale analysis on the web and you don't want to have to actually run a large scale crawl, use the CommonCrawl dataset[1]! Common Crawl is a non profit organization
235.
by Smerity 12y ago
Browsers only complain if you go from HTTPS=>HTTP, not the other way around, so there is no mixed content warning. The article itself, hosted on Blogger, demonstrates this if you check the source code -- whilst the website is HTTP, it us
236.
by Smerity 12y ago
Considering the importance of HTTPS to, in Google's words, "[making the] Internet safer more broadly", this seems like a good time to again suggest that Google enable HTTPS for Google Analytics by default[1]. Google Analytics
237.
by Smerity 12y ago
There is a major improvement to privacy in the larger scale: the Google Analytics servers are a funnel for a substantial percentage of all web traffic. Read the write-up I linked to, specifically "How does that help the NSA?". Usa
238.
by Smerity 12y ago
Browsers tend to only complain if doing so breaks security, such as using Javascript from a HTTP source on a HTTPS page: the JS could be swapped out by a man-in-the-middle. If you use or communicate using HTTPS on a HTTP page, there's
239.
by Smerity 12y ago
I really appreciated this story. It's so hard to make people care about "small" data leaks when they have no idea what many "small" data leaks can lead to. The bug the journalists discovered that revealed Skype&#x27
240.
by Smerity 12y ago
It honestly disturbs me that Myspace, Justin.tv, and these other companies don't seem to ever consider shipping their soon-to-be-deceased data to the Internet Archive. Why not help preservation a little? It'd likely be simpler to

More ›