6 ms·
Show HN: Browser extension to read HN comments for any url, in ClojureScript
- jdormit 9y agoWell this is awkward. Seems that shortly after I posted this here both Mozilla and Google rejected the extension from their stores. Mozilla's primary concern was that the analytics were opt-out, not opt-in, so I'm patching that in and resubmitting. Typically, Google just said that it "did not comply with their policies" with no additional information, but I'm guessing their concerns are along the same lines. Meanwhile, if anyone is interested in trying it out, the README has instructions on running the extension locally in Firefox or Chrome.
- nerdponx 9y agoOut of curiosity, why did you choose to set up analytics as opt-out? What do you even need them for?
- jdormit 9y agoI built this as an exercise in product management - even though it is just a little extension, I want to run it as if it were a revenue-generating product so that I can gain experience in that area. Part of that is looking at usage data to identify what parts are working, what needs improvement, retention statistics, etc. I made them opt-out because I would be pissed if someone made a cool thing that I wanted to use but insisted on tracking me with it :)
- neolefty 9y agoThe build instructions say that it will create ext/js/generated, but I'm only seeing ext/js, with a few .js files, but no "generated" directory.
- staunch 9y agoIt's not awkward to violate people's privacy, it's wrong. Maybe you didn't know that before, but you still don't seem to get it.
- jdormit 9y agoIt's not a violation if there is consent. I'll grant that I didn't think through the consent process for the extension very well, though.
- staunch 9y agoYou did not get consent. You did violate people's privacy. Your extension got removed by Google and Mozilla.
- bluejekyll 9y agoPerson posts cool tool they wrote, that they find useful. Shows it to the community, this community, which many may really appreciate. Then gets ripped for not thinking through the complex privacy issues with said tool. Not everything is malicious. It would be better to give constructive criticism, like: This is a really cool tool! I might even choose to use it, but I have a few concerns: 1) it sends too much data unfiltered to the algolia search servers. Could you instead make it a button that only then triggers the request or opens another browser window? 2) the analytics are also a concern. In general, you should always make these opt-in; better, ask the user through a dialogue to enable them; even better, request feedback through some other system (github for example). Let’s try and be nice here and give people the benefit of the doubt.
- jdormit 9y agoHeh, thanks bluejekyll. I appreciate this. FWIW, I am taking the privacy and security concerns raised in this thread seriously. In fact, this type of candid feedback is one of the best things about HN. I anticipate it will take me a few days to choose and implement some of the fixes suggested here.
- staunch 9y ago
- JepZ 9y agoAnybody knows how long it will take before Mozilla will make the new version available (avg. review time)?
- jdormit 9y agoI'm still working on making the changes they requested - once I've finished that and submitted a new version, I anticipate it will be a couple days at most until they make the new version available.
- jdormit 9y ago@JepZ, the new version has been approved by Mozilla and is available at https://addons.mozilla.org/en-US/firefox/addon/looped-in https://addons.mozilla.org/en-US/firefox/addon/looped-in
- JepZ 9y agoThx for the hint. One suggestion that came to my mind: As most pages I visit have no comments, I closed the sidebar again. But that way I don't see the comments when there are some. How about adding a setting to let the sidebar open/close automatically if there are comments for a page?
- jdormit 9y agoThe extension icon should have a number on it that corresponds to the number of comments. My thinking was that you could use that to see if a page had comments or not.
- chmod775 9y agoWill this instantly look up any URL I visit, or does it only look up HN posts for the current URL when I click some icon? I'd rather not have a browser extension that sends everything I visit to some US servers - but I do like the idea behind this.
- jdormit 9y agoThe current implementation will look up any URL you visit so that it can populate the "number of comments" text that appears over the button. The data only goes to 1 server - hn.algolia.com, a HN search API provided by the company Algolia.
- chmod775 9y agoJust.... don't. There's so much that can go wrong with that, especially if this stuff eventually shows up in some data leak. Here's some examples: 1. URLs can contain sensitive data, which will tie your session to your person (facebook URLs etc) 2. This could create vulnerabilities in (for example) checkout flows that pass data via URLs 3. I'd rather not make the job of intelligence services easier than it already is 4. Sometimes I clear my browser history for a reason
- jdormit 9y agoAlthough I trust Algolia not to abuse their service, I understand why you do not. Maybe the "number of comments" display could be opt-in as well? Unfortunately, I don't think there's a way to get that particular UI feature without sending all URLs to some external server.
- nerdponx 9y agoYou can at least start by stripping all query parameters and anchors off of the URL. And instead of automatically piping everything to Algolia, just give the user a damn button.
- 9y ago
- bfred_it 9y agoDoes this have to be an extension with `<all_urls>` permission? Can it be a bookmarklet that will just open the full-fledged HN in a new tab?
- jdormit 9y agoIt could do that - I built it to satisfy my own use case, which was that I wanted to see the HN comments side-by-side with the content while I was browsing.
- mistakevin 9y agoHere's a bookmarklet I use to launch a quick search for a page I'm looking at. javascript:(function()%7Bwindow.open('https%3A%2F%2Fhn.algolia.com%2F%3Fquery%3D' %2B (window.location.hostname %2B window.location.pathname %2B window.location.hash).split('%2F').join(' ').split('%23').join(' ')%2C '_blank')%7D)()
- tedchs 9y agoWhy send each URL to a server to be checked, instead of doing a periodic download of the (very small) list of HN links and comment counts, to be checked offline like an ad blocker? It's probably 5kb compressed.
- diggan 9y agoWhat makes you think it's a very small list? HN has been around since 2007, with a lot of submissions. Any guesses on the size of that? I think it's bigger than
- xiphias 9y agoAnother option is using a bloom filter (just like how Chrome does it for malware URL detection)
- vijayp 9y agoYou can sort HN by date, and few URLs are updated every day. So you can push a new bloom filter every day and a different list of updates every 5m. Then just check your URL against both of them.
- tbirrell 9y agoWell... including comments it looks like we are pushing 16.3 million posts. The id in the url is sequential. If you are saving url, HN id, and comment count, that's probably no more than a couple megs, if even that.
- Ajedi32 9y ago~391 MB if we store SHA-1 hashes of the URLs (160 bits each) and HN ids and assume 16.3 million posts[1]. (Probably less, since, as you said, some posts are just comments.) If we're okay submitting one out of every hundred URLs as a SHA-1 hashed value to an external server, we can reduce that further to ~18 MB with a bloom filter[2]. [1]: https://www.google.com/search?q=(160+bits+%2B+32+bits)+*+16.3+million&oq=(160+bits+%2B+32+bits)+*+16.3+million https://www.google.com/search?q=(160+bits+%2B+32+bits)+*+16.... [2]: https://hur.st/bloomfilter?n=16300000&p=0.01 https://hur.st/bloomfilter?n=16300000&p=0.01
- dustingetz 9y agosick nice job Jeremy
- Gys 9y agoThis comment refers to a similar extension but was asked by HN to stop doing it: https://news.ycombinator.com/item?id=15938700 https://news.ycombinator.com/item?id=15938700 And there is also: https://github.com/powerpak/hn-sidebar https://github.com/powerpak/hn-sidebar which was in the Chrome Store but not anymore...
- jdormit 9y ago:fingerscrossed: this one doesn't end up in the same bucket. Any idea why the earlier extension was shut down by YC?
- tosh 9y agoReminds me of hoodwink.d by why the lucky stiff. Anyone remembers it?
- ComodoHacker 9y agoAre there any other websites/apps that are known to respect DNT flag?
- pault 9y agoI use DNT with Firefox 59 and I often see DNT notices, but usually for YouTube embeds.
- jdormit 9y agoI believe that medium.com respects DNT. I'm sure there are others, but I haven't looked into it too much.
- LinuxBender 9y agoDo any countries have laws that enforce compliance and have serious consequences for violations?
- ungzd 9y agoDo you have REPL both for background and content scripts? I tried to use clojurescript for Chrome extensions few years ago but failed to configure fast reload cycle (either repl or figwheel).
- jdormit 9y agoNo. In fact, the development cycle on this was pretty awful - due to browser extensions' strict CSP, I couldn't even compile the CLJS with {:optimizations :none}, so every code change took ~20 seconds to recompile. Do you know any good resources on REPL-driven ClojureScript development? Preferably with a REPL that lives in the page environment so that browser variables etc. can be accessed.
- ungzd 9y agoChromex-sample (https://github.com/binaryage/chromex-sample https://github.com/binaryage/chromex-sample) mentions :optimizations :none and figwheel support (but with disabled repl), but I didn't try it yet.
- jdormit 9y agoInteresting, looks like they got :optimizations :none to work for background scripts but not content scripts (which makes sense). Figwheel support sounds great, I'll look into adding that to Looped In.
- deleted 9y ago[deleted]
- yread 9y agoThere is also Kiwi https://chrome.google.com/webstore/detail/kiwi-conversations/pkifhlefpamigmobjmjjjnjglpebflhp?hl=en https://chrome.google.com/webstore/detail/kiwi-conversations... Also "researches" Reddit, Product Hunt and Google News. And only on demand
- nkurz 9y agoAs other comments here point out, there are significant privacy issues with sending every visited URL to a "trusted" server to check for comments. There are also load issues for the server that is getting all the unwanted requests. So if doing this, you'd probably want at least the initial lookup to be local. So, can this be done with an initial download of some small number of megabytes, then incremental updates of a few kilobytes as often as desired? I think so. Guessing at numbers (see other comments here), there are probably fewer than 1,000,000 URL's that have comments associated with them on HN. For each, you store hash of the canonicalized URL and a comment count. Collisions aren't deadly, so you could probably get down to 8B per hash (7B for hash, 1B for count). Updates can be a list of all new and revised URL-hash-counts since a given date. Lookup the hash of the URL of each visited page with binary search on the 7B prefix in the ~8MB ordered list of data. If found, report the number of comments in something clickable that loads a sidebar. The only data that leaves the machine is based on the active click to load the comments. Maybe store a "false positive list" so that the rare collisions are only visible to the user once. Maybe use a bit for "visited" so you can distinguish pages with new comments? The numbers seem surprisingly manageable.
- jdormit 9y agoExcellent idea, and a nice summary of the discussion of this from further down the thread. I've opened an issue to implement this here: https://github.com/jdormit/looped-in/issues/4 https://github.com/jdormit/looped-in/issues/4 As time allows, I will implement this later this week.
- latte 9y agoThank you for posting this! What does your development workflow for CLJS browser extensions look like? Do you have to rebuild and reload the extension on your development machine each time you change the code? Is it possible to use Figwheel when writing browser extensions?
- jdormit 9y agoI'm still working out the kinks in the workflow. I believe it is possible to use Figwheel while writing browser extensions [0], but I haven't set that up. I have `lein cljsbuild auto` running in the background to automatically recompile the JS when I change the source code, and use Mozilla's web-ext utility [1] to automatically reload the web extension once the JS has compiled. This system has some disadvantages, though. The main one is the glacial feedback loop - due to strict CSP restrictions in web extensions, I had trouble compiling the CLJS with {:optimizations :none}, so each save-compile-reload cycle takes ~30 seconds from when I save the source file to when I see the results of the change in the browser. I also did not figure out how to set up a REPL environment connected to the code running in the web extension. Lots of room for improvement here, basically. [0]: https://github.com/binaryage/chromex-sample#chromex-sample-project-has-following-configuration https://github.com/binaryage/chromex-sample#chromex-sample-p... [1]: https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Getting_started_with_web-ext https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Ge...