8 ms·
How do dat:// sites interact with servers?
- wesleytodd 8y agodat://pfrazee.hashbase.io/blog/dat-and-servers
- panarky 8y ago"Cross-origin resource sharing (CORS) is a policy that prevents a webpage from connecting to a server, unless that server has given that webpage permission to connect." CORS only applies to requests for restricted resources like fonts or XHR requests that aren't simple GETs and POSTs. "So, while this is typically not possible: foo.com/index.html ---GET---> bar.com/pic.jpg" Typically this is possible. Images, stylesheets, scripts, iframes, and videos aren't subject to CORS. "You can solve it by routing the request through your host server: foo.com/index.html ---GET---> foo.com ---GET---> bar.com/pic.jpg Not necessary, the client's browser can get bar.com/pic.jpg just fine all by itself. "Pinning tools like Hashbase and Homebase help keep dat:// sites online" If you publish a dat archive, how do you notify Hashbase to pin it? Can you do it through dat? To keep my dat alive with Hashbase, do I have to set up an account, provide and confirm an email address, link a credit card, etc.? Are centralized servers, financial institutions and surveillance all required components of the anonymous, decentralized, peer-to-peer web?
- wesleytodd 8y ago> If you publish a dat archive, how do you notify Hashbase to pin it? This is provided by hashbase in their ui. Sign up for an account and add your dat. It is email based, no credit card/address unless you go over the data cap. If you are uncomfortable using the service you can run your own node, it is open source: https://github.com/beakerbrowser/hashbase https://github.com/beakerbrowser/hashbase
- pfraze 8y ago> Typically this is possible. Images aren't subject to CORS. You're right, that was a misleading example. I changed it to data.json to be more clear. > If you publish a dat archive, how do you notify Hashbase to pin it? Can you do it through dat? Does it require setting up an account on Hashbase, providing an email address, linking a credit card, etc.? The "Pinning" system is very similar to Git remotes. You can pin using any endpoint that complies with https://www.datprotocol.com/deps/0003-http-pinning-service-api/ https://www.datprotocol.com/deps/0003-http-pinning-service-a.... So, similar to a git remote, you do need some kind of authentication with the pinning service - unless somebody writes one that's open for anybody to push to. The UX flow will be similar to a git remote as well, you use an HTTPS to tell the server to sync the dat. So, it's an explicit user action. We've got two nodejs implementations of the pinning service API you can self-deploy, https://github.com/beakerbrowser/hashbase https://github.com/beakerbrowser/hashbase and https://github.com/beakerbrowser/homebase https://github.com/beakerbrowser/homebase, and then we run a Hashbase instance at hashbase.io
- panarky 8y ago>> Typically this is possible. Images aren't subject to CORS. > You're right, that was a misleading example. I changed it to data.json to be more clear. CORS doesn't preflight GET, POST or HEAD methods unless they have custom headers or content-type other than application/x-www-form-urlencoded, multipart/form-data or text/plain. So a simple GET bar.com/data.json works just fine in today's browsers. https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
- pfraze 8y agoI'd be happy to be corrected on this, but here's what I understand: While fetch doesn't preflight for GET, it does require an Access-Control-Allow-Origin header. You can specify `no-cors` in the mode to circumvent this, but then you cant access the response body (https://developer.mozilla.org/en-US/docs/Web/API/Request/mode https://developer.mozilla.org/en-US/docs/Web/API/Request/mod...)
- panarky 8y agoHere's an example using your own site. Open the console in an empty Chrome window and paste this: fetch('https://pfrazee.hashbase.io/feed.xml') .then(response => response.text()) .then(str => console.log(str)) You'll get your RSS feed, no CORS tricks required.
- pfraze 8y agoMy site has the Access-Control-Allow-Origin:* header set. Try the same thing on the Beaker site in the devtools for https://example.com https://example.com. fetch('https://beakerbrowser.com/dat.json') .then(response => response.text()) .then(str => console.log(str)) It should fail
- bhritchie 8y agoIt is failing for me because of the content security policy, which is a different thing.
- lachenmayer 8y ago> Are centralized servers, financial institutions and surveillance all required components of the anonymous, decentralized, peer-to-peer web? Servers and financial institutions will never go away, no matter how hard we try. I'm hoping that we can make surveillance go away by sharing much less data with organisations who rely on surveillance to survive. Hashbase is not "centralized" in the sense that you are always free to choose a different provider of its hosting services. You can host your own Hashbase: https://github.com/beakerbrowser/hashbase https://github.com/beakerbrowser/hashbase You can even choose multiple providers at once, providing you with resilience in case one of your chosen providers violates your trust, eg. by losing your data or using it to spy on you.
- lucideer 8y ago> CORS only applies to requests for restricted resources like fonts or XHR requests that aren't simple GETs and POSTs. CORS applies to XHR. Including GETs and POSTs, and including fetching images over XHR. There's some other sibling comments here discussing preflight requests, which it sounds like you might be referring to, but CORS is not limited to just preflight requests.
- inglor 8y agoCORS applies to XHR but fetch can make opaque requests by setting {mode: 'no-cors'} as the second parameter.
- jsjohnst 8y ago> CORS only applies to requests for restricted resources like fonts or XHR requests that aren't simple GETs and POSTs. This is completely incorrect. You can not make any XHR GET or POST requests cross-origin without CORS. Fonts also do not require CORS, you can link to them in your CSS/styles/etc without technical restrictions (might be legal restrictions of course).
- inglor 8y agoSure you can, you can make any request you want without CORS, the content is just opaque to you (for example you can display an image but not manipulate its pixels). You do this for example with `fetch('./any-resource', {mode: 'no-cors'})`. You can then for example do a `.then(x => x.blob())` and then use the resulting blob as an image.
- inglor 8y agoExactly! This is a little backwards, the goal of CORS isn't just to protect the _user_ it is also to protect the _third party website_. All it takes for a website to opt-in to this is just adding a single header - it's possible for bar.com to allow the request from foo.com by opting into it.
- olt 8y ago> Images, stylesheets, scripts, iframes, and videos aren't subject to CORS. It depends on what you want to do with the image: https://developer.mozilla.org/en-US/docs/Web/HTML/CORS_enabled_image#What_is_a_tainted_canvas https://developer.mozilla.org/en-US/docs/Web/HTML/CORS_enabl...
- 3pt14159 8y agoI still wish these guys all the best, but if you want to start doing 3rd party like this then everything will eventually devolve like it did for the normal web. We do need a new web and way of moving information, but once you start directly connecting to servers they need to know who you are.
- chatmasta 8y agoFirefox is in the early phase of implementing support for non-HTTP protocols, starting with Tor. My understanding (from reading some Mozilla comments here maybe?) is that their goal is to make this pluggable so that eventually Firefox can support all kinds of non-standard transports, including p2p ones like dat:// We’re not there yet, but Mozilla is making a very positive step in that direction, and hopefully other browser vendors will follow. If I were creating a new protocol, I would be doing it now, to “skate where the puck is going” as they say.
- wesleytodd 8y agoI think the really interesting things here are more than just "support new p2p transports/protocols". The things Beaker is doing around changing the way we author the web are the real important pieces, and finding the "middle ground" or transition path is one of the keys to success. So, to answer the OP as well, supporting loading assets from traditional servers is pretty much a requirement for authors I think.
- pfraze 8y agoI don’t hold this binary view that a “decentralized Web” has to avoid certain technologies. I believe we should aim to use peer-to-peer systems where it’s impactful and practical, and in the rare case you do need a third-party server, there are things you can do to limit your dependence and make them easy to reconfigure. That’s our approach with Beaker. We use peer-to-peer systems as much as possible, and then plug in servers as minimally as possible.
- 3pt14159 8y agoI upvoted you and wish you all the best, but the core problem I see with this is that it makes it hard to make policy around. If I'm making a new system or setting policy for a government or other high-security minded client (like a political campaign, military contractor, activist group, or private intelligence corp) I need off the shelf stuff with zero known attack surface OR I need to individual vet every single offering within that protocol suite. This is why you can email members that work for The Government of Ontario, but they won't click on links to non-whitelisted places. The attack surface when clicking a link is fucking huuuuuge (pdf 0days anyone?), while the attack surface for loading an email is much smaller. There are a ton of interesting web-replacements that hackers are playing around with right now, but the one that wins for the next web is the one that lets stupid people do whatever they want without worrying. In my opinion, 3rd party means worrying, and in an ideal world it would go away. The irony of this whole thing is that I'm actively arguing against my own long-term interests. A structural change of the kind I advocate for would dramatically reduce the profitability of being in either data science or cybersecurity; both fields I have a foot in. But I don't care. Securing the flow of information between people is too important to humanity's long term survival.
- symlock 8y agoReally neat to see beaker broswer support native P2P with progressive enhancements like github.com/beakerbrowser/hashbase providing the cute URL's we all like. I'm sure hashbase.io will be blocked really quick, so it's important that the core P2P address system stay in the forefront. Transports also need to find many ways to communicate over https, shadowsocks, tor, DNS, and others.
- Sol- 8y agoDoes the dat:// scheme have any history or did the developers of this Beaker browser invent it? Just curious since I haven't seen the protocol before.
- repolfx 8y agoIt's been around for a year or so. I was wondering when it'd make it to HN. It's very nicely done but unfortunately the only implementations are in javascript.
- pfraze 8y agoRust implementation is in the works with grant funding, https://github.com/datrs https://github.com/datrs
- repolfx 8y agoI'm not sure Rust is an especially great language for interop purposes or that the productivity will be high enough to keep up with the JavaScript implementation. If you want to make a library with a C compatible API I'd be tempted to explore SubstrateVM. It can export C symbols to the generated standalone .so / .dll files, maybe you can even reuse some of the JS code, or failing that, a Java/Kotlin implementation would be compiled down to native code or be usable from other scripting languages like Ruby.
- woah 8y agoIt's been around for more than 4 years.
- warent 8y agoI doubt it. datproject.org was registered two years ago, and their whitepaper was published last year https://github.com/datproject/docs/blob/master/papers/dat-paper.pdf https://github.com/datproject/docs/blob/master/papers/dat-pa... Where are you getting 4 years from?
- k__ 8y agoI like the ideas of dat and IPFS, but I can't quite understand the difference. What I can understand, is, they use new protocols and this is an issue in the Web today. I think the only way they can succeed would be with some laws or misstakes by big corps that would drive customers away from them. What I also liked was remoteStorage [0] it is a bit like localStorage, but the data is managed independendly from the application itself. [0] https://remotestorage.io/ https://remotestorage.io/
- wesleytodd 8y agoI think "laws or misstakes by big corps" are exactly what we are seeing happen #Facebook #GDPR
- jimpick 8y agoThere's a lot of similarities, as they are both peer-to-peer and decentralized. I've mostly done Dat. I want to do a bit more IPFS. Dat feels a bit more like git for files - you can create a local file archive using the command line tools, and it's a separate step to sync it to the peer-to-peer network. There's a global discovery service for advertising archive keys, but it doesn't work at the level of single files. It's very lightweight. IPFS supports many of the same operations, but you're mostly interacting with a local gateway server which is continuously connected to the network. I believe IPFS tries to content hash every single file so they are de-duplicated globally.
- k__ 8y agoDoes anyone has build-in realtime features?
- lachenmayer 8y agoYes, Dat's core data structure is an append-only log that is designed for extremely fast real-time replication. http://github.com/mafintosh/hypercore http://github.com/mafintosh/hypercore If you use Dat through its command-line app, you get real-time updates to the files shared using it.
- 8y ago
- fjabre 8y agoDoesn't answer your original question but I moved to syncthing from dat. It just works out of the box, no manual setup, and a very active community around it. Also opensource and in production for years with rave reviews, and being used in plenty of big scale production projects.
- seymour333 8y agoI had to read the title 10 times before I realized it wasn't someone asking: "how do dat?"
- quanticle 8y ago>For each new origin that the site contacts, a permission prompt will be presented I don't think this is an adequate approach to security. When the browser presents me with a prompt to load data from a third party site, I don't know what data is being loaded, what it's being used for, or whether this prompt is expected (as part of the regular functioning of the application) or unexpected (indicating that the application has been compromised, and I should navigate away from it). In general, I've noticed that users react in one of two ways to these sorts of prompts. Naive users will blanket allow -- allowing all sites to access all of the capabilities of their browsers, regardless of the reason or necessity of that access. More sophisticated users will blanket deny. If it's not immediately apparent why a site needs the permission that it requests, that request will get denied, even if it's a valid requirement. Very very few users will think about why a site is requesting the permissions that it is requesting and consider those requests on a case by case basis.
- EGreg 8y agoI never understood how Beaker browser can act as a server listening on a port. It sounds like you always need relays on the internet because your router and ISP is gonna block all ports unless requested not to. https://www.scuttlebutt.nz/stories/design-challenge-avoid-centralization-and-singletons.html https://www.scuttlebutt.nz/stories/design-challenge-avoid-ce...
- jimpick 8y agoFirewalls are a pain. UDP hole punching (using the UTP protocol) and the discovery network works a lot of the time. Much of the people publishing public content for access by Beaker are using hashbase.io to "pin" the content and to act as a public peer, and those ports aren't behind a firewall, so the data can be directly replicated easily.
- nottorp 8y agoArticles like these would benefit from a link summarizing what that technology is about... At least add to the title on HN something about what a dat:// link is :)