9 ms·
Show HN: patchbay.pub – Poor man's ngrok/IFTTT/serverless
- timvdalen 7y agoThis is incredibly interesting and super clever. I love all the little examples of what you could do with this and am definitely going to play with this.
- anderspitman 7y agoThanks!
- whois 7y agoI like this. Have you considered releasing the source so people can run it on their own servers?
- dnpp123 7y ago^^^ This. The kind of people who use this are power users who want to run this on their server. Or use IFTTT otherwise.
- edoceo 7y agoNow that we've all seen the awesome implementation I think a clone in Go is not far off. If I don't see one by EOD I'll have to make - is thing is super cool
- openthc 7y agoYou'd better post the code
- anderspitman 7y agoHere's the extra poor man's version to get ya'll started. It doesn't have any of the auth/rate limiting/etc or pubsub, but it should basically work for the MPMC queue stuff: https://github.com/patchbay-pub/patchbay-simple-server https://github.com/patchbay-pub/patchbay-simple-server Note that this will leak memory since there's no logic for removing old unused channels.
- deleted 7y ago[deleted]
- edoceo 7y agoI did one too -- https://github.com/edoceo/httpubsub https://github.com/edoceo/httpubsub
- anderspitman 7y agoNice!
- olup 7y agoAnd, a bit late, me too, of the pub/sub option that sounded more appealing to me. https://github.com/olup/patchcreek https://github.com/olup/patchcreek
- anderspitman 7y agoI would love to make the source available. But there's no getting that genie back in the bottle once you go that route. I wanted to release it and get some feedback before deciding what to do with it. That said, a server that supports the examples on this page is pretty trivial to implement (especially if you skip the pubsub stuff). I had a working prototype in a weekend. Multiple implementations would probably be a good thing. EDIT: also, the current state of the code base is not something I'd be excited about having my name attached to ;)
- whois 7y agoAh yea I totally understand. I love this thing though. So simple but powerful.
- mathfailure 7y agoAnders, please, opensauce it!
- henryfjordan 7y agoYou can think about releasing the code with no license or with some form of the newer BSL or something. It won't stop bad actors from stealing your code and using it but you will stop most corporate actors from touching it (and they are the ones you'd want to pay in the future anyway).
- sairamkunala 7y agoI made a similar project when starting to learn Go. It's opensource and it works. Just don't host it out on the internet. Intranet should be good because no security audit has happened. This was sort of an MVP https://github.com/sairam/daata-portal/tree/master/daata-server https://github.com/sairam/daata-portal/tree/master/daata-ser...
- jedimastert 7y agoI really like this. I think it'd be interesting to implement on my own server as well (You know, every non-cloud reliance or whatever). It's super simple and super easy to grok. I'm guessing you aren't looking to put this out open source if you're looking to make a product of it, but I'd love to see the implementation.
- fenwick67 7y agoThe name (including the .pub part) made me think this was related to the Secure Scuttlebutt client Patchbay: https://github.com/ssbc/patchbay https://github.com/ssbc/patchbay
- tylerchilds 7y agoSame, I was super confused and I was trying to find out the relation the entire time and there is none.
- anderspitman 7y agoNo confusion was intended; sorry about that. The domains seem to be fairly orthogonal. Or am I not understanding what ssbc/patchbay does?
- ChristianBundy 7y agoPatchbay is an interface for Secure Scuttlebutt. Secure Scuttlebutt is a P2P pubsub system for sending files, messages (chat/notifications), and everything else you can implement with pubsub-like systems. I think it's just confusing because of the overlap with: - Patchbay - Pub - Channel If you're curious about SSB, I've posted a link to your site on the network. You can view that (and other content) here: https://oasis-demo.fraction.io/thread/%25IGLa%2F3NDG3cj5gYOYmeBL93m%2FJyxEAKGp65IksEZHfY%3D.sha256#%25IGLa%2F3NDG3cj5gYOYmeBL93m%2FJyxEAKGp65IksEZHfY%3D.sha256 https://oasis-demo.fraction.io/thread/%25IGLa%2F3NDG3cj5gYOY...
- anderspitman 7y agoAh gotcha. I've heard about Scuttlebutt, but didn't realize there was so much terminology overlap.
- ISL 7y agoThere also used to be an IoT network called Pachube.
- 7y ago
- lsferreira42 7y agoWell this is amazing, it's simple and functional, solves a wide variety of problems with a very simple approach. Congratulations for shipping it!
- anderspitman 7y agoThank you!
- moose333 7y agoThis is a fantastic idea. Great useful examples, the little chat client is such a novel use of a simple concept. Well done!
- kylek 7y agoThis is super weird. The "job queue" example made me facepalm, I'd probably use a one-liner like- ls -1 *.mp3 | xargs -P4 -n1 -i ffmpeg -i "{}" "{}.ogg" (yah the filenames would end up .mp3.ogg, but so does the example...should throw basename in there somewhere)
- Mic92 7y agoIt makes sense if you have multiple machines to actually distribute the load (files could be on a NFS).
- emj 7y agoYou can send the files through the pubsub directly and just convert a stream of bytes. It is a beautiful pattern, but needs some tools to make it really easy to use (e.g. how to I metadata and bytestream to a pipe)
- anderspitman 7y agoSending the actual files would probably be really slow on the free server, due to bandwidth limiting. As for metadata, you might consider multipart/form-data, or just packing it into the query string (obviously depends on how much data you're talking about). Of course you'd probably need to write that in something other than bash. EDIT: It's pretty easy to send form-data with curl: https://ec.haxx.se/http-multipart.html https://ec.haxx.se/http-multipart.html
- emj 7y agoYes but it is the receiving that mess you up, if you want to use standard shell tools and just stream it. Everything can be solved by using other tools. # preserve filename for i in a b; do gzip -9 $i; done # does not preserve filenames while true; done curl $patchbay| mangle | gzip -9 | curl -d@- $patchbay; done
- WhatIsDukkha 7y agoYou might be better served with gnu parallel - #create the directory structure first find . -type f -name '*.flac' |ionice -c3 parallel -j1 'mkdir -p $transcodes{//}' && #now the encodes find . -type f -name '*.flac' |ionice -c3 parallel -j4 "ffmpeg -n -v 0 -i {} -c:a libopus -b:a 128k $transcodes{.}.opus" Note the {.} which would actually only grab the basename.
- phlyingpenguin 7y agoadoc formatted mirror from before/while this got destroyed from its chat getting DOS'd: https://pastebin.com/tAWNuRL7 https://pastebin.com/tAWNuRL7
- anderspitman 7y agoHm I'm not sure it was actually a load problem. The server load never spiked. I'm guessing the hosting got in a weird state, either on the server side or the producer process. EDIT: But in case it happens again here's an internet archive link: https://web.archive.org/web/20191126170030/https://patchbay.pub/ https://web.archive.org/web/20191126170030/https://patchbay....
- deleted 7y ago[deleted]
- sairamkunala 7y agoI made a similar open source self hosted tool for sharing logs or analysis from several machines, hosting internal versioned documentation, counters and time-series graphs etc., This was from a few years ago - https://github.com/sairam/daata-portal/tree/master/daata-server https://github.com/sairam/daata-portal/tree/master/daata-ser... When people working in ops or with remote CLIs would have a great need for such a tool.
- davidito 7y agoGood luck with your project, it sounds interesting. FYI, your domain is blocked by Cisco Umbrella (OpenDNS) as "This site is blocked due to a security threat that was discovered by the Cisco Umbrella security researchers." Not sure why, you might want to contact them.
- anderspitman 7y agoThanks for the heads up. That's fascinating. Has anyone else ran into this before?
- parliament32 7y agoProbably just a tainted IP from Digital Ocean.
- zrail 7y agoComplete wild guess, but maybe it's because it matches an aggressive regex meant to match the pirate bay and derivatives.
- ryanlol 7y agoRandom cisco employee I annoedon IRC blocked my domain once, I complained to another cisco employee on another IRC and the block was promptly removed. I don’t think they have much oversight over this stuff.
- royragsdale 7y agoI ran into it this as well and the response from the folks running our block list said it triggered a "newly seen across all of OpenDNS" heuristic. This matches other similar blocks I've seen for newly registered gTLD domains. Seems that this is the relevant documentation: https://support.umbrella.com/hc/en-us/articles/235911828-Newly-Seen-Domains-Security-Category https://support.umbrella.com/hc/en-us/articles/235911828-New...
- fiatjaf 7y agoAdded to https://github.com/fiatjaf/awesome-loginless https://github.com/fiatjaf/awesome-loginless.
- anderspitman 7y agoThanks! That's a sweet list.
- ee3456 7y agosomeone DDoS'ed the chat again, site is down :(
- shark0der 7y agoHN hug of death?
- deleted 7y ago[deleted]
- anderspitman 7y agoNot quite sure. The server load is surprisingly low, but it's gotten locked up a couple different ways.
- shark0der 7y agoCheck the amount of traffic you've got: number of packets, number of connections. If it's under attack you might have LOTS of packets (for example SYN) coming your way which explains the slow performance.
- anderspitman 7y agoOnly a few hundred connections, and iftop is showing very low traffic (does that capture SYN floods?). I get the feeling people might be DOSing as a proof-of-concept then backing off, which I appreciate.
- anderspitman 7y agoAuthor here. I implemented[0] a stripped-down server that includes the MPMC functionality. Should be a good starting point for alternative implementations. It really shows how simple the concept is under the hood. All it really does is translate HTTP requests into golang channels. [0] https://github.com/patchbay-pub/patchbay-simple-server https://github.com/patchbay-pub/patchbay-simple-server
- yoru-sulfur 7y agoThis is great, I've been thinking of building something similar myself for exactly the use case you mentioned.
- thojest 7y agoHow does this technically work? Is this a server which just tunnels through tcp traffic? So does all data pass through your server or is it a rendezvous server, which uses ICE to establish a P2P connection?
- anderspitman 7y agoYep everything goes through my server (see the security and privacy sections at the bottom of the page). It would be awesome if someone could figure out a peer-to-peer way to accomplish the same thing. I'm not familiar with it, but I think that might essentially be what Tor hidden services are.
- thojest 7y agoThank you very much for your answer, and great project :) I am also not very familiar with P2P, but I know that it is a bit complicated because of NATS. ICE is a protocol to make it possible for clients to establish a P2P connection, but it is not always possible because of symmetric NATS. Thus, for example in WebRTC there exist STUN and TURN servers.
- zomglings 7y agoThis is excellent - a big fan of its scriptability via curl. I think you have the beginnings of a truly amazing developer experience here. If you're interested in monetizing this: I would pay a fee for this - say $4/month. The only features I would like at this price: 1. Authentication (-H "Authorization: Bearer <token>" would be good enough) - at the level of the routes. This sounds like what you have done for routes like '/' anyway. 2. A CLI to tell which routes I "own". 3. Persistence of messages on the patchbay server up to some reasonable time limit T so that consumers can replay messages since time T without the publishers being blocked on their side. I would pay $8 per month if: 1. You hosted consumers (go binaries or docker containers which would accept the body and headers of the HTTP response on the consumer side and do something with it), ran them for me in a loop 2. Guaranteed me some level of isolation for things like API keys, etc. which would be passed to my consumers in some fashion. Even if it were open source, at these prices, I would buy. Don't want the headache of hosting it myself.
- anderspitman 7y agoThanks for the detailed suggestions. A few questions: 1/2. Why not a private instance for $4/month, accessible only to authenticated requests (unless of course you choose to open up certain routes for GET and/or POST)? Not only does this mean you can use semantic URLs instead of random ones (which helps offset the annoyance of managing API tokens), but you also get a unique domain that you can do things like put behind CloudFlare or another CDN. 3. This is an interesting idea. However, by far the greatest strength of the current implementation is that it has 0 (well ok, very little) explicit state. What's your use case for the semi-persistent producers? 1. It's interesting that you request hosted consumers and not hosted producers (a la for hosting a website). But I see the appeal. It would be nice to be able to set up something like a text notification trigger and not have to worry about it being tied to your own physical machine. I totally agree about the open source bit. I don't see open source as a threat to a business model here. With something so simple it's going to come down to execution and customer service. As a staunch supporter of self-hosting, after 1 or 2 services it simply becomes too much to manage, unless you enjoy it as a hobby. I just want to get work done without giving my digital self over to the big guys. Even paying others per service doesn't really work, because at 5USD/service (which I paid for a Mastodon instance for a while), it quickly becomes too much money. Part of the inspiration for patchbay was trying to make a dead-simple substrate on top of which I could implement many other things, hopefully saving me some self-hosting trouble.
- qrv3w 7y agoThis is so cool! I really like your project. Its so simple and powerful and very succinct. Do you have an example for the "Poor man's file sharing"? I tried doing the simple thing: > curl -F 'data=@test.txt' https://patchbay.pub/test.txt https://patchbay.pub/test.txt And on the other computer: > curl https://patchbay.pub/test.txt https://patchbay.pub/test.txt But I'm seeing that the output is being wrapped with "Content-Disposition" and "Content-Type" at the top. I'm no HTTP expert, but is there a way to use your server (or namely, curl) to transfer the file so that it doesn't wrap this info?
- codetrotter 7y agoChange the command on the sending computer to curl -X POST --data-binary "@test.txt" https://patchbay.pub/test.txt The command on the other computer remains the same as what you had curl https://patchbay.pub/test.txt
- qrv3w 7y agoThanks!
- anderspitman 7y agoAlso want to mention that in general you can add mime=whatever on the consumer side to override the Content-Type the server uses. I haven't set anything up for Content-Disposition yet, so that's going to be whatever golang defaults to for the file extension. Feel free to open an issue if you'd like specific behavior
- notliketherest 7y agoJust want to point out how easy it is to do this using SSH tunnels. ssh -nNT -R :<server port>:localhost:<local port> user@domain
- anderspitman 7y agoNoticed someone working on a rather interesting exploit in the logs: POST /.well-known/acme-challenge/huhn2?mime=text%2Fplain Plugged that one. If anyone thinks of similar holes I should consider, please shoot me an email at info@patchbay.pub. There are a whole new set of security issues that crop up with this sort of system. Definitely be aware of that before using it!
- brianzelip 7y agoHow did you "plug" that one? And what is "that one"? I assume "that one" is posting to /.well-known/.
- anderspitman 7y agoPOSTing or GETing currently
- jehna1 7y agoOk, the first thing in my mind is how to share passwords with your teammates. Sending a password: echo 'hunter2' | openssl bf - | curl -X POST --data-binary @- https://patchbay.pub/mysecretpasswordhandle Receiving a password: curl --silent https://patchbay.pub/mysecretpasswordhandle | openssl bf -d - This uses end-to-end encryption (Blowfish cipher) that relies on a shared password.
- anderspitman 7y agoBut how do you securely share the patchbay channel? ;)
- g4d 7y agoinvisible ink and paper
- qbaqbaqba 7y agoIt would be nice to have read-only endpoints.
- anderspitman 7y agoI thought about a couple different ways that could be implemented, without requiring a login. For example, you could make the channel a public key signature, then only accept POSTs that can prove they own the private key. The problem is that it makes it trivial for people to anonymously host whatever they want on my server... As it stands now, there's a bit of game theory involved. You can shared a channel publicly, but if even one person doesn't like what you're publishing, they can bring it down. This essentially forces people to keep their channels for private use. This is also one good use case for charging people for private namespaces/channels.
- qbaqbaqba 7y agoSimplest solution would be a different name for broadcasting channel. But you are right in your concerns about people being assholes.
- anderspitman 7y agoMain problem with simply using a different name is that the server has to keep track of the mapping between the producer port and the consumer port. As it is currently the server has almost no state.
- billhathaway 7y agoThe term 'poor man' isn't very informative to me. Does that mean it has all the functionality but is free or does it mean has basic functionality and is free, or is cheaper?
- anderspitman 7y agoI'm using it in the idiomatic English[0] sense, meaning a cheaper/inferior alternative. For any one of the examples, there are way better tools out there. But how many of those individual tools can be used to accomplish many different things? [0] https://www.merriam-webster.com/dictionary/poor%20man%27s https://www.merriam-webster.com/dictionary/poor%20man%27s