9 ms·

Pigshell – Unix the Web

333 points by nmcfarl 12y ago

nmcfarl 12y ago
More in this comment by the author: https://news.ycombinator.com/item?id=8436716 https://news.ycombinator.com/item?id=8436716
bilalhusain 12y ago
The psty links in README file are broken. A working link is http://pigshell.com/v/0.6.2/psty.py http://pigshell.com/v/0.6.2/psty.py As mentioned in the README, you need to run psty server locally and establish websocket connection (mount in pigshell). This will enable you to harness local filesystem from pigshell.
kolev 12y ago
This is beautiful!
tomw1808 12y ago
I just had a glance about the functionality, but for my understanding, this is the best idea right behind ifttt. Pipelining all the web's things with well designed interfaces on a shell. Even if the product itself would be unstable, alone the idea is great enough for a gold-medal. Keep it up!
tubelite 12y ago
Author here. - Please use a desktop browser (Chrome preferred). While mobile browsers kinda-sorta work if you're just clicking through the examples, keyboard input doesn't work on mobiles (yet). - Pigshell has few users (in fact, using the plural is a slight marketing exaggeration) so please bear with me for any teething issues, broken links, inadequate documentation... - One of the practical use cases is copying files across Google accounts, and backing it up to your desktop. (Details here: http://pigshell.com/v/0.6.2/doc/gdrive.html http://pigshell.com/v/0.6.2/doc/gdrive.html) The nice thing about a unix approach is that the exact same cp command can be used to copy GDrive to Dropbox, or any other supported filesystem. - Some more D3 visualization examples here, using Mike Bostock's gist viewer: http://bl.ocks.org/ganeshv http://bl.ocks.org/ganeshv (Do use the "Open in a new window" links, like this one: http://bl.ocks.org/ganeshv/raw/46cc526d41d16995ccdc/ http://bl.ocks.org/ganeshv/raw/46cc526d41d16995ccdc/).
- sprayk 12y ago
  I looked through the four documentation links on the right and was unable to find out how to cat a Google spreadsheet as anything consumable by the shell. I took a guess and tried cat -o gdrive.fmt=txt and csv, but those were not fruitful. Is this possible currently? edit: found mimeMap.js and saw tsv was an option. That didn't work either.
  - tubelite 12y ago
    IIRC the Drive API does not support those conversion options. The way forward would be for an xlsx parsing JS library to convert it into CSV or a stream of Javascript objects. I didn't find any GPL-compatible library last time I checked. See https://developers.google.com/drive/web/manage-downloads https://developers.google.com/drive/web/manage-downloads for a list of export format possibilities. For spreadsheets, the only options are xlsx, odt and pdf.
    - deleted 12y ago
      [deleted]
    - fiatjaf 12y ago
      Google Drive document can be embedded as iframes. Wouldn't this solve this problem? Here's a proof: http://www.cellmaster.com.au/example.html http://www.cellmaster.com.au/example.html
      - sprayk 12y ago
        not for retrieving the juicy data inside the spreadsheet as objects.
        fiatjaf 12y ago
        Oh, I didn't read the whole thread, I was just thinking about my own experience with `cat gdrive/something` showing me a lot of unreadable bytes when it could show an iframe.
- 12y ago
linvin 12y ago
Ah! Nice feeling that whole of internet as a single (unix) machine.
- serf 12y ago
  reminds me of Plan9 sorta.
crater 12y ago
I like the idea. If I'm playing with some facebook data, right now I use whatever language I'm confortable with and use its HTTP request functionality, and than manipulate the data how I want in that language. To achieve the same thing in pigshell I have to write a shell script that manipulates the data. What can pigshell do that I can't just do in another language?
- tubelite 12y ago
  In theory, the availability of APIs should be enough for the technical user to write their own data extraction routines, connect them to other APIs and so on, using their preferred language and toolchain. In practice, reading up the documentation, writing and debugging code sets up enough of a barrier to entry that it discourages casual usage without a powerful incentive. The availability of a file abstraction covering many popular cloud services, web resources etc. enables casual, low cost experimentation across these domains, without having to read up each one of the APIs. If something clicks, then it may make sense to write a better "native" implementation.
fiatjaf 12y ago
I'm curious about how are people using this. Is it just a visualization tool for data or what else? Is anyone using it as their primary navigation tool?
- tubelite 12y ago
  Like I said in an earlier comment, there are very few users right now (cough OK just me cough). Mostly I have fun with it, e.g. trying out visualizations of interesting data (e.g. http://bl.ocks.org/ganeshv/raw/fada13e56d2c5b7b1a78/ http://bl.ocks.org/ganeshv/raw/fada13e56d2c5b7b1a78/) My Serious Use Case is full and incremental backups of the contents of my Google Drive, both personal and company, to my desktop. Running the Drive desktop app does not backup Google Docs, and you can only sign in as a single user at a time.
- basantrajan 12y ago
  perhaps stating the obvious here, but with the abstraction pigshell provides of resources on the web, it should be reasonably straight forward to build arbitrary "internet applications" too. with deferred pipes, split and joins (that pigshell already provides/supports) it should be possible to implement arbitrary work/data flow definitions one desires. coupled with psty, one could build/harness special purpose "transformers" on the web pushing it further, one could conceive of setting up pipes between transformers on the web. of course getting there will require significant work on a security model as well. [and then someone will figure out a way to interject ads]
  - fiatjaf 12y ago
    CouchDB is already compatible.
  - progman 12y ago
    > of course getting there will require significant work on a security model as well. It could probably be easier than you imagine. The security model could be implemented in psty completely by granting/denying requested resources whose permissions are stored in a database on the psty server, and managed by using long security IDs in the URLs. This way the security model could even be fine-grained. In other words, pigshell itself could be embedded into a secure environment, covered by the permissions of psty.
mgrassotti 12y ago
holy crap this is awesome
jayseeg 12y ago
With some practice, or even better, some support libraries, this looks like a great way to quickly explore viz prototypes. Great work!
roywiggins 12y ago
Attempting to copy photos out of Facebook albums into GDrive gives me "Exception: SecurityError: The operation is insecure." I'm not running psty locally. This is super cool, though. There's an undocumented template engine built in it seems? (see /bin/mostliked) which is neat.
Erwin 12y ago
Interesting. I'm trying to really go out all the API for my project and was thinking about something like a Mathemathica-like notebook for the local API. E.g. say you want to execute an API call which normally does GET /api/v1/x/y/z; in the "API shell" you type x/y/z and the JSON collection is summarized as a small datatable (not showing all of the 500 rows / 40 columns that might have, but an interesting summary) that you can later interact with. I'm thinking that such an approach can give the user their own way to explore/modify the data ,especially if you combine this with some kind of pipe lines that let you analyze/mutate/visualize the data without having to write some Python code to access the API (even though with a REST API plus e.g. requests it's simple). I like the psty idea to integrate with the local system. Sounds like it could also be used to start e.g. emacs on a remote page. Say you have a site with an API that lets you read/write files, you mount it and then can fire up emacs on it via pigshell.
- privong 12y ago
  Is this approach similar to something like flickrfs[0], to make one's flickr account locally accessible? It always seemed like a cool idea (though I never had much luck with flickrfs itself), and it is neat to see the broader utility of psty. [0] https://launchpad.net/flickrfs https://launchpad.net/flickrfs
talles 12y ago
> Error loading root filesystem at http://pigshell.com/v/0.6.2/root.tar http://pigshell.com/v/0.6.2/root.tar: Forbidden :(
dllthomas 12y ago
Have you given thought to exposing this to the regular shell with FUSE?
- tubelite 12y ago
  The concept of lazy-stream-of-JS-objects and HTML as a display engine underpin pigshell. I guess it would be possible to adapt the filesystem code to fuse4js. It won't be a drop-in; unix FS ops differ from pigshell's, but it shouldn't be too hard. Personally, I find psty (http://pigshell.com/v/0.6.2/doc/psty.html http://pigshell.com/v/0.6.2/doc/psty.html) a better way to incorporate (a part of) the unix filesystem into the web rather than fuse - which is the other way round.
  - clebio 12y ago
    Isn't this -- at least in spirit -- what IPython notebook tries to do? In IPython, you can use shell commands with a bang, e.g. `!ls -l`. And `ipython notebook` uses ZMQ to connect the server and client. At any rate, awesome work. I've been thinking for a while, offhandedly, about the idea of piping web URLs around. But I don't know how in fact it would work under the hood. I write a lot of API interfaces in my work, and it's all in the details -- every API is different in small but important ways.
    - tubelite 12y ago
      Yes, it is similar in spirit to IPython. We have been developing pigshell off and on since ~2012, and I was unaware of IPython notebooks until ~6 months ago. When I'd last used IPython, it didn't have any of the notebook stuff. So it was a bit of a jar to see that they had been there, done that, got the T-shirt, published the book, etc. etc. :) Pigshell can run in the browser with zero installation, and has a low barrier for casual usage. "Notebooks" aka gists can be shared without requiring a backend. The focus of pigshell is more around providing file adapters for web/cloud data stores. That said - more power to IPython! I love the conversational, exploratory, CLI style of interacting with a computing environment. The more ecosystems of this sort, the better.
tubelite 12y ago
So if you are running psty, here is a fun sequence of commands to try out: cd http://dilbert.com/strips/comic/1995-06-24/ ls cat 21021.strip.zoom.gif Never really liked the colorization of Dilbert. It was so much nicer as a line drawing. Best we can do is remove the colors. cat 21021.strip.zoom.gif | to canvas | pixastic -d And to save your result, cat 21021.strip.zoom.gif | to canvas | pixastic -d >/home/dilbert-unix.png Or to GDrive or Dropbox or even the local RamFS by changing the target directory. (Scott Adams is fine with personal usage. I checked. And you need psty because it proxies HTTP requests for sites which don't have CORS enabled)
bespoke_engnr 12y ago
Wow, this is a really interesting way to think about web resources. After about two minutes of playing around with this, it gave me a more pleasant experience of the web than I've had in a while.
- anigbrowl 12y ago
  This is what it was like when I first got on the Internet (around 1992); there was no web then, instead you FTPed everything and used resources like Gopher, Archie, and Veronica to find documents. I find PigShell interesting from a historic point of view but think it also points to a flaw in how the web currently works, with far too much glue and eye candy, which is a direct result of the ad business model.
  - bespoke_engnr 12y ago
    I completely agree. I actually deleted a bunch of my comment before posting it, because I started getting carried away pondering about what the right level of abstraction is for dealing with web resources. At the risk of sounding like one of the semantic-web xml-standards people, it really does make me wonder if all this horrible, incompatible, site-specific frontend crap is just a fad that will give way to tools (like psty) that ignore all that and just deal with the raw semantic content (and HTTP verbs) at a higher level of abstraction. Woah, man. We're like...surfing the Matrix, man.
    - svalorzen 12y ago
      Things like this really make me miss the old Firefox experiment "Ubiquity". It was truly a marvel for the time it was up, and I often wonder why they did discontinue it. You could search websites, send emails, look at maps and much more (you could build your own commands) without ever actually opening a webpage. It had 200.000 users when they shut it down.
      - jacobwcarlson 12y ago
        Wouldn't you just be able to fork it and continue as a hobby project of your own? Who knows, maybe you find other enthusiasts and it becomes a real project again. Mozilla licenses are (I believe and could be wrong), quite lenient about these things.
deleted 12y ago
[deleted]
TimWolla 12y ago
Does not work in Internet Explorer 11: Exception: The property "slice" of an undefined or null referance cannot be accessed (Exception: Die Eigenschaft "slice" eines undefinierten oder Nullverweises kann nicht abgerufen werden.)
- tubelite 12y ago
  Sorry, Internet Explorer is not supported currently. Please try Chrome or Firefox, in that order.
dullcrisp 12y ago
I do love this idea, but it bothers me from a security standpoint. Is there a way to do something like this without giving the author explicit control of all of my online data?
- tubelite 12y ago
  The author has NO control whatsoever of your data. - Pigshell is a 100% client-side Javascript app. There is no server side. pigshell.com is a dumb static html/js/css server. - All data (username/password as well as user data like photos and files) travels directly between the user's browser and the provider (e.g. google.com, facebook.com). pigshell.com is not involved at all. - Pigshell is free software - you are free to download the source code, examine, modify and run it locally yourself. (https://github.com/pigshell/pigshell https://github.com/pigshell/pigshell) - Our privacy policy is minimal because we know NOTHING about you - not your name, email address, facebook id, nothing. Only your IP address is recorded in apache logs, which is the case with every website you visit. What is more, we don't want to know. Pigshell has been deliberately designed so as to afford maximum privacy and freedom to the end user.
  - dullcrisp 12y ago
    I don't really doubt you or your intentions, but I guess I might just not understand how authentication works for something like this. When I try to sign in with Google Drive, it tells me that PigShell (developer email: xxxx@gmail.com) would like access to my Google Drive, as well as my photos and videos. Does that give your API key access to my account, or does only code that I run that's hosted on pigshell.com have access? What do the permissions get tied to? And how would it work if I were to host it locally?
    - basantrajan 12y ago
      one way to look at it ... the "application" Pigshell (a copy of the javascript downloaded from the "site" pigshell.com and running in your browser is an instance of the "application") requests and say is granted access to your GDrive. so all permissions/rights are associated with the running instance of the "application" pigshell in your browser - cookies apart, end of story. other than having served the javascript, the "site" pigshell.com has no further role to play here. so you could just as well have sourced/hosted these javascripts locally.
fiatjaf 12y ago
How is the procedure for "installing" new programs? Adding a local directory with javascript commands to the PATH will do it? What about saving the state of things. I want, for example, to `mount` my local directory with psty everytime I open pigshell, and I wanna have my PATH set automatically. Will a `/local/.config/.pigrc` do this? (This raises another question: file editing. Where is the file editing tool? Maybe a web-based editor somewhere). Well, maybe a gist will do it.
- tubelite 12y ago
  There are two kinds of commands: Javascript commands (like ls, cat) and scripts (ps, kill, man, csv2js, table2js). Currently, Javascript commands have to be built-in at "compilation" time. (I need to add a load command which will evaluate a new JS command inside the pigshell closure) Scripts can be anywhere in PATH. There is an rc mechanism for doing exactly what you want. At boot time, we run /etc/profile, followed by /local/rc.sh - this is a file saved in the browser's localStorage. Mine is very simple: HOME=/home mount http://localhost:50937/ $HOME You can create it simply by echo "HOME=/home\nmount http://localhost:50937/ $HOME" >/local/rc.sh or running "edit /local/rc.sh". Edit provides a simple inline CodeMirror-based editor (actually even the CLI is a CodeMirror editor instance). I use vim on the desktop to do any serious editing. If you run psty, /home is a desktop directory visible inside pigshell, so edit on desktop/run in shell is a low-friction process. If you manage to mess up /local/rc.sh to the point where pigshell is not "booting", then you can boot it in "safe mode" by appending #norc to the URL, i.e. http://pigshell.com/#norc http://pigshell.com/#norc
  - progman 12y ago
    I downloaded http://pigshell.com/v/0.6.2/psty.py http://pigshell.com/v/0.6.2/psty.py, started it with "python psty.py -a" and did "mount http://localhost:50937/ http://localhost:50937/ /dir" successfully. Unfortunately only "ls /dir/file" works but neither "cat /dir/file" nor "cp /dir/file ." although "file" was marked with "chmod a+r". I get the message "Cross-origin request denied (check if psty is running)". What do I wrong?
    - 12y ago
aidos 12y ago
Kudos, tubelite. This is really cool. I'm not sure I'd ever really use it but it's been fun to play with. Definitely made me think that I'd like to use the web in a more command based fashion.
vezzy-fnord 12y ago
So in many ways, this is like having a FUSE/Hurd translator implementation directly from the browser, though it also provides a mini-Unix environment due to needing to implement corollaries to some Unix utilities. The scope of this project is definitely what makes it nifty.
drivingmenuts 12y ago
I have no idea what I would use this for, but I really wish I did. Very cool.
omouse 12y ago
I like the sentiment, however I think it'd be cooler if you could get the Facebook, Google, and other APIs to behave in the "everything is a file" tradition of UNIX.
joseph8th 12y ago
A new toy! Thanks!
codehero 12y ago
I typed in `oink` but the command was not found...
altrego99 12y ago
How do I know what are all the fields available on a file?
- tubelite 12y ago
  Here is one way: ls -d /path/to/file_or_dir | printf This command prints the stack of file objects. The last one printed is at the "top" of the stack and tells you the fields available. You can then print a field ls -d /gdrive/user@gmail.com/somedoc | printf "%(raw.ownerNames)s" Sorry, a lot more documentation is needed. Working on it.
shykes 12y ago
Looks a lot like webshell (http://webshell.io http://webshell.io). I wonder if there are other similar tools out there?
itazula 12y ago
Wow, this was like the nutshell cracked and the universe spilled out ...
deleted 12y ago
[deleted]
ConSeannery 12y ago
This is awesome. It reminds me of a talk I saw recently, linked from hackernews at some point in the last year or so. Something about re-thinking internet infrastructure into a "directory-tree" like model. Does anyone remember it?
- oftenwrong 12y ago
  I don't, but I am interested if you find it. We already have a hierarchy in a way. The domain tree goes from top to bottom this way <--- The resource tree goes from top to bottom this way ---> So, using numbering for order, you could look at a URI like: 0002.0001.0000/0003/0004/0005 Meanwhile, in an alternate reality... $ ls /net | shuf -n 16 coffee net рф 한국 org dating pw 中國 institute இலங்கை ninja com みんな su 政务
progman 12y ago
Congratulations! The way how you implemented "man" ("man ls" for instance) shows how a convenient shell can look like. Pigshell could be a foundation for a completely new kind of toolchain for web development. I am thinking of seemless integration of JS editors, D3 graphics for pipe output, etc. ... Wait - it's already working :-) http://bl.ocks.org/ganeshv/raw/b971989c337e958c0531/ http://bl.ocks.org/ganeshv/raw/b971989c337e958c0531/ If it just were possible to access the bare bone hardware so that we wouldn't be restricted to the JS world (in a closed environment, of course).
- pmontra 12y ago
  The way the man page is formatted is great but I don't like the scrollbar inside a scrollbar effect. It's been traditionally a worst practice. It's better to have the man page take all the vertical space it needs in the main frame and add a Close button to the end of the man. By the way, from http://pigshell.com/v/0.6.2/doc/psty.html http://pigshell.com/v/0.6.2/doc/psty.html "Websocket server: Every unix app on the desktop which uses stdin/stdout can participate in a pigshell pipeline" so maybe your request has already been implemented.
  - progman 12y ago
    > By the way, ... Thanks, meanwhile I have realized that also :-)
damian2000 12y ago
My connection is shaped at 128 kbps at the moment - am grateful that this loads really quickly. Awesome stuff. HN itself is really low bandwidth too.
kybernetyk 12y ago
rm -r facebook ah, that feels good.
diltonm 12y ago
It's very cool, gets back to using the power of the simplicity of UNIX. I found though, that X middle click paste doesn't work. I made a directory at /mnt/mccomsoft then ran Firefox and didn't see it which is good. :) Then I tried mounting that using your HttpFS with: mount http://www.mccomsoft.com/index.html /mnt/mccomsoft It gave me: Cross-origin request denied (check if psty is running) Which is also good. :) If I load Pigshell from my domain though then it should mount fine. One user commented below it's like the Internet was a shell and you've cracked it open. I agree.

333 points by nmcfarl 12y ago

More in this comment by the author: https://news.ycombinator.com/item?id=8436716 https://news.ycombinator.com/item?id=8436716

bilalhusain 12y ago

The psty links in README file are broken. A working link is http://pigshell.com/v/0.6.2/psty.py http://pigshell.com/v/0.6.2/psty.py As mentioned in the README, you need to run psty server locally and establish websocket connection (mount in pigshell). This will enable you to harness local filesystem from pigshell.

kolev 12y ago

This is beautiful!

tomw1808 12y ago

I just had a glance about the functionality, but for my understanding, this is the best idea right behind ifttt. Pipelining all the web's things with well designed interfaces on a shell. Even if the product itself would be unstable, alone the idea is great enough for a gold-medal. Keep it up!

tubelite 12y ago

Author here. - Please use a desktop browser (Chrome preferred). While mobile browsers kinda-sorta work if you're just clicking through the examples, keyboard input doesn't work on mobiles (yet). - Pigshell has few users (in fact, using the plural is a slight marketing exaggeration) so please bear with me for any teething issues, broken links, inadequate documentation... - One of the practical use cases is copying files across Google accounts, and backing it up to your desktop. (Details here: http://pigshell.com/v/0.6.2/doc/gdrive.html http://pigshell.com/v/0.6.2/doc/gdrive.html) The nice thing about a unix approach is that the exact same cp command can be used to copy GDrive to Dropbox, or any other supported filesystem. - Some more D3 visualization examples here, using Mike Bostock's gist viewer: http://bl.ocks.org/ganeshv http://bl.ocks.org/ganeshv (Do use the "Open in a new window" links, like this one: http://bl.ocks.org/ganeshv/raw/46cc526d41d16995ccdc/ http://bl.ocks.org/ganeshv/raw/46cc526d41d16995ccdc/).

sprayk 12y ago
I looked through the four documentation links on the right and was unable to find out how to cat a Google spreadsheet as anything consumable by the shell. I took a guess and tried cat -o gdrive.fmt=txt and csv, but those were not fruitful. Is this possible currently? edit: found mimeMap.js and saw tsv was an option. That didn't work either.
- tubelite 12y ago
  IIRC the Drive API does not support those conversion options. The way forward would be for an xlsx parsing JS library to convert it into CSV or a stream of Javascript objects. I didn't find any GPL-compatible library last time I checked. See https://developers.google.com/drive/web/manage-downloads https://developers.google.com/drive/web/manage-downloads for a list of export format possibilities. For spreadsheets, the only options are xlsx, odt and pdf.
  - deleted 12y ago
    [deleted]
  - fiatjaf 12y ago
    Google Drive document can be embedded as iframes. Wouldn't this solve this problem? Here's a proof: http://www.cellmaster.com.au/example.html http://www.cellmaster.com.au/example.html
    - sprayk 12y ago
      not for retrieving the juicy data inside the spreadsheet as objects.
      - fiatjaf 12y ago
        Oh, I didn't read the whole thread, I was just thinking about my own experience with `cat gdrive/something` showing me a lot of unreadable bytes when it could show an iframe.
12y ago

linvin 12y ago

Ah! Nice feeling that whole of internet as a single (unix) machine.

serf 12y ago
reminds me of Plan9 sorta.

crater 12y ago

I like the idea. If I'm playing with some facebook data, right now I use whatever language I'm confortable with and use its HTTP request functionality, and than manipulate the data how I want in that language. To achieve the same thing in pigshell I have to write a shell script that manipulates the data. What can pigshell do that I can't just do in another language?

tubelite 12y ago
In theory, the availability of APIs should be enough for the technical user to write their own data extraction routines, connect them to other APIs and so on, using their preferred language and toolchain. In practice, reading up the documentation, writing and debugging code sets up enough of a barrier to entry that it discourages casual usage without a powerful incentive. The availability of a file abstraction covering many popular cloud services, web resources etc. enables casual, low cost experimentation across these domains, without having to read up each one of the APIs. If something clicks, then it may make sense to write a better "native" implementation.

fiatjaf 12y ago

I'm curious about how are people using this. Is it just a visualization tool for data or what else? Is anyone using it as their primary navigation tool?

tubelite 12y ago
Like I said in an earlier comment, there are very few users right now (cough OK just me cough). Mostly I have fun with it, e.g. trying out visualizations of interesting data (e.g. http://bl.ocks.org/ganeshv/raw/fada13e56d2c5b7b1a78/ http://bl.ocks.org/ganeshv/raw/fada13e56d2c5b7b1a78/) My Serious Use Case is full and incremental backups of the contents of my Google Drive, both personal and company, to my desktop. Running the Drive desktop app does not backup Google Docs, and you can only sign in as a single user at a time.
basantrajan 12y ago
perhaps stating the obvious here, but with the abstraction pigshell provides of resources on the web, it should be reasonably straight forward to build arbitrary "internet applications" too. with deferred pipes, split and joins (that pigshell already provides/supports) it should be possible to implement arbitrary work/data flow definitions one desires. coupled with psty, one could build/harness special purpose "transformers" on the web pushing it further, one could conceive of setting up pipes between transformers on the web. of course getting there will require significant work on a security model as well. [and then someone will figure out a way to interject ads]
- fiatjaf 12y ago
  CouchDB is already compatible.
- progman 12y ago
  > of course getting there will require significant work on a security model as well. It could probably be easier than you imagine. The security model could be implemented in psty completely by granting/denying requested resources whose permissions are stored in a database on the psty server, and managed by using long security IDs in the URLs. This way the security model could even be fine-grained. In other words, pigshell itself could be embedded into a secure environment, covered by the permissions of psty.

mgrassotti 12y ago

holy crap this is awesome

jayseeg 12y ago

With some practice, or even better, some support libraries, this looks like a great way to quickly explore viz prototypes. Great work!

roywiggins 12y ago

Attempting to copy photos out of Facebook albums into GDrive gives me "Exception: SecurityError: The operation is insecure." I'm not running psty locally. This is super cool, though. There's an undocumented template engine built in it seems? (see /bin/mostliked) which is neat.

Erwin 12y ago

Interesting. I'm trying to really go out all the API for my project and was thinking about something like a Mathemathica-like notebook for the local API. E.g. say you want to execute an API call which normally does GET /api/v1/x/y/z; in the "API shell" you type x/y/z and the JSON collection is summarized as a small datatable (not showing all of the 500 rows / 40 columns that might have, but an interesting summary) that you can later interact with. I'm thinking that such an approach can give the user their own way to explore/modify the data ,especially if you combine this with some kind of pipe lines that let you analyze/mutate/visualize the data without having to write some Python code to access the API (even though with a REST API plus e.g. requests it's simple). I like the psty idea to integrate with the local system. Sounds like it could also be used to start e.g. emacs on a remote page. Say you have a site with an API that lets you read/write files, you mount it and then can fire up emacs on it via pigshell.

privong 12y ago
Is this approach similar to something like flickrfs[0], to make one's flickr account locally accessible? It always seemed like a cool idea (though I never had much luck with flickrfs itself), and it is neat to see the broader utility of psty. [0] https://launchpad.net/flickrfs https://launchpad.net/flickrfs

talles 12y ago

> Error loading root filesystem at http://pigshell.com/v/0.6.2/root.tar http://pigshell.com/v/0.6.2/root.tar: Forbidden :(

dllthomas 12y ago

Have you given thought to exposing this to the regular shell with FUSE?

tubelite 12y ago
The concept of lazy-stream-of-JS-objects and HTML as a display engine underpin pigshell. I guess it would be possible to adapt the filesystem code to fuse4js. It won't be a drop-in; unix FS ops differ from pigshell's, but it shouldn't be too hard. Personally, I find psty (http://pigshell.com/v/0.6.2/doc/psty.html http://pigshell.com/v/0.6.2/doc/psty.html) a better way to incorporate (a part of) the unix filesystem into the web rather than fuse - which is the other way round.
- clebio 12y ago
  Isn't this -- at least in spirit -- what IPython notebook tries to do? In IPython, you can use shell commands with a bang, e.g. `!ls -l`. And `ipython notebook` uses ZMQ to connect the server and client. At any rate, awesome work. I've been thinking for a while, offhandedly, about the idea of piping web URLs around. But I don't know how in fact it would work under the hood. I write a lot of API interfaces in my work, and it's all in the details -- every API is different in small but important ways.
  - tubelite 12y ago
    Yes, it is similar in spirit to IPython. We have been developing pigshell off and on since ~2012, and I was unaware of IPython notebooks until ~6 months ago. When I'd last used IPython, it didn't have any of the notebook stuff. So it was a bit of a jar to see that they had been there, done that, got the T-shirt, published the book, etc. etc. :) Pigshell can run in the browser with zero installation, and has a low barrier for casual usage. "Notebooks" aka gists can be shared without requiring a backend. The focus of pigshell is more around providing file adapters for web/cloud data stores. That said - more power to IPython! I love the conversational, exploratory, CLI style of interacting with a computing environment. The more ecosystems of this sort, the better.

tubelite 12y ago

So if you are running psty, here is a fun sequence of commands to try out: cd http://dilbert.com/strips/comic/1995-06-24/ ls cat 21021.strip.zoom.gif Never really liked the colorization of Dilbert. It was so much nicer as a line drawing. Best we can do is remove the colors. cat 21021.strip.zoom.gif | to canvas | pixastic -d And to save your result, cat 21021.strip.zoom.gif | to canvas | pixastic -d >/home/dilbert-unix.png Or to GDrive or Dropbox or even the local RamFS by changing the target directory. (Scott Adams is fine with personal usage. I checked. And you need psty because it proxies HTTP requests for sites which don't have CORS enabled)

bespoke_engnr 12y ago

Wow, this is a really interesting way to think about web resources. After about two minutes of playing around with this, it gave me a more pleasant experience of the web than I've had in a while.

anigbrowl 12y ago
This is what it was like when I first got on the Internet (around 1992); there was no web then, instead you FTPed everything and used resources like Gopher, Archie, and Veronica to find documents. I find PigShell interesting from a historic point of view but think it also points to a flaw in how the web currently works, with far too much glue and eye candy, which is a direct result of the ad business model.
- bespoke_engnr 12y ago
  I completely agree. I actually deleted a bunch of my comment before posting it, because I started getting carried away pondering about what the right level of abstraction is for dealing with web resources. At the risk of sounding like one of the semantic-web xml-standards people, it really does make me wonder if all this horrible, incompatible, site-specific frontend crap is just a fad that will give way to tools (like psty) that ignore all that and just deal with the raw semantic content (and HTTP verbs) at a higher level of abstraction. Woah, man. We're like...surfing the Matrix, man.
  - svalorzen 12y ago
    Things like this really make me miss the old Firefox experiment "Ubiquity". It was truly a marvel for the time it was up, and I often wonder why they did discontinue it. You could search websites, send emails, look at maps and much more (you could build your own commands) without ever actually opening a webpage. It had 200.000 users when they shut it down.
    - jacobwcarlson 12y ago
      Wouldn't you just be able to fork it and continue as a hobby project of your own? Who knows, maybe you find other enthusiasts and it becomes a real project again. Mozilla licenses are (I believe and could be wrong), quite lenient about these things.

deleted 12y ago

[deleted]

TimWolla 12y ago

Does not work in Internet Explorer 11: Exception: The property "slice" of an undefined or null referance cannot be accessed (Exception: Die Eigenschaft "slice" eines undefinierten oder Nullverweises kann nicht abgerufen werden.)

tubelite 12y ago
Sorry, Internet Explorer is not supported currently. Please try Chrome or Firefox, in that order.

dullcrisp 12y ago

I do love this idea, but it bothers me from a security standpoint. Is there a way to do something like this without giving the author explicit control of all of my online data?

tubelite 12y ago
The author has NO control whatsoever of your data. - Pigshell is a 100% client-side Javascript app. There is no server side. pigshell.com is a dumb static html/js/css server. - All data (username/password as well as user data like photos and files) travels directly between the user's browser and the provider (e.g. google.com, facebook.com). pigshell.com is not involved at all. - Pigshell is free software - you are free to download the source code, examine, modify and run it locally yourself. (https://github.com/pigshell/pigshell https://github.com/pigshell/pigshell) - Our privacy policy is minimal because we know NOTHING about you - not your name, email address, facebook id, nothing. Only your IP address is recorded in apache logs, which is the case with every website you visit. What is more, we don't want to know. Pigshell has been deliberately designed so as to afford maximum privacy and freedom to the end user.
- dullcrisp 12y ago
  I don't really doubt you or your intentions, but I guess I might just not understand how authentication works for something like this. When I try to sign in with Google Drive, it tells me that PigShell (developer email: xxxx@gmail.com) would like access to my Google Drive, as well as my photos and videos. Does that give your API key access to my account, or does only code that I run that's hosted on pigshell.com have access? What do the permissions get tied to? And how would it work if I were to host it locally?
  - basantrajan 12y ago
    one way to look at it ... the "application" Pigshell (a copy of the javascript downloaded from the "site" pigshell.com and running in your browser is an instance of the "application") requests and say is granted access to your GDrive. so all permissions/rights are associated with the running instance of the "application" pigshell in your browser - cookies apart, end of story. other than having served the javascript, the "site" pigshell.com has no further role to play here. so you could just as well have sourced/hosted these javascripts locally.

fiatjaf 12y ago

How is the procedure for "installing" new programs? Adding a local directory with javascript commands to the PATH will do it? What about saving the state of things. I want, for example, to `mount` my local directory with psty everytime I open pigshell, and I wanna have my PATH set automatically. Will a `/local/.config/.pigrc` do this? (This raises another question: file editing. Where is the file editing tool? Maybe a web-based editor somewhere). Well, maybe a gist will do it.

tubelite 12y ago
There are two kinds of commands: Javascript commands (like ls, cat) and scripts (ps, kill, man, csv2js, table2js). Currently, Javascript commands have to be built-in at "compilation" time. (I need to add a load command which will evaluate a new JS command inside the pigshell closure) Scripts can be anywhere in PATH. There is an rc mechanism for doing exactly what you want. At boot time, we run /etc/profile, followed by /local/rc.sh - this is a file saved in the browser's localStorage. Mine is very simple: HOME=/home mount http://localhost:50937/ $HOME You can create it simply by echo "HOME=/home\nmount http://localhost:50937/ $HOME" >/local/rc.sh or running "edit /local/rc.sh". Edit provides a simple inline CodeMirror-based editor (actually even the CLI is a CodeMirror editor instance). I use vim on the desktop to do any serious editing. If you run psty, /home is a desktop directory visible inside pigshell, so edit on desktop/run in shell is a low-friction process. If you manage to mess up /local/rc.sh to the point where pigshell is not "booting", then you can boot it in "safe mode" by appending #norc to the URL, i.e. http://pigshell.com/#norc http://pigshell.com/#norc
- progman 12y ago
  I downloaded http://pigshell.com/v/0.6.2/psty.py http://pigshell.com/v/0.6.2/psty.py, started it with "python psty.py -a" and did "mount http://localhost:50937/ http://localhost:50937/ /dir" successfully. Unfortunately only "ls /dir/file" works but neither "cat /dir/file" nor "cp /dir/file ." although "file" was marked with "chmod a+r". I get the message "Cross-origin request denied (check if psty is running)". What do I wrong?
  - 12y ago

aidos 12y ago

Kudos, tubelite. This is really cool. I'm not sure I'd ever really use it but it's been fun to play with. Definitely made me think that I'd like to use the web in a more command based fashion.

vezzy-fnord 12y ago

So in many ways, this is like having a FUSE/Hurd translator implementation directly from the browser, though it also provides a mini-Unix environment due to needing to implement corollaries to some Unix utilities. The scope of this project is definitely what makes it nifty.

drivingmenuts 12y ago

I have no idea what I would use this for, but I really wish I did. Very cool.

omouse 12y ago

I like the sentiment, however I think it'd be cooler if you could get the Facebook, Google, and other APIs to behave in the "everything is a file" tradition of UNIX.

joseph8th 12y ago

A new toy! Thanks!

codehero 12y ago

I typed in `oink` but the command was not found...

altrego99 12y ago

How do I know what are all the fields available on a file?

tubelite 12y ago
Here is one way: ls -d /path/to/file_or_dir | printf This command prints the stack of file objects. The last one printed is at the "top" of the stack and tells you the fields available. You can then print a field ls -d /gdrive/user@gmail.com/somedoc | printf "%(raw.ownerNames)s" Sorry, a lot more documentation is needed. Working on it.

shykes 12y ago

Looks a lot like webshell (http://webshell.io http://webshell.io). I wonder if there are other similar tools out there?

itazula 12y ago

Wow, this was like the nutshell cracked and the universe spilled out ...

deleted 12y ago

[deleted]

ConSeannery 12y ago

This is awesome. It reminds me of a talk I saw recently, linked from hackernews at some point in the last year or so. Something about re-thinking internet infrastructure into a "directory-tree" like model. Does anyone remember it?

oftenwrong 12y ago
I don't, but I am interested if you find it. We already have a hierarchy in a way. The domain tree goes from top to bottom this way <--- The resource tree goes from top to bottom this way ---> So, using numbering for order, you could look at a URI like: 0002.0001.0000/0003/0004/0005 Meanwhile, in an alternate reality... $ ls /net | shuf -n 16 coffee net рф 한국 org dating pw 中國 institute இலங்கை ninja com みんな su 政务

progman 12y ago

Congratulations! The way how you implemented "man" ("man ls" for instance) shows how a convenient shell can look like. Pigshell could be a foundation for a completely new kind of toolchain for web development. I am thinking of seemless integration of JS editors, D3 graphics for pipe output, etc. ... Wait - it's already working :-) http://bl.ocks.org/ganeshv/raw/b971989c337e958c0531/ http://bl.ocks.org/ganeshv/raw/b971989c337e958c0531/ If it just were possible to access the bare bone hardware so that we wouldn't be restricted to the JS world (in a closed environment, of course).

pmontra 12y ago
The way the man page is formatted is great but I don't like the scrollbar inside a scrollbar effect. It's been traditionally a worst practice. It's better to have the man page take all the vertical space it needs in the main frame and add a Close button to the end of the man. By the way, from http://pigshell.com/v/0.6.2/doc/psty.html http://pigshell.com/v/0.6.2/doc/psty.html "Websocket server: Every unix app on the desktop which uses stdin/stdout can participate in a pigshell pipeline" so maybe your request has already been implemented.
- progman 12y ago
  > By the way, ... Thanks, meanwhile I have realized that also :-)

damian2000 12y ago

My connection is shaped at 128 kbps at the moment - am grateful that this loads really quickly. Awesome stuff. HN itself is really low bandwidth too.

kybernetyk 12y ago

rm -r facebook ah, that feels good.

diltonm 12y ago

It's very cool, gets back to using the power of the simplicity of UNIX. I found though, that X middle click paste doesn't work. I made a directory at /mnt/mccomsoft then ran Firefox and didn't see it which is good. :) Then I tried mounting that using your HttpFS with: mount http://www.mccomsoft.com/index.html /mnt/mccomsoft It gave me: Cross-origin request denied (check if psty is running) Which is also good. :) If I load Pigshell from my domain though then it should mount fine. One user commented below it's like the Internet was a shell and you've cracked it open. I agree.