4 ms·
Glad to see people are still making better window managers! Is this open source? I'd love to see how it works by reading the code. Which Windows APIs does it u
by DonHopkins 10y ago
Glad to see people are still making better window managers!
Is this open source? I'd love to see how it works by reading the code. Which Windows APIs does it use? How is it customizable in XAML? Can it be extended in JavaScript?
I think extensibility and accessibility are extremely important for window managers.
There is a window manager for the Mac called Slate that is extensible in JavaScript -- it makes a hidden WebView and uses its JS interpreter by extending it with some interfaces to the app to do window management, using the Mac Accessibility API.
"Slate: A window management application (replacement for Divvy/SizeUp/ShiftIt)": https://github.com/jigish/slate https://github.com/jigish/slate
I'd like to take that idea a lot further, so I wrote up some ideas about programming window management, accessibility, screen scraping, pattern recognition and automation in JavaScript.
"aQuery: like jQuery for Accessibility": http://donhopkins.com/mediawiki/index.php/AQuery http://donhopkins.com/mediawiki/index.php/AQuery
Check out Morgan Dixon's and James Fogarty's amazing work on user interface customization with Prefab, about which they've published several excellent CHI papers:
"Prefab: What if We Could Modify Any Interface?": https://www.youtube.com/watch?v=lju6IIteg9Q https://www.youtube.com/watch?v=lju6IIteg9Q
Imagine if every interface was open source. Any of us could modify the software we use every day. Unfortunately, we don't have the source.
Prefab realizes this vision using only the pixels of everyday interfaces. This video shows the use of Prefab to add new functionality to Adobe Photoshop, Apple iTunes, and Microsoft Windows Media Player. Prefab represents a new approach to deploying HCI research in everyday software, and is also the first step toward a future where anybody can modify any interface.
Presented by Morgan Dixon and James Fogarty at CHI 2010.
"Content and Hierarchy in Prefab": https://www.youtube.com/watch?v=w4S5ZtnaUKE https://www.youtube.com/watch?v=w4S5ZtnaUKE
Imagine if every interface was open source. Any of us could modify the software we use every day. Unfortunately, we don't have the source.
Prefab realizes this vision using only the pixels of everyday interfaces. This video shows how we advanced the capabilities of Prefab to understand interface content and hierarchy. We use Prefab to add new functionality to Microsoft Word, Skype, and Google Chrome. These demonstrations show how Prefab can be used to translate the language of interfaces, add tutorials to interfaces, and add or remove content from interfaces solely from their pixels. Prefab represents a new approach to deploying HCI research in everyday software, and is also the first step toward a future where anybody can modify any interface.
Presented by Morgan Dixon and James Fogarty at CHI 2010.
"A General-Purpose Bubble Cursor": https://www.youtube.com/watch?v=46EopD_2K_4 https://www.youtube.com/watch?v=46EopD_2K_4
We present a general-purpose implementation of Grossman and Balakrishnan's Bubble Cursor (http://www.tovigrossman.com/BubbleCursor/ http://www.tovigrossman.com/BubbleCursor/), the fastest general pointing facilitation technique in the literature. Our implementation functions with any application on the Windows 7 desktop. Our implementation functions across this infinite range of applications by analyzing pixels and by leveraging human corrections when it fails.
"Prefab: Implementing Advanced Behaviors Using Pixel-Based Reverse Engineering of Interface Structure." at CHI2009: https://redesign.cs.washington.edu/sites/default/files/hci/papers/CHI2009-Prefab-Final.pdf https://redesign.cs.washington.edu/sites/default/files/hci/p...
"Prefab: What if Every GUI Were Open-Source?" at CHI2010: https://homes.cs.washington.edu/~jfogarty/publications/workshop-chi2010.pdf https://homes.cs.washington.edu/~jfogarty/publications/works...
"Abstract:
Current methods for implementing graphical user interfaces
create fundamental challenges for HCI research and
practice. Researchers are often unable to demonstrate or
evaluate new techniques beyond small toy applications, and
practitioners are often unable to adopt methods from the
literature in new and existing applications. This position
statement examines a vision in which anybody can modify
any GUI of any application, similar to a scenario where
every GUI of every application is open-source. We are
currently working to enable this vision through our
development of Prefab, using pixel-based interpretation of
GUIs to enable modification of those GUIs without any
cooperation from the underlying application. We see
participation in the FLOSS HCI workshop as valuable in at
least two regards. First, fully realizing this vision will likely
require a community-based approach, so we are interested
in Prefab as a platform for collaboration between HCI
researchers and the FLOSS community. Second, enabling
arbitrary modification of any GUI would seem to blur many
current distinctions between open and closed applications,
introducing new research questions and further magnifying
the importance of the workshop’s focus."
"Content and Hierarchy in Pixel-Based Methods for Reverse-Engineering Interface Structure" at CHI2011: http://homes.cs.washington.edu/~jfogarty/publications/chi2011.pdf http://homes.cs.washington.edu/~jfogarty/publications/chi201...
"A General-Purpose Target-Aware Pointing Enhancement Using Pixel-Level Analysis of Graphical Interfaces" at CHI2012: https://homes.cs.washington.edu/~jfogarty/publications/chi2012-prefab.pdf https://homes.cs.washington.edu/~jfogarty/publications/chi20...
And if you're into the history of window management, and would like to read some interesting discussions about many great ideas, check out this classic Springer Verlag book that's now available free online:
"Methodology of Window Management": http://www.chilton-computing.org.uk/inf/literature/books/wm/overview.htm http://www.chilton-computing.org.uk/inf/literature/books/wm/...
By F R A Hopgood, D A Duce, E V C Fielding, K Robinson, A S Williams. 29 April 1985.
This is the Proceedings of the Alvey Workshop at Cosener's House, Abingdon that took place from 29 April 1985 until 1 May 1985. It was input into the planning for the MMI part of the Alvey Programme.
The Proceedings were later published by Springer-Verlag in 1986.
My favorite chapters:
"Ten Years of Window Systems - A Retrospective View" by Warren Teitelman: http://www.chilton-computing.org.uk/inf/literature/books/wm/p004.htm http://www.chilton-computing.org.uk/inf/literature/books/wm/...
"SunDew - A Distributed and Extensible Window System" by Games Gosling: http://www.chilton-computing.org.uk/inf/literature/books/wm/p005.htm http://www.chilton-computing.org.uk/inf/literature/books/wm/...
"User Interface Working Group Discussions": http://www.chilton-computing.org.uk/inf/literature/books/wm/p017.htm http://www.chilton-computing.org.uk/inf/literature/books/wm/...
"User Interface Working Group Final Report": http://www.chilton-computing.org.uk/inf/literature/books/wm/p018.htm http://www.chilton-computing.org.uk/inf/literature/books/wm/...
- lostmsu 10y agoWow, that's a really long comment. Stack is not open source, however, some libs are (though only non-window management related). You can find them at https://github.com/losttech/ https://github.com/losttech/ There's really not many WinAPI to call. SetWindowHookEx for handling global mouse events (there's an open source lib I use actually). MoveWindow to (guess what?) move windows. The rest is mostly hand-made. There is no JavaScript extensibility right now, but I could think about one for the future versions. Basically, extensibility right now is limited to decorations in XAML. However, that will be significantly extended when some features I mentioned above will land. What I also though about is an open source library of pre-made screen layouts.
- nunb 10y agoHello, nice to see another big fan of Fogarty & Dixon's work ... such a pity that Microsoft or others didn't make these ideas mainstream! Dixon pointed me to his repo on github if you're interested...
- DonHopkins 10y agoYes please I'd love to see their code! It needs to be integrated with Electron as a V8 extension.
- DonHopkins 10y agoFound it! https://github.com/prefab/code https://github.com/prefab/code Prefab is a set of software tools for reverse engineering graphical interfaces from their pixels. Prefab supports a variety of features for modeling the appearance of interface elements, identifying those elements in pixels, and applying semantic metadata to those elements. Prefab reverse engineers interfaces many times per second, making it possible to completely modify existing applications at runtime, independent of their underlying source code. Prefab is written in C# and runs on Windows. However, because it only relies on the pixels of an interface, it can be combined with a remote desktop or virtual machine client to reverse engineer applications that run in Mac, Linux, or any other OS. Prefab is the creation of the University of Washington Dub Group http://dub.washington.edu http://dub.washington.edu.