5 ms·

MIT creates picture-driven programming for the masses

27 points by prat 17y ago

physcab 17y ago
There already is a visual scripting environment--its called LabView and its terrible. LabView is used mostly by scientists who want to control their experiments but don't want to learn how to program. It's very effective when dealing with hardware and you just want to get it working. But if it doesn't work, you're screwed because its very difficult to debug. That's actually the problem I forsee with the visual scripting environment. There is no better single point of instruction than a compiler error or syntax error highlighting. If you try to circumvent that process to make it more "accessible" to the public, you're going to keep your visual programmers in the dark as to what is actually occurring.
- sp332 17y ago
  Did you even read the article? This isn't a visual scripting environment, it's an intuitive way to use Python to do image processing. Edit: here's a paper with examples. http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-uist2009.pdf http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-uist2...
  - physcab 17y ago
    Ah. You're right. I misread it. However, I stand by my points on visual scripting in general. Even if there is a way to automate certain tasks via GUI elements, you are placing a great deal of trust on the system to operate as expected. I do quite a bit of research in machine learning, and I can say that these systems are never perfect. Building an abstraction on top of an image processing technique is interesting, but I really wonder how a system like that can be debugged if something goes wrong.
- mtomczak 17y ago
  For the record (though as other commenters have pointed out, this rant on LabView is off-topic): LabView is a very successful tool that is used by many programmers in various fields, not just instrumentation. LabView does have compiler errors and syntax error highlighting; they don't look the same as in a text-focused language, but since G is a programming language, it has its own syntax and errors can occur. Programming languages are an abstraction. G is just another abstraction. I've used BASIC and C compilers with terribly cryptic, useless errors too; the errors had more to do with flaws in the compiler and IDE than the language itself. If errors in LabView are cryptic or obtuse, I would assume something similar.
Maciek416 17y ago
There's more to this project than is hinted at by the "picture-driven programming" label. This scripting system is driven not just by pictures, but by screenshot snippets of the GUI elements that you want to click on / control. The language uses computer vision techniques to match things on the screen (and I'm guessing to prevent programs from becoming overly brittle if minor visual changes happen to the screen). Check out the video on the project's actual page, it's actually kind of interesting: http://sikuli.csail.mit.edu/ http://sikuli.csail.mit.edu/
- yannis 17y ago
  I read the paper and frankly speaking, I am very skeptical that this is an application for the masses. I found this piece interesting: >The Sikuli Search prototype has a database of 102 popular computer books covering various operating systems (e.g., Windows XP, MacOS) and applications (e.g., Photoshop, Office), all represented in PDF4. This database contains more than 50k screenshots. The three-feature indexing scheme is written in C++ to index these screenshots, using SIFT [11] to extract visual features, Tesseract5 for OCR, and Ferret6 for indexing the text surrounding the screenshots. All other server-side functionality, such as accepting queries and formatting search results, is implemented in Ruby on Rails7 with a SQL database. On the client side, the interfaces for searching and annotating screenshots are implemented in Java. ... and the scripting engine is in Python. Sometimes I wonder if Computer Science wouldn't have been better off with a constantly evolving C rather than all this mess. Every time I build a web application I feel as if I am building Frankenstein's body with different animal parts. There were two major attempts to bring computers to the masses COBOL and SQL and they both managed to produce the opposite result rather than the masses using them a demand was created for very highly paid Programmer jobs! Electronic Engineers never attempted to teach the masses how to build TV's. In the late 80's I was teaching laymen things like 'How to Use MSDOS' and 'Introduction to Wordperfect' which paid for the rent but overall was a depressing experience. I also taught at the same time 'Pascal' and 'C' to Mechanical Engineers, my conclusion was that some people take to programming - like some people take to maths, they enjoy it and after an initial push they will teach themselves the rest and the rest will never touch programming even if their life depended on it. In my mind the only successful 'programming for the masses' products are Excel and partly HTML.
wallflower 17y ago
Wow. This might be perfect for advanced screen scraping. "GUI Testing Using Computer Vision" http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-chi2010.pdf http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-chi20... EDIT: It appears it cannot grab the actual text using the Native APIs so you would have to include a pipeline to OCR black-on-white text
SandB0x 17y ago
I thought this might be similar to the incredible Baltie programming language: http://en.wikipedia.org/wiki/File:Baltik3-screenshot1.PNG http://en.wikipedia.org/wiki/File:Baltik3-screenshot1.PNG (it isn't)

27 points by prat 17y ago

There already is a visual scripting environment--its called LabView and its terrible. LabView is used mostly by scientists who want to control their experiments but don't want to learn how to program. It's very effective when dealing with hardware and you just want to get it working. But if it doesn't work, you're screwed because its very difficult to debug. That's actually the problem I forsee with the visual scripting environment. There is no better single point of instruction than a compiler error or syntax error highlighting. If you try to circumvent that process to make it more "accessible" to the public, you're going to keep your visual programmers in the dark as to what is actually occurring.

sp332 17y ago
Did you even read the article? This isn't a visual scripting environment, it's an intuitive way to use Python to do image processing. Edit: here's a paper with examples. http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-uist2009.pdf http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-uist2...
- physcab 17y ago
  Ah. You're right. I misread it. However, I stand by my points on visual scripting in general. Even if there is a way to automate certain tasks via GUI elements, you are placing a great deal of trust on the system to operate as expected. I do quite a bit of research in machine learning, and I can say that these systems are never perfect. Building an abstraction on top of an image processing technique is interesting, but I really wonder how a system like that can be debugged if something goes wrong.
mtomczak 17y ago
For the record (though as other commenters have pointed out, this rant on LabView is off-topic): LabView is a very successful tool that is used by many programmers in various fields, not just instrumentation. LabView does have compiler errors and syntax error highlighting; they don't look the same as in a text-focused language, but since G is a programming language, it has its own syntax and errors can occur. Programming languages are an abstraction. G is just another abstraction. I've used BASIC and C compilers with terribly cryptic, useless errors too; the errors had more to do with flaws in the compiler and IDE than the language itself. If errors in LabView are cryptic or obtuse, I would assume something similar.

Maciek416 17y ago

There's more to this project than is hinted at by the "picture-driven programming" label. This scripting system is driven not just by pictures, but by screenshot snippets of the GUI elements that you want to click on / control. The language uses computer vision techniques to match things on the screen (and I'm guessing to prevent programs from becoming overly brittle if minor visual changes happen to the screen). Check out the video on the project's actual page, it's actually kind of interesting: http://sikuli.csail.mit.edu/ http://sikuli.csail.mit.edu/

yannis 17y ago
I read the paper and frankly speaking, I am very skeptical that this is an application for the masses. I found this piece interesting: >The Sikuli Search prototype has a database of 102 popular computer books covering various operating systems (e.g., Windows XP, MacOS) and applications (e.g., Photoshop, Office), all represented in PDF4. This database contains more than 50k screenshots. The three-feature indexing scheme is written in C++ to index these screenshots, using SIFT [11] to extract visual features, Tesseract5 for OCR, and Ferret6 for indexing the text surrounding the screenshots. All other server-side functionality, such as accepting queries and formatting search results, is implemented in Ruby on Rails7 with a SQL database. On the client side, the interfaces for searching and annotating screenshots are implemented in Java. ... and the scripting engine is in Python. Sometimes I wonder if Computer Science wouldn't have been better off with a constantly evolving C rather than all this mess. Every time I build a web application I feel as if I am building Frankenstein's body with different animal parts. There were two major attempts to bring computers to the masses COBOL and SQL and they both managed to produce the opposite result rather than the masses using them a demand was created for very highly paid Programmer jobs! Electronic Engineers never attempted to teach the masses how to build TV's. In the late 80's I was teaching laymen things like 'How to Use MSDOS' and 'Introduction to Wordperfect' which paid for the rent but overall was a depressing experience. I also taught at the same time 'Pascal' and 'C' to Mechanical Engineers, my conclusion was that some people take to programming - like some people take to maths, they enjoy it and after an initial push they will teach themselves the rest and the rest will never touch programming even if their life depended on it. In my mind the only successful 'programming for the masses' products are Excel and partly HTML.

wallflower 17y ago

Wow. This might be perfect for advanced screen scraping. "GUI Testing Using Computer Vision" http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-chi2010.pdf http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-chi20... EDIT: It appears it cannot grab the actual text using the Native APIs so you would have to include a pipeline to OCR black-on-white text

SandB0x 17y ago

I thought this might be similar to the incredible Baltie programming language: http://en.wikipedia.org/wiki/File:Baltik3-screenshot1.PNG http://en.wikipedia.org/wiki/File:Baltik3-screenshot1.PNG (it isn't)