4 ms·

There's more to this project than is hinted at by the "picture-driven programming" label. This scripting system is driven not just by pictures, but by screensho

There's more to this project than is hinted at by the "picture-driven programming" label. This scripting system is driven not just by pictures, but by screenshot snippets of the GUI elements that you want to click on / control. The language uses computer vision techniques to match things on the screen (and I'm guessing to prevent programs from becoming overly brittle if minor visual changes happen to the screen). Check out the video on the project's actual page, it's actually kind of interesting: http://sikuli.csail.mit.edu/ http://sikuli.csail.mit.edu/

yannis 17y ago
I read the paper and frankly speaking, I am very skeptical that this is an application for the masses. I found this piece interesting: >The Sikuli Search prototype has a database of 102 popular computer books covering various operating systems (e.g., Windows XP, MacOS) and applications (e.g., Photoshop, Office), all represented in PDF4. This database contains more than 50k screenshots. The three-feature indexing scheme is written in C++ to index these screenshots, using SIFT [11] to extract visual features, Tesseract5 for OCR, and Ferret6 for indexing the text surrounding the screenshots. All other server-side functionality, such as accepting queries and formatting search results, is implemented in Ruby on Rails7 with a SQL database. On the client side, the interfaces for searching and annotating screenshots are implemented in Java. ... and the scripting engine is in Python. Sometimes I wonder if Computer Science wouldn't have been better off with a constantly evolving C rather than all this mess. Every time I build a web application I feel as if I am building Frankenstein's body with different animal parts. There were two major attempts to bring computers to the masses COBOL and SQL and they both managed to produce the opposite result rather than the masses using them a demand was created for very highly paid Programmer jobs! Electronic Engineers never attempted to teach the masses how to build TV's. In the late 80's I was teaching laymen things like 'How to Use MSDOS' and 'Introduction to Wordperfect' which paid for the rent but overall was a depressing experience. I also taught at the same time 'Pascal' and 'C' to Mechanical Engineers, my conclusion was that some people take to programming - like some people take to maths, they enjoy it and after an initial push they will teach themselves the rest and the rest will never touch programming even if their life depended on it. In my mind the only successful 'programming for the masses' products are Excel and partly HTML.
- abdulhaq 17y ago
  I don't believe that COBOL was designed for the masses - after all, this was 1959, when there were so few computers and it was believed by some that ultimately we would only need one! Also, SQL came out of academia and I don't believe that was for the masses either. I would say however that BASIC was intended for the masses, and as such it was successful. A generation in the UK learnt BASIC on the ZX80, ZX81, Spectrum, BBC Micro etc.
  - yannis 17y ago
    Sadly, when CODASYL committee was formed to develop a programming language ideally suited for business data processing needs they were looking at a language that 'normal operators' could learn and that it would be similar to a human language that is why you get Cobol syntax like: ADD YEARS TO AGE A similar motivation was for SQL. We will give you a database and a query language that anyone can use: SELECT title FROM Book ..... During the early days of computing the thought prevailed - pretty much like now - that it was a matter of 10 years before computers would be able to talk in natural languages and AI was around the corner. But, you are right about BASIC that was my first language as well, quickly dropped for Turbo Pascal which was the sort of the in-thing for a while.

yannis 17y ago

I read the paper and frankly speaking, I am very skeptical that this is an application for the masses. I found this piece interesting: >The Sikuli Search prototype has a database of 102 popular computer books covering various operating systems (e.g., Windows XP, MacOS) and applications (e.g., Photoshop, Office), all represented in PDF4. This database contains more than 50k screenshots. The three-feature indexing scheme is written in C++ to index these screenshots, using SIFT [11] to extract visual features, Tesseract5 for OCR, and Ferret6 for indexing the text surrounding the screenshots. All other server-side functionality, such as accepting queries and formatting search results, is implemented in Ruby on Rails7 with a SQL database. On the client side, the interfaces for searching and annotating screenshots are implemented in Java. ... and the scripting engine is in Python. Sometimes I wonder if Computer Science wouldn't have been better off with a constantly evolving C rather than all this mess. Every time I build a web application I feel as if I am building Frankenstein's body with different animal parts. There were two major attempts to bring computers to the masses COBOL and SQL and they both managed to produce the opposite result rather than the masses using them a demand was created for very highly paid Programmer jobs! Electronic Engineers never attempted to teach the masses how to build TV's. In the late 80's I was teaching laymen things like 'How to Use MSDOS' and 'Introduction to Wordperfect' which paid for the rent but overall was a depressing experience. I also taught at the same time 'Pascal' and 'C' to Mechanical Engineers, my conclusion was that some people take to programming - like some people take to maths, they enjoy it and after an initial push they will teach themselves the rest and the rest will never touch programming even if their life depended on it. In my mind the only successful 'programming for the masses' products are Excel and partly HTML.

abdulhaq 17y ago
I don't believe that COBOL was designed for the masses - after all, this was 1959, when there were so few computers and it was believed by some that ultimately we would only need one! Also, SQL came out of academia and I don't believe that was for the masses either. I would say however that BASIC was intended for the masses, and as such it was successful. A generation in the UK learnt BASIC on the ZX80, ZX81, Spectrum, BBC Micro etc.
- yannis 17y ago
  Sadly, when CODASYL committee was formed to develop a programming language ideally suited for business data processing needs they were looking at a language that 'normal operators' could learn and that it would be similar to a human language that is why you get Cobol syntax like: ADD YEARS TO AGE A similar motivation was for SQL. We will give you a database and a query language that anyone can use: SELECT title FROM Book ..... During the early days of computing the thought prevailed - pretty much like now - that it was a matter of 10 years before computers would be able to talk in natural languages and AI was around the corner. But, you are right about BASIC that was my first language as well, quickly dropped for Turbo Pascal which was the sort of the in-thing for a while.