19 ms·

Show HN: Don't code your UI, draw it

302 points by karanchahal 8y ago

KaoruAoiShiho 8y ago
What are people's thoughts on when this will actually be usable? UI eng is just another labor intensive automatable step for creatives. Next will be to automate the design part too. Don't draw your UI, generate it from user stories.
- deleted 8y ago
  [deleted]
- DonHopkins 8y ago
  Pff! Yeah. And generating users stories is the PERFECT application of artificial intelligence. /s You could automate the entire software design and development process! Could't be any worse that what we have now, amiright?
  - la6470 8y ago
    Yes if you consider that once upon a time only specialists could write word documents , the software development process is in its infancy. Remember when one had to know macros to write a wordstar doc.
- aaaaaaaaaab 8y ago
  Ah, the recurring pipe dream of business people: software without coding! A few billion dollars must have been flushed down this toilet bowl already; I wonder how much is yet to come...
atulvi 8y ago
Why doodle? What difference does it make than to simply type the name of the widget you want?
- alangpierce 8y ago
  As one example, you might use a tablet and tablet pen to draw up a few alternative sketches for the same UI, each of which might have tens of UI elements. With that input mechanism, it seems a bit tedious to have to tell the computer "this is a text box" for each text box, especially when a human (or smart algorithm) looking at the drawing can see that pretty clearly.
tyingq 8y ago
Haven't used it, but Yotako seems similar: https://www.yotako.io/ https://www.yotako.io/
akx 8y ago
This is not the sort of repo I would present to the public. * There are no instructions on how to actually run the thing. * There is no requirements.txt or similar, so I have no idea which version of dependencies I'd need. * The repository is strewn with unnecessary files (.pyc/.ds_store/.so...), random-looking images with names like "plswork.png", a HTML file from some "starter kit"... * I can't seem to find the React frontend that is mentioned in the readme -- on the other hand, it looks like `server2.py` is looking for them outside the repository (`".././reactExperiments"`).
- karanchahal 8y ago
  Hello ! Thank you for your comments, it is true that this repository is not in ideal shape. I shall work on making this much more user friendly by tomorrow. I had no idea that this would be seen by this many people. The react frontend is currently not in this repo, I shall add that along with documentation soon :) Thank you for your comments and your review.
- DevX101 8y ago
  > This is not a prodution worthy piece of software,it is only meant for demo purposes It's a pretty cool proof of concept ;) Go easy!
- senthil_rajasek 8y ago
  From the readme, "This is not a prodution worthy piece of software,it is only meant for demo purposes"
  - scarface74 8y ago
    It’s not that the code is not production ready. It’s more about first impressions. There are some really fundamental issues with the presentation. The first thing you do when starting a new git repo is create your .gitignore. I’m also not a Pythonista and I’ve only been working with Python for about a year, but including required packages in the requirements.txt is like Python 102.
    - 8y ago
jakelazaroff 8y ago
Codeless UI is so exciting! This reminds me of Ivan Sutherland's Sketchpad from the early 60s https://en.wikipedia.org/wiki/Sketchpad https://en.wikipedia.org/wiki/Sketchpad
- sizzle 8y ago
  Exactly what came to my mind! Along with airbnb's take on it: https://airbnb.design/sketching-interfaces/ https://airbnb.design/sketching-interfaces/
- zeroname 8y ago
  Don't get excited. The thing that this demo shows in minutes could've been typed up in text in seconds. It is not a productive way to do things. There's a reason that these systems never catched on, they are unnecessary. You may say, but "non-programmers" will use it! No, they won't. Designers will use real design tools to create (non-functional) visual designs. Programmers will bring those visual designs to functionality. That procedure works. It'll keep working. These systems are diversions, not improvements. Worthy of investigation, but not practical.
  - swsieber 8y ago
    Airplanes were once worthy of investigation but practical. As were automobiles. Generally, the fact that something works isn't a compelling argument that something else won't succeed. I do aggree that it's pretty high bar in this case though - it's changing the flow, not just improving it. So it'd have to get very polished to be able to compete, which I just don't think it will. Still, maybe someday.
    - zeroname 8y ago
      > I do aggree that it's pretty high bar in this case though - it's changing the flow, not just improving it. My whole point is that this is not an improvement, it's actually a worse way to enter a simple datastructure into the computer. It's even worse than using the already established UI paradigm of programs like Paint. Picking a tool and dragging out a box is faster, because you don't have to learn the visual language of how to draw these widgets. It does look cool, because it makes the computer appear smart, but it's just not a good interface for actual use.
mpolichette 8y ago
If you already have a consistent design language,and those pre-baked pieces, it seems like assembling them visually is the easy part. This would still leave the behavior logic an data flows. I’d be interested to see new things like this though!
deleted 8y ago
[deleted]
platz 8y ago
Why not create a text file that encodes the structure of the desired output instead of having a CNN inferior it
treve 8y ago
7 comments in, and it's all ragging on this. Good job everyone
- scarface74 8y ago
  What’s the purpose of developing in the open if not to get feedback? A pat on the back and a attaboy doesn’t help anyone improve.
  - narenchoudhary 8y ago
    I feel people who are only willing to point out negatives (without any appreciation for good parts) should refrain from giving feedback. Frankly speaking, such feedback isn't very motivating.
    - wild_preference 8y ago
      Nor is it actionable. Exists solely for the "critic" to signal how unimpressed they are, as if seeing the defects in something is some challenging feat. We're in this obnoxious age of confusing useful criticism with any reaction one can come up with on the fly, no matter how superficial. Like it's their destiny to weigh in on something as rapidly as they can, and as if they're doing some critical service for the universe. The comment above exemplifies this when they say "what, we're supposed to pat them on the head and say attaboy?" No, the problem is that you think you need to fire off some undigested response at all. If you have nothing meaningful to say, then just say nothing. It's okay.
      - scarface74 8y ago
        The feedback was actionable - add a requirements.txt to list dependencies - create a .gitignore file.
- yen223 8y ago
  Half the comments here bring to mind the infamous Dropbox comment: https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224 Reinforces my personal rule-of-thumb that any comment HN has about UX can be safely ignored.
nebulous1 8y ago
Interesting idea, but I'm not sure I see any advantage over dragging movable/resizable components from a toolbar, but there are several obvious disadvantages.
- mcbutterbunz 8y ago
  What if you could sketch your UI with a stencil, paper, and pencil. Then take a picture of it and have it translated into native components?
  - jfolkins 8y ago
    Now that is a super interesting idea. Create a flow scheme too so you can process transitions? flow:success f1:s1->f1:s2->f1:s3 flow:error f1:e1->f1:e2 edit: another thought is that this concept could encourage people in your org who struggle with wireframe technology to express their ideas. Generationally and across culture, smartphone use is now accepted. People also know how to draw on pencil and paper. Now all you are asking them is a final DSL to express their thoughts. Lower barrier? edit2: there is also something to be said for having someone step through their wireframe and flow control by taking pictures. It may take the abstract and create something tangible as they can logically piece their work together with actual pieces of paper?
    - mcbutterbunz 8y ago
      Yeah, a flow scheme would be killer. You could easily come up with many different prototypes and get quick feedback on what works or doesn't work.
  - taeric 8y ago
    Odds are high, mine would look worse. I'm not exactly an artist. Unless you mean just sketch using graph paper and translating coordinates. Not sure why I need a camera for that. :(
    - mcbutterbunz 8y ago
alangpierce 8y ago
karanchahal, not sure if this can be renamed retroactively, but this may have been a good "Show HN" candidate: https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html Regardless, it's a cool idea that made me laugh a bit at first (seems almost absurd at first glance) but then got me thinking about possibilities. Good job! It would be great if you could get it in a state where people could really try it out (even in an unpolished state), either locally or ideally with a web-based demo.
- karanchahal 8y ago
  Hello, thank you for your comment. I have updated the code to a somewhat polished state with instructions on how to run and build the project. I hope this is fine. Please tell me if anything else is required of me in the issues section of the repo! Again, thanks :)
johnjohnsmith 8y ago
Aaaaand we've come full circle back to Microsoft Foundation Class builder that was all the rage in the early 1990s.
catacombs 8y ago
Thanks, but I'll stick with Sketch.
GijsjanB 8y ago
I think this is great. I see a future where designers can draw and create an interactive prototype. Anything beyond that is, my (educated) guess, long way off. Anybody hoping this will remove the need for designers or front end devs will be disappointed.
- vogt 8y ago
  >I think this is great. I see a future where designers can draw and create an interactive prototype. We are already there with Framer X and competitors pending launch in the near future. There is a learning curve that most designers are not super comfortable with yet, but I expect that will improve quickly. We also are limited currently to React for Framer X but I think opening it up to other front-end frameworks is on the horizon. Exciting times!
- DonHopkins 8y ago
  I'm not disappointed because I don't think removing the benefits of thought and effort by people who have studied and understand user interface design, interaction, usability, accessibility, Fitts' Law, etc, is such a great idea or good for users. Drawing where to put the widgets (and not using constraints or grids or automatic layout or adaptive rules or responsive design, or user testing and performance measurement and empirical evaluation) isn't the hard or important part of user interface design. Who is supposed to benefit from this? A company who refuses to hire a competent user interface designer and wants to crank something out really quick regardless of quality? Users spend much more time using an interface than you spend designing and implementing it, so optimizing the time and amount of mental effort you have to put into making a user interface isn't worth it if it doesn't result in a better, easier to use interface.
  - GijsjanB 8y ago
    I imagine a WordPress-like site, where you draw widgets and bring it to life with a theme. Maybe nothing too rocket sciency, but I reckon there's a market for it.
rglover 8y ago
This is a fun idea! Congrats on getting it working. It may not seem like much right now, but the fundamental idea behind this sort of stuff is the future, especially for front-end code.
laurentb 8y ago
this is pretty cool! one thing on the image drawing one is that there are "industry-wide" common practices when using a placeholder/drawing one to represent an image as a square/rectangle crossed from each angle (if I explain myself correctly). this dates back to the print design days, but would be a shape that's a lot easier to draw than the pseudo picture that's common in OS UI's. Yotako, which has been linked somewhere in the thread by tyingq (which reminds me of the Palm Pilot days) uses a similar approach as well. It makes it easier to draw and faster too
GuB-42 8y ago
Maybe I am not a UI designer but I much prefer rule based approaches than drawing. Rule based means I tell use instruction like "this button must be on top on that button horizontally centered", "this label must fit that text", "this image must be between this and that", etc... and let the layout engine deal with it. UIs are usually not paintings, window sizes vary, text length changes with localization, decorations change depending on the environment, etc... Approaching a UI like a canvas will certainly yield good results on the designer machine, but will look out of place everywhere else, if it is usable at all. I think it is the basis or what they call "responsive web design".
- Retric 8y ago
  This really depends on how the UI designer does layout. Nothing says you need to allow users to do absolute positioning vs rule based when someone drags stuff around. You can help this along with multiple common views at different resolutions, so users don’t try and force a pixel perfect version.
  - blattimwind 8y ago
    It's funny how web development still doesn't offer the ease and productivity of early 2000s RAD environments. Even this software is actually not simpler, on the contrary: It is far more complex, because you don't know how to create a given widget [assuming you already learned what widgets there are, because the software doesn't tell you]. You have to learn what the software recognizes and how you need to draw it in multiple strokes. Since recognition is ML-based, it is difficult to tweak and a black box to both user and developer ("why doesn't it recognize this...?"). Contrast with 90s form designers with a simple drag-and-drop palette. (They didn't have layouts just yet, that came a bit later). You can immediately see what widgets are offered and to instantiate them you simply drag them from their "reservoir" to the active area. Simplicity itself.
    - pjmlp 8y ago
      Actually there are a couple of companies doing it, but they have commited the capital crime of asking for money, as such they remain a niche product. WebFlow and OutSystems are two examples that come to my mind. I hope that WebComponents will make it easier to adopt such tooling.
steveharman 8y ago
Neat, but wouldn't it just be easier to have a pallet of controls / components that user could drag into a view and cut out the need for interpreting a drawing?
Someone1234 8y ago
Microsoft also offers this as Sketch2Code: https://github.com/Microsoft/ailab/tree/master/Sketch2Code https://github.com/Microsoft/ailab/tree/master/Sketch2Code
stevew20 8y ago
pix2code
zeroname 8y ago
> "Don't code your UI" Do code your UI.
jack9 8y ago
Wireframe Sketcher (my fav out of them all) does this in a more efficient manner.
qwerty456127 8y ago
For my whole life I've been "drawing" real GUIs and writing the logic code, not GUI code then. It started with VisualBasic 1.0 for DOS, then there were different versions of VisualBasic for Windows, then Borland C++ Builder and Delphi (I was using the former), then NetBeans Swing designer and WinForms designer in VisualStudio. And now building a GUI is such a problem that a huge number of almost-useless (I really prefer pen and paper over drawing with a mouse) "mockup tools" has emerged and we have to use artificial intelligence to produce the actual UI code or just code it manually...
- sien 8y ago
  It really is disappointing how modern web GUI environments are harder to use than what was around 20+ years ago. There are still Visual Basic Apps from the 1990s floating around in large corporations and in government. Users who were expert in the process but not expert developers could easily code up tools to their exact specifications. https://en.wikipedia.org/wiki/Visual_Basic#1990s https://en.wikipedia.org/wiki/Visual_Basic#1990s Admittedly the deployment of web applications is much better and they can run on different devices. Hopefully when web assembly takes off easier to use tools will reappear.
  - andyidsinga 8y ago
    just curious - why would web assembly make it easier for those tools to reappear? I used to use hose 90s GUI tools - VB's, VC's, Symantec's and Borland's java gui tools. Although they did work well for fixed UIs (absolute positioning) -- they were rather hard to get non-fixed uis working. IMHO Bootstrap's grid system was a real leap in this regard and (to me) still a pleasure to work with.
    - sien 8y ago
      Perhaps because I live in hope... The main reason WASM and whatnot might make it easier is that instead of depending on thousands of Javascript libraries the way that so many modern web apps do perhaps a more controlled environment might include a better set of standard libraries and controls. Also Javascript has evolved at a lightning pace which is remarkable but also makes any complex tooling for it tricky. 10 years ago jquery was the main thing, today React,Vue,Angular and so many other things are front end tools of choice. But all that may be totally wishful thinking.
andyidsinga 8y ago
its a great concept & prototype - really reminds me of the palm pilot graffiti writing system and alphabet [0]. after just watching the video - seems like adding a plugin system for targeting various UI library (ex bootstrap) would be really cool (I didn't read all of the text - maybe he suggested or already has this..) [0] https://en.wikipedia.org/wiki/Graffiti_(Palm_OS) https://en.wikipedia.org/wiki/Graffiti_(Palm_OS)
mygo 8y ago
I still create my UI’s with a sketchbook and pen / pencil. For me there’s nothing that matches it yet. Even though there are a bunch of digital product design tools nowadays, I still tend to begin with my sketchbook. It’s fast and expressive, and always there and always on, just a single tool / interface (pen to paper), which is a huge advantage when just trying to get concepts down visually. The clincher in this decade though, is the necessity to think responsively while sketching, understanding that there are all sorts of device sizes now. Then either mock it up in a design tool later or straight code it up once I have the general concepts down. I’ve tried all sorts of things (was really disappointed when SubForm shut down, that one was kind of interesting). From concept to product, starting off with paper and pen is still the quickest route for me.
qazqwert 8y ago
it looks so ineffective and useless... idk it's might be good for not programmers who try to learn or... but i really don't see the point i mean how hard it is to learn css and design a website by a draw/... your given -and how hard it is to draw on a Little piece of paper of what you want the site to look alike. i just don't the point
punnerud 8y ago
How was the PyTorch model trained? (Most interested in the training data)
rcarmo 8y ago
Only the other day I was looking at https://sketch2code.azurewebsites.net https://sketch2code.azurewebsites.net, which does something similar atop Azure cognitive APIs (I work on Azure).
omvek 8y ago
Here is another example of this type of approach from a top-tier, peer reviewed software engineering journal: https://arxiv.org/abs/1802.02312 https://arxiv.org/abs/1802.02312
LifeQuestioner 8y ago
woooooa this is so cool!!!
kapauldo 8y ago
That's really cool.
zachguo 8y ago
Too bad nowadays frontend dev is more about state management than layout and boxes. Designers have tons of tools turning mockups to interactable prototype. Non-technical people have drag and drop tools.
johannesbeil 8y ago
Congrats! Can you maybe say a bit about how you trained it?
wonderofworld 8y ago
While i am a fan, we took it up a notch by allowing users to upload their UI and we write the supporting code using A.I. https://crane.ai/ https://crane.ai/
- karanchahal 8y ago
  Do you use some generative AI or is it still quite rule based with AI doing the classification bit ?

302 points by karanchahal 8y ago

What are people's thoughts on when this will actually be usable? UI eng is just another labor intensive automatable step for creatives. Next will be to automate the design part too. Don't draw your UI, generate it from user stories.

deleted 8y ago
[deleted]
DonHopkins 8y ago
Pff! Yeah. And generating users stories is the PERFECT application of artificial intelligence. /s You could automate the entire software design and development process! Could't be any worse that what we have now, amiright?
- la6470 8y ago
  Yes if you consider that once upon a time only specialists could write word documents , the software development process is in its infancy. Remember when one had to know macros to write a wordstar doc.
aaaaaaaaaab 8y ago
Ah, the recurring pipe dream of business people: software without coding! A few billion dollars must have been flushed down this toilet bowl already; I wonder how much is yet to come...

atulvi 8y ago

Why doodle? What difference does it make than to simply type the name of the widget you want?

alangpierce 8y ago
As one example, you might use a tablet and tablet pen to draw up a few alternative sketches for the same UI, each of which might have tens of UI elements. With that input mechanism, it seems a bit tedious to have to tell the computer "this is a text box" for each text box, especially when a human (or smart algorithm) looking at the drawing can see that pretty clearly.

tyingq 8y ago

Haven't used it, but Yotako seems similar: https://www.yotako.io/ https://www.yotako.io/

akx 8y ago

This is not the sort of repo I would present to the public. * There are no instructions on how to actually run the thing. * There is no requirements.txt or similar, so I have no idea which version of dependencies I'd need. * The repository is strewn with unnecessary files (.pyc/.ds_store/.so...), random-looking images with names like "plswork.png", a HTML file from some "starter kit"... * I can't seem to find the React frontend that is mentioned in the readme -- on the other hand, it looks like `server2.py` is looking for them outside the repository (`".././reactExperiments"`).

karanchahal 8y ago
Hello ! Thank you for your comments, it is true that this repository is not in ideal shape. I shall work on making this much more user friendly by tomorrow. I had no idea that this would be seen by this many people. The react frontend is currently not in this repo, I shall add that along with documentation soon :) Thank you for your comments and your review.
DevX101 8y ago
> This is not a prodution worthy piece of software,it is only meant for demo purposes It's a pretty cool proof of concept ;) Go easy!
senthil_rajasek 8y ago
From the readme, "This is not a prodution worthy piece of software,it is only meant for demo purposes"
- scarface74 8y ago
  It’s not that the code is not production ready. It’s more about first impressions. There are some really fundamental issues with the presentation. The first thing you do when starting a new git repo is create your .gitignore. I’m also not a Pythonista and I’ve only been working with Python for about a year, but including required packages in the requirements.txt is like Python 102.
  - 8y ago

jakelazaroff 8y ago

Codeless UI is so exciting! This reminds me of Ivan Sutherland's Sketchpad from the early 60s https://en.wikipedia.org/wiki/Sketchpad https://en.wikipedia.org/wiki/Sketchpad

sizzle 8y ago
Exactly what came to my mind! Along with airbnb's take on it: https://airbnb.design/sketching-interfaces/ https://airbnb.design/sketching-interfaces/
zeroname 8y ago
Don't get excited. The thing that this demo shows in minutes could've been typed up in text in seconds. It is not a productive way to do things. There's a reason that these systems never catched on, they are unnecessary. You may say, but "non-programmers" will use it! No, they won't. Designers will use real design tools to create (non-functional) visual designs. Programmers will bring those visual designs to functionality. That procedure works. It'll keep working. These systems are diversions, not improvements. Worthy of investigation, but not practical.
- swsieber 8y ago
  Airplanes were once worthy of investigation but practical. As were automobiles. Generally, the fact that something works isn't a compelling argument that something else won't succeed. I do aggree that it's pretty high bar in this case though - it's changing the flow, not just improving it. So it'd have to get very polished to be able to compete, which I just don't think it will. Still, maybe someday.
  - zeroname 8y ago
    > I do aggree that it's pretty high bar in this case though - it's changing the flow, not just improving it. My whole point is that this is not an improvement, it's actually a worse way to enter a simple datastructure into the computer. It's even worse than using the already established UI paradigm of programs like Paint. Picking a tool and dragging out a box is faster, because you don't have to learn the visual language of how to draw these widgets. It does look cool, because it makes the computer appear smart, but it's just not a good interface for actual use.

mpolichette 8y ago

If you already have a consistent design language,and those pre-baked pieces, it seems like assembling them visually is the easy part. This would still leave the behavior logic an data flows. I’d be interested to see new things like this though!

deleted 8y ago

[deleted]

platz 8y ago

Why not create a text file that encodes the structure of the desired output instead of having a CNN inferior it

treve 8y ago

7 comments in, and it's all ragging on this. Good job everyone

scarface74 8y ago
What’s the purpose of developing in the open if not to get feedback? A pat on the back and a attaboy doesn’t help anyone improve.
- narenchoudhary 8y ago
  I feel people who are only willing to point out negatives (without any appreciation for good parts) should refrain from giving feedback. Frankly speaking, such feedback isn't very motivating.
  - wild_preference 8y ago
    Nor is it actionable. Exists solely for the "critic" to signal how unimpressed they are, as if seeing the defects in something is some challenging feat. We're in this obnoxious age of confusing useful criticism with any reaction one can come up with on the fly, no matter how superficial. Like it's their destiny to weigh in on something as rapidly as they can, and as if they're doing some critical service for the universe. The comment above exemplifies this when they say "what, we're supposed to pat them on the head and say attaboy?" No, the problem is that you think you need to fire off some undigested response at all. If you have nothing meaningful to say, then just say nothing. It's okay.
    - scarface74 8y ago
      The feedback was actionable - add a requirements.txt to list dependencies - create a .gitignore file.
yen223 8y ago
Half the comments here bring to mind the infamous Dropbox comment: https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224 Reinforces my personal rule-of-thumb that any comment HN has about UX can be safely ignored.

nebulous1 8y ago

Interesting idea, but I'm not sure I see any advantage over dragging movable/resizable components from a toolbar, but there are several obvious disadvantages.

mcbutterbunz 8y ago
What if you could sketch your UI with a stencil, paper, and pencil. Then take a picture of it and have it translated into native components?
- jfolkins 8y ago
  Now that is a super interesting idea. Create a flow scheme too so you can process transitions? flow:success f1:s1->f1:s2->f1:s3 flow:error f1:e1->f1:e2 edit: another thought is that this concept could encourage people in your org who struggle with wireframe technology to express their ideas. Generationally and across culture, smartphone use is now accepted. People also know how to draw on pencil and paper. Now all you are asking them is a final DSL to express their thoughts. Lower barrier? edit2: there is also something to be said for having someone step through their wireframe and flow control by taking pictures. It may take the abstract and create something tangible as they can logically piece their work together with actual pieces of paper?
  - mcbutterbunz 8y ago
    Yeah, a flow scheme would be killer. You could easily come up with many different prototypes and get quick feedback on what works or doesn't work.
- taeric 8y ago
  Odds are high, mine would look worse. I'm not exactly an artist. Unless you mean just sketch using graph paper and translating coordinates. Not sure why I need a camera for that. :(
  - mcbutterbunz 8y ago

alangpierce 8y ago

karanchahal, not sure if this can be renamed retroactively, but this may have been a good "Show HN" candidate: https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html Regardless, it's a cool idea that made me laugh a bit at first (seems almost absurd at first glance) but then got me thinking about possibilities. Good job! It would be great if you could get it in a state where people could really try it out (even in an unpolished state), either locally or ideally with a web-based demo.

karanchahal 8y ago
Hello, thank you for your comment. I have updated the code to a somewhat polished state with instructions on how to run and build the project. I hope this is fine. Please tell me if anything else is required of me in the issues section of the repo! Again, thanks :)

johnjohnsmith 8y ago

Aaaaand we've come full circle back to Microsoft Foundation Class builder that was all the rage in the early 1990s.

catacombs 8y ago

Thanks, but I'll stick with Sketch.

GijsjanB 8y ago

I think this is great. I see a future where designers can draw and create an interactive prototype. Anything beyond that is, my (educated) guess, long way off. Anybody hoping this will remove the need for designers or front end devs will be disappointed.

vogt 8y ago
>I think this is great. I see a future where designers can draw and create an interactive prototype. We are already there with Framer X and competitors pending launch in the near future. There is a learning curve that most designers are not super comfortable with yet, but I expect that will improve quickly. We also are limited currently to React for Framer X but I think opening it up to other front-end frameworks is on the horizon. Exciting times!
DonHopkins 8y ago
I'm not disappointed because I don't think removing the benefits of thought and effort by people who have studied and understand user interface design, interaction, usability, accessibility, Fitts' Law, etc, is such a great idea or good for users. Drawing where to put the widgets (and not using constraints or grids or automatic layout or adaptive rules or responsive design, or user testing and performance measurement and empirical evaluation) isn't the hard or important part of user interface design. Who is supposed to benefit from this? A company who refuses to hire a competent user interface designer and wants to crank something out really quick regardless of quality? Users spend much more time using an interface than you spend designing and implementing it, so optimizing the time and amount of mental effort you have to put into making a user interface isn't worth it if it doesn't result in a better, easier to use interface.
- GijsjanB 8y ago
  I imagine a WordPress-like site, where you draw widgets and bring it to life with a theme. Maybe nothing too rocket sciency, but I reckon there's a market for it.

rglover 8y ago

This is a fun idea! Congrats on getting it working. It may not seem like much right now, but the fundamental idea behind this sort of stuff is the future, especially for front-end code.

laurentb 8y ago

this is pretty cool! one thing on the image drawing one is that there are "industry-wide" common practices when using a placeholder/drawing one to represent an image as a square/rectangle crossed from each angle (if I explain myself correctly). this dates back to the print design days, but would be a shape that's a lot easier to draw than the pseudo picture that's common in OS UI's. Yotako, which has been linked somewhere in the thread by tyingq (which reminds me of the Palm Pilot days) uses a similar approach as well. It makes it easier to draw and faster too

GuB-42 8y ago

Maybe I am not a UI designer but I much prefer rule based approaches than drawing. Rule based means I tell use instruction like "this button must be on top on that button horizontally centered", "this label must fit that text", "this image must be between this and that", etc... and let the layout engine deal with it. UIs are usually not paintings, window sizes vary, text length changes with localization, decorations change depending on the environment, etc... Approaching a UI like a canvas will certainly yield good results on the designer machine, but will look out of place everywhere else, if it is usable at all. I think it is the basis or what they call "responsive web design".

Retric 8y ago
This really depends on how the UI designer does layout. Nothing says you need to allow users to do absolute positioning vs rule based when someone drags stuff around. You can help this along with multiple common views at different resolutions, so users don’t try and force a pixel perfect version.
- blattimwind 8y ago
  It's funny how web development still doesn't offer the ease and productivity of early 2000s RAD environments. Even this software is actually not simpler, on the contrary: It is far more complex, because you don't know how to create a given widget [assuming you already learned what widgets there are, because the software doesn't tell you]. You have to learn what the software recognizes and how you need to draw it in multiple strokes. Since recognition is ML-based, it is difficult to tweak and a black box to both user and developer ("why doesn't it recognize this...?"). Contrast with 90s form designers with a simple drag-and-drop palette. (They didn't have layouts just yet, that came a bit later). You can immediately see what widgets are offered and to instantiate them you simply drag them from their "reservoir" to the active area. Simplicity itself.
  - pjmlp 8y ago
    Actually there are a couple of companies doing it, but they have commited the capital crime of asking for money, as such they remain a niche product. WebFlow and OutSystems are two examples that come to my mind. I hope that WebComponents will make it easier to adopt such tooling.

steveharman 8y ago

Neat, but wouldn't it just be easier to have a pallet of controls / components that user could drag into a view and cut out the need for interpreting a drawing?

Someone1234 8y ago

Microsoft also offers this as Sketch2Code: https://github.com/Microsoft/ailab/tree/master/Sketch2Code https://github.com/Microsoft/ailab/tree/master/Sketch2Code

stevew20 8y ago

pix2code

zeroname 8y ago

> "Don't code your UI" Do code your UI.

jack9 8y ago

Wireframe Sketcher (my fav out of them all) does this in a more efficient manner.

qwerty456127 8y ago

For my whole life I've been "drawing" real GUIs and writing the logic code, not GUI code then. It started with VisualBasic 1.0 for DOS, then there were different versions of VisualBasic for Windows, then Borland C++ Builder and Delphi (I was using the former), then NetBeans Swing designer and WinForms designer in VisualStudio. And now building a GUI is such a problem that a huge number of almost-useless (I really prefer pen and paper over drawing with a mouse) "mockup tools" has emerged and we have to use artificial intelligence to produce the actual UI code or just code it manually...

sien 8y ago
It really is disappointing how modern web GUI environments are harder to use than what was around 20+ years ago. There are still Visual Basic Apps from the 1990s floating around in large corporations and in government. Users who were expert in the process but not expert developers could easily code up tools to their exact specifications. https://en.wikipedia.org/wiki/Visual_Basic#1990s https://en.wikipedia.org/wiki/Visual_Basic#1990s Admittedly the deployment of web applications is much better and they can run on different devices. Hopefully when web assembly takes off easier to use tools will reappear.
- andyidsinga 8y ago
  just curious - why would web assembly make it easier for those tools to reappear? I used to use hose 90s GUI tools - VB's, VC's, Symantec's and Borland's java gui tools. Although they did work well for fixed UIs (absolute positioning) -- they were rather hard to get non-fixed uis working. IMHO Bootstrap's grid system was a real leap in this regard and (to me) still a pleasure to work with.
  - sien 8y ago
    Perhaps because I live in hope... The main reason WASM and whatnot might make it easier is that instead of depending on thousands of Javascript libraries the way that so many modern web apps do perhaps a more controlled environment might include a better set of standard libraries and controls. Also Javascript has evolved at a lightning pace which is remarkable but also makes any complex tooling for it tricky. 10 years ago jquery was the main thing, today React,Vue,Angular and so many other things are front end tools of choice. But all that may be totally wishful thinking.

andyidsinga 8y ago

its a great concept & prototype - really reminds me of the palm pilot graffiti writing system and alphabet [0]. after just watching the video - seems like adding a plugin system for targeting various UI library (ex bootstrap) would be really cool (I didn't read all of the text - maybe he suggested or already has this..) [0] https://en.wikipedia.org/wiki/Graffiti_(Palm_OS) https://en.wikipedia.org/wiki/Graffiti_(Palm_OS)

mygo 8y ago

I still create my UI’s with a sketchbook and pen / pencil. For me there’s nothing that matches it yet. Even though there are a bunch of digital product design tools nowadays, I still tend to begin with my sketchbook. It’s fast and expressive, and always there and always on, just a single tool / interface (pen to paper), which is a huge advantage when just trying to get concepts down visually. The clincher in this decade though, is the necessity to think responsively while sketching, understanding that there are all sorts of device sizes now. Then either mock it up in a design tool later or straight code it up once I have the general concepts down. I’ve tried all sorts of things (was really disappointed when SubForm shut down, that one was kind of interesting). From concept to product, starting off with paper and pen is still the quickest route for me.

qazqwert 8y ago

it looks so ineffective and useless... idk it's might be good for not programmers who try to learn or... but i really don't see the point i mean how hard it is to learn css and design a website by a draw/... your given -and how hard it is to draw on a Little piece of paper of what you want the site to look alike. i just don't the point

punnerud 8y ago

How was the PyTorch model trained? (Most interested in the training data)

rcarmo 8y ago

Only the other day I was looking at https://sketch2code.azurewebsites.net https://sketch2code.azurewebsites.net, which does something similar atop Azure cognitive APIs (I work on Azure).

omvek 8y ago

Here is another example of this type of approach from a top-tier, peer reviewed software engineering journal: https://arxiv.org/abs/1802.02312 https://arxiv.org/abs/1802.02312

LifeQuestioner 8y ago

woooooa this is so cool!!!

kapauldo 8y ago

That's really cool.

zachguo 8y ago

Too bad nowadays frontend dev is more about state management than layout and boxes. Designers have tons of tools turning mockups to interactable prototype. Non-technical people have drag and drop tools.

johannesbeil 8y ago

Congrats! Can you maybe say a bit about how you trained it?

wonderofworld 8y ago

While i am a fan, we took it up a notch by allowing users to upload their UI and we write the supporting code using A.I. https://crane.ai/ https://crane.ai/

karanchahal 8y ago
Do you use some generative AI or is it still quite rule based with AI doing the classification bit ?