7 ms·

Translate screenshot into HTML using neural networks

71 points by adamnemecek 9y ago

wrangler99 9y ago
I’d love to see a proof of concept of a network that generates the bootstrap front-end and connects a backend. Bonus, if the model applies attention to visualize the focus on the front-end markup as it builds the back-end.
- sova 9y ago
  Neural networks that accomplish intense categorization techniques rely on multiple layers at varying precisions. It's fully possible to seed a neural network with good names for elements (like a CSS file) and layouts. It's a neat idea, but dynamic pages would require more finesse than simply being able to generate a static sheet, if you wanted to use instantaneous transitions there'd have to be some sort of state-caching for transitions.
partycoder 9y ago
Translating requirements into code is what a programmer does. Automating some parts of the software development lifecycle pipeline will probably not fully replace a developer, but decrease the workload enough to require fewer developers for the same tasks.
darepublic 9y ago
People are commenting on whether this replaces a developer or not, but I don't see the readme for this making any such claim. I believe it is just a static website (?) in HTML which is A LOT less ambitious than trying to have a machine create a full stack web application.
- peterchon 9y ago
  I agree. Also, most development happens with ambiguous requirements and wants that can never be translated/interpreted correctly by machine (yet?) Also, most work is some form of addition to an existing code base that simply cannot be re-written every time.
  - darepublic 9y ago
    That's why we need to have ML for project management first
orliesaurus 9y ago
First of all, this is a repost from literally 4 days ago [1] The article title also, is a bit clickbait - as one of the comments in the original submission mentions: A lot of people in this thread seem to think that this is a neural network that takes an image and produces HTML, when that's not the case here at all. [1] https://news.ycombinator.com/item?id=16115353 https://news.ycombinator.com/item?id=16115353
hokus 9y ago
At first the challenge should be to have any generated tag soup that matches the input but I sure hope it will eventually crunch on to have the absolute fewest number of html tags and style rules. Also, making a design from a screenshot is rather easy as you don't have to worry about infinite view-port sizes. A series of screenies would fit the goal better. Nevertheless: Fun stuff! looking forwards to the future of it.
sova 9y ago
So my ideal IDE for webpages would be a canvas I can draw on and have elements tied to backend functionality automatically. This is definitely a step in the right direction. A little bit of react here and there and this could be a feasible beginning point for project generation. Are there enough classy and elegant dynamic web pages that one could train a neural network on (GitHub repos) and end up with something similar?
alex19-84 9y ago
Does any one used it for random screenshot, or it only for boostrap things?

71 points by adamnemecek 9y ago

I’d love to see a proof of concept of a network that generates the bootstrap front-end and connects a backend. Bonus, if the model applies attention to visualize the focus on the front-end markup as it builds the back-end.

sova 9y ago
Neural networks that accomplish intense categorization techniques rely on multiple layers at varying precisions. It's fully possible to seed a neural network with good names for elements (like a CSS file) and layouts. It's a neat idea, but dynamic pages would require more finesse than simply being able to generate a static sheet, if you wanted to use instantaneous transitions there'd have to be some sort of state-caching for transitions.

partycoder 9y ago

Translating requirements into code is what a programmer does. Automating some parts of the software development lifecycle pipeline will probably not fully replace a developer, but decrease the workload enough to require fewer developers for the same tasks.

darepublic 9y ago

People are commenting on whether this replaces a developer or not, but I don't see the readme for this making any such claim. I believe it is just a static website (?) in HTML which is A LOT less ambitious than trying to have a machine create a full stack web application.

peterchon 9y ago
I agree. Also, most development happens with ambiguous requirements and wants that can never be translated/interpreted correctly by machine (yet?) Also, most work is some form of addition to an existing code base that simply cannot be re-written every time.
- darepublic 9y ago
  That's why we need to have ML for project management first

orliesaurus 9y ago

First of all, this is a repost from literally 4 days ago [1] The article title also, is a bit clickbait - as one of the comments in the original submission mentions: A lot of people in this thread seem to think that this is a neural network that takes an image and produces HTML, when that's not the case here at all. [1] https://news.ycombinator.com/item?id=16115353 https://news.ycombinator.com/item?id=16115353

hokus 9y ago

At first the challenge should be to have any generated tag soup that matches the input but I sure hope it will eventually crunch on to have the absolute fewest number of html tags and style rules. Also, making a design from a screenshot is rather easy as you don't have to worry about infinite view-port sizes. A series of screenies would fit the goal better. Nevertheless: Fun stuff! looking forwards to the future of it.

sova 9y ago

So my ideal IDE for webpages would be a canvas I can draw on and have elements tied to backend functionality automatically. This is definitely a step in the right direction. A little bit of react here and there and this could be a feasible beginning point for project generation. Are there enough classy and elegant dynamic web pages that one could train a neural network on (GitHub repos) and end up with something similar?

alex19-84 9y ago

Does any one used it for random screenshot, or it only for boostrap things?