4 ms·

Is there a standard tool that does this? Especially for, shall we say, highly interconnected webpages? I need it for Overcoming Bias.

by Eliezer 18y ago

alexwg 18y ago
Yup, Graphviz: http://www.graphviz.org http://www.graphviz.org
- jyothi 18y ago
  Graphviz only renders a graph which is represented as per dot format. There are other programs: dot, neato etc which can produce jpeg,png,gif... formats through command line tools. I guess what he meant was tools which can do this for a particular site. I can't think of any but as a previous poster said - a crawl of desired depth for all local pages, then feed the access log to "statviz" to generate a neat dot file along with the links.
jjs 18y ago
Yes, it's called python. Write a crawler that examines all local links, and builds a graph (it needn't be anything fancy; a collection of nodes and edges is enough, but you'll have to detect cycles when building the graph), and then print it out in graphviz .dot language. Then you can feed it into neato [either through a command-line pipe, or construct one using Python's subprocess module], et voilà, instant pagemap! :) You can also use some of the rendering modes and scrape position data for the rendered nodes, if you want to generate a clickable imagemap.
- jey 18y ago
  Here's a first cut implementation in ruby. Requires the "hpricot" gem. Outputs a dot file to stdout. http://gist.github.com/35841 http://gist.github.com/35841 Output from the above is at: http://jey.kottalam.net/tmp/obgraph.out http://jey.kottalam.net/tmp/obgraph.out I tried rendering this with "dot -Tcmapx -oob.map -Tgif -oob.gif" but dot segfaulted after 70 minutes. The code as posted at gist only outputs nodes for articles written by Eliezer in an attempt to make the dot file a manageable size. Tips/fixes appreciated.
  - jjs 18y ago
    Not sure if it will work, but try: 1) Rendering an undirected graph via graphviz's neato tool. 2) Rendering to svg, which might let graphviz not have to keep as much rendering information in memory at any one time. 3) Asking about it on the grapviz-interest mailing list. https://mailman.research.att.com/mailman/listinfo/graphviz-interest https://mailman.research.att.com/mailman/listinfo/graphviz-i...
  - Eliezer 18y ago
    Yeah, it's that thing with the segfault that I worried about. "Highly interconnected." Thanks for the graph, which I may be able to use for my own purposes even if I can't show it. There's got to be standard graphing tools for things that are highly connected...
    - khafra 18y ago
      When Daniel J. Bernstein made his DAGs of standard crypto algorithms (http://cr.yp.to/cipherdag/cipherdag-20070630.pdf http://cr.yp.to/cipherdag/cipherdag-20070630.pdf) he mentioned that they'd crashed every standard drawing tool he tried. He makes a reference to "...my own drawing tools, which are much more careful in their use of memory." Unfortunately, I'm not sure where to obtain DJB's drawing tools.
zack 18y ago
I am home for winter break and would be willing to write this for you in the name of science. Plus, it seems that you know Marcello Herreshoff, with whom I took Computer Science in high school. Email me the requirements for your tool to <zackster>:at:<gmail.com> please.
- deleted 18y ago
  [deleted]
deleted 18y ago
[deleted]

Is there a standard tool that does this? Especially for, shall we say, highly interconnected webpages? I need it for Overcoming Bias.

by Eliezer 18y ago

Is there a standard tool that does this? Especially for, shall we say, highly interconnected webpages? I need it for Overcoming Bias.

alexwg 18y ago

Yup, Graphviz: http://www.graphviz.org http://www.graphviz.org

jyothi 18y ago
Graphviz only renders a graph which is represented as per dot format. There are other programs: dot, neato etc which can produce jpeg,png,gif... formats through command line tools. I guess what he meant was tools which can do this for a particular site. I can't think of any but as a previous poster said - a crawl of desired depth for all local pages, then feed the access log to "statviz" to generate a neat dot file along with the links.

jjs 18y ago

Yes, it's called python. Write a crawler that examines all local links, and builds a graph (it needn't be anything fancy; a collection of nodes and edges is enough, but you'll have to detect cycles when building the graph), and then print it out in graphviz .dot language. Then you can feed it into neato [either through a command-line pipe, or construct one using Python's subprocess module], et voilà, instant pagemap! :) You can also use some of the rendering modes and scrape position data for the rendered nodes, if you want to generate a clickable imagemap.

jey 18y ago
Here's a first cut implementation in ruby. Requires the "hpricot" gem. Outputs a dot file to stdout. http://gist.github.com/35841 http://gist.github.com/35841 Output from the above is at: http://jey.kottalam.net/tmp/obgraph.out http://jey.kottalam.net/tmp/obgraph.out I tried rendering this with "dot -Tcmapx -oob.map -Tgif -oob.gif" but dot segfaulted after 70 minutes. The code as posted at gist only outputs nodes for articles written by Eliezer in an attempt to make the dot file a manageable size. Tips/fixes appreciated.
- jjs 18y ago
  Not sure if it will work, but try: 1) Rendering an undirected graph via graphviz's neato tool. 2) Rendering to svg, which might let graphviz not have to keep as much rendering information in memory at any one time. 3) Asking about it on the grapviz-interest mailing list. https://mailman.research.att.com/mailman/listinfo/graphviz-interest https://mailman.research.att.com/mailman/listinfo/graphviz-i...
- Eliezer 18y ago
  Yeah, it's that thing with the segfault that I worried about. "Highly interconnected." Thanks for the graph, which I may be able to use for my own purposes even if I can't show it. There's got to be standard graphing tools for things that are highly connected...
  - khafra 18y ago
    When Daniel J. Bernstein made his DAGs of standard crypto algorithms (http://cr.yp.to/cipherdag/cipherdag-20070630.pdf http://cr.yp.to/cipherdag/cipherdag-20070630.pdf) he mentioned that they'd crashed every standard drawing tool he tried. He makes a reference to "...my own drawing tools, which are much more careful in their use of memory." Unfortunately, I'm not sure where to obtain DJB's drawing tools.

zack 18y ago

I am home for winter break and would be willing to write this for you in the name of science. Plus, it seems that you know Marcello Herreshoff, with whom I took Computer Science in high school. Email me the requirements for your tool to <zackster>:at:<gmail.com> please.

deleted 18y ago
[deleted]

deleted 18y ago

[deleted]