4 ms·

So you can load this from a schematic format, parse it, get all the element symbols loaded in the right places, grab and render the fonts and place them appropr

So you can load this from a schematic format, parse it, get all the element symbols loaded in the right places, grab and render the fonts and place them appropriately, and construct a well-formed output like https://www.nerdkits.com/media/circuitlab/20120620-democircuit.pdf https://www.nerdkits.com/media/circuitlab/20120620-democircu... , all in 1/30th of a core-second? With a bunch of disk seeks in there too? Of course, we could devote engineering effort to bring that 10 seconds down by an order of magnitude, writing a fully custom PDF (and EPS, SVG, PNG, ...) rendering engine, but part of the beauty of elastic cloud computing is that we can make better use of existing components -- even if it's less efficient from a CPU time perspective -- because the CPU time is cheap and programmer time is expensive (and better used elsewhere). There's a balance, of course, but making well-informed tradeoffs is what engineering is all about.

revelation 14y ago
Uh.. yes? Obviously you should have all the elements, symbols, fonts already cached in memory. Its not like you have thousands of these, anyway. Layouting was already done by the user, so I don't see what part of rendering is incurring the heavy costs here. As it stands, you have now spent lots and lots of programmer time building a batch system from scratch.
- compumike 14y ago
  Premature optimization can be an expensive mistake to make... It may feel "wrong" to use 10X or even 100X CPU cycles when you "know" that you could do it in X, but if the 100X cycle solution still fits the business requirements of the overall system, and takes a lot less programmer time, which path would you choose?
  - revelation 14y ago
    If, as a progammer, I see something like this render taking 10 seconds, I will shake my head in confusion and think "what the hell is taking up all this time". Stuff like loading the same font a hundred times, allocating memory for all the same symbols - identifying and getting rid of that is not premature optimization, its plain fixing the code. Of course you might have done all of that - no way to know unless you tell us more about the actual pipeline.
  - bazzargh 14y ago
    But this sounds more like 'no optimisation', and a lot of complexity to maintain. I'm scratching my head over the toolchain chosen. The output is PNG, EPS, PDF, SVG, but you're using Inkscape and Ghostscript? Something seems off there. Cairo has all of those backends already, it's fast and you can call it from python (via pycairo). What's more, Inkscape already uses it as a backend. There's even another schematic editor (gEDA) that uses it, AFAIK. But, I presume you didn't use cairo for a reason - or you are using it, but need those other tools to post-process its output somehow. I'd be interested to know why?
  - judofyr 14y ago
    Premature optimization is not about "optimizing before you need it", it's about optimizing before you've measured it. In this case, you have already measured it; you know the cost. There's nothing premature of optimizing at this moment. I would consider spending a day profiling the current setup to figure out exactly how much you could optimize. > and takes a lot less programmer time I don't see how this is true. You've already spent time setting up a distributed system from scratch (pretty much at least), and you also need to maintain it. Distributed systems are complicated stuff; lots of stuff that can go wrong. If the optimization made it possible to use a single machine, it would be much easier to grasp and debug.
colanderman 14y ago
writing a fully custom PDF (and EPS, SVG, PNG, ...) rendering engine You mean like this? http://cairographics.org/ http://cairographics.org/ (Which, BTW, has Python bindings…) because the CPU time is cheap and programmer time is expensive (and better used elsewhere) Yes, but programmer time to implement rendering using a fast library such as Cairo is O(1), while the CPU time is O(n) where n is the amount of business you do. (Disclaimer: I have written a schematic renderer before using Cairo. It took me maybe a couple days.)

revelation 14y ago

Uh.. yes? Obviously you should have all the elements, symbols, fonts already cached in memory. Its not like you have thousands of these, anyway. Layouting was already done by the user, so I don't see what part of rendering is incurring the heavy costs here. As it stands, you have now spent lots and lots of programmer time building a batch system from scratch.

compumike 14y ago
Premature optimization can be an expensive mistake to make... It may feel "wrong" to use 10X or even 100X CPU cycles when you "know" that you could do it in X, but if the 100X cycle solution still fits the business requirements of the overall system, and takes a lot less programmer time, which path would you choose?
- revelation 14y ago
  If, as a progammer, I see something like this render taking 10 seconds, I will shake my head in confusion and think "what the hell is taking up all this time". Stuff like loading the same font a hundred times, allocating memory for all the same symbols - identifying and getting rid of that is not premature optimization, its plain fixing the code. Of course you might have done all of that - no way to know unless you tell us more about the actual pipeline.
- bazzargh 14y ago
  But this sounds more like 'no optimisation', and a lot of complexity to maintain. I'm scratching my head over the toolchain chosen. The output is PNG, EPS, PDF, SVG, but you're using Inkscape and Ghostscript? Something seems off there. Cairo has all of those backends already, it's fast and you can call it from python (via pycairo). What's more, Inkscape already uses it as a backend. There's even another schematic editor (gEDA) that uses it, AFAIK. But, I presume you didn't use cairo for a reason - or you are using it, but need those other tools to post-process its output somehow. I'd be interested to know why?
- judofyr 14y ago
  Premature optimization is not about "optimizing before you need it", it's about optimizing before you've measured it. In this case, you have already measured it; you know the cost. There's nothing premature of optimizing at this moment. I would consider spending a day profiling the current setup to figure out exactly how much you could optimize. > and takes a lot less programmer time I don't see how this is true. You've already spent time setting up a distributed system from scratch (pretty much at least), and you also need to maintain it. Distributed systems are complicated stuff; lots of stuff that can go wrong. If the optimization made it possible to use a single machine, it would be much easier to grasp and debug.

colanderman 14y ago

writing a fully custom PDF (and EPS, SVG, PNG, ...) rendering engine You mean like this? http://cairographics.org/ http://cairographics.org/ (Which, BTW, has Python bindings…) because the CPU time is cheap and programmer time is expensive (and better used elsewhere) Yes, but programmer time to implement rendering using a fast library such as Cairo is O(1), while the CPU time is O(n) where n is the amount of business you do. (Disclaimer: I have written a schematic renderer before using Cairo. It took me maybe a couple days.)