6 ms·

pretty neat. But what I need the most right now is a way to visualize information. I work on computer fluid dynamics and there I have: - super intricate algori

by wiz21c 3y ago

pretty neat. But what I need the most right now is a way to visualize information. I work on computer fluid dynamics and there I have: - super intricate algorithms with lots of variables running around - lots of huge numpy arrays storing the bulk of the data I find the current debuggers lacking here: - showing numpy arrays in colours, easily changing color scales, etc. is not easy (you can do your own visualization, but then it's code); explore specific parts of the array in detail; link that to other arrays, etc. - build watches panels where you can mix analysis and values extracted from code. I can't count the number of time I freeze the execution, export data to R, then do analysis there, then fix the code. It'd be nice to have that integrated in the IDE. - have watches that record values as execution runs (I guess that's one of the stuff omniscient debugging does) And while I'm at it: - why can't I cut paste formulas, pictures, etc. in my code ?? (no Jupyter, you're ok for a nice paper like presentation but not for code with hundreds of thousands of lines)

heavenlyblue 3y ago
Good software development implies writing code that is easy to debug, including writing your own tools to debug your own niche-specific code!
- wiz21c 3y ago
  Yeah sure but I'd like to sprinkle lots of little codes in many different places, ad hoc. This happens while I'm debugging issues 'cos I have to integrate information from various places. Note also that it's not like trying to figure a clearly visible bug: sometime sI have to wander through literally hundreds of data arrays just to find what's going wrong. Since I do that close to the code (modifying bits here and there to see how it influences things), it'd be nice to have a "data analysis debug context" very close to the code. I'm sure there's a space here where one can provide tools. For example a drop in "record_this(object)" method integrated with the debugger so that I can look at the results, without the need to build a logging structure to do it myself.
  - vaughan 3y ago
    I had an idea to trace all usages of values. So you can see the entire history of how a value came to be. All the places it went. All the transformations too. Also, I think code should be written so that it can be easily visualized. Nothing else! Code is for humans to read, not machines - that's what compilers are for. Code instrumentation is usually for code coverage...but for debugging it can be great too.
    - tracnar 3y ago
      Indeed, on top of stacktraces it would be great to have 'value traces' (with functional programming I suppose those would be the same). Especially in testing, what's important are the inputs with the system under test.
      - db48x 3y ago
        Pernosco has that!
        vaughan 3y ago
        I wish it was open-source.
        db48x 3y ago
        True. Thankfully RR, which Pernosco is built upon, is open source. You can do a very simple form of value tracing with just RR by setting a watchpoint on the memory where the value is stored, and then using the “reverse continue” command to run the program backwards(!). The watchpoint will trigger when the memory changes, when will happen whenever it is “overwritten” by the previous value. This is exactly the point where the value you are tracing was written into memory. This is not as precise as Pernosco’s value traces, but it is still immensely useful. It makes debugging buffer overruns, use after free, stack or heap corruption, etc, etc extremely easy. Since traditional methods of detecting these types of bugs are a lot more work, you can earn an astounding rate as a contractor fixing people's C++ code using RR or Pernosco. While I also wish Pernosco were open source, there is at least reason to be glad that it is a managed service rather than something we run on our laptops. I talked to Roc about self–hosting (which is available) once and he mentioned that a trace of an application like Firefox gets turned into a database taking up dozens or hundreds of gigabytes of disk space. The only reason Pernosco can turn them around in just a minute or two is that each trace uploaded is given to it’s own 36–core instance for the conversion. Even if it were open source, we wouldn’t exactly be running it on our laptops at the beach.
pizza 3y ago
What if Jupyter was ok for code with hundreds of thousands of lines? I suspect it might be.. I think nbdev ameliorates my major gripes with jupyter: it lets you easily maintain a bidirectional jupyter <-> (library as standard python files) mapping, while also giving you the excellent presentational flexibility of Jupyter. Plus a good looking docs site, for free. Since the code is exported as scripts, you can just `pip install -e .` and get all your IDE integration as normal.
- wiz21c 3y ago
  I'll look at it. My major gripe with jupyter is that it's hard to document classes because a class must be inside one code block and you can't mix in comments block... (unless I didn't see some obvious thing)
  - pizza 3y ago
    Could you have special markdown cells that have mixed python code blocks and other inline media that gets converted to stripped python cells? You might be able to write an ipython ‘magic’ for something like that
corysama 3y ago
Charles Simonyi's Intentional Software was doing that back in the days of Windows NT https://www.youtube.com/watch?v=tSnnfUj1XCQ https://www.youtube.com/watch?v=tSnnfUj1XCQ But, they stayed insular and enterprise right up until Microsoft bought them back. We mortals barely got to see or hear anything about it. I heard one podcast interview of a developer there who said they basically represent the code as an s-expression tree with a bunch of metadata on each node.
- vaughan 3y ago
  Whoa this is incredible. The things you find on HN! I've been researching these kind of things for ages...and this is the first time I find out about this. Code as a database.

by wiz21c 3y ago

heavenlyblue 3y ago

Good software development implies writing code that is easy to debug, including writing your own tools to debug your own niche-specific code!

wiz21c 3y ago
Yeah sure but I'd like to sprinkle lots of little codes in many different places, ad hoc. This happens while I'm debugging issues 'cos I have to integrate information from various places. Note also that it's not like trying to figure a clearly visible bug: sometime sI have to wander through literally hundreds of data arrays just to find what's going wrong. Since I do that close to the code (modifying bits here and there to see how it influences things), it'd be nice to have a "data analysis debug context" very close to the code. I'm sure there's a space here where one can provide tools. For example a drop in "record_this(object)" method integrated with the debugger so that I can look at the results, without the need to build a logging structure to do it myself.
- vaughan 3y ago
  I had an idea to trace all usages of values. So you can see the entire history of how a value came to be. All the places it went. All the transformations too. Also, I think code should be written so that it can be easily visualized. Nothing else! Code is for humans to read, not machines - that's what compilers are for. Code instrumentation is usually for code coverage...but for debugging it can be great too.
  - tracnar 3y ago
    Indeed, on top of stacktraces it would be great to have 'value traces' (with functional programming I suppose those would be the same). Especially in testing, what's important are the inputs with the system under test.
    - db48x 3y ago
      Pernosco has that!
      - vaughan 3y ago
        I wish it was open-source.
        db48x 3y ago
        True. Thankfully RR, which Pernosco is built upon, is open source. You can do a very simple form of value tracing with just RR by setting a watchpoint on the memory where the value is stored, and then using the “reverse continue” command to run the program backwards(!). The watchpoint will trigger when the memory changes, when will happen whenever it is “overwritten” by the previous value. This is exactly the point where the value you are tracing was written into memory. This is not as precise as Pernosco’s value traces, but it is still immensely useful. It makes debugging buffer overruns, use after free, stack or heap corruption, etc, etc extremely easy. Since traditional methods of detecting these types of bugs are a lot more work, you can earn an astounding rate as a contractor fixing people's C++ code using RR or Pernosco. While I also wish Pernosco were open source, there is at least reason to be glad that it is a managed service rather than something we run on our laptops. I talked to Roc about self–hosting (which is available) once and he mentioned that a trace of an application like Firefox gets turned into a database taking up dozens or hundreds of gigabytes of disk space. The only reason Pernosco can turn them around in just a minute or two is that each trace uploaded is given to it’s own 36–core instance for the conversion. Even if it were open source, we wouldn’t exactly be running it on our laptops at the beach.

pizza 3y ago

What if Jupyter was ok for code with hundreds of thousands of lines? I suspect it might be.. I think nbdev ameliorates my major gripes with jupyter: it lets you easily maintain a bidirectional jupyter <-> (library as standard python files) mapping, while also giving you the excellent presentational flexibility of Jupyter. Plus a good looking docs site, for free. Since the code is exported as scripts, you can just `pip install -e .` and get all your IDE integration as normal.

wiz21c 3y ago
I'll look at it. My major gripe with jupyter is that it's hard to document classes because a class must be inside one code block and you can't mix in comments block... (unless I didn't see some obvious thing)
- pizza 3y ago
  Could you have special markdown cells that have mixed python code blocks and other inline media that gets converted to stripped python cells? You might be able to write an ipython ‘magic’ for something like that

corysama 3y ago

Charles Simonyi's Intentional Software was doing that back in the days of Windows NT https://www.youtube.com/watch?v=tSnnfUj1XCQ https://www.youtube.com/watch?v=tSnnfUj1XCQ But, they stayed insular and enterprise right up until Microsoft bought them back. We mortals barely got to see or hear anything about it. I heard one podcast interview of a developer there who said they basically represent the code as an s-expression tree with a bunch of metadata on each node.

vaughan 3y ago
Whoa this is incredible. The things you find on HN! I've been researching these kind of things for ages...and this is the first time I find out about this. Code as a database.