4 ms·

"The point is I wish we could just ask the computer to sort things out for us, show different views, etc. this would require a software system able to work with

by jerf 3y ago

"The point is I wish we could just ask the computer to sort things out for us, show different views, etc. this would require a software system able to work with more granular code units, maybe something like [1]." The problem I've seen with this idea is that in order for it to work, you have to be willing to add more metadata to your program, because programs themselves don't have the necessary data in them. You can make it mandatory to your language, but than just raises the friction for people trying to use that language. But, if you're willing to add the metadata to a hypothetical new system for slicing and dicing code, why aren't you using the existing mechanisms in your system for organizing and documenting things? If you are, then you make the delta between "it may not be perfect but here's some documented and organized code" and the perfect vision small enough to not be worth a lot of effort to chase. If you aren't, then you aren't going to give the perfect all-singing slicer and dicer what it needs either. Replace "you" with "coworker(s)" for the same dilemma, only sadder. And your generic strawman coworkers are going to be very upset when you try to get them to use this language... "it's so hard, it's always complaining about perfectly good code, I'm missing my deadlines because it's making me document things I'll never use, I'm just putting 'a' as the documentation for these things anyhow, oh look here's our boss ordering us to go back to our previous language because we're all spending too much time documenting rather than building new features". Some things can't really be fixed by process.

emmanueloga_ 3y ago
I agree on the human friction part... I like this idea and yet I struggle to implement it in my own projects. I think the reason is that there's no good language yet to write that metadata. Even advanced type systems mainly focus on guiding how components connect, but they don't explain _why_ a program behaves a certain way or how the data is supposed to flow. Documentation is often unreliable as it becomes stale easily. As developers we spend and order of magnitude more time building a graph of data flow in our heads than actually writing code. Could we reify that graph? IMO that crucial metadata would answer the questions we would ask the Senior developer in our team when we need to work in the codebase: * When dealing with feature X, what are the involved software components Y and Z, how do we create instances of those? * What are the actual features the system implements? Is this the correct behavior? Is this what the system was intended to do? * If I point to a specific file and line in the project, what features does that code support? Why is that file there? Expressing the answers in a machine-readable format, in a language agnostic way, could look something like: System { Dependencies - Database depends on Logging - UI depends on Database - ... Feature A - Subtopic 1 - Subtopic 2 Feature B - ... } Feature A { requires database queries X if (database has data) produces Y, Z else doTheThing(X, "blah") } This is a broad idea, and the devil is in the details... This would not replace type systems or traditional languages. I'm thinking this spec-lang would have the minimum amount of flowchart-like constructs, and everything even slightly more complicated would rely on an interface. The system would implement all boilerplate and layout the project entirely. It would be amazing if we could also express state machines or even statecharts. Teams could use off-the shelf generators, or create a custom one, to produce the interfaces and boilerplate implementation. Since the system has a lot of information we could query it to answer questions like "give me all code of feature A", or even ask it to create symlinks so we can focus our attention in a single folder of files. Also different generators could create the structure that is more appropriate for different programming languages. It could be a good tool to assist rewrites, if needed for whatever reason. Tests/specs could be generated automatically, to a certain extent. I'm sure some people here will have PTSD about 4th gen languages and BPEL/UML nightmares :-). But even though BPEL is horrible I think the core of the idea is good and worth revisiting.
solarkraft 3y ago
> order for it to work, you have to be willing to add more metadata to your program, because programs themselves don't have the necessary data in them I'm kind of stumped by this assertion. What data is missing? If it is required or necessary, doesn't it make it just data, as in an integral part of the code?
trealira 3y ago
> The problem I've seen with this idea is that in order for it to work, you have to be willing to add more metadata to your program, because programs themselves don't have the necessary data in them. You can make it mandatory to your language, but than just raises the friction for people trying to use that language. I'm confused by what you mean. Can you be more specific as to what sort of metadata that has to be embedded in the code would be needed? I've never used it, but watching this Smalltalk-80 IDE demonstration seems like it could be a workable idea. https://youtu.be/JLPiMl8XUKU?si=Hly_8m4GiaDSQmpn&t=269 https://youtu.be/JLPiMl8XUKU?si=Hly_8m4GiaDSQmpn&t=269 Edit: A more modern example using Pharo. You can see in the video that he's not accessing files, rather everything is sorted into packages and classes. https://youtu.be/HOuZyOKa91o?si=qibMeMOkKy7ao5Hb https://youtu.be/HOuZyOKa91o?si=qibMeMOkKy7ao5Hb
- jerf 3y ago
  I mean that if you are asking for something we don't currently have, it pretty much by construction has to be something that involves more data and metadata. IDEs already do a lot of this sort of thing with all languages. Slicing and dicing by functions and such is not that hard and I've seen any number of variations on it, but coherent views of any sizable code base do not arise from any amount of that. If you want a coherent view of the code base, you're going to need to provide more. One such example would be literate programming, something that a lot of people look over and pine for the end results, but aren't actually willing to put the effort in that is irreducibly necessary to obtain such results.

by jerf 3y ago

emmanueloga_ 3y ago

I agree on the human friction part... I like this idea and yet I struggle to implement it in my own projects. I think the reason is that there's no good language yet to write that metadata. Even advanced type systems mainly focus on guiding how components connect, but they don't explain _why_ a program behaves a certain way or how the data is supposed to flow. Documentation is often unreliable as it becomes stale easily. As developers we spend and order of magnitude more time building a graph of data flow in our heads than actually writing code. Could we reify that graph? IMO that crucial metadata would answer the questions we would ask the Senior developer in our team when we need to work in the codebase: * When dealing with feature X, what are the involved software components Y and Z, how do we create instances of those? * What are the actual features the system implements? Is this the correct behavior? Is this what the system was intended to do? * If I point to a specific file and line in the project, what features does that code support? Why is that file there? Expressing the answers in a machine-readable format, in a language agnostic way, could look something like: System { Dependencies - Database depends on Logging - UI depends on Database - ... Feature A - Subtopic 1 - Subtopic 2 Feature B - ... } Feature A { requires database queries X if (database has data) produces Y, Z else doTheThing(X, "blah") } This is a broad idea, and the devil is in the details... This would not replace type systems or traditional languages. I'm thinking this spec-lang would have the minimum amount of flowchart-like constructs, and everything even slightly more complicated would rely on an interface. The system would implement all boilerplate and layout the project entirely. It would be amazing if we could also express state machines or even statecharts. Teams could use off-the shelf generators, or create a custom one, to produce the interfaces and boilerplate implementation. Since the system has a lot of information we could query it to answer questions like "give me all code of feature A", or even ask it to create symlinks so we can focus our attention in a single folder of files. Also different generators could create the structure that is more appropriate for different programming languages. It could be a good tool to assist rewrites, if needed for whatever reason. Tests/specs could be generated automatically, to a certain extent. I'm sure some people here will have PTSD about 4th gen languages and BPEL/UML nightmares :-). But even though BPEL is horrible I think the core of the idea is good and worth revisiting.

solarkraft 3y ago

> order for it to work, you have to be willing to add more metadata to your program, because programs themselves don't have the necessary data in them I'm kind of stumped by this assertion. What data is missing? If it is required or necessary, doesn't it make it just data, as in an integral part of the code?

trealira 3y ago

> The problem I've seen with this idea is that in order for it to work, you have to be willing to add more metadata to your program, because programs themselves don't have the necessary data in them. You can make it mandatory to your language, but than just raises the friction for people trying to use that language. I'm confused by what you mean. Can you be more specific as to what sort of metadata that has to be embedded in the code would be needed? I've never used it, but watching this Smalltalk-80 IDE demonstration seems like it could be a workable idea. https://youtu.be/JLPiMl8XUKU?si=Hly_8m4GiaDSQmpn&t=269 https://youtu.be/JLPiMl8XUKU?si=Hly_8m4GiaDSQmpn&t=269 Edit: A more modern example using Pharo. You can see in the video that he's not accessing files, rather everything is sorted into packages and classes. https://youtu.be/HOuZyOKa91o?si=qibMeMOkKy7ao5Hb https://youtu.be/HOuZyOKa91o?si=qibMeMOkKy7ao5Hb

jerf 3y ago
I mean that if you are asking for something we don't currently have, it pretty much by construction has to be something that involves more data and metadata. IDEs already do a lot of this sort of thing with all languages. Slicing and dicing by functions and such is not that hard and I've seen any number of variations on it, but coherent views of any sizable code base do not arise from any amount of that. If you want a coherent view of the code base, you're going to need to provide more. One such example would be literate programming, something that a lot of people look over and pine for the end results, but aren't actually willing to put the effort in that is irreducibly necessary to obtain such results.