5 ms·

Parent post is getting down voted to oblivion but it seems a reasonable belief for someone who is not highly engaged with AI. I have only the vaguest understand

Parent post is getting down voted to oblivion but it seems a reasonable belief for someone who is not highly engaged with AI. I have only the vaguest understanding of how it works (and it's probably wrong) and to my layman mind it also seems like a totally fair assumption, based on experience as a user and the constant flood of news. Please explain why the suggestion that a future AI / sufficiently advanced LLM could refactor a complex codebase is so preposterous.

not_the_fda 3y ago
Because refactoring requires understanding, which LLMs completely lack.
- ctoth 3y ago
  > Because refactoring requires understanding, which LLMs completely lack. Interesting. Here's a sample question I just asked the AI: We are working on a React mapping application. I am going to give you two components from this application. Our goal is to extract functionality to deduplicate code We can make a single base component, extract code to hooks, or any other strategy which you think is appropriate Here is travel.tsx: <snip> Here is heatmap.tsx: <snip> And here is the response: To deduplicate code and improve maintainability between the `TravelMap` and `Heatmap` components in your React mapping application, we can identify common functionalities and extract them into reusable hooks or components. Here's a strategy considering your code: 1. *Extract Common Hooks*: Both components use similar hooks for managing markers, popups, and map references. We can create custom hooks that encapsulate these functionalities. 2. *Create Shared Utility Functions*: Functions like `initialViewState`, which are identical in both components, can be moved to a separate utility file and imported where needed. 3. *Refactor Map Layers and Sources*: The way map layers and sources are handled is quite similar in both components. A custom hook or component to manage map layers and sources would be beneficial. 4. *Generalize Popup Handling*: The popup handling in both components can be abstracted into a custom hook or a higher-order component. 5. *Combine Common Styling*: Shared styles, such as `MAP_CONTAINER_STYLE`, can be moved to a common CSS module or a styled-component that can be reused. 6. *Refactor Event Handlers*: Functions like `onMapClick`, `onMouseMove`, and `onMouseLeave` have similar structures. These can be part of a custom hook or a utility function with configurable parameters. 7. *Legend and Live Region Components*: These components are used in a similar way in both `TravelMap` and `Heatmap`. Ensure they are designed to be reusable with different props. 8. *Refactor Map Initialization*: The way the map is initialized and configured is quite similar. This can be abstracted into a custom component or hook. ### Example Implementations: #### 1. Custom Hook for Map Markers: ```typescript // useMapMarkers.ts import { useState, useCallback } from 'react'; export const useMapMarkers = (initialLat, initialLng, world) => { // Similar marker logic from your components // ... return { marker, onMarkerDrag, onMarkerDragEnd, updateMarkerPosition, // other marker related states and functions }; }; ``` #### 2. Shared Utility Function: ```typescript // utils/mapUtils.ts export const initialViewState = (map, mapType) => { // Implementation }; ``` #### 3. Custom Hook for Map Layers and Sources: ```typescript // useMapLayers.ts import { useState, useEffect } from 'react'; export const useMapLayers = (props) => { // Logic to handle layers and sources // ... return { dataLayer, patternLayer, // other layer related states and functions }; }; ``` #### 4. Generalized Popup Handling: ```typescript // useMapPopup.ts import { useState } from 'react'; export const useMapPopup = () => { // Popup logic from your components // ... return { popupState, openPopup, closePopup, setPopupData, // other popup related states and functions }; }; ``` ### Integration: Once these hooks and utilities are created, integrate them into your `TravelMap` and `Heatmap` components. Replace the duplicated logic with calls to these shared resources. This will significantly reduce code duplication and make future maintenance easier. ### Testing and Refinement: After refactoring, thoroughly test both components to ensure that all functionalities work as expected. Watch out for any edge cases or specific behaviors that might be affected by the refactoring. For those suggestions, I might use five out of eight of them, and probably do one or two things differently. But you cannot, with a straight face, say the model did not understand. It clearly did. It suggested reasonable refactors. If being able to refactor means understanding, I guess we have understanding! I could continue with this conversation, ask it to produce the full code for the hooks (I have in my custom prompt to provide outlines) and once the hooks are complete, ask it to rewrite the components using the shared code. Have you ever used one of these models?
  - never_inline 3y ago
    Eliminating duplication and cleaning code is a different type of refactoring than supporting concurrency, which is much much harder. Cleaning up code also follows some well established patterns, performance work is much less pattern-y. Codebases like FFMPEG are one of the kind. I bet you need 10 or 100 times more understanding than the react thing you mentioned above. One day maybe AI can do it, but it probably won't be LLM. It would be something which can understand symbols and math.
    - ctoth 3y ago
      Ah, we're having some classic goalpost moving! > Because refactoring requires understanding, which LLMs completely lack. <demonstration that an LLM can refactor code> > Cleaning up code also follows some well established patterns, performance work is much less pattern-y. Just as writing shitty react apps follow patterns, low-level performance and concurrency work also follow patterns. See [0] for a sample. > I bet you need 10 or 100 times more understanding Okay, so a 10 or 100 times larger model? Sounds like something we'll have next year, and certainly within a decade. > One day maybe AI can do it, but it probably won't be LLM. It would be something which can understand symbols and math. You do understand that the reason some of the earlier GPTs had trouble with symbols and math was the tokenization scheme, completely separate from how they work in general, right? [0]: C++ Concurrency in Action: Practical Multithreading 1st Edition https://www.amazon.com/C-Concurrency-Action-Practical-Multithreading/dp/1933988770 https://www.amazon.com/C-Concurrency-Action-Practical-Multit...
      - kcbanner 3y ago
        > Because refactoring requires understanding, which LLMs completely lack. It's obvious from context here that the refactoring that was mentioned was specifically around concurrency, not simply cleaning up code.
        ctoth 3y ago
        So if I show you an LLM implementing concurrency, will you concede the point? Is this your true objection? https://chat.openai.com/share/7c41f59a-c21c-4abd-876c-c95647d68026 https://chat.openai.com/share/7c41f59a-c21c-4abd-876c-c95647...
- atrus 3y ago
  Chess requires understanding, which computers lack. Go requires understanding, which computers lack. X requires Y which AI technology today lacks. AI is a constantly moving goalpost it seems.
  - satvikpendem 3y ago
    > AI is a constantly moving goalpost it seems. alwayshasbeen.png > The AI effect occurs when onlookers discount the behavior of an artificial intelligence program by arguing that it is not "real" intelligence.[1] > Author Pamela McCorduck writes: "It's part of the history of the field of artificial intelligence that every time somebody figured out how to make a computer do something—play good checkers, solve simple but relatively informal problems—there was a chorus of critics to say, 'that's not thinking'."[2] Researcher Rodney Brooks complains: "Every time we figure out a piece of it, it stops being magical; we say, 'Oh, that's just a computation.'"[3] > "AI is whatever hasn't been done yet." > —Larry Tesler https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect
  - The_Colonel 3y ago
    It was always clear that games like chess or go can be played by computers well, even with simple algorithms, because they were completely formalized. The only issue was with performance / finding more efficient algorithms. That's very different from code which (perhaps surprisingly) isn't well formalized. The goals are often vague and it's difficult to figure out what is intentional and what incidental behavior (esp. with imperative code).
    - apetresc 3y ago
      > It was always clear that games like chess or go can be played by computers well As someone who was deeply involved in the Go scene since the early 2000s let me emphatically assure you it was not at all clear. Indeed it was a major point of pride among Go enthusiasts that computers could not play it well, for various reasons (some, like the branching factor, one could potentially grant that advances in hardware and software could solve for eventually. Others, like the inherent difficulty in constructing an evaluation function, seemed intractable). Betting markets at the time of the AlphaGo match still had favorable odds for Sedol, even with the knowledge that Google was super-confident baked in. It is extreme hindsight-bias of exactly the type the grandparent was talking about to suggest that obviously everybody knew all along that Go was very beatable by "non-real AI".
astrange 3y ago
The ffmpeg tests take a lot more than a few seconds to run, and an AI god is still going to have trouble debugging multithreaded code.
dataangel 3y ago
AI is not very good at single threaded code which is widely regarded as much easier. The breathless demos don't generalize well when you truly test on data not in the training set, it's just that most people don't come up with good tests because they take something from the internet, which is the training set. But the code most people need to write is to do tasks that are bespoke to individual businesses/science-experiments/etc not popular CS problems that there are 1000 tutorials online for. When you get into those areas it becomes apparent really quickly that the AI only gets the "vibes" of what code should look like, it doesn't have any mechanistic understanding.

not_the_fda 3y ago

Because refactoring requires understanding, which LLMs completely lack.

ctoth 3y ago
> Because refactoring requires understanding, which LLMs completely lack. Interesting. Here's a sample question I just asked the AI: We are working on a React mapping application. I am going to give you two components from this application. Our goal is to extract functionality to deduplicate code We can make a single base component, extract code to hooks, or any other strategy which you think is appropriate Here is travel.tsx: <snip> Here is heatmap.tsx: <snip> And here is the response: To deduplicate code and improve maintainability between the `TravelMap` and `Heatmap` components in your React mapping application, we can identify common functionalities and extract them into reusable hooks or components. Here's a strategy considering your code: 1. *Extract Common Hooks*: Both components use similar hooks for managing markers, popups, and map references. We can create custom hooks that encapsulate these functionalities. 2. *Create Shared Utility Functions*: Functions like `initialViewState`, which are identical in both components, can be moved to a separate utility file and imported where needed. 3. *Refactor Map Layers and Sources*: The way map layers and sources are handled is quite similar in both components. A custom hook or component to manage map layers and sources would be beneficial. 4. *Generalize Popup Handling*: The popup handling in both components can be abstracted into a custom hook or a higher-order component. 5. *Combine Common Styling*: Shared styles, such as `MAP_CONTAINER_STYLE`, can be moved to a common CSS module or a styled-component that can be reused. 6. *Refactor Event Handlers*: Functions like `onMapClick`, `onMouseMove`, and `onMouseLeave` have similar structures. These can be part of a custom hook or a utility function with configurable parameters. 7. *Legend and Live Region Components*: These components are used in a similar way in both `TravelMap` and `Heatmap`. Ensure they are designed to be reusable with different props. 8. *Refactor Map Initialization*: The way the map is initialized and configured is quite similar. This can be abstracted into a custom component or hook. ### Example Implementations: #### 1. Custom Hook for Map Markers: ```typescript // useMapMarkers.ts import { useState, useCallback } from 'react'; export const useMapMarkers = (initialLat, initialLng, world) => { // Similar marker logic from your components // ... return { marker, onMarkerDrag, onMarkerDragEnd, updateMarkerPosition, // other marker related states and functions }; }; ``` #### 2. Shared Utility Function: ```typescript // utils/mapUtils.ts export const initialViewState = (map, mapType) => { // Implementation }; ``` #### 3. Custom Hook for Map Layers and Sources: ```typescript // useMapLayers.ts import { useState, useEffect } from 'react'; export const useMapLayers = (props) => { // Logic to handle layers and sources // ... return { dataLayer, patternLayer, // other layer related states and functions }; }; ``` #### 4. Generalized Popup Handling: ```typescript // useMapPopup.ts import { useState } from 'react'; export const useMapPopup = () => { // Popup logic from your components // ... return { popupState, openPopup, closePopup, setPopupData, // other popup related states and functions }; }; ``` ### Integration: Once these hooks and utilities are created, integrate them into your `TravelMap` and `Heatmap` components. Replace the duplicated logic with calls to these shared resources. This will significantly reduce code duplication and make future maintenance easier. ### Testing and Refinement: After refactoring, thoroughly test both components to ensure that all functionalities work as expected. Watch out for any edge cases or specific behaviors that might be affected by the refactoring. For those suggestions, I might use five out of eight of them, and probably do one or two things differently. But you cannot, with a straight face, say the model did not understand. It clearly did. It suggested reasonable refactors. If being able to refactor means understanding, I guess we have understanding! I could continue with this conversation, ask it to produce the full code for the hooks (I have in my custom prompt to provide outlines) and once the hooks are complete, ask it to rewrite the components using the shared code. Have you ever used one of these models?
- never_inline 3y ago
  Eliminating duplication and cleaning code is a different type of refactoring than supporting concurrency, which is much much harder. Cleaning up code also follows some well established patterns, performance work is much less pattern-y. Codebases like FFMPEG are one of the kind. I bet you need 10 or 100 times more understanding than the react thing you mentioned above. One day maybe AI can do it, but it probably won't be LLM. It would be something which can understand symbols and math.
  - ctoth 3y ago
    Ah, we're having some classic goalpost moving! > Because refactoring requires understanding, which LLMs completely lack. <demonstration that an LLM can refactor code> > Cleaning up code also follows some well established patterns, performance work is much less pattern-y. Just as writing shitty react apps follow patterns, low-level performance and concurrency work also follow patterns. See [0] for a sample. > I bet you need 10 or 100 times more understanding Okay, so a 10 or 100 times larger model? Sounds like something we'll have next year, and certainly within a decade. > One day maybe AI can do it, but it probably won't be LLM. It would be something which can understand symbols and math. You do understand that the reason some of the earlier GPTs had trouble with symbols and math was the tokenization scheme, completely separate from how they work in general, right? [0]: C++ Concurrency in Action: Practical Multithreading 1st Edition https://www.amazon.com/C-Concurrency-Action-Practical-Multithreading/dp/1933988770 https://www.amazon.com/C-Concurrency-Action-Practical-Multit...
    - kcbanner 3y ago
      > Because refactoring requires understanding, which LLMs completely lack. It's obvious from context here that the refactoring that was mentioned was specifically around concurrency, not simply cleaning up code.
      - ctoth 3y ago
        So if I show you an LLM implementing concurrency, will you concede the point? Is this your true objection? https://chat.openai.com/share/7c41f59a-c21c-4abd-876c-c95647d68026 https://chat.openai.com/share/7c41f59a-c21c-4abd-876c-c95647...
atrus 3y ago
Chess requires understanding, which computers lack. Go requires understanding, which computers lack. X requires Y which AI technology today lacks. AI is a constantly moving goalpost it seems.
- satvikpendem 3y ago
  > AI is a constantly moving goalpost it seems. alwayshasbeen.png > The AI effect occurs when onlookers discount the behavior of an artificial intelligence program by arguing that it is not "real" intelligence.[1] > Author Pamela McCorduck writes: "It's part of the history of the field of artificial intelligence that every time somebody figured out how to make a computer do something—play good checkers, solve simple but relatively informal problems—there was a chorus of critics to say, 'that's not thinking'."[2] Researcher Rodney Brooks complains: "Every time we figure out a piece of it, it stops being magical; we say, 'Oh, that's just a computation.'"[3] > "AI is whatever hasn't been done yet." > —Larry Tesler https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect
- The_Colonel 3y ago
  It was always clear that games like chess or go can be played by computers well, even with simple algorithms, because they were completely formalized. The only issue was with performance / finding more efficient algorithms. That's very different from code which (perhaps surprisingly) isn't well formalized. The goals are often vague and it's difficult to figure out what is intentional and what incidental behavior (esp. with imperative code).
  - apetresc 3y ago
    > It was always clear that games like chess or go can be played by computers well As someone who was deeply involved in the Go scene since the early 2000s let me emphatically assure you it was not at all clear. Indeed it was a major point of pride among Go enthusiasts that computers could not play it well, for various reasons (some, like the branching factor, one could potentially grant that advances in hardware and software could solve for eventually. Others, like the inherent difficulty in constructing an evaluation function, seemed intractable). Betting markets at the time of the AlphaGo match still had favorable odds for Sedol, even with the knowledge that Google was super-confident baked in. It is extreme hindsight-bias of exactly the type the grandparent was talking about to suggest that obviously everybody knew all along that Go was very beatable by "non-real AI".

astrange 3y ago

The ffmpeg tests take a lot more than a few seconds to run, and an AI god is still going to have trouble debugging multithreaded code.

dataangel 3y ago

AI is not very good at single threaded code which is widely regarded as much easier. The breathless demos don't generalize well when you truly test on data not in the training set, it's just that most people don't come up with good tests because they take something from the internet, which is the training set. But the code most people need to write is to do tasks that are bespoke to individual businesses/science-experiments/etc not popular CS problems that there are 1000 tutorials online for. When you get into those areas it becomes apparent really quickly that the AI only gets the "vibes" of what code should look like, it doesn't have any mechanistic understanding.