3 ms·

I built Slopo to solve one specific problem: finding similar code that is hardest to detect by other tools, coding AI agents, and humans. It finds similar-look

I built Slopo to solve one specific problem: finding similar code that is hardest to detect by other tools, coding AI agents, and humans. It finds similar-looking code with embeddings. This detects more than just copy-paste clones or even clones with minor changes. Similar code is often not a clone to refactor, and this is a trade-off. Initial results need to be verified, but coding agents can do this quickly. Example prompts are available on https://slopo.dev https://slopo.dev Additionally, similar code distant in the codebase is ranked higher to focus on less obvious duplication. The results differ a lot depending on the codebase. I noticed that sometimes most of the detected duplicates are false positives, but the remaining ones are strong candidates to refactor or even bugs. Sometimes it reveals much more real duplication.

realxrobau 3mo ago
If it did PHP I would love to run it over WordPress. What would it take to add that?
- rkochanowski 3mo ago
  PHP support can be easily added, I will release a new version soon.
  - raro11 3mo ago
    Thank you
- rkochanowski 3mo ago
  PHP was added in the latest release.
klibertp 3mo ago
Correct me if I'm wrong, but looking at [1] it seems to be specifically using function definitions (I'm guessing this works with functions, methods, and lambdas (the "<unknown>" part)?) as units of repetition. If yes, that's fine, but I would seriously consider adding some settings to allow the user to control that granularity. Sometimes, the repeated code is a conditional branch within larger functions (i.e., "every else:" or "every except Ex:" looks the same). If the functions are large enough, the dissimilarity of the rest of the body would (probably?) cause such things to be missed. I would also consider - perhaps as a separate pass, with scoring set differently - to analyze comments (especially docstrings in Python). If I read the code correctly, you're currently just stripping them, which is the right thing to do when looking for code duplication, but duplicated docstrings are also often a signal that something is wrong in the codebase. The "different scoring" is because we expect docstring to be structured similarly (at least more than normal code), so some tweaking would be needed. Finally: very nice project, congrats! :) [1] https://github.com/rafal-qa/slopo/blob/main/src/slopo/indexing/parsing/lang/python.py https://github.com/rafal-qa/slopo/blob/main/src/slopo/indexi...
- rkochanowski 3mo ago
  Currently, only whole functions (including function-like constructs depending on language) are considered as unit. Skipping the extraction of conditional branches was my decision to not overcomplicate the first versions, which was intended to validate the idea. I will add this in future versions because I agree it's needed for large functions. I don't think it needs configurable granularity. In the current version, there is an analogous mechanism: when functions are nested, both outer and inner are embedded separately. When both are similar to each other, this pair is excluded. Inner or outer functions can appear in results depending on similarity to other units. Regarding comments, they are removed and I will think about handling them. The challenge is not with extraction, but with how to present this in a report. This may be a nice addition because coding agents often add comments. Thanks for the feedback.
nttylock 3mo ago
[flagged]
- rkochanowski 3mo ago
  My solution for false positives is simpler: 1. The tool uses only cosine similarity plus boost depending on distance in the codebase. 2. Classification with LLM. This can be done by coding agent used with project giving better results than integrating this pass in the tool. LLMs used for coding are pretty good. I assumed that this is not a problem I need to solve inside the tool. I'm aware this is not deterministic, but this is by design. Regarding information about raw similarity: currently, the score (raw similarity + boost) is visible in the report, so this value can be configured based on data. The raw similarity threshold can also be configured, but it's not displayed. I will think about how to handle this.
  - AlexeyBelov 3mo ago
    You're replying to an LLM bot.
jadbox 3mo ago
What a clever little tool. This is exactly the kind of pragmatic AI tools I want to see more of: linux-y single purpose tools!

realxrobau 3mo ago

If it did PHP I would love to run it over WordPress. What would it take to add that?

rkochanowski 3mo ago
PHP support can be easily added, I will release a new version soon.
- raro11 3mo ago
  Thank you
rkochanowski 3mo ago
PHP was added in the latest release.

klibertp 3mo ago

Correct me if I'm wrong, but looking at [1] it seems to be specifically using function definitions (I'm guessing this works with functions, methods, and lambdas (the "<unknown>" part)?) as units of repetition. If yes, that's fine, but I would seriously consider adding some settings to allow the user to control that granularity. Sometimes, the repeated code is a conditional branch within larger functions (i.e., "every else:" or "every except Ex:" looks the same). If the functions are large enough, the dissimilarity of the rest of the body would (probably?) cause such things to be missed. I would also consider - perhaps as a separate pass, with scoring set differently - to analyze comments (especially docstrings in Python). If I read the code correctly, you're currently just stripping them, which is the right thing to do when looking for code duplication, but duplicated docstrings are also often a signal that something is wrong in the codebase. The "different scoring" is because we expect docstring to be structured similarly (at least more than normal code), so some tweaking would be needed. Finally: very nice project, congrats! :) [1] https://github.com/rafal-qa/slopo/blob/main/src/slopo/indexing/parsing/lang/python.py https://github.com/rafal-qa/slopo/blob/main/src/slopo/indexi...

rkochanowski 3mo ago
Currently, only whole functions (including function-like constructs depending on language) are considered as unit. Skipping the extraction of conditional branches was my decision to not overcomplicate the first versions, which was intended to validate the idea. I will add this in future versions because I agree it's needed for large functions. I don't think it needs configurable granularity. In the current version, there is an analogous mechanism: when functions are nested, both outer and inner are embedded separately. When both are similar to each other, this pair is excluded. Inner or outer functions can appear in results depending on similarity to other units. Regarding comments, they are removed and I will think about handling them. The challenge is not with extraction, but with how to present this in a report. This may be a nice addition because coding agents often add comments. Thanks for the feedback.

nttylock 3mo ago

[flagged]

rkochanowski 3mo ago
My solution for false positives is simpler: 1. The tool uses only cosine similarity plus boost depending on distance in the codebase. 2. Classification with LLM. This can be done by coding agent used with project giving better results than integrating this pass in the tool. LLMs used for coding are pretty good. I assumed that this is not a problem I need to solve inside the tool. I'm aware this is not deterministic, but this is by design. Regarding information about raw similarity: currently, the score (raw similarity + boost) is visible in the report, so this value can be configured based on data. The raw similarity threshold can also be configured, but it's not displayed. I will think about how to handle this.
- AlexeyBelov 3mo ago
  You're replying to an LLM bot.

jadbox 3mo ago

What a clever little tool. This is exactly the kind of pragmatic AI tools I want to see more of: linux-y single purpose tools!