3 ms·

The magic is testing. Having locally available testing and high throughput testing with high amount of test cases now unlocks more speed. The test cases themse

The magic is testing. Having locally available testing and high throughput testing with high amount of test cases now unlocks more speed. The test cases themselves becomes the foci - the LLM usually can't get them right.

neonbrain 7mo ago
The word "Testing" is a very loaded term. Few non-professionals, or even many professionals, fully understand what is meant by it. Consider the the following: Unit, Integration, System, UAT, Smoke, Sanity, Regression, API Testing, Performance, Load, Stress, Soak, Scalability, Reliability, Recovery, Volume Testing, White Box Testing, Mutation Testing, SAST, Code Coverage, Control Flow, Penetration Testing, Vulnerability Scanning, DAST, Compliance (GDPR/HIPAA), Usability, Accessibility (a11y), Localization (L10n), Internationalization (i18n), A/B Testing, Chaos Engineering, Fault Injection, Disaster Recovery, Negative Testing, Fuzzing, Monkey Testing, Ad-hoc, Guerilla Testing, Error Guessing, Snapshot Testing, Pixel-Perfect Testing, Compatibility Testing, Canary Testing, Installation Testing, Alpha/Beta Testing... ...and I'm certain I've missed dozens of other test approaches.
- megous 7mo ago
  You forgot a hope-driven development and release process and other optimism based ("i'm sure it's fine" method), or faith based approaches to testing (ship and pray, ...). Customer driven invluntary beta testing also comes to mind and "let's see what happens" 0-day testing before deployment. We also do user-driven error discovery, frequently.
- worik 7mo ago
  There is no science to testing, no provable best way, despite many people's vehement opinions
  - neonbrain 7mo ago
    Why did you assume I'm talking about a "provable best way"? I meant that it doesn't make sense to talk simply about "testing" without clarifying what one means by it. If you assume that the absence of a "provable best way" implies a lack of utility, let me remind you that there is no "provable best way" for training LLMs either. Does that matter in practice?
robhlt 7mo ago
How does that test suite get built and validated? A comprehensive and high quality test suite is usually much larger than the codebase it tests. For example, the sqlite test suite is 590x [1] the size of the library itself 1. https://sqlite.org/testing.html https://sqlite.org/testing.html
- nswango 7mo ago
  sqlite is an extreme outlier not a typical example, with regard to test suite size and coverage.
- Aperocky 7mo ago
  By sweat and tears, and unfortunately, AI can only help so much in those cases. You'll have to have a really concrete idea about what your product is and how it should work.
worik 7mo ago
> The magic is testing. No it is not. There os no amount of testing that can fix a flawed design
- Aperocky 7mo ago
  That is a given, similar to how no amount of implementation can fix a wrong product. Both statements are not very meaningful.

neonbrain 7mo ago

The word "Testing" is a very loaded term. Few non-professionals, or even many professionals, fully understand what is meant by it. Consider the the following: Unit, Integration, System, UAT, Smoke, Sanity, Regression, API Testing, Performance, Load, Stress, Soak, Scalability, Reliability, Recovery, Volume Testing, White Box Testing, Mutation Testing, SAST, Code Coverage, Control Flow, Penetration Testing, Vulnerability Scanning, DAST, Compliance (GDPR/HIPAA), Usability, Accessibility (a11y), Localization (L10n), Internationalization (i18n), A/B Testing, Chaos Engineering, Fault Injection, Disaster Recovery, Negative Testing, Fuzzing, Monkey Testing, Ad-hoc, Guerilla Testing, Error Guessing, Snapshot Testing, Pixel-Perfect Testing, Compatibility Testing, Canary Testing, Installation Testing, Alpha/Beta Testing... ...and I'm certain I've missed dozens of other test approaches.

megous 7mo ago
You forgot a hope-driven development and release process and other optimism based ("i'm sure it's fine" method), or faith based approaches to testing (ship and pray, ...). Customer driven invluntary beta testing also comes to mind and "let's see what happens" 0-day testing before deployment. We also do user-driven error discovery, frequently.
worik 7mo ago
There is no science to testing, no provable best way, despite many people's vehement opinions
- neonbrain 7mo ago
  Why did you assume I'm talking about a "provable best way"? I meant that it doesn't make sense to talk simply about "testing" without clarifying what one means by it. If you assume that the absence of a "provable best way" implies a lack of utility, let me remind you that there is no "provable best way" for training LLMs either. Does that matter in practice?

robhlt 7mo ago

How does that test suite get built and validated? A comprehensive and high quality test suite is usually much larger than the codebase it tests. For example, the sqlite test suite is 590x [1] the size of the library itself 1. https://sqlite.org/testing.html https://sqlite.org/testing.html

nswango 7mo ago
sqlite is an extreme outlier not a typical example, with regard to test suite size and coverage.
Aperocky 7mo ago
By sweat and tears, and unfortunately, AI can only help so much in those cases. You'll have to have a really concrete idea about what your product is and how it should work.

worik 7mo ago

> The magic is testing. No it is not. There os no amount of testing that can fix a flawed design

Aperocky 7mo ago
That is a given, similar to how no amount of implementation can fix a wrong product. Both statements are not very meaningful.