3 ms·

We are using OpenAI to only to create schema, column matching and Regex generation.No CSV data are sent to open AI..

by yosai 4y ago

nerdponx 4y ago
So you ask OpenAI to generate a schema and computer code that will clean input data to conform to that schema, and then run the user's data through that program? Is it possible for users to obtain and audit the generated code for correctness, performance, etc.? How do you prevent things like the AI from generating catastrophically-backtracking regex, O(N!) algorithms, or outright mistakes?
- yosai 4y ago
  @nerdponx Open AI only generate a validation schema.We have our own schema generation engine for any custom validation which can be used where Open AI is not able to understand or generate correct schema.Yes you are absolutely right.Open AIs output is not always right.We have integrated a JSON parser which validate Open AI output and currently developing a regex parser which validates Open AIs output.Hope it answers your query..Happy to understand your pain point more.
  - nerdponx 4y ago
    Interesting, thanks for explaining how it works. Would it also be possible to construct one of these JSON schemas by hand, without OpenAI? The core data cleaning system sounds like just as interesting a piece of technology as the AI schema generation.
    - yosai 4y ago
      @nerdponx Yes YoBulk provides a way to write JSON schemas by hand..We have added some custom keyword like validate which is not defined in JSON standard..But our validation engine can understand that..You can pass a javascript function through validate keyword..It's a game changer.. "first_name":{ "type":"string", "format": "first_name", "validate": "(x) => {\r\n let regex = new RegExp(\"([a-z][a-zA-Z]*)\");\r\n return regex.test(x);\r\n }" },
      - nerdponx 4y ago
        This is a very interesting combination of technologies. Thank you again for explaining how they work! I tend to prefer open-source solutions for my own work, but I can see this being highly useful and valuable for many businesses.
        yosai 4y ago
        Thanks keep following YoBulk..We will keep on doing some path breaking work in CSV importing space.
justeleblanc 4y ago
So what exactly is sent to OpenAI?
- yosai 4y ago
  @justeleblanc we do auto column matching through open AI..Ex:If you have defined a template with column name Date of Joining and the CSV data donor upload a CSV with a field DOJ,then your validation engine will skip the importing as it does not match with expected column names..Here Open AI comes handy..It gets the context and smartly identify that DOJ is same as Date of Joining and does the importing..You can go through https://doc.yobulk.dev/YoBulk%20AI/AI%20usecases https://doc.yobulk.dev/YoBulk%20AI/AI%20usecases to understand more on AI usecases of YoBulk.
  - dang 4y ago
    Sorry for the offtopicness but can you please email me at hn@ycombinator.com?
    - cdolan 4y ago
      Cheers to OP for whatever magic this interaction has unlocked!
      - dang 4y ago
        I resort to comments like that when I don't have another way of contacting the user. If it happens in a Show HN thread, it's probably that I want to ask if they've considered applying to YC with the project. People often underestimate what YC might be interested in. Open-source startups are a big part of what gets funded these days. I particularly love it when startups make it in to YC through HN—it's great for both YC and HN, and if a big success ever came out of it, it would easily fund HN for another century :) If I post a please-email-me outside of a Show HN thread, it might be anything! but the most common reason is that I want to send them a repost invite for some cool article they posted long ago. Invited reposts make it into the second-chance pool (see https://news.ycombinator.com/pool https://news.ycombinator.com/pool and https://news.ycombinator.com/item?id=26998308 https://news.ycombinator.com/item?id=26998308).

We are using OpenAI to only to create schema, column matching and Regex generation.No CSV data are sent to open AI..

by yosai 4y ago

We are using OpenAI to only to create schema, column matching and Regex generation.No CSV data are sent to open AI..

So you ask OpenAI to generate a schema and computer code that will clean input data to conform to that schema, and then run the user's data through that program? Is it possible for users to obtain and audit the generated code for correctness, performance, etc.? How do you prevent things like the AI from generating catastrophically-backtracking regex, O(N!) algorithms, or outright mistakes?

yosai 4y ago
@nerdponx Open AI only generate a validation schema.We have our own schema generation engine for any custom validation which can be used where Open AI is not able to understand or generate correct schema.Yes you are absolutely right.Open AIs output is not always right.We have integrated a JSON parser which validate Open AI output and currently developing a regex parser which validates Open AIs output.Hope it answers your query..Happy to understand your pain point more.
- nerdponx 4y ago
  Interesting, thanks for explaining how it works. Would it also be possible to construct one of these JSON schemas by hand, without OpenAI? The core data cleaning system sounds like just as interesting a piece of technology as the AI schema generation.
  - yosai 4y ago
    @nerdponx Yes YoBulk provides a way to write JSON schemas by hand..We have added some custom keyword like validate which is not defined in JSON standard..But our validation engine can understand that..You can pass a javascript function through validate keyword..It's a game changer.. "first_name":{ "type":"string", "format": "first_name", "validate": "(x) => {\r\n let regex = new RegExp(\"([a-z][a-zA-Z]*)\");\r\n return regex.test(x);\r\n }" },
    - nerdponx 4y ago
      This is a very interesting combination of technologies. Thank you again for explaining how they work! I tend to prefer open-source solutions for my own work, but I can see this being highly useful and valuable for many businesses.
      - yosai 4y ago
        Thanks keep following YoBulk..We will keep on doing some path breaking work in CSV importing space.

justeleblanc 4y ago

So what exactly is sent to OpenAI?

yosai 4y ago
@justeleblanc we do auto column matching through open AI..Ex:If you have defined a template with column name Date of Joining and the CSV data donor upload a CSV with a field DOJ,then your validation engine will skip the importing as it does not match with expected column names..Here Open AI comes handy..It gets the context and smartly identify that DOJ is same as Date of Joining and does the importing..You can go through https://doc.yobulk.dev/YoBulk%20AI/AI%20usecases https://doc.yobulk.dev/YoBulk%20AI/AI%20usecases to understand more on AI usecases of YoBulk.
- dang 4y ago
  Sorry for the offtopicness but can you please email me at hn@ycombinator.com?
  - cdolan 4y ago
    Cheers to OP for whatever magic this interaction has unlocked!
    - dang 4y ago
      I resort to comments like that when I don't have another way of contacting the user. If it happens in a Show HN thread, it's probably that I want to ask if they've considered applying to YC with the project. People often underestimate what YC might be interested in. Open-source startups are a big part of what gets funded these days. I particularly love it when startups make it in to YC through HN—it's great for both YC and HN, and if a big success ever came out of it, it would easily fund HN for another century :) If I post a please-email-me outside of a Show HN thread, it might be anything! but the most common reason is that I want to send them a repost invite for some cool article they posted long ago. Invited reposts make it into the second-chance pool (see https://news.ycombinator.com/pool https://news.ycombinator.com/pool and https://news.ycombinator.com/item?id=26998308 https://news.ycombinator.com/item?id=26998308).