3 ms·

Hey guys, looking for feedback on the essential concept of the idea. I'd like to hear from real programmers whether they understand what DitaBase is and whether

Hey guys, looking for feedback on the essential concept of the idea. I'd like to hear from real programmers whether they understand what DitaBase is and whether they think they would use, either now or in the future. It's been difficult to explain DitaBase, and I still don't know if that's because its actually a terrible idea, and much too complicated. If it is a good idea, then I think sometimes good ideas look complicated like this until people figure out how best to explain them. You wouldn't have expected the internet, or say, object orientation, to be instantly and clearly obvious what it is when they were first being talked about, and that is how I see DitaBase. Its basically that DitaBase is 4 essential concepts: schemas, validation scripts, conversion scripts, and public version control. If you take any single one out, then the really useful part, the dit file type, doesn't work. But it's also true that these concepts have never been applied in this way (that I have found), and so there isn't any default, easy language to call upon to explain DitaBase to people readily and easily. The answer to the question "What is DitaBase?" feels like it has to be "1, 2, 3, 4" where I list out the essential components. Perhaps, its a "common language for data", or a "general data accuracy system", but those concepts sound totally meaningless by themselves. I really would love to hear some feedback on how you understand the idea and whether my thinking on this is sound, or if I am totally off base, being caught up too close to the idea.

rubyn00bie 7y ago
I think I like the idea well enough you're trying to express; I just am, unsure, of like how you're conretely proposing to do it. All the code examples I see are just JSON schemas; which, I totally get, but they don't at all show me what its like to implement anything with it... Specifically... * How do I pull this into a project? * How do I access new dits? * How do I write a dit and publish it? * Where is the source code? Is it going to be open source and free?
- IsaiahShiner 7y ago
  The first thing I'll say is that if it would be helpful, I can have an example of a real dit (showing real syntax, the way labels change on the fly, etc.) on https://www.ditabase.io/example https://www.ditabase.io/example in a few hours. I only didn't do that to save some time. Thank you for expressing interest! So these are all questions I can answer: 1. How do I pull this into a project? Assuming you have a dit that you can use (full of scripts and such that you need), all you have to do to use it in a project is use existing infrastructure to tell the dit to validate itself, convert to something, query the data inside, etc. The plan is that this would all work with a linux app. I suppose a repl code interpreter would also be convenient? dit validate workspace/assets/example.dit dit convert workspace/assets/example.dit "sy7QvPp9" //UUID of the schema you want to convert to. dit query workspace/assets/example.dit "LengthEnglish" //Lots of options, etc. 2. How do I access new dits? All public dits would be available for free on the website. Anyone can view, reproduce, fork, pull request, etc. The starting dits would almost certainly be based on schema.org's object model. If you want something specifc, you could start at say, the DitaBase fork of https://schema.org/Organization https://schema.org/Organization and then look at the most popular child schemas until you find what you're looking for. The whole concept is to function the way open source software already does. If someone wants to announce they've made something new, or ask for help finding something, they use exising channels. Naturally, some people will want to keep their dits closed. In this case, they fork public dits and keep their own closed, using whatever project managment stuff they prefer. They never have to inform anyone how they are using dits, the same way you can start a new git project without ever informing GitHub or BitBucket or something. 3. How do I write a dit and publish it? The same way you would start a new GitHub project, but on DitaBase instead. Create an account, start a new dit, and beginning adding whatever you need for your project. Every dit gets its own project page, and dits probably will not really belong exclusively to the original owner of the project, because for really popular open dits, they will get used by lots of people. Of course, dits are not really like regular code projects, since having a public dit that isn't on the DitaBase system isn't all that useful. You can also imagine that you might want to use tools available on the website specific to dits, like pasting a dit and asking the site to check it for you, or attempt to convert it. It might show you a graphical map of the many different components of your dit and how those relate to where you want to convert to, and if a converter is missing or bugged, you can jump right to that page and start adding one, right on the website (many converters will be extremely simple). 4. Where is the source code? Is it going to be open source and free? At the moment, there is no product. I'm only about a month into this project and I want to get feedback before I start going nuts making a product without hearing from anyone. I intend for the overwhelming majority of vital DitaBase features to be open source. The Linux app for example, obviously open source. Any dits someone makes, if they want them to be, totally free and open source. DitaBase can make its money in other ways, such as providing all of this stuff as a service, and capitalizing on the new market advantages of DitaBase. In the long run, I envision a search engine that queries public dit repos, rather than just webpages, and can return data directly instead of just saying "Well, here are some webpages which probably contain what you want." So, yes, I want to challenge Google search.
thawkins 7y ago
I love the idea, but i'm a little wary of the implementation, we are heavy json-schema users, so i worry that this schema data is not going to track the json-schema standard. we got burned with swagger, where in V2 they where pretty much fully J-S compliant, but then for some reason dumped compliance in v3, so we had to kiss them goodbye. I don't need my schema's tied up in yet another propriety format, even if it is close to J-S. I cant see any compliance statements on the site that tell me what level of compliance exists.
- thawkins 7y ago
  Bad form replying to my own response, but i have inspected it some more and have some comments. 1. The site says schema.org is included, but schema.org is mainly just documentation, I woukd like something more like json-schema that I can actualy use the schema data directly. 2. I dont get why the validators have function names, surely the validator is anonymous, and its name should be a function of the property path, the argumant is always "value", and it always returns a boolean. So why do you need to declare a name at all. 3. If you cant support json-schema then you need an export function that allows you to export valid draft v8 json-schema. 4. Some way of anotating properties for graphql endpoint generation would be cool, ie generating .graphql files, or mongodb bson.
  - IsaiahShiner 7y ago
    Let me see if I can boil down your comments to specifc questions that I can answer. "Is DitaBase compliant with JSON-Schema?" The answer to this question is critical to understand about how DitaBase is different: if you want it to, then yes! The validators and converters are fully Turing complete, and not limited to Javascript either. If you can make a language interpreter in C/C++ (think Javascript V8), then you can use it with DitaBase. So whatever format you want, just rebuild it in (or port it to) dit schemas. Therefore, it isn't possible for DitaBase to not be compliant with something (at least, as far as I can think) because if someone thinks its important, they'll rebuild it in dit schemas. "Schema.org is just documentation. Are there programatic, queryable schemas?" Yes, but only optionally. In theory, all that is required to be a dit is a validator. You could inherit from "thing" and just directly have some text and a validator. But if you want to use Controllors (CRUD for a specific schema), Containers (abstract the shape of the data from the content) and other things like that, then yes, you need programatically defined schemas, just like in J-S. If someone has them, then you can use them. If not, in theory, if their schema is public, then you could just make a pull request with the relavent additions. DitaBase is very open compared to something like Swagger. "Uh, anonymous functions? Duh?" Oh, yes that makes much more sense that the ridiculous names I was giving those in the example. That actually just never occured to me. Of course these things will have to get slowly fleshed out over time, and I suppose there's no reason you couldn't have both. "GraphQL, BSON, custom tools, cool things" This is the great thing about DitaBase. Since you know the exact structure of any dit, and everything about dits is generic, you can just build all of this on top, without any overhead. DitaBase will almost certainly build some things like this in house, especially something like GraphQL, and some data analysis and statistical plotting utilites. Or perhaps, similar to how VScode absorbs open source projects into itself when they get really popular, DitaBase could probably do the same.
Vinnl 7y ago
I'm not entirely clear yet on what problem this solves, but I'm just going to share my first impressions in the hope that it might be helpful. So the first term that comes to my mind is RDF. How does it relate to Dit? Schema.org uses it, does Dit? The problem I think it might be supposed to solve: checking whether data is in a particular shape, and converting from one such shape to another (including from older versions of a shape to newer versions). Then I think that's primarily a problem of data from one source that should be interoperable with data from another source? As in: if I control the database and the software interacting with it, then I can simply ("simply") run a migration and update the software. If I control the API and the front-end talking to it, I can update both at the same time, or at least make the front-end able to deal with both the old version and new version without the overhead of adding Dit to my stack. So the use case would be if I don't control both the producer and consumer side of data? And in fact, there might be multiple parties on either side? If that's the case, then that sounds like the problem the Linked Data community also wants to solve - as far as I know, not extremely successful (but not entirely unsuccessful either). Is there anything in Dit that improves upon a problem that Linked Data is unable to solve?
- IsaiahShiner 7y ago
  You are correct, DitaBase is in some ways extremely similar to Linked Data/Semantic Web/RDF etc. In fact, I would say that DitaBase is, in some ways, just another attempt at making the Semantic Web deliver on its original promise. DitaBase obviously couldn't exist without HTTP. "Does dit use RDF?" The answer to this question is critical to understand about how DitaBase is different: if you want it to, then yes! The validators and converters are fully Turing complete, and not limited to Javascript either. If you can make a language interpreter in C/C++ (think Javascript V8), then you can use it with DitaBase. So whatever format you want, just rebuild it in (or port it to) dit schemas. Therefore, it isn't possible for DitaBase to not be compliant with something (at least, as far as I can think) because if someone thinks its important, they'll rebuild it in dit schemas. The reason DitaBase will succeed where Linked Data failed is by making DitaBase something that people in their day jobs want to use too, because the tools are just so good. Right now, a company with a data problem is extremely incentivized to seek a solution that solves only their specific problem. Obviously, because it's cheaper. But end-to-end solutions usually are not compatible with things like Linked Data, which needs solutions to be more general. If a company sees that the best solution to that problem is becoming compliant with DitaBase, then obviously they're going to do it. But surprise surprise, now they're compliant with Linked Data as well.
grenoire 7y ago
example.html shows an embedded JPEG as binary. How exactly does this work in terms of parsing? This question also goes for embedded JS and whatnot. How can you tell where a block ends when the contents of a block do not conform to the block's specification?
- IsaiahShiner 7y ago
  For block length, if you can't use context or control character/symbols, then the label would be required to specific the length of the payload in bytes, or similar. When you reach the end of the given length, then the next token should be the closing payload tag/brace. The important thing here is that everything gets described. The structures within a field, such as label, meta, validators, etc. always follow the specification set by the previous label. That label then sets the specifications for the next payload. And since the validators and such are Turing complete, the specifications can be anything, rebuilt within dit schemas.

rubyn00bie 7y ago

I think I like the idea well enough you're trying to express; I just am, unsure, of like how you're conretely proposing to do it. All the code examples I see are just JSON schemas; which, I totally get, but they don't at all show me what its like to implement anything with it... Specifically... * How do I pull this into a project? * How do I access new dits? * How do I write a dit and publish it? * Where is the source code? Is it going to be open source and free?

IsaiahShiner 7y ago
The first thing I'll say is that if it would be helpful, I can have an example of a real dit (showing real syntax, the way labels change on the fly, etc.) on https://www.ditabase.io/example https://www.ditabase.io/example in a few hours. I only didn't do that to save some time. Thank you for expressing interest! So these are all questions I can answer: 1. How do I pull this into a project? Assuming you have a dit that you can use (full of scripts and such that you need), all you have to do to use it in a project is use existing infrastructure to tell the dit to validate itself, convert to something, query the data inside, etc. The plan is that this would all work with a linux app. I suppose a repl code interpreter would also be convenient? dit validate workspace/assets/example.dit dit convert workspace/assets/example.dit "sy7QvPp9" //UUID of the schema you want to convert to. dit query workspace/assets/example.dit "LengthEnglish" //Lots of options, etc. 2. How do I access new dits? All public dits would be available for free on the website. Anyone can view, reproduce, fork, pull request, etc. The starting dits would almost certainly be based on schema.org's object model. If you want something specifc, you could start at say, the DitaBase fork of https://schema.org/Organization https://schema.org/Organization and then look at the most popular child schemas until you find what you're looking for. The whole concept is to function the way open source software already does. If someone wants to announce they've made something new, or ask for help finding something, they use exising channels. Naturally, some people will want to keep their dits closed. In this case, they fork public dits and keep their own closed, using whatever project managment stuff they prefer. They never have to inform anyone how they are using dits, the same way you can start a new git project without ever informing GitHub or BitBucket or something. 3. How do I write a dit and publish it? The same way you would start a new GitHub project, but on DitaBase instead. Create an account, start a new dit, and beginning adding whatever you need for your project. Every dit gets its own project page, and dits probably will not really belong exclusively to the original owner of the project, because for really popular open dits, they will get used by lots of people. Of course, dits are not really like regular code projects, since having a public dit that isn't on the DitaBase system isn't all that useful. You can also imagine that you might want to use tools available on the website specific to dits, like pasting a dit and asking the site to check it for you, or attempt to convert it. It might show you a graphical map of the many different components of your dit and how those relate to where you want to convert to, and if a converter is missing or bugged, you can jump right to that page and start adding one, right on the website (many converters will be extremely simple). 4. Where is the source code? Is it going to be open source and free? At the moment, there is no product. I'm only about a month into this project and I want to get feedback before I start going nuts making a product without hearing from anyone. I intend for the overwhelming majority of vital DitaBase features to be open source. The Linux app for example, obviously open source. Any dits someone makes, if they want them to be, totally free and open source. DitaBase can make its money in other ways, such as providing all of this stuff as a service, and capitalizing on the new market advantages of DitaBase. In the long run, I envision a search engine that queries public dit repos, rather than just webpages, and can return data directly instead of just saying "Well, here are some webpages which probably contain what you want." So, yes, I want to challenge Google search.

thawkins 7y ago

I love the idea, but i'm a little wary of the implementation, we are heavy json-schema users, so i worry that this schema data is not going to track the json-schema standard. we got burned with swagger, where in V2 they where pretty much fully J-S compliant, but then for some reason dumped compliance in v3, so we had to kiss them goodbye. I don't need my schema's tied up in yet another propriety format, even if it is close to J-S. I cant see any compliance statements on the site that tell me what level of compliance exists.

thawkins 7y ago
Bad form replying to my own response, but i have inspected it some more and have some comments. 1. The site says schema.org is included, but schema.org is mainly just documentation, I woukd like something more like json-schema that I can actualy use the schema data directly. 2. I dont get why the validators have function names, surely the validator is anonymous, and its name should be a function of the property path, the argumant is always "value", and it always returns a boolean. So why do you need to declare a name at all. 3. If you cant support json-schema then you need an export function that allows you to export valid draft v8 json-schema. 4. Some way of anotating properties for graphql endpoint generation would be cool, ie generating .graphql files, or mongodb bson.
- IsaiahShiner 7y ago
  Let me see if I can boil down your comments to specifc questions that I can answer. "Is DitaBase compliant with JSON-Schema?" The answer to this question is critical to understand about how DitaBase is different: if you want it to, then yes! The validators and converters are fully Turing complete, and not limited to Javascript either. If you can make a language interpreter in C/C++ (think Javascript V8), then you can use it with DitaBase. So whatever format you want, just rebuild it in (or port it to) dit schemas. Therefore, it isn't possible for DitaBase to not be compliant with something (at least, as far as I can think) because if someone thinks its important, they'll rebuild it in dit schemas. "Schema.org is just documentation. Are there programatic, queryable schemas?" Yes, but only optionally. In theory, all that is required to be a dit is a validator. You could inherit from "thing" and just directly have some text and a validator. But if you want to use Controllors (CRUD for a specific schema), Containers (abstract the shape of the data from the content) and other things like that, then yes, you need programatically defined schemas, just like in J-S. If someone has them, then you can use them. If not, in theory, if their schema is public, then you could just make a pull request with the relavent additions. DitaBase is very open compared to something like Swagger. "Uh, anonymous functions? Duh?" Oh, yes that makes much more sense that the ridiculous names I was giving those in the example. That actually just never occured to me. Of course these things will have to get slowly fleshed out over time, and I suppose there's no reason you couldn't have both. "GraphQL, BSON, custom tools, cool things" This is the great thing about DitaBase. Since you know the exact structure of any dit, and everything about dits is generic, you can just build all of this on top, without any overhead. DitaBase will almost certainly build some things like this in house, especially something like GraphQL, and some data analysis and statistical plotting utilites. Or perhaps, similar to how VScode absorbs open source projects into itself when they get really popular, DitaBase could probably do the same.

Vinnl 7y ago

I'm not entirely clear yet on what problem this solves, but I'm just going to share my first impressions in the hope that it might be helpful. So the first term that comes to my mind is RDF. How does it relate to Dit? Schema.org uses it, does Dit? The problem I think it might be supposed to solve: checking whether data is in a particular shape, and converting from one such shape to another (including from older versions of a shape to newer versions). Then I think that's primarily a problem of data from one source that should be interoperable with data from another source? As in: if I control the database and the software interacting with it, then I can simply ("simply") run a migration and update the software. If I control the API and the front-end talking to it, I can update both at the same time, or at least make the front-end able to deal with both the old version and new version without the overhead of adding Dit to my stack. So the use case would be if I don't control both the producer and consumer side of data? And in fact, there might be multiple parties on either side? If that's the case, then that sounds like the problem the Linked Data community also wants to solve - as far as I know, not extremely successful (but not entirely unsuccessful either). Is there anything in Dit that improves upon a problem that Linked Data is unable to solve?

IsaiahShiner 7y ago
You are correct, DitaBase is in some ways extremely similar to Linked Data/Semantic Web/RDF etc. In fact, I would say that DitaBase is, in some ways, just another attempt at making the Semantic Web deliver on its original promise. DitaBase obviously couldn't exist without HTTP. "Does dit use RDF?" The answer to this question is critical to understand about how DitaBase is different: if you want it to, then yes! The validators and converters are fully Turing complete, and not limited to Javascript either. If you can make a language interpreter in C/C++ (think Javascript V8), then you can use it with DitaBase. So whatever format you want, just rebuild it in (or port it to) dit schemas. Therefore, it isn't possible for DitaBase to not be compliant with something (at least, as far as I can think) because if someone thinks its important, they'll rebuild it in dit schemas. The reason DitaBase will succeed where Linked Data failed is by making DitaBase something that people in their day jobs want to use too, because the tools are just so good. Right now, a company with a data problem is extremely incentivized to seek a solution that solves only their specific problem. Obviously, because it's cheaper. But end-to-end solutions usually are not compatible with things like Linked Data, which needs solutions to be more general. If a company sees that the best solution to that problem is becoming compliant with DitaBase, then obviously they're going to do it. But surprise surprise, now they're compliant with Linked Data as well.

grenoire 7y ago

example.html shows an embedded JPEG as binary. How exactly does this work in terms of parsing? This question also goes for embedded JS and whatnot. How can you tell where a block ends when the contents of a block do not conform to the block's specification?

IsaiahShiner 7y ago
For block length, if you can't use context or control character/symbols, then the label would be required to specific the length of the payload in bytes, or similar. When you reach the end of the given length, then the next token should be the closing payload tag/brace. The important thing here is that everything gets described. The structures within a field, such as label, meta, validators, etc. always follow the specification set by the previous label. That label then sets the specifications for the next payload. And since the validators and such are Turing complete, the specifications can be anything, rebuilt within dit schemas.