4 ms·
This looks very neat! I like the approach of including all data (API, messaging, database, files) as I went through similar thoughts and implementation recently
by pitah1 3y ago
This looks very neat! I like the approach of including all data (API, messaging, database, files) as I went through similar thoughts and implementation recently with my project.
A growing pain for data integrations is data validations. The schema can be correct but the data itself may have incorrect values. I noticed you had a section about the structure and semantics of the data here (https://orbitalhq.com/docs/background/intro-to-semantic-integration https://orbitalhq.com/docs/background/intro-to-semantic-inte...) but don't see where you could validate records using it. How do you handle these scenarios?
- martypitt 3y agoCurrently, we don't have validations implemented. We've spiked it in the past, and had some nice designs that attached validation rules semantically to the scalar types (which are very sharable), and models (which are more system-specific). The early designs felt nice (albeit early), but the deal we were working on didn't eventuate, so we didn't push it through. I think Validations become really interesting in both evaluation flows (ie., ingesting in a pipeline or through a mutating API), as well as quality scoring for data-at-rest. I'm excited to explore these in more detail, given a use-case.