4 ms·

DocArray: a data structure for unstructured data

83 points by jonbaer 4y ago

mdaniel 4y ago
That linked document must be written for a very specific target audience because it seems to use a lot of words without providing context of the problem it's solving. The example in their GH repo is a little more "this does what?": https://github.com/jina-ai/docarray#example-1-represent-multimodal-data-in-dataclass https://github.com/jina-ai/docarray#example-1-represent-mult... although even that is a little "don't worry where there seemingly random methods on Document came from" which https://docarray.jina.ai/api/docarray.document/#docarray.document.Document https://docarray.jina.ai/api/docarray.document/#docarray.doc... seems to fill in a little more Then again, I guess folks were similarly confused in the past: https://news.ycombinator.com/from?site=jina.ai https://news.ycombinator.com/from?site=jina.ai https://news.ycombinator.com/from?site=github.com/jina-ai https://news.ycombinator.com/from?site=github.com/jina-ai
- jonbaer 4y ago
  Thanks for pointing that out, I came across it via @ https://qdrant.tech/documentation/install/#docarray https://qdrant.tech/documentation/install/#docarray so I was not sure what other libs were integrating it
- danbrooks 4y ago
  Thanks, the GitHub readme is much clearer.
- scott_s 4y ago
  I felt the same. The most important thing for a new library or language to do in its introduction is to show meaningful examples that solve a problem in the target domain in the canonical way.
- vladsanchez 4y ago
  If you or others need that much explanation it's probably not for you. Perhaps read [Multimodal Search](https://en.wikipedia.org/wiki/Multimodal_search https://en.wikipedia.org/wiki/Multimodal_search) and see whether it clarifies you.
jbverschoor 4y ago
> What is DocArray? Still don't know what it is.
- nsxwolf 4y ago
  It's a data structure for unstructured data.
  - BiteCode_dev 4y ago
    And it gots electrolites.
  - deleted 4y ago
    [deleted]
  - jbverschoor 4y ago
    But it's also like a library and like protobufs, but it's named after arrays. I don't see how the format works, etc. In other words, it's everything, which usually means it's really bad at everything, and in reality it's nothing. So again, what is it? The first screenshot is some python code, and then it's talking about hugging face. It'd call it 0clarityArray
    - nsxwolf 4y ago
      I have the same impression. "If you are a deep learning engineer who works on scalable deep learning services, you should use DocArray: it can be the basic building block of your system." I mean, wow. The basic building block of your system. The very nucleus of any scalable deep learning service! But, what is it?
Q6T46nT668w6i3m 4y ago
I’m excited to look at this but the comparisons are misleading! E.g., NPY has existed since 2007 (I believe it was the first NumPy RFC) and is exceedingly popular; JSON can, of course, represent multi-dimensional arrays (you can’t get much simpler or readable than [[1, 2], [3, 4]]); I also don’t understand “Pythonic experience” for JSON since dictionaries are so ubiquitous; I could go on …
fhaltmayer 4y ago
The description of this is kind of confusing but I think the easiest way to understand it is that it is a data processing pipeline of sorts. Take unstructured data and apply transformation and computation. A similar project to this is Towhee (https://github.com/towhee-io/towhee https://github.com/towhee-io/towhee). This project tries to simplify unstructured data processing and provides pretrained models and pipelines from their hub.
hnaccountme 4y ago
There is no such thing as unstructured data. If it were unstructured it would be noise. What people call unstructured data is, when they had little to no foresight when designing the original data structures. Any new data fields are just pilled on later and called unstructured

83 points by jonbaer 4y ago

That linked document must be written for a very specific target audience because it seems to use a lot of words without providing context of the problem it's solving. The example in their GH repo is a little more "this does what?": https://github.com/jina-ai/docarray#example-1-represent-multimodal-data-in-dataclass https://github.com/jina-ai/docarray#example-1-represent-mult... although even that is a little "don't worry where there seemingly random methods on Document came from" which https://docarray.jina.ai/api/docarray.document/#docarray.document.Document https://docarray.jina.ai/api/docarray.document/#docarray.doc... seems to fill in a little more Then again, I guess folks were similarly confused in the past: https://news.ycombinator.com/from?site=jina.ai https://news.ycombinator.com/from?site=jina.ai https://news.ycombinator.com/from?site=github.com/jina-ai https://news.ycombinator.com/from?site=github.com/jina-ai

jonbaer 4y ago
Thanks for pointing that out, I came across it via @ https://qdrant.tech/documentation/install/#docarray https://qdrant.tech/documentation/install/#docarray so I was not sure what other libs were integrating it
danbrooks 4y ago
Thanks, the GitHub readme is much clearer.
scott_s 4y ago
I felt the same. The most important thing for a new library or language to do in its introduction is to show meaningful examples that solve a problem in the target domain in the canonical way.
vladsanchez 4y ago
If you or others need that much explanation it's probably not for you. Perhaps read [Multimodal Search](https://en.wikipedia.org/wiki/Multimodal_search https://en.wikipedia.org/wiki/Multimodal_search) and see whether it clarifies you.

jbverschoor 4y ago

> What is DocArray? Still don't know what it is.

nsxwolf 4y ago
It's a data structure for unstructured data.
- BiteCode_dev 4y ago
  And it gots electrolites.
- deleted 4y ago
  [deleted]
- jbverschoor 4y ago
  But it's also like a library and like protobufs, but it's named after arrays. I don't see how the format works, etc. In other words, it's everything, which usually means it's really bad at everything, and in reality it's nothing. So again, what is it? The first screenshot is some python code, and then it's talking about hugging face. It'd call it 0clarityArray
  - nsxwolf 4y ago
    I have the same impression. "If you are a deep learning engineer who works on scalable deep learning services, you should use DocArray: it can be the basic building block of your system." I mean, wow. The basic building block of your system. The very nucleus of any scalable deep learning service! But, what is it?

Q6T46nT668w6i3m 4y ago

I’m excited to look at this but the comparisons are misleading! E.g., NPY has existed since 2007 (I believe it was the first NumPy RFC) and is exceedingly popular; JSON can, of course, represent multi-dimensional arrays (you can’t get much simpler or readable than [[1, 2], [3, 4]]); I also don’t understand “Pythonic experience” for JSON since dictionaries are so ubiquitous; I could go on …

fhaltmayer 4y ago

The description of this is kind of confusing but I think the easiest way to understand it is that it is a data processing pipeline of sorts. Take unstructured data and apply transformation and computation. A similar project to this is Towhee (https://github.com/towhee-io/towhee https://github.com/towhee-io/towhee). This project tries to simplify unstructured data processing and provides pretrained models and pipelines from their hub.

hnaccountme 4y ago

There is no such thing as unstructured data. If it were unstructured it would be noise. What people call unstructured data is, when they had little to no foresight when designing the original data structures. Any new data fields are just pilled on later and called unstructured