gschoeni

8 ms·

31.
by gschoeni 3y ago
Super cool! Would love to see an integration with Oxen and their data version control https://github.com/Oxen-AI/oxen-release
32.
by gschoeni 3y ago
We have been working on a data version control tool called Oxen that is tackling many of your needs. Feel free to check it out here: https://github.com/Oxen-AI/oxen-release#-oxen Going down your list of requirements, O
33.
by gschoeni 4y ago
Will add a local network benchmark as well! Many reasons to upload your data to the cloud...but agree that there are use cases where you might just want to version on your local network.
34.
by gschoeni 4y ago
Oxen dev here - let me add some benchmarks for DVC backed by an S3 bucket. I did it awhile back and we were still faster, but agree it's a good benchmark to have. Fundamentally even adding and committing data locally is slower, even be
35.
by gschoeni 4y ago
We've been working on some open source tooling called "oxen" that was built for large datasets of images, video, audio, text etc. We wanted to solve the exact problem you're flagging here with git. Feel free to check it
36.
by gschoeni 4y ago
Good feedback, we're working on more streaming features as well as supporting different backends for the CLI. Any other features you would find useful or a dealbreaker?
37.
by gschoeni 4y ago
We have working with an S3 backend in the upcoming features, agree it's essential.
38.
by gschoeni 4y ago
Oops, thanks for catching! Updated.
39.
by gschoeni 4y ago
Good catch, thank you! Updated
40.
by gschoeni 4y ago
Good call out, didn't realize how hard to see they were. Updated!
41.
by gschoeni 4y ago
Dolthub is more about versioning structured SQL data tables. Oxen handles more large unstructured sets of images, video, audio, text, and DataFrame (parquet, csv, arrow, etc) files
42.
by gschoeni 4y ago
We definitely have some of these features on our roadmap! Anything particularly helpful in DVC's workflow that you think we should prioritize?
43.
by gschoeni 4y ago
Creator of the project here, would love to look at what an LFS style integration with git hooks could look like. Let me know if you have any ideas there!
44.
by gschoeni 4y ago
Raw speed on large datasets of images, video, audio, etc is one factor, some performance numbers can be found here: https://github.com/Oxen-AI/oxen-release/blob/main/Performanc...
45.
by gschoeni 4y ago
Creators of the project here, the first difference is the raw speed if you have many images, video, audio, files etc. We did some benchmarking here: https://github.com/Oxen-AI/oxen-release/blob/main/Perfo
46.
by gschoeni 4y ago
We've been working on an open source tool called Oxen to help store large ML datasets. It's optimized for large sets of unstructured data ie images, video, audio, text, as well as parquet or arrow style DataFrames. Would love to g
47.
by gschoeni 4y ago
We're more focused on the data use case right now, but I could see it being useful for large code bases as well.
48.
by gschoeni 4y ago
For fast hashing of the content, can hash at 30gb/sec over sha1 or md5 which is less than 1gb/sec
49.
Show HN: Oxen.ai – Fast Unstructured Data Version Control(github.com)
16 points by gschoeni 4y ago|5 comments
50.
by gschoeni 4y ago
At Oxen.ai we have built fast, efficient, version control for machine learning datasets. It can handle large datasets of images, video, audio, text, data frames, etc. Would love to get your feedback!
51.
We Version Our Code, Why Not Our Data?(medium.com)
4 points by gschoeni 4y ago|1 comments
52.
by gschoeni 4y ago
Where are you hosting the dataset? Would love to help out, I'm building an open source data version control tool to help iterate on ML datasets. https://github.com/Oxen-AI/oxen-release Would be cool if we could ge
53.
by gschoeni 4y ago
Any plans on releasing the actual audio data and not just a csv with links to YouTube IDs? Also maybe a silly question, but what's the legal ramifications of downloading these YouTube videos and training on them yourself? Google must h
54.
by gschoeni 4y ago
If you ever want to add more data...and it is too much to check into your GitHub repository, I'm working on a "data version control" system that is built to be complimentary to your code repo, without taking up space next to
55.
by gschoeni 4y ago
Early days, but trying to solve the lack of data problem with a project called "Oxen". At it's core it's a data version control system. Built from the ground up to handle the size and scale of some of these ML datasets (
56.
by gschoeni 4y ago
Let me know if you want a good place to host and iterate on the dataset, I'm working on a version control system optimized for deep learning datasets. Free to host public datasets right now, just looking for feedback. It's called
57.
by gschoeni 4y ago
Has somebody crawled and made a corpus out of hacker news? Is it maintained?
58.
by gschoeni 4y ago
Been working on a version control system to solve this large binary file problem as well. It's called Oxen and is initially targeted towards unstructured machine learning datasets, but could be good for this use case as well. Would lo