8 ms·

Only 39 days since the last "GitHub for data" was announced: https://news.ycombinator.com/item?id=22375774 https://news.ycombinator.com/item?id=22375774 I'll s

Only 39 days since the last "GitHub for data" was announced: https://news.ycombinator.com/item?id=22375774 https://news.ycombinator.com/item?id=22375774 I'll say what I said in February: I started a company with the same premise 9 years ago, during the prime "big data" hype cycle. We burned through a lot of investor money only to realize that there was not a market opportunity to capture. That is, many people thought it was cool - we even did co-sponsored data contests with The Economist - but at the end of the day, we couldn't find anyone with an urgent problem that they were willing to pay to solve. I wish these folks luck! Perhaps things have changed; we were part of a flock of 5 or 10 similar projects and I'm pretty sure the only one still around today is Kaggle. https://www.youtube.com/watch?v=EWMjQhhxhQ4 https://www.youtube.com/watch?v=EWMjQhhxhQ4

sjtindell 7y ago
Just want to say, really appreciate this food for thought. Where do you go and see someone say "my company tried it and...". This site is a godsend.
- austinjp 7y ago
  Agreed, it's always refreshing to read candid posts like this. A useful term for Google is "startup post mortem" although you still have to do plenty of sifting.
timsehn 7y ago
I'll check it out. We think the world is a little more ready for this now, given how widely Git is adopted and the advances in other data tooling (like ML). But, as we're all aware, starting a business is hard :-)
zyang 7y ago
The need is definitely there. My day job involves such need. But we simply cannot trust a drive-by startup to fill the gap. It's safer just to roll your own.
- timsehn 7y ago
  Dolt is and always will be open source. Feel free to fork a copy and make it your own.
philipov 7y ago
Git succeeded because it was free, and then business models were able to be built up around the open-source ecosystem after a market evolved naturally. There is a need, but if you go into it trying to build a business from scratch, you're going to have a bad time.
- TylerE 7y ago
  Git succeeded because of Linus. Sure as hell wasn't because of the UX, else Mercurial would have won, or even DARCS. 99.99999% of projects are not the Linux kernel
  - sdan 7y ago
    I'd argue it succeeded (at least more recently) because of the UX that companies like Github and Gitlab gave, not particularly linus or because it was free.
    - TylerE 7y ago
      If Linus hasn’t pushed it, it never would have caught on.
      - bryanrasmussen 7y ago
        Sure lots of good things would never catch on except an influential person in the field the good thing targets says: hey this is really useful for us! This gets lots of people to look at it. But those people still have to decide in the end of the day whether it is actually useful for them.
        sdan 7y ago
        Reminds me of sports: Just because Steph Curry uses Under Armor doesn't mean everyone will too. I think obviously something created by Linus was deemed to be of great value, but what made Git as profound as it is today is: UX for the majority of people who are beginners, which was Github backed by millions of other developers.
drewmol 7y ago
Do you consider your effort a mistake looking back? Were you able to pay yourself fairly or was it a losing investment?
- peteforde 7y ago
  The answer to that can only be nuanced. On one hand, startups are as exciting as bank heists. You put together an amazing team, do your homework as best you can, then get killed trying to execute your perfect plan. I'm proud of what we built and we all learned a lot the hard way. On the down-side, building a company is emotionally, physically, mentally exhausting. It wasn't really a matter of whether I could pay myself fairly; I drew a typical programmer salary along with everyone else. However, the important detail is that you're spending someone else's money and every penny of it represents someone you respect putting their trust in you, and you feel the weight of that every day. Ultimately, I don't exactly regret it but I certainly wish that we weren't so convincing that we convinced ourselves of a market opportunity that we couldn't access or didn't exist at all. There was so much heat for "data" in 2011 it really seemed like we just had to show up with an amazing product. We were wrong.
  - dan00 7y ago
    > Ultimately, I don't exactly regret it but I certainly wish that we weren't so convincing that we convinced ourselves of a market opportunity that we couldn't access or didn't exist at all. How should you really know this without trying? If you aren't convinced, where should the motivation come from?
    - peteforde 7y ago
      Actually, I do have a pretty good answer to this. Every business has about 100 questions where if you know the answer to those questions, you are quite likely to be successful. The hard part is knowing which questions need to be asked of each business. To be clear: my co-founders and I were all seasoned, multi-startup people. We had extremely high-calibre investors with finely tuned bullshit meters. In the end, our own pitching skill undid us because I think less-convincing founders would have undergone 25% more scrutiny and that would have required us to demonstrate that we had signed LOIs from 3-5 real customers before starting. There was a subconscious misdirection around the fact that we didn't actually have anyone beating down our door. We let the excitement for "data", our personalities and our track records carry the moment. Of course founders have to drink their own Kool-Aid to some extent or they won't make it. But there's real power and value in the customer development mindset. People want this? Prove it.
ken 7y ago
That's GitHub for data. It's a service, and they still haven't launched anything yet. This is Git for data. It's a program, and it appears to be an open-source one you can download and use today.
- pavlov 7y ago
  The OP points to a site called DoltHub.com, so it’s not like they don’t have ambitions to commercialize as another “GitHub for data”.
  - ken 7y ago
    Sure, but the existence of GitHub has only helped Git grow, not hurt it.
- enos_feedler 7y ago
  There is actually an old git for data project too: https://github.com/datproject/dat https://github.com/datproject/dat It's ~5 years old and I really wanted it to be huge. Hoping this new project is a success. Especially since I notice I went to high school with one of the founders of Dolt (Hey Tim!)
  - visarga 7y ago
    Can it remove a file from the repo history? It's a GDPR feature that makes git hard to use for data.
  - cbenz 7y ago
    Dat is more about distribution (décentralized, distributed, P2P) but it's not possible to make queries.
    - enos_feedler 7y ago
      Yes, the project used to associate itself as a git for data, but I guess not in the sense of a db.
  - 7y ago
staflow 7y ago
1. Develop a solution with catchy name. 2. ONLY and only after it’s finished, start looking for problem that fits that solution. 3. Realize there is none. 4. ????? 5. Profit! Gee I wonder why our economy is so ineffective..
- chmaynard 7y ago
  Step 4 is where you create a snazzy website with lots of seductive images, clever animations, and "Free Trial" buttons.
kidintech 7y ago
Palantir is successful as well, but no clue what the hateboner on HN for them is, since it never even gets mentioned in these threads.
- th3iedkid 7y ago
  Is it because much of palantir success is with large enterprises and not startups?
  - kidintech 7y ago
    OP says "we couldn't find anyone with an urgent problem that they were willing to pay to solve" and "there was no market to capture". Wouldn't a counter example contradict those statements, regardless of the size of Palantir's clients?
roystonvassey 7y ago
In our group we use git for code repos and cloud for storage and actual compute. It works seamlessly and git APIs work fantastically with almost any service, IDE or whatever your tool of choice. I suspect with the increasing cloud adaption, accessing data is getting easier by the day and I see no real need for a “git for data” tool. Plus, as a data scientist, it allows me to keep code and data separate, especially if I’m working with confidential data.
deforciant 7y ago
We also started "Git for data" several years ago but since then pivoted to data science/ML tooling (https://dotscience.com/ https://dotscience.com/) by building features that people actually want on the original product. Since then the "git for data" accounts only probably for 5% of the total functionality :) I guess "Git for data" is not very useful if you don't have the whole platform built around it to actually use the features. We mainly use it for data synchronization between the nodes and provenance tracking so people can see what data was used to build specific models and to track how the project evolves itself without forcing people to "commit" their changes manually (as we have seen that often data scientists don't even use git, just files on their Jupyter notebooks).
NicoJuicy 7y ago
> that there was not a market opportunity to capture Weird, there is an internal project ( already live) at the company ( multinational) where i work for that it's used.
solatic 7y ago
I wonder how much of the market opportunity here is contingent upon market education. Everybody can clearly see the value of having a personal automobile, but how successful can you be at selling automobiles to people who don't know how to drive? Do people desire cars enough to buy one if they don't know how to drive? Everyone can see how FAANG companies are growing wealthy off the mountains of data they are amassing, so everyone understands how data can be desirable. But what if your potential market base doesn't understand how to "drive" data - how to identify which data would be valuable for them and how best to exploit it? It seems to me that part of a go-to-market strategy needs, at least in the short term, to help potential customers transition from "that's a really shiny bauble" to "I understand how this is going to make me money."
- peteforde 7y ago
  Maybe, but also maybe there just isn't a huge demographic of data scientists with discretionary purchasing capacity and a hair-on-fire problem that they are desperately searching for someone to take their money and fix. I think that a lot of the data VIP types we met with honestly wanted to know why they needed it, but the more they thought about it, the more it just seemed like a shiny thing. It's telling that dozens of similar companies with smart people behind them have thrown their talents at this solution, and none of them have located the problem people are eager to pay to solve.
dmitryminkovsky 7y ago
You may be right in the big picture, but not all "GitHubs for data" are the same. This product seems super cool: > Dolt is Git for data. Instead of versioning files, Dolt versions tables. DoltHub is a place on the internet to share Dolt repositories. As far as we can tell, Dolt is the only database with branches. I find it hard to believe no other database has branches, but if that's true and if this product works like you'd imagine, that is really cool. Given your historical observation, I think you're right that this will not lead to a market revolution, but sometimes you need the right product to change the landscape.
nailer 7y ago
> we were part of a flock of 5 or 10 similar projects and I'm pretty sure the only one still around today is Kaggle. Dat is still around, on version 14, last update in git was 5 days ago. https://github.com/datproject/dat https://github.com/datproject/dat https://dat.foundation/ https://dat.foundation/
- moondowner 7y ago
  Impressed by the documentation https://datprotocol.github.io/how-dat-works/ https://datprotocol.github.io/how-dat-works/
specialist 7y ago
Maybe just too early? Grant funders are starting to require teams to publish their code and data. Maybe they're the target audience? Data repos vendors could get on the list of approved vendors for teams receiving funding.

sjtindell 7y ago

Just want to say, really appreciate this food for thought. Where do you go and see someone say "my company tried it and...". This site is a godsend.

austinjp 7y ago
Agreed, it's always refreshing to read candid posts like this. A useful term for Google is "startup post mortem" although you still have to do plenty of sifting.

timsehn 7y ago

I'll check it out. We think the world is a little more ready for this now, given how widely Git is adopted and the advances in other data tooling (like ML). But, as we're all aware, starting a business is hard :-)

zyang 7y ago

The need is definitely there. My day job involves such need. But we simply cannot trust a drive-by startup to fill the gap. It's safer just to roll your own.

timsehn 7y ago
Dolt is and always will be open source. Feel free to fork a copy and make it your own.

philipov 7y ago

Git succeeded because it was free, and then business models were able to be built up around the open-source ecosystem after a market evolved naturally. There is a need, but if you go into it trying to build a business from scratch, you're going to have a bad time.

TylerE 7y ago
Git succeeded because of Linus. Sure as hell wasn't because of the UX, else Mercurial would have won, or even DARCS. 99.99999% of projects are not the Linux kernel
- sdan 7y ago
  I'd argue it succeeded (at least more recently) because of the UX that companies like Github and Gitlab gave, not particularly linus or because it was free.
  - TylerE 7y ago
    If Linus hasn’t pushed it, it never would have caught on.
    - bryanrasmussen 7y ago
      Sure lots of good things would never catch on except an influential person in the field the good thing targets says: hey this is really useful for us! This gets lots of people to look at it. But those people still have to decide in the end of the day whether it is actually useful for them.
      - sdan 7y ago
        Reminds me of sports: Just because Steph Curry uses Under Armor doesn't mean everyone will too. I think obviously something created by Linus was deemed to be of great value, but what made Git as profound as it is today is: UX for the majority of people who are beginners, which was Github backed by millions of other developers.

drewmol 7y ago

Do you consider your effort a mistake looking back? Were you able to pay yourself fairly or was it a losing investment?

peteforde 7y ago
The answer to that can only be nuanced. On one hand, startups are as exciting as bank heists. You put together an amazing team, do your homework as best you can, then get killed trying to execute your perfect plan. I'm proud of what we built and we all learned a lot the hard way. On the down-side, building a company is emotionally, physically, mentally exhausting. It wasn't really a matter of whether I could pay myself fairly; I drew a typical programmer salary along with everyone else. However, the important detail is that you're spending someone else's money and every penny of it represents someone you respect putting their trust in you, and you feel the weight of that every day. Ultimately, I don't exactly regret it but I certainly wish that we weren't so convincing that we convinced ourselves of a market opportunity that we couldn't access or didn't exist at all. There was so much heat for "data" in 2011 it really seemed like we just had to show up with an amazing product. We were wrong.
- dan00 7y ago
  > Ultimately, I don't exactly regret it but I certainly wish that we weren't so convincing that we convinced ourselves of a market opportunity that we couldn't access or didn't exist at all. How should you really know this without trying? If you aren't convinced, where should the motivation come from?
  - peteforde 7y ago
    Actually, I do have a pretty good answer to this. Every business has about 100 questions where if you know the answer to those questions, you are quite likely to be successful. The hard part is knowing which questions need to be asked of each business. To be clear: my co-founders and I were all seasoned, multi-startup people. We had extremely high-calibre investors with finely tuned bullshit meters. In the end, our own pitching skill undid us because I think less-convincing founders would have undergone 25% more scrutiny and that would have required us to demonstrate that we had signed LOIs from 3-5 real customers before starting. There was a subconscious misdirection around the fact that we didn't actually have anyone beating down our door. We let the excitement for "data", our personalities and our track records carry the moment. Of course founders have to drink their own Kool-Aid to some extent or they won't make it. But there's real power and value in the customer development mindset. People want this? Prove it.

ken 7y ago

That's GitHub for data. It's a service, and they still haven't launched anything yet. This is Git for data. It's a program, and it appears to be an open-source one you can download and use today.

pavlov 7y ago
The OP points to a site called DoltHub.com, so it’s not like they don’t have ambitions to commercialize as another “GitHub for data”.
- ken 7y ago
  Sure, but the existence of GitHub has only helped Git grow, not hurt it.
enos_feedler 7y ago
There is actually an old git for data project too: https://github.com/datproject/dat https://github.com/datproject/dat It's ~5 years old and I really wanted it to be huge. Hoping this new project is a success. Especially since I notice I went to high school with one of the founders of Dolt (Hey Tim!)
- visarga 7y ago
  Can it remove a file from the repo history? It's a GDPR feature that makes git hard to use for data.
- cbenz 7y ago
  Dat is more about distribution (décentralized, distributed, P2P) but it's not possible to make queries.
  - enos_feedler 7y ago
    Yes, the project used to associate itself as a git for data, but I guess not in the sense of a db.
- 7y ago

staflow 7y ago

1. Develop a solution with catchy name. 2. ONLY and only after it’s finished, start looking for problem that fits that solution. 3. Realize there is none. 4. ????? 5. Profit! Gee I wonder why our economy is so ineffective..

chmaynard 7y ago
Step 4 is where you create a snazzy website with lots of seductive images, clever animations, and "Free Trial" buttons.

kidintech 7y ago

Palantir is successful as well, but no clue what the hateboner on HN for them is, since it never even gets mentioned in these threads.

th3iedkid 7y ago
Is it because much of palantir success is with large enterprises and not startups?
- kidintech 7y ago
  OP says "we couldn't find anyone with an urgent problem that they were willing to pay to solve" and "there was no market to capture". Wouldn't a counter example contradict those statements, regardless of the size of Palantir's clients?

roystonvassey 7y ago

In our group we use git for code repos and cloud for storage and actual compute. It works seamlessly and git APIs work fantastically with almost any service, IDE or whatever your tool of choice. I suspect with the increasing cloud adaption, accessing data is getting easier by the day and I see no real need for a “git for data” tool. Plus, as a data scientist, it allows me to keep code and data separate, especially if I’m working with confidential data.

deforciant 7y ago

We also started "Git for data" several years ago but since then pivoted to data science/ML tooling (https://dotscience.com/ https://dotscience.com/) by building features that people actually want on the original product. Since then the "git for data" accounts only probably for 5% of the total functionality :) I guess "Git for data" is not very useful if you don't have the whole platform built around it to actually use the features. We mainly use it for data synchronization between the nodes and provenance tracking so people can see what data was used to build specific models and to track how the project evolves itself without forcing people to "commit" their changes manually (as we have seen that often data scientists don't even use git, just files on their Jupyter notebooks).

NicoJuicy 7y ago

> that there was not a market opportunity to capture Weird, there is an internal project ( already live) at the company ( multinational) where i work for that it's used.

solatic 7y ago

I wonder how much of the market opportunity here is contingent upon market education. Everybody can clearly see the value of having a personal automobile, but how successful can you be at selling automobiles to people who don't know how to drive? Do people desire cars enough to buy one if they don't know how to drive? Everyone can see how FAANG companies are growing wealthy off the mountains of data they are amassing, so everyone understands how data can be desirable. But what if your potential market base doesn't understand how to "drive" data - how to identify which data would be valuable for them and how best to exploit it? It seems to me that part of a go-to-market strategy needs, at least in the short term, to help potential customers transition from "that's a really shiny bauble" to "I understand how this is going to make me money."

peteforde 7y ago
Maybe, but also maybe there just isn't a huge demographic of data scientists with discretionary purchasing capacity and a hair-on-fire problem that they are desperately searching for someone to take their money and fix. I think that a lot of the data VIP types we met with honestly wanted to know why they needed it, but the more they thought about it, the more it just seemed like a shiny thing. It's telling that dozens of similar companies with smart people behind them have thrown their talents at this solution, and none of them have located the problem people are eager to pay to solve.

dmitryminkovsky 7y ago

You may be right in the big picture, but not all "GitHubs for data" are the same. This product seems super cool: > Dolt is Git for data. Instead of versioning files, Dolt versions tables. DoltHub is a place on the internet to share Dolt repositories. As far as we can tell, Dolt is the only database with branches. I find it hard to believe no other database has branches, but if that's true and if this product works like you'd imagine, that is really cool. Given your historical observation, I think you're right that this will not lead to a market revolution, but sometimes you need the right product to change the landscape.

nailer 7y ago

> we were part of a flock of 5 or 10 similar projects and I'm pretty sure the only one still around today is Kaggle. Dat is still around, on version 14, last update in git was 5 days ago. https://github.com/datproject/dat https://github.com/datproject/dat https://dat.foundation/ https://dat.foundation/

moondowner 7y ago
Impressed by the documentation https://datprotocol.github.io/how-dat-works/ https://datprotocol.github.io/how-dat-works/

specialist 7y ago

Maybe just too early? Grant funders are starting to require teams to publish their code and data. Maybe they're the target audience? Data repos vendors could get on the list of approved vendors for teams receiving funding.