5 ms·

Here's an unpopular answer - don't store 10PB of data. Find a way for your startup to work without needlessly having to store insane amounts of data that will l

by sgt 6y ago

Here's an unpopular answer - don't store 10PB of data. Find a way for your startup to work without needlessly having to store insane amounts of data that will likely never be needed.

boffinism 6y ago
They're a data storage service.
asadlionpk 6y ago
Excellent advice for a data backup startup.
- CyberDildonics 6y ago
  Doesn't this imply that they started a company without actually having a plan for the most fundamental part of what they are selling? This is like an ISP asking how they can get hooked up to the internet.
  - asadlionpk 6y ago
    No. They are already managing 10PB, planning for which would be very stupid when just starting up.
    - CyberDildonics 6y ago
      Why would planning to be able to execute the single focus of your startup be stupid?
      - spookthesunset 6y ago
        I tend to agree. If you are a storage company, I’d think that part of your secret sauce should be how to store tens or hundreds of petabytes of customer backups economically. Maybe I’m wrong though. Perhaps the real secret sauce is the end user experience and the kind of storage you use on the backend doesn’t matter at all. However I bet that the “cloud storage space” is pretty crowded and lots of people shop on price more than anything. If your business model is all about price, then finding economical storage is critical to your company and needs to be part of your core competency. If price isn’t that important, perhaps it doesn’t matter... the “winners” would win no matter how expensive their storage solution is. But honestly.... I feel like part of your core competency needs to be managing the storage system.
      - lmeyerov 6y ago
        A rule of thumb for performance is every 10-100X involves changing up your fundamentals It's a bit different nowadays that a lot of scaling tech is commoditized, but still means things like negotiating new contracts, finding & fixing the odd pieces that weren't stressed before, etc. (congrats on hitting the new usage levels + good luck! we're at a much smaller scale, but trying to figure out some similar questions for stuff like web-scale publishing of data journalism without dying on egress $, so it's an interesting thread...)
      - matwood 6y ago
        I also tend to agree. I think AWS is great and use it as my default solution, but if I was starting a company that had high bandwidth and/or storage requirements, I would be looking for other solutions from day one.
  - qeternity 6y ago
    It's more like a company that has validated product fit now needing to figure out how to scale economically. Apple didn't start manufacturing with mega Foxconn contracts. They had to figure that out along the way as their scale demanded. However I share your sentiment: doing things the same way but cheaper is usually not the solution. Doing things differently (in-sourcing) might be the path forward.
    - CyberDildonics 6y ago
      Figuring out how to scale is not just a part of a storage startup, it's the whole thing. Apple created something people wanted and sold at a price that would still make money if it was assembled by hand. They didn't form a company around a commodity like data storage. Data storage is a commodity. Everyone already has some, online storage companies already exist. If you don't know how to store a lot of data and your company's whole purpose is to store a lot of data, it sounds like something that should have been worked out before making the company.
philippb 6y ago
If you can solve that for me without affecting revenue I have $1m in cash for you right there. We are a photo/video storage service.
- lostcolony 6y ago
  So from a completely evil (well, capitalist) perspective, do you have data on how often people retrieve backups, and at what 'age' they do so? Because there may be an inflection point that offering monetary compensation for data loss, rather than actually trying to store the data, would make more financial sense. I.e., "All data > than 2 years gets silently expunged, and anyone trying to retrieve it at that point gets $10 per gig in compensation for 'our mistake'". Please don't actually consider that though.
  - lostcolony 6y ago
    (A less evil approach that might still lead to reduced costs would be detecting old, unaccessed data of sufficient size and flagging it for users, with a small refund or service discount if they purge it. Though that assumes you have 'power users' who are storing massive amounts, to where the savings in storage costs would be worth it) (And if you don't already, I would also consider making it so items that are in the trash for some period of time, say 30 days, get deleted automatically as well, possibly with a reminder email a few days before) (And lastly, depending on user profiles and usage, incentives around reducing resolution/quality of photos and video, and automating that in the app as part of the sync process, might provide some opportunities to reduce costs of storage > the lost revenue of cheaper plans.)
  - european321 6y ago
    What about storing the older things in some slower backup service that's slow to access but cheap? If the user eventually accesses those some day you would kick of some background job to get them fresh again. Not super UX friendly of course, but could reduce costs. Or is this already the standard thing to do?
  - saalweachter 6y ago
    "We used advanced machine learning algorithms to predict which users will need to retrieve which pieces of data in the future, and silently delete everything else."
hansvm 6y ago
You could probably put a fun marketing spin on that. "We use ML to ensure we only store the highest quality data, freeing you from the chains of having too much worthless data and nothing to do with it."
- qeternity 6y ago
  GPT-3 that convinces you why you don't need to store this thing
  - saalweachter 6y ago
    Image processing to label each image ("baby with spaghetti on head", "cat playing with string", "naked person"), and then only save one image with each label.
    - giomasce 6y ago
      Just save the label. Then you use generative techniques when they want to retrieve the image.

by sgt 6y ago

Here's an unpopular answer - don't store 10PB of data. Find a way for your startup to work without needlessly having to store insane amounts of data that will likely never be needed.

boffinism 6y ago

They're a data storage service.

asadlionpk 6y ago

Excellent advice for a data backup startup.

CyberDildonics 6y ago
Doesn't this imply that they started a company without actually having a plan for the most fundamental part of what they are selling? This is like an ISP asking how they can get hooked up to the internet.
- asadlionpk 6y ago
  No. They are already managing 10PB, planning for which would be very stupid when just starting up.
  - CyberDildonics 6y ago
    Why would planning to be able to execute the single focus of your startup be stupid?
    - spookthesunset 6y ago
      I tend to agree. If you are a storage company, I’d think that part of your secret sauce should be how to store tens or hundreds of petabytes of customer backups economically. Maybe I’m wrong though. Perhaps the real secret sauce is the end user experience and the kind of storage you use on the backend doesn’t matter at all. However I bet that the “cloud storage space” is pretty crowded and lots of people shop on price more than anything. If your business model is all about price, then finding economical storage is critical to your company and needs to be part of your core competency. If price isn’t that important, perhaps it doesn’t matter... the “winners” would win no matter how expensive their storage solution is. But honestly.... I feel like part of your core competency needs to be managing the storage system.
    - lmeyerov 6y ago
      A rule of thumb for performance is every 10-100X involves changing up your fundamentals It's a bit different nowadays that a lot of scaling tech is commoditized, but still means things like negotiating new contracts, finding & fixing the odd pieces that weren't stressed before, etc. (congrats on hitting the new usage levels + good luck! we're at a much smaller scale, but trying to figure out some similar questions for stuff like web-scale publishing of data journalism without dying on egress $, so it's an interesting thread...)
    - matwood 6y ago
      I also tend to agree. I think AWS is great and use it as my default solution, but if I was starting a company that had high bandwidth and/or storage requirements, I would be looking for other solutions from day one.
- qeternity 6y ago
  It's more like a company that has validated product fit now needing to figure out how to scale economically. Apple didn't start manufacturing with mega Foxconn contracts. They had to figure that out along the way as their scale demanded. However I share your sentiment: doing things the same way but cheaper is usually not the solution. Doing things differently (in-sourcing) might be the path forward.
  - CyberDildonics 6y ago
    Figuring out how to scale is not just a part of a storage startup, it's the whole thing. Apple created something people wanted and sold at a price that would still make money if it was assembled by hand. They didn't form a company around a commodity like data storage. Data storage is a commodity. Everyone already has some, online storage companies already exist. If you don't know how to store a lot of data and your company's whole purpose is to store a lot of data, it sounds like something that should have been worked out before making the company.

philippb 6y ago

If you can solve that for me without affecting revenue I have $1m in cash for you right there. We are a photo/video storage service.

lostcolony 6y ago
So from a completely evil (well, capitalist) perspective, do you have data on how often people retrieve backups, and at what 'age' they do so? Because there may be an inflection point that offering monetary compensation for data loss, rather than actually trying to store the data, would make more financial sense. I.e., "All data > than 2 years gets silently expunged, and anyone trying to retrieve it at that point gets $10 per gig in compensation for 'our mistake'". Please don't actually consider that though.
- lostcolony 6y ago
  (A less evil approach that might still lead to reduced costs would be detecting old, unaccessed data of sufficient size and flagging it for users, with a small refund or service discount if they purge it. Though that assumes you have 'power users' who are storing massive amounts, to where the savings in storage costs would be worth it) (And if you don't already, I would also consider making it so items that are in the trash for some period of time, say 30 days, get deleted automatically as well, possibly with a reminder email a few days before) (And lastly, depending on user profiles and usage, incentives around reducing resolution/quality of photos and video, and automating that in the app as part of the sync process, might provide some opportunities to reduce costs of storage > the lost revenue of cheaper plans.)
- european321 6y ago
  What about storing the older things in some slower backup service that's slow to access but cheap? If the user eventually accesses those some day you would kick of some background job to get them fresh again. Not super UX friendly of course, but could reduce costs. Or is this already the standard thing to do?
- saalweachter 6y ago
  "We used advanced machine learning algorithms to predict which users will need to retrieve which pieces of data in the future, and silently delete everything else."

hansvm 6y ago

You could probably put a fun marketing spin on that. "We use ML to ensure we only store the highest quality data, freeing you from the chains of having too much worthless data and nothing to do with it."

qeternity 6y ago
GPT-3 that convinces you why you don't need to store this thing
- saalweachter 6y ago
  Image processing to label each image ("baby with spaghetti on head", "cat playing with string", "naked person"), and then only save one image with each label.
  - giomasce 6y ago
    Just save the label. Then you use generative techniques when they want to retrieve the image.