4 ms·

I worked for a storage startup in the South Bay in the late 00s which did something like this. After staff crunched to deliver the base operating system and ker

I worked for a storage startup in the South Bay in the late 00s which did something like this. After staff crunched to deliver the base operating system and kernel controllers, management gave across-the-board poor reviews to all IC staff to justify no raise. Those with the ability quickly left for greener pastures; Google hired some of the best. Only those who had to (work visas, lack of experience, etc.) remained. The product, being a v0.1 release implemented in C, was of course plagued with data corruption and poor performance. It read /etc/password from disk on every file access. An uninitialized int caused indirect inode corruption every ~millionth write. And so on. Because of the poor product quality, they lost multiple opportunities, including a bid for early Facebook photo storage. The company never had more than 250 customers and ended up being sold to a Valley stalwart. The CEO and VP Eng were paid out handsomely, of course; preferred converted stock tranches or whatever. My options from 4 years of employment ended up worth $500.

sterlind 3y ago
Sounds like the Igneous playbook, from what I've heard. Immediately after launch they laid off a majority of the dev team, as they'd outlived their usefulness after completing the crunch. I interviewed there but fortunately declined the offer - I felt like I dodged a bullet when my friend (who survived the layoffs, only to be conscripted as a sales grunt) told me about it. I also gather that the CEO had pulled similar shenanigans with Isilon in the past. If it's any consolation, Igneous died and the execs didn't make out like bandits this time.
andsoitis 3y ago
”The product, being a v0.1 release implemented in C, was of course plagued with data corruption and poor performance. It read /etc/password from disk on every file access. An uninitialized int caused indirect inode corruption every ~millionth write. And so on. Because of the poor product quality, they lost multiple opportunities, …” It sounds like the poor reviews for the ICs was warranted, wouldn’t you say?
- smsm42 3y ago
  Not necessarily. Big codebase in something like C, written in a rush, pretty much is guaranteed to have such things. We still are finding high-grade security issues in glibc (if you want some library to be clean, it's the one practically everybody is using), and it has been around for 37 years. v0.1 prototype is often plagued with various issues and often doesn't have best performance. You either spend years on polishing it and never release, of you start with v0.1 and fix the issues. Or, having fired all the dev team, don't fix the issues.
- sangnoir 3y ago
  > It sounds like the poor reviews for the ICs was warranted, wouldn’t you say How many crunches have you survived that resulted in a high-quality initial product? If there is enough time to get things right, it wouldn't be a crunch, by definition.
foobiekr 3y ago
Smells like Igneous or TinTri. Story shows too much competence for Springpath or Whiptail. Storage is particularly rife with stuff like this for some reason.
- ericbarrett 3y ago
  Was a long time ago, way before them. I agree though, glad to have changed fields.
  - foobiekr 3y ago
    You would be pleased to know that Whiptail exceeded whatever low bar you might have had. I like to see people achieve the things they set out to do, and I am assuming that was their goal. Rare to see a product completely canceled and removed from the market as soon as customers started reporting minor things like total data loss due to corruption..
gspetr 3y ago
This makes me even more incredulous than the GP commenter. How did people who "read /etc/password from disk on every file access." get hired by Google? Or perhaps the right question is for which roles, because with that level of DS&A they shouldn't have passed Google's DS&A whiteboard interview.
- ericbarrett 3y ago
  I think you give the whiteboard far too much credit. In the succeeding two decades I have worked at many companies filled with past, present, and future FAANG engineers, and I can definitely say knowledge of algorithms isn’t a defense against writing silly bugs when implementing a complex, novel project from scratch. Just a dumb oversight that would have been caught quickly by user reports—had all the engineers who cared not quit from misanthropic management. There were many such issues, and I only remember these two because I found them myself. I was in support and only had two good resources for tracking this kind of stuff down, a burnt out but kind SCSI specialist and a brilliant, massively overworked Russian kernel dev on visa. They both lasted an extra year. Bless them both for answering my pestering questions, I learned a lot. Good times, but all this was about a year too late to make a difference, then the Great Recession hit and it was done.
- me_me_me 3y ago
  google hires 27k devs (as per google search) Do you honestly think all 27k devs are 10x certified geniuses?
bitcharmer 3y ago
> The CEO and VP Eng were paid out handsomely, of course It's the prevalence of scenarios like this that makes me hate the MBA caste with a pasison

sterlind 3y ago

Sounds like the Igneous playbook, from what I've heard. Immediately after launch they laid off a majority of the dev team, as they'd outlived their usefulness after completing the crunch. I interviewed there but fortunately declined the offer - I felt like I dodged a bullet when my friend (who survived the layoffs, only to be conscripted as a sales grunt) told me about it. I also gather that the CEO had pulled similar shenanigans with Isilon in the past. If it's any consolation, Igneous died and the execs didn't make out like bandits this time.

andsoitis 3y ago

”The product, being a v0.1 release implemented in C, was of course plagued with data corruption and poor performance. It read /etc/password from disk on every file access. An uninitialized int caused indirect inode corruption every ~millionth write. And so on. Because of the poor product quality, they lost multiple opportunities, …” It sounds like the poor reviews for the ICs was warranted, wouldn’t you say?

smsm42 3y ago
Not necessarily. Big codebase in something like C, written in a rush, pretty much is guaranteed to have such things. We still are finding high-grade security issues in glibc (if you want some library to be clean, it's the one practically everybody is using), and it has been around for 37 years. v0.1 prototype is often plagued with various issues and often doesn't have best performance. You either spend years on polishing it and never release, of you start with v0.1 and fix the issues. Or, having fired all the dev team, don't fix the issues.
sangnoir 3y ago
> It sounds like the poor reviews for the ICs was warranted, wouldn’t you say How many crunches have you survived that resulted in a high-quality initial product? If there is enough time to get things right, it wouldn't be a crunch, by definition.

foobiekr 3y ago

Smells like Igneous or TinTri. Story shows too much competence for Springpath or Whiptail. Storage is particularly rife with stuff like this for some reason.

ericbarrett 3y ago
Was a long time ago, way before them. I agree though, glad to have changed fields.
- foobiekr 3y ago
  You would be pleased to know that Whiptail exceeded whatever low bar you might have had. I like to see people achieve the things they set out to do, and I am assuming that was their goal. Rare to see a product completely canceled and removed from the market as soon as customers started reporting minor things like total data loss due to corruption..

gspetr 3y ago

This makes me even more incredulous than the GP commenter. How did people who "read /etc/password from disk on every file access." get hired by Google? Or perhaps the right question is for which roles, because with that level of DS&A they shouldn't have passed Google's DS&A whiteboard interview.

ericbarrett 3y ago
I think you give the whiteboard far too much credit. In the succeeding two decades I have worked at many companies filled with past, present, and future FAANG engineers, and I can definitely say knowledge of algorithms isn’t a defense against writing silly bugs when implementing a complex, novel project from scratch. Just a dumb oversight that would have been caught quickly by user reports—had all the engineers who cared not quit from misanthropic management. There were many such issues, and I only remember these two because I found them myself. I was in support and only had two good resources for tracking this kind of stuff down, a burnt out but kind SCSI specialist and a brilliant, massively overworked Russian kernel dev on visa. They both lasted an extra year. Bless them both for answering my pestering questions, I learned a lot. Good times, but all this was about a year too late to make a difference, then the Great Recession hit and it was done.
me_me_me 3y ago
google hires 27k devs (as per google search) Do you honestly think all 27k devs are 10x certified geniuses?

bitcharmer 3y ago

> The CEO and VP Eng were paid out handsomely, of course It's the prevalence of scenarios like this that makes me hate the MBA caste with a pasison