25 ms·
What it was like working for Gitlab
- RyeCombinator 3y ago".. also had a tendency to listen more to complaints on Hacker News than internal complaints" This all sounds too familiar.
- sevagh 3y ago>Location based salaries are discriminatory This one seems to be contentious in the comments section but I've had issues with this before. At the end of the day it's a balance between accepting that reality involves a lot of factors that result in a Canadian coworker making half the pay of their U.S. colleagues, and chafing at the injustice of the same fact. I've not been able to be reasonable about this issue. If corp A pays me X and pays somebody else 2*X, I will always, always have a discomfort around this setup (even if X is still great).
- ganarajpr 3y agoCompanies attempting to pay an engineer according to location, in my opinion, is another kind of discrimination. You are supposed to pay an employee based on his/her abilities, not her location. We have rules in govts that companies should not discriminate against employees based on sex, religion, sexual orientation etc etc.. But it is fair to discriminate the salary of an employee based on location? For ex: I know a few friends who have moved from Europe to Asia with the same role and are getting paid less compared to what they were getting paid in Europe. Its the same role, its the same person, but getting paid less just because of location ?
- jblox 3y agoYeah I absolutely hate that. I get why they do it from a business point of view but as an engineer, I'm instantly put off when I see something along the lines of "up to $x (depending on location)".
- hawk_ 3y agoDiscrimination is around things that an individual can't choose (religion being the weird elephant in the room). Fair or not, this isn't discrimination.
- thewakalix 3y agoI don't think that rule holds in general. For another example besides religion, you can choose whether or not to marry interracially.
- simonbarker87 3y agoMany people can’t choose where they live either. Getting a visa to the US is a ludicrous process and even if they wanted to they maybe tied by family obligations.
- hawk_ 3y agoBut people don't choose to be rich or poor exactly either. As a general rule, discrimination is around things for which there's no choice. Having a choice over where they live or whether they are rich doesn't mean it's easy or practical. But that can't make it grounds for discrimination, even if unfair.
- oldkinglog 3y agoIn the UK, the Equality Act (2010) protects: age, disability, gender reassignment, marriage or civil partnership (in employment only), pregnancy and maternity, race, religion or belief, sex, sexual orientation. Pregnancy, maternity, marriage, civil partnerships and gender re-assignment are usually chosen by individuals, not forced upon them.
- Scarblac 3y agoDisagree on gender re-assignment, people don't choose to have gender dysphoria just like they don't choose sexual orientation.
- 3y ago
- patcon 3y agoI disagree. I believe a company should pay based on ability of employees to have comfort and wellness, not some universal measure of value (which I believe to be impossible). What you are advocating for inadvertently breaks community and exacerbates gentrification and destruction of social fabric via inequality. Location matters.
- ta8645 3y ago> You are supposed to pay an employee based on his/her abilities, not her location. I don't believe that is a legal requirement, anywhere. Remuneration is based on many factors, which can include the cost of living. A company will not be able to hire someone in New York City, for the same price as someone in a less expensive jurisdiction. This isn't discrimination, it's simple economic reality.
- samsonradu 3y agoIt s based on suply and demand indeed. The legal requirements might affect minimum pay.
- hw 3y ago> Companies attempting to pay an engineer according to location, in my opinion, is another kind of discrimination. You are supposed to pay an employee based on his/her abilities, not her location. So you’re saying, we should be paying engineers in Europe and in the US the same as an engineer in LATAM or India or Asia that has the same level of experience and skill. The only way to be non discriminatory is to have a standardized formula of compensation that takes into account cost of living (rent, food, healthcare, taxes etc) where the final take home pay in locations around the world are equivalent - which I believe should be the case at most companies
- mjr00 3y agoA worker in country X or country Y are very different for companies' balance sheets. For instance, my company is Canadian, and we are eligible for significant tax credits through SR&ED[0] for software developers. If a software developer moved their permanent residence to outside Canada, even if we could magically pin exchange rates to pay them the CAD equivalent in local currency, it would be a significantly different financial impact on the company as they aren't eligible for that program. I'm not an expert, but I imagine there are many equivalent programs in every country, state, and even municipality. It works both ways, anyway. If those friends had moved from Thailand to Switzerland, would it be discrimination to pay them more? [0] https://www.canada.ca/en/revenue-agency/services/scientific-research-experimental-development-tax-incentive-program.html https://www.canada.ca/en/revenue-agency/services/scientific-...
- supafastcoder 3y ago> A worker in country X or country Y are very different for companies' balance sheets Yes, but quite often, workers are in the same country (or even same state!) and still get paid differently based on CoL.
- neoberg 3y agoI agree with this in theory but I can't see how it will work in practice. There isn't a global "value of ability" to base the pay on. It's valued differently in every location.
- ako 3y agoWhy are you supposed to pay based on abilities? Where is that stated? As a company you need certain abilities, and you pay whatever the market decides these abilities are worth, and nothing more. Depending on location, the market will set a different price on these abilities, so you pay different.
- randomdata 3y ago> You are supposed to pay an employee based on his/her abilities, not her location. You are supposed to pay them the minimum amount it takes to get them to show up to work. When someone moves to a less competitive market, where getting another job is harder, then they are more likely to show up for lesser pay. And remember that a country may have a less competitive market, even if the workers are remote and not seemly bound by a local market, because governments often love to put up huge roadblocks when it comes to international hiring. If you are being paid less than you were in another country doing the same job for the same employer, this is almost certainly why you have agreed to take a pay cut.
- stavros 3y agoExactly. This isn't a "cost of living" adjustment, it's a "we're lucky you have fewer options, so we don't have to pay as much" adjustment. If I get hired in such a company, I'm moving to SF or Zurich the next day.
- myaccountonhn 3y agoHow would you move there without a visa?
- verve_rat 3y agoFemales generally get paid less for the same work that males do. If someone transitions male -> female should they get a pay cut?
- randomdata 3y agoThat’s up to them. It’s the worker who chooses how much it takes to show up. I suppose if they want to truly play the part of being female they may want to accept less.
- bigstrat2003 3y agoAlso, the gender pay gap is a myth so there's no reason to consider it in the first place.
- ako 3y agoYou could also argue that difference in pay is less discriminatory. You are paying employees to have the same quality of life, same type of housing, same opportunity to provide for family, send your kids to the same type of schooling. These things cost differently in different countries, so require different income.
- DandyDev 3y agoExactly this! Location-based pay is not so much about cost of living as it is about buying power. In the end, money is just a place holder for real value which comes in the form of goods and services. And the real value of the same amount of dollars wildly differs per location. So if you want to pay fairly and not discriminate, you have to try and make sure people can roughly buy the same things in their differing locations for the money you give them.
- drewcoo 3y agoWhile I'm sure it's very kind of companies to care about my quality of life and the type of my housing, it's honestly none of their business. Even if they tell me I'm "family." Fair pay to me, at least, means paying for results. Not paying for hours spent toiling. Not paying for where I am on the planet. Not paying for how I get the results, just for the results. Instead, there are all of these gamey factors inserted into the mix. They're emotional. They're manipulative. Yuck!
- ako 3y agoThat’s not how free markets/capitalism is supposed to work. Companies are expected to just shop around for the lowest price on the capabilities they need. Are you suggesting we should adopt something else then capitalism/free markets?
- filleokus 3y ago> You are supposed to pay an employee based on his/her abilities, not her location. I don't think this make any sense on so many levels. First, "abilities" are not a good way to think about wages. If you hire a neurosurgeon to do your gardening, you won't pay them more than a run of the mill gardener. Rather, you as the employer compete against other employers on different markets in a fairly classical supply and demand situation. The "abilities" of an compliance expert with tech skills did not change much when GDPR was introduced, but as all EU companies scrambled to figure out the regulation (and the DPO role was popularised by fiat), the compensation went way up. If the employee can participate in e.g the SF labour market, you have to pay a competitive salary in that market, if not you don't have to. As long as there are barriers, e.g a on-location worker in SF has more opportunities for whatever reason, the location premium makes sense. To take your example in the opposite direction. Let's say a east-european company want to expand into the US and open up a sales engineering office in SF, and want their best sales engineers to go work their, it would be completely insane to not raise their wage. "We pay people after ability here you have 40k USD, have fun finding housing".
- silisili 3y ago> Location based salaries are discriminatory I used to feel this way, but largely grown out of it. In the end, you're asking for SF salaries, and those in India are asking for NL salaries(or SF salaries). You'd largely find a race to the bottom, I think. The simple fact is you made a good living for your area, and they made a good living for theirs. If you want more, move. If you won't move, there's likely a reason why. Everyone wants a SF salary with an Indiana or even India cost of living. For obvious reasons, that can't work.
- rldjbpin 3y agothe issue comes when the pay does not match the living standards. despite having the least in absolute terms, it is possible to have a great life with the mid-to-high level income in IT there. can't say the same with what most people earn in europe.
- gbil 3y agoBig discussion, let me touch another point >It doesn't matter whether you're paying somebody in the Bay Area $100 000 per year, or somebody in the Philippines, because the cost for you as a business is the same. The cost for you as business is NOT the same. Start from the taxes part, that 100k is what the employee gets but the cost for the business is higher depending on the country/area because they also pay tax on top, social benefits etc. Also need to have in many cases a local business, doesn't matter if that is virtual etc, since they need to adhere to local laws thus having resources supporting that etc. So while I'm on the employee side here as I'm also working on a multinational coorp with global role yet paid with local standards, there is more than meets the eye
- mjr00 3y agoyep. People vastly underestimate how difficult the logistics of "pay an employee for services" can be, particularly when you don't have a legal corporate presence in the country where the employee resides. There are services that handle this for you and they charge a 30-40% premium on top of the employee's salary. And sometimes this is still worth it, because many countries charge an absolutely obscene incorporation fee for foreign-owned businesses.
- rockyj 3y agoRuby's entire programming model was based on the premise that language speed does not matter since most of the time you are waiting on IO / Network. Well now, both on Node.js and JVM we have programming models which say that - while I wait for IO / Network let me do some other work or service more requests (Async / Webflux / Coroutines). IMHO - using Ruby/Rails in 2024 can be wasteful, but of-couse for the right situation it can be a good choice. (Just) For example an enterprise app where you know the number of users will be limited, or when you know the development speed is paramount or where you want to build a quick proptype to test the market out. Rails is a great framework and the productivity is unmatched, but with time a 2-3 years old Rails project is always tricky to maintain.
- hw 3y agoRuby now has Fibers and other constructs for async like Ractor. > Rails is a great framework and the productivity is unmatched, but with time a 2-3 years old Rails project is always tricky to maintain. It isn’t any less tricky than a Django or Express project. With any codebase discipline and regular tending to the garden of code is important to prevent weeds from growing. If anything that Ruby (and Rails) has going against it is still the raw language performance and higher memory requirements/usage than its counterparts, which makes it less desirable for workloads that require low latency or small memory footprint
- sosodev 3y agoMeh, it's fine for 99% of use cases. Rails only becomes a pain in my experience if you have random, huge swings in traffic or a ton of active users with websocket stuff (mostly solved by anycable). But very few products are going to reach those limits and if they do those problems can be solved too. Stripe, GitHub, and Shopify are the shining examples of Rails scaling in all kinds of ways without problem.
- jb3689 3y agoStripe doesn't use Rails. Stripe's application of Ruby is far removed from your typical Rails app. A lot of custom stuff was built into Ruby to make the multi-million line codebase work as well as it does.
- jurgenaut23 3y ago> Location based salaries are discriminatory The article is nice and well articulated, but I'd argue exactly the opposite on this point. The author seems to confuse equality and equity. He takes the Netherlands and Bay Area as an example and don't make a compelling argument as to how and why paying differently those employees is akin to discrimination based on race. Were you to pay the same salary in Netherlands and Bay Area, you would effectively either (i) pay absurdly high salaries to your dutch engineers, similar to what the prime minister earns or (ii) pay absurdly low salaries to your Bay Area engineers (to the point that you'd basically have none of them).
- XiS 3y agoWhat about (iii) pay a salary that would be in between absurdly high and absurdly low?
- vouwfietsman 3y agoBottom line you want the best people, which are spread around the globe, but within each country is a different price point for what you pay the best people. Bidding too high is wasteful to your resources, too low makes it impossible to compete in higher pay areas, and middle ground is the worst of both worlds: loose out on high pay areas, be wasteful in low pay areas. I hope future generations will better understand that accumulating wealth should not be the goal of life.
- ksplicer 3y agoMaybe future generations will not have to worry as much about accumulating wealth when countries work harder to ensure a high quality of life in the middle class. I doubt that will happen for the west until they figure out how to relieve all the pressure the housing market is putting on their citizens.
- globular-toast 3y agoWho cares if you don't have Bay Area engineers? Their ability isn't based on where they live, is it? If they want to lower their costs of living so they can compete they can "just move".
- karolist 3y agoMy conclusion after reading this is never give it your all unless it's your own company. Burnout after working for someone else in tech should not happen at all. The post overall has negative undertones so it feels like the burnout was really bad after all this time, I hope the author recovers and finds joy and peace working on their own projects.
- imafish 3y agoAlso, from my understanding, the burnout did not seem to stem from the actual work but more from interpersonal relations and disagreements. I am certain that is not uncommon
- redrove 3y agoRarely was my burnout caused by actual work. It’s usually caused by management being asshats.
- CipherThrowaway 3y agoThis is what burnout is. Burnout is not caused by working, but by stress. The root cause of this stress is personal and interpersonal factors. People can burn out on very small workloads if the ratio of interpersonal toxicity to hours of work is high enough. Sometimes it looks like people are burning out because of extreme workloads. In these situations, I've always asked "why is the workload so extreme to begin with?" This gets you to the real source of stress.
- karolist 3y agoIt's amazing how many issues in life are caused by the act of thinking of these possible issues, than the issues themselves. My friend had horrible issues sleeping, he tried many things before going to psychiatrist. The conclusion was quick and accurate - your issues sleeping are caused by your fear of not falling asleep, basically anxiety. Anti-anxiety meds fixed it immediately.
- jacquesm 3y ago
- usr1106 3y agoRegarding gitlab performance: I moved from using bitbucket, first own servers and later SaaS to gitlab.com SaaS in early 2018. I cannot say anything else than gitlab performance and UI being much better. I was permanently complaining about bitbucket, but very happy with gitlab most of the time. They had rather frequent partial outages, but resolution was always quick. Maybe starting 1 year ago I noticed gitlab.com reponse times going down. It's not that things hang or are super slow, but they seem to consistently take 2 seconds more than what they used to. (The number is a complete guess, I have no measurements either before or after.) It's still usable, but no longer the pleasant experience it used to be.
- usr1106 3y agoAnd following up to myself: Being able to work from where I happen to live and I have thought about whether I would like to work for them. That it's a good product would certainly be a motivation. However, frequent incidents being resolved fast must be incredibly stressful for the staff. I prefer to stay where we make heavy releases only a few times a year for regulatory reasons. (Even though stress levels can raise until the last MR for the release has been merged.)
- sph 3y agoGreat retrospective, and apart from the personal burnout story, could be titled "what happens to a startup when managers start to run the show"
- hassmo 3y agoAs opposed to what, engineers running the show?
- deleted 3y ago[deleted]
- DandyDev 3y agoI read this meme so often on HN, so I genuinely want to know: how do you organize work and get people to act on a common vision if there are no managers? I think people mistake “manager” as being people without tech skills. I think a tech company should have managers to create alignment and those managers should spend part of their time “managing” and part of their time still honing their tech skills.
- gtirloni 3y agoIn my experience, people will use "managers" as a proxy for complaining about all the other things they don't like. Of course clueless managers and extreme bureaucracy will destroy a company, but I very often see engineers that just want to do X and ignore everything around it as if they were working in a isolated environment. It's often the managers calling these engineers back to reality, which creates tension.
- sph 3y agoThe issue is you literally cannot be an engineer if you don't know how to build things. On the other hand, there are plenty of managers that do not know how to manage people, that do not even know what is expected of them. Should they optimize for cost? For productivity? For employee happiness? Competent managers are a minority. They tend to rise quickly, leaving the inept ones to run the day-to-day.
- jumploops 3y ago> The time it takes to deploy code is vital to the success of an organization I've found this to be one of the most important metrics (if not the most important) for maintaining developer velocity as a startup scales. Context switching is one of the most expensive operations in a developer's day-to-day work, and also one of the most ignored. Managers love to build team schedules around their personal schedules, but this is often disruptive to ICs' "maker schedules." Many orgs identify this and focus on the scheduling cost w.r.t. context switching, but the build-to-deploy time is equally (if not more) important for developer ICs.
- mkl95 3y agoI recently spent a few minutes applying to a Gitlab position, only to get a link to some Gitlab wiki explaining the list of EU countries they hire at is limited, and mine isn't among them. I wish whoever posted the ad had spent a few seconds copying and pasting that list from their wiki.
- csallen 3y ago> No matter how you try to spin this, it's by all accounts an act of discrimination to pay one person less than another purely based on where they live.. Companies aren't gods charged with dispensing fair wages to us mortals. They are participants in the capitalist market. And in that market, all prices are a matter of negotiation. In any transaction, the buyer wants to pay as little as possible, and the seller wants to earn as much as possible. This isn't evil or greedy. It's a basic incentive that comes from the fact that people/companies/literally everyone prefers to have more money than less money. Thus, your company wants to pay you as little money as possible, and you want them to pay you as much as possible. The same is true when the roles are reversed and you're the one in the buyer position. For example, when you're shopping, you want to pay as low a price as possible, and the seller wants to charge as high a price as possible. Again, all prices are a matter of negotiation.. Power in a negotiation comes down to your BATNA, i.e. your best alternative if negotiation fails. Put simply, whoever cares the least has the most power. In a capitalist market, that BATNA for the buyer is the next cheapest option. The next furniture store that's selling this couch for cheaper. The next adequately qualified software engineer in the Bay Area who's selling their services for cheaper. And the BATNA for the seller is obviously the next most generous spender. The customer who's willing to pay a little more. The company that's willing to pay higher salaries. If it sounds like I'm conflating two concepts by comparing customers with employers, and businesses with employees, it's because you've been brainwashed into thinking these things are different. They're not. When money changes hands, whoever receives it is the business. Whoever pays it is the customer. That means, in the relationship between you and your employer, you are the business selling a service, and they are a customer paying for your service. Your salary is just a price. Your resume was an advertisement. Posting it everywhere was marketing. Interviewing was your sales pitch. As an employee, you are a business. (Someone just came along and renamed all the terms to confuse you.) Which raises the question: Why on earth would your customer voluntarily pay a higher price than they need to? If all the stores in your area are cheap, why would they pay you more for your services than your neighbor is asking for? When you travel to a country cheaper than yours, do you pay every restaurant, store, and taxi driver the same way you would at home? No, you don't. Because that would be foolish. You pay local rates, which are less. That's the same thing companies are doing when they pay lower salaries in cheaper areas.
- cousin_it 3y agoSo, he joined as employee #28 reporting directly to the CEO, then gradually more managers were slotted in above him, then one of the managers put him on a PEP, then he got the hint and left. I wanna say this story is pretty standard, and probably a big part of the reason why people in big companies often don't do great things. Heck, even if you start a project within a company and it gets successful, the company can just slot in people above you, so you become employee #n in a project that you started, and then these people can say you underperform and so on.
- spense 3y agoare you suggesting a better way?
- nip 3y agoManagement is mostly needed for two things [*]: - Organising the work and steering it in the right direction - Ensuring that people work well together, help them grow, deal with "people problems" If and when both of the above is achieved without a person holding the title "Manager", you don't need them. This can be achieved by hiring 51%ers for example [1] and by actively monitoring the health of your organisation. [1] https://news.ycombinator.com/item?id=39333921 https://news.ycombinator.com/item?id=39333921 [1-1] https://www.amazon.com/Setting-Table-Transforming-Hospitality-Business/dp/0060742763 https://www.amazon.com/Setting-Table-Transforming-Hospitalit... [*] YMMV: the hardest problem in any organisation is the "people" aspect, there's no silver bullet. EDIT: Added the link to my other comment about 51%ers
- higeorge13 3y ago51%ers?
- nip 3y agoI hope I'm not violating any copyrights – page 143 of Setting the Table from Danny Meyer [1] > To me, a 51 percenter has five core emotional skills. I’ve learned that we need to hire employees with these skills if we’re to be champions at the team sport of hospitality.They are: 1. Optimistic warmth (genuine kindness, thoughtfulness, and a sense that the glass is always at least half full) 2. Intelligence (not just “smarts” but rather an insatiable curiosity to learn for the sake of learning) 3. Work ethic (a natural tendency to do something as well as it can possibly be done) 4. Empathy (an awareness of, care for, and connection to how others feel and how your actions make others feel) 5. Self-awareness and integrity (an understanding of what makes you tick and a natural inclination to be accountable for doing the right thing with honesty and superb judgment) [1] https://www.amazon.com/Setting-Table-Transforming-Hospitality-Business/dp/0060742763 https://www.amazon.com/Setting-Table-Transforming-Hospitalit...
- tjpnz 3y agoI've seen some recent data on the salaries they're offering in my area. I like Gitlab but wouldn't be prepared to make those kinds of sacrifices to apply there.
- nitwit005 3y ago> This then lead to the discovery that we didn't have any backups as a result of the system not working for a long time, as well as the system meant to notify us of any backup errors not working either. This seems to be entirely normal for startups. I've asked about it during interviews.
- gtirloni 3y ago> This seems to be entirely normal for startups I have trouble understanding this. Do these startups consciously decide not to have backups (for time/money reasons perhaps)? It makes no sense to me. I've worked at some startups and most senior engineers know what's necessary to keep production systems running reliably. That includes monitoring and backups. Could it be that this is entirely normal for startup with less experienced engineers doing things for the first time?
- nitwit005 3y agoThey have backups. They haven't tested actually restoring data from a backup. That is, they haven't ensured their backup system is functional.
- woadwarrior01 3y agoTerrible managers are the bane of our industry. OTOH, many startups owe their existence to their founders having had to deal with terrible managers in their previous jobs. The story of the traitorous eight and Fairchild semiconductors comes to mind. In my own life, if it weren't for getting a non-technical but political and vindictive manager in my last job, I probably wouldn't have had the motivation to start my own startup. In hindsight, I wish I'd been assigned a manager as bad as the last one I had, earlier in my career. :)
- mawadev 3y agoI wish you could flag companies that have toxic management without compromising your privacy or getting outright censored. It would also be great to find companies that truly need your skills, so you can hop more easily without the HR overhead.
- deleted 3y ago[deleted]
- pooper 3y agoMicrosoft has toxic management but it also has M1 who are just trying to navigate the maze and do the right thing. It is never straightforward like this.
- 3y ago
- higeorge13 3y agoI wish the ‘More people doesn't equal better results’ would be put into every manager door and desk. I would also add more results to the quote to make it even more accurate.
- DeathArrow 3y ago>For example, sharding is useful if writes heavily outnumber reads If you distribute your data carefully and don't need to do joins across shards, I fail to see how sharding doesn't help with reads.
- YorickPeterse 3y agoEvery time we'd look into this, we'd look into the challenges we'd have to face and see if they changed since last time. One of the problems GitLab faces is that it JOINs all over the place. In addition, while most data can be shared on a group/namespace basis, GitLab also does a bunch of cross-group queries in frequently accessed pages. To put it differently, try to think of an application that does all the things you _don't_ want to be doing if you're going to (or want to) shard. Then imagine that that's basically GitLab, coupled with the load patterns simply not justifying it.
- isbvhodnvemrwvn 3y agoIt adds a huge amount of complexity due to those limitations, and unlike writes, you can handle reads off of read replicas, as they did.
- ahofmann 3y agoRead replicas are better suited for this situation and way less complicated. And you said it yourself: you need to be careful and you loose the ability to do joins.
- evanelias 3y agoTotally agree. Overall the author's arguments against sharding seem short-sighted, at best. All large successful user-generated content applications eventually need to shard, and most of them are read-heavy just like gitlab.com. The two primary motivators for sharding in this case are database size and blast radius, rather than needing to scale writes. For database size, even if your DB still can fit on one beefy server, operational activities like backup/restore and new replica cloning take an increasingly long time. This in turn becomes a major risk to the business -- for example he even discusses how long the restore took when he had to restore the prod DB from the staging DB. For blast radius, it's the ability to have outages (e.g. hardware failure) only affect a small subset of users. Or for a more extreme situation, the massive difference between "I accidentally dropped the database" and "I accidentally dropped a database". Sharding does indeed add terrible complexity to the application as well as operations. But in any case, eventually the data size motivator becomes unavoidable for user generated content: you eventually must shard, and the longer you wait, the more difficult it will become.
- DeathArrow 3y ago>A SaaS and self-hosting don't go well together There are countless counterexamples.
- DeathArrow 3y ago>In spite of all this, I'm not sure what alternative I would recommend instead of the combination of Ruby and Ruby on Rails. Languages such as Go, Rust or Node.js might be more efficient than Ruby, but none have a framework as capable as Ruby on Rails. Go would be a sensible choice. Both .NET and Java have very good frameworks and are good choices for large scale development. Rails is nice for small to medium websites, but for large microservice based apps will present some issues.
- gls2ro 3y agoGithub, Gitlab and Shopify are built with Ruby on Rails. That seems big enough for me.
- DeathArrow 3y ago>Location based salaries are discriminatory How is that true? If a company pays $30k monthly in Bay Area and $10k in Netherlands, what should the unique, non discriminatory wage should be? $30k or $10k?
- michaelt 3y agoSome people believe that every employee's productivity and contribution to value can be precisely quantified, and fairly split between employer and employee. I've always been sceptical of this myself - how do you value that code review where I stopped a junior engineer's XSS getting into production? Incredible value? Or 10 minutes of work?
- raccoonDivider 3y agoAlways wondered that. It means everyone's salary is determined by the highest-cost of living area you want to hire from. Doesn't really make any sense seen that way.
- DeathArrow 3y agoReading this was quite refreshing, as I realized I am not the worst developer out there.
- ahofmann 3y agoDo you think that the author is a bad developer? Why?
- DeathArrow 3y agoNo, I don't think the author is a bad developer. I was thinking more at the general situation at Gitlab.
- tovarisch 3y agoEverywhere I’ve worked there’s always been one burnout engineer like this who doesn’t really know what anyone else does or why but is absolutely convinced that everyone else except for him is an irredeemable moron.
- skrebbel 3y agoLoved this read. One part didn’t resonate with me though: > In practice this lead to GitLab building many features over the years that just weren't useful: a serverless platform nobody asked for and that was ultimately killed off, support for managing Kubernetes clusters that didn't work for three weeks without anybody noticing, a chatops solution we had to build on top of our CI offering (thus introducing significant latency) instead of using existing solutions, or a requirements management feature that only supported creating and viewing data (not even updating or deleting); these are just a few examples from recent years. If my company ever grows to be as successful as Gitlab, I’m going to much more worried about innovation grinding to a halt than about people trying stuff that doesn’t work, or trying stuff the wrong way. It’s so much easier to shoot an idea down than to believe in it long enough for it to have a chance. I strongly agree with the author about making engineers be the decision makers on how to build out an engineering product, but I’m not convinced that this list of failed ideas is a bad sign at all.
- pgeorgi 3y agoIt's a matter of focus: Having some time to tinker and try out stuff that might just fail is great. If it consumes so much development time that the must-haves deteoriate, you let down your customers.
- aragilar 3y agoI think it's less innovation and more failing to understand the requirements (i.e. only implementing half of what's needed, and then moving to the next thing). E.g. adding labels to gmail, but no way to have filters, so you have to manually apply them.
- Aeolun 3y agoYeah, some of Gitlab’s features are hilariously half assed. Of the ‘one crud form but missing the delete and update part’ level.
- ehmc 3y agoThis part of the article resonated with me, as I used to work as a tech lead for a very similar startup to Gitlab. We were always pushed to "ship MVP's" and "move fast and break things" by product and design, which always resulted in a half-broken features getting pushed to production and left there once the sprint was over. This also led to a constant inexorable growth of tech debt, that made me feel like we'd never be able to get things remotely stable in our corner of the product. I've since come to the conclusion that it's much better to build small, flexible, feature-complete systems that can be used for multiple iterations of ideas.
- jatins 3y agoStory of bad management aside, this was also a highlight for me on the risks of early stage startup even when the startup goes big > In my case the amount of taxes would be so high I wouldn't be able to afford it, forcing me to wait until GitLab went public I have seen a bunch of cases where people really want to leave the company but can't due to short exercise windows and taxes that come along with exercise. One of the companies I know had to partner with a lender just so that their employees could exercise stocks. So now if the company IPOs you owe the lender a load compounding at 15% Thankfully many newer companies seem to be going towards 10 year exercise windows, but they are still a minority
- kevin_nisbet 3y ago> Thankfully many newer companies seem to be going towards 10 year exercise windows, but they are still a minority I'm actually a bit on the fence for the 10 year exercise window, as I'm not sure it's always a benefit. As I understand it, startup exits timelines are on an upward trend, so in 10 years will the average exit still be below 10 years, and even then are you working for a company that meets the average timeline. Then, from a taxation perspective, if you do want to exercise before the 10 year expiry, if it's cost prohibitive now, it should be bankrupting after a few more investment rounds and increases to the 409A valuation. So I'm curious if there is a side effect here that the 10 year exercise window actually leads to many more shares becoming even more cost prohibitive to exercise right before the company reaches a liquidity event and as such not exercised (if you're in the early stage employee group).
- jatins 3y agoYeah, you are right on the point that if you are not able to exercise now you might not be able exercise later also. It's still lesser of the 2 evils imo because: 1/ many companies have liquidity events as they mature. So you may not have to wait for IPO/exit 2/ In 10 years you often know if the companies gonna go anywhere. So if you feel the company's ngmi, might as well not exercise. I have also heard some companies fixing their exercise price (as in its not tied to 409A) to $1 or some small value like this. That seems like the best option
- intellectronica 3y agoThis is an interesting read and I am grateful for the author sharing their experience. One thing that I could pick up is that the author is somewhat bitter about their experience, but isn't really taking responsibility for their own contribution to getting into the situation. At most companies that are not evil and not exceptionally dysfunctional (and I presume GitLab is neither) getting to a state where one is put on a PEP/PIP is preceded by a longer period where they had indications that they are not performing in accordance with what is expected of them. Also, at most larger companies consistently "meeting expectations" is not hard, if you're in a role you're skilled for (which it sounds like the author was) and willing to receive the explicit and implicit signals from your environment, especially the hierarchy above you, and adjust. Being able to make these adjustments and work effectively as part of a complex environment is an important part of working at a larger company. Not taking responsibility for the need to learn how this system works and adjust accordingly is counter-productive. It sounds like the author realised that rather late. Perhaps they could have had a better experience if they figured that out earlier.
- kunley 3y agoVery interesting to read as a former employee of a company being Gitlab's customer, self-hosting EE production and solving everyday problems, with their support also, during the described years
- tnolet 3y agoInteresting that the OP puts “caring about performance” so high where he admits that it had very little impact on business results. If the data shows it does not matter for success of the company, why care so much and stress it is important?
- qznc 3y agoMy guess for a reason would be that developers rarely care for business results. The technical challenge of scalability looks more like fun than fixing yet another enterprise permission issue.
- eadmund 3y ago> A mistake GitLab made, and continued to make when I left, was not caring enough about scalability. Yes, directors would say it was important and improvements were certainly made, but it was never as much of a priority as other goals. At the heart of this problem lies the way GitLab makes money: it primarily earns money from customers self-hosting GitLab Enterprise Edition, not GitLab.com. In fact, GitLab.com always cost much more money than it brought in. Is that really a mistake, then? As a first approximation, company has to invest in the thing which makes money, not the thing which loses money. Now, higher-order thinking might suggest that a more-performant gitlab.com might bring in more customers, build a stronger reputation and so forth. Still, if the company’s real business is selling self-hosted software, it makes sense to focus development resources on self-hosted software.
- zero_ 3y agoBut to be honest I do see the point of the author. Either stop "half-assing" GitLab.com or remove the product (GitLab.com) entirely (and focus only on self-hosting customers (GitLab Enterprise Edition)) might be a better decision in the long run. But I am sure no one on management level wants to do this decision. So you half-ass a product. This kind of situation can be frustrating for people working on the product itself (at least when they care). And each new developer that joins the product will probably suggest: "Oh we should care more about scalability." And each senior will be: "yeah yeah, i know..."
- jacquesm 3y agoGitlab.com is a marketing instrument and if you 'remove the product entirely' then the enterprise edition will sell only a small fraction of what it sells today.
- zero_ 3y agoI do not know if your statement holds true. But you shouldn't half-ass a marketing instrument either IMHO.
- 3y ago
- gnfargbl 3y agoI find the bit about employees being permitted to use their own computers almost astounding. However small an organization is, it should be insisting on providing a company-owned machine and having all company business be conducted on that machine. This has massive benefits to the company in terms of control of corporate IP, huge benefits to the employee in terms of separation of work time and personal time, and it doesn't even cost that much to do.
- askonomm 3y agoFrom an employers perspective, maybe. From an employee perspective? Yeah I'd much rather use my (much more powerful) machine where I have everything set-up exactly how I like instead. Tons more convenient.
- FireBeyond 3y ago> I find the bit about employees being permitted to use their own computers almost astounding. However small an organization is, it should be insisting on providing a company-owned machine and having all company business be conducted on that machine. I've worked with small and startup healthcare/healthcare-tech companies who not only permit it, but have formal BYOD policies... when they are dealing with PHI.
- pas 3y agoit's GitLab, open source / open core / source available, MRs in the open, etc. there's basically zero difference between internal/external contributors. (unless the employee was working on some embargoed/security/backoffice/infra stuff)
- hnarn 3y agoNot to mention the inability to enforce full disk encryption and (hopefully) avoid leakage when someone quits. It’s obviously not a silver bullet but at least then leakage must be hostile action or incompetence by transferring sensitive data outside of authorized channels.
- littlestymaar 3y ago
- hnarn 3y ago> This then lead to the discovery that we didn't have any backups as a result of the system not working for a long time, as well as the system meant to notify us of any backup errors not working either. Reminder that validating backups for critical systems is everyones[1] job in organizations where there’s not a literal backup team working on it full time. This is probably one of the worst experiences a developer or sysadmin can have, but in no situation can it just be one persons fault. If multiple lines of defense have failed (backup validation, monitoring etc.) and nobody noticed, it’s simply a question of when. [1]: all technical personnel that can be remotely considered stakeholders in the backups not failing
- Aeolun 3y agoI really enjoy RDS for not having to worry about this. Point-in-time rollbacks for the past 24 hours, and if we accidentally drop the whole database we can restore it from the periodical snapshots.
- hnarn 3y agoZFS snapshots are really nice too, if ZFS is an option.
- andrewaylett 3y agoYou should probably still try it occasionally :). For one of my team's services, we don't just back up the data: we then also restore it into a parallel environment. If the backup fails, the restore fails, and then the environment fails, and while our monitoring might miss some instances of spurious success, it's less likely that the whole chain will look like it's working if the actual backup is not actually working.
- wlll 3y agoI noticed that we weren't validating backups at a company I used to work at so I did a /really/ simple hack where I checked that the backups were within a certain number of bytes in size of the previous dump. Literally a couple of weeks later after putting this live (I think it was a Nagios check) it alerted that the backups had gone down from many 10s of GB to a few k in size. I can't remember what had gone wrong now, it was years ago, but we /did/ catch the issue. Obviously this isn't a comprehensive check, I later wrote a tool that pulled backups down, restored them, and did some sanity checks on the data, but the quick hack worked as an interim.
- deng 3y ago> Think about it: if a company pays a person less because of the color of their skin or their gender, the company would be in big trouble. But somehow it's OK to pay a person less based on their location? I'd suggest the author thinks yet a bit more about this. "color of the skin", "gender", "location of living", I'm sure the author can figure out what differentiates the first two from the latter. For a start, one could think about which of these things one is able to change. Or you might ask the question: which of these things have actually something to do with my job? And yes, location matters: the company usually needs to create a subsidiary in the country of the employee, get familiar with their judicial and tax system, then there's also the issue of time zones, travel cost to company meetings, etc. Don't get me wrong: one can surely discuss whether differentiating pay based on location is OK or not. However, comparing this to discrimination because of skin color or gender will not help this discussion at all, as in effect the author is comparing GitLab to a racist and sexist organization, which will end any discussion fairly quickly.
- rmk 3y agoYeah, his statement makes about as much sense as the expectation that all people must be charged the same for something all over the world. i.e., 50c-equivalent hamburger in, say, South Africa should have the same price in San Francisco. Ain't gonna happen because economic realities do not bow to idealism, however well-intentioned.
- shutupnerd0000 3y agoThis topic is as tired and cliche on this forum as "single page apps vs progressive enhancement"
- paulddraper 3y ago> one can surely discuss whether differentiating pay based on location is OK or not. However, comparing this to discrimination because of skin color or gender will not help this discussion at all Exactly. Compare it to something else, like paying people more if they own a sports car.
- dijit 3y agoThats a somewhat uncharitable interpretation. As you are almost certainly aware, the majority of racial problems in the US boil down to the very simple fact that black people are on average of low socio-economic standing; and if there is one thing the US system is geared against its any person of low socioeconomic standing. Put succinctly: You might consider that its ok to pay differently based on if you’re in Los Angeles or Bryan, Ohio- but would you feel the same if pay was different depending on if you lived in Inglewood vs Santa Monica?
- skc 3y agoAlso interesting that he publishes his new projects on github.
- christkv 3y agoI never had a problem with location causing differences in pay. However that it extends to options grants always seemed to me as utterly incomprehensible. Why should you living in the Bay Area mean you get a massively bigger lottery ticket than somebody in Europe or India.
- PeterStuer 3y ago"Location based salaries are discriminatory" You have to understand companies will pay you the minimum possible to get you to join/stay.
- y2hhcmxlcw 3y agoI can truly relate to poor management. I'm unsure though if the author is referring at times to poor engineering/HR managers, or poor product managers. At my company they call the latter Product Owners but it's the same thing I believe. I deal so often with both being terrible. I've had terrible Engineering Managers, who simply don't understand the tech, and in the worst case they think they do but don't. Add in some office politics and they can make life truly unbearable. Same for Product Managers but it's a whole other array of chaos. I've had a tough time with poor managers, and so I can really relate to that aspect of this. What I wonder though reading between the lines is if Gitlab has reached a point where they suffer from poor Product Managers/Product Owners. I see that in later stage startups and especially large orgs. One would hope Gitlab, by nature of its product space, would never suffer from that, so if they are that's disheartening. I'm unsure what at this point would really differentiate Gitlab from Github unless it's devex, and to have good devex they'd need engineering centric product vision, I hope Gitlab is not losing that by hiring an army of "Business Analysts" or MBA's. On the positive, in some respects I like hearing that HN comments drive the vision at Gitlab :)
- Waterluvian 3y agoThe paid by location argument is a challenge for me. On one hand I really like meritocracy. Be paid what the job is worth. Or rather, what the market bears. But then the tone changes the moment the market expands internationally. Lots of people ready to do the job for far less. Sometimes I sense a hypocrisy. A demand for things to be fair… but only within this artificially delineated geography. On the other hand, nothing will make it not feel freakishly weird when HR needs to “approve” you, a 100% remote worker, moving somewhere, as a mechanism for cutting your salary.
- languagehacker 3y agoLots to disagree about here, but I'd prefer to treat the moral of this story as, not all the same people will thrive in a small company after it's hit a growth inflection point. This is okay! Without some of those folks who are willing to go broad and wear a lot of hats, companies can't get to where they need to be to start growing in the first place. The responsibility of leadership and managers in this situation involves creating venues where early, impactful employees can be adequately rewarded without becoming impediments to their personal growth or the growth trajectory of the company. Imagine how much less burnout the author would have felt if the stock situation didn't keep him locked into the company far beyond when he didn't feel welcome. I've worked at enough places where early contributors white-knuckled their way through changing values until the first possible liquidity event, or just left a big financial upside on the table because they couldn't make the numbers and their sanity work at the same time. These "growing pains" result in burnout, poor quality and communication from individuals we've previously trusted as veterans, and a tumultuous culture with conflict between the old guard and new layers of management. I believe we can minimize the friction on both sides by significantly increasing the exercise period for stock options, prioritizing hiring growth in duplicative roles for individuals showing early signs of culture clash and disengagement, and empathetic coaching that names the problem (terminal burnout) and includes resources for teams or companies that better fit that person's disposition.
- mschuster91 3y ago> No matter how you try to spin this, it's by all accounts an act of discrimination to pay one person less than another purely based on where they live. Think about it: if a company pays a person less because of the color of their skin or their gender, the company would be in big trouble. But somehow it's OK to pay a person less based on their location? Yes, yes it absolutely is. My favorite example is a train conductor in the deepest netherworld of Saxony and a train conductor in Munich. Both perform the absolutely same work, both in complexity and time, and yet with the usual "everyone gets paid the same" collective agreement, either the train conductor in Munich can't even rent a 1br apartment, or the train conductor in Munich barely scrapes by while his colleague in Saxony lives in a mansion. Wage schedules in any multihomed organization have to be CoL based, there's no way around that.
- moribvndvs 3y ago> A SaaS and self-hosting don't go well together About two months ago, I was commenting on my struggles working at a place that offered cloud and self-hosted solutions where it was not going well. I was mostly convinced that the problem was organizational, but thinking more about it, it’s more complicated than that. To be clear, I don’t think it’s impossible to do both. If your application is very simple and self contained, then of course; we use such services every day in a variety of environments. However, you have to design the software from the ground up for simplicity, and to carefully dogfood your self-hosted experience via your cloud offering to ensure both models are aligned. And of course, complex system architecture is a significant obstacle; assuming your customers are going to wrangle dozens of microservices, multiple database and distributed caches, etc. is not reasonable for most organizations. So, if you strictly and carefully design your product to be self-hosted-first, I think these two models can coexist… up to a point. If you’re fortunate enough that your cloud offering becomes huge, you might reach a point where the simplicity of self-hosted-first becomes a restriction that turns into a liability. At that point, you can simply make a decision: who butters your bread? That sucks for your customers who lose in that decision. If I select self hosted, it’s often because I am concerned about lack of agency, control, and partitioning, or I need to flatten my costs, or I have an external requirement that is incompatible with the cloud offering. Is it not a risk, then, to hitch my wagon to a company that offers both, and at some point may make a decision to rescind or water down their self-hosted solution? If it’s closed-source or open-core, then I would say so. From that perspective, I think it would be better to go with a vendor that just picks a lane, and avoid those that use watered down self-hosted to funnel customers into more expensive cloud offerings (in other words, open-core model is a big red flag for me). I suppose this is maybe getting into whether you should rely on proprietary software whatsoever for vital parts of your business, but let’s not get into that. Like I said, I think the question is complicated.
- hankchinaski 3y ago>Location based salaries are discriminatory While I understand the feeling, I would be pressed to say that this is purely driven by market dynamics. Companies that hire in a certain market have no incentive to overpay compared to the local market rates. Software engineering has now become commoditised. This holds true for remote companies. Wages are a function of local supply/demand and labour costs in that location. If you want to follow the same logic, you would expect to pay a coffee $10 in Brazil as you would pay in NYC. Following the same logic a coffee shop in Rio de Janeiro would say “location based pricing is discriminatory”. Markets don’t work that way
- rwmj 3y ago> I think the idea of product managers needs to go in favour of giving team leads more power and having them interact more with users. To me, that's ultimately what a "product manager" should do: help build the product at a technical level, but also act as a liaison between the team and its users. Yes! In my experience product managers bring very little to the table, and the most enlightening discussions I have are when I'm allowed to talk directly to customers. Very often I find out about pain points with software (often ones which are easily and quickly fixed) by talking to customers, which I'd never heard of before through product management.
- mastax 3y ago> That's not an exaggeration by the way: the only service running at the time was a New Relic trial account that only allowed monitoring of one, maybe two servers out of the (I think) total of 15-20 servers we had at the time. I don’t feel so bad about my monitoring setup now.
- zzzeek 3y ago"Many of the performance problems solved during my first few years at GitLab were N+1 query problems." "Other frameworks have learned from this over the years and provide better alternatives. The usual approach is that instead of being able to arbitrarily query associated data, you have to pass in the data ahead of time. The benefit here is that if you were to forget passing the data in, you'd run into some sort of error rather than the code querying the data for you on a per-row basis, introducing performance problems along the way." can someone...translate this for me into human? If I want to query for some data I have to ....first pass the data in? Where did I get the data? I can't parse this paragraph at all. Rubyists....
- YorickPeterse 3y agoIn Rails, you can write something like this: Project.limit(10).each do |project| project.members.count end Here `project.members` returns some sort of query object that produces a query along the lines of this: [...] FROM members WHERE members.project_id = X [...] In other words, you can just query associated data on a per-project basis, without needing to pass any additional arguments. The problem this results in is that in the above code you'd run a COUNT query for _each_ project. Instead what you'd want is a single COUNT that groups data per project, such that you can then pass that data along with the `each` call. This setup would only require 2 queries, instead of 20. To eager load data in Rails you have to explicitly opt-in, resulting in something like this: Project.includes(:members).limit(10).each do |project| project.members.count end The problem here is that an opt-in mechanism is too easy to forget (as is evident by how common these N+1 query problems are), and even if you include it there are certain cases where you still end up running extra queries for each row. The solution here is to separate querying from the row instance types, e.g. a "Project" type can't query data itself and instead requires it to be passed in. This makes it much more difficult to create N+1 query problems.
- zzzeek 3y agooh OK , then I can translate: 1. have eager loading, but then also 2. prohibit lazy loading, have it raise rather than implicitly load voila haha, downmods
- deleted 3y ago[deleted]
- samlambert 3y agoThis post is interesting overall. But saying the org needs to care about scalability and then not understanding sharding is an interesting vibe.
- dalyons 3y agoI think the author understands sharding very well. And the operational complexities of it and thus why it should be a last resort.
- samlambert 3y ago1. it's not as complex as stated. 2. sharding scales read heavy workloads just as well as write heavy.
- dalyons 3y agook lets look at it. Start with the absolute best case for sharding - a perfectly segregated data model, say either by user or customer id. No shared data, no joins between tenents. Some relatively straightforward framework code to redirect queries between N shards. cool. Operationally now you need N database clusters (at least a primary and a replica per cluster). You need to manage and monitor all those, figure cluster local and shard promotion schemes, figure shard based backups and restores. Probably figure out shard rebalancing. None of it rocket science, but a lot to get right, test and maintain. Now do all that operational stuff in a way that works with on prem installs (gitlabs primary customer type). And then add in the fact that practically noones data model is that cleanly shardable, so add support for cross shard joins, and global tables. Its pretty easy to see how a couple of read replicas (that are near zero cost operationally if you're using a cloud db) are a VASTLY simpler solution.
- toomanyrichies 3y ago> Or as the Dutch saying goes: "Lekker gewerkt, pik!" (good luck translating that). Google Translate says this just means "Nice work, dude!" Out of curiosity, is there some culturally-specific subtext which is missing from that translation?
- sdwvit 3y agoProbably the company grew, and standards grew as well. It is difficult to do layoffs / fire people based on perf in Netherlands, so it seems that's why it was only hinted for the person to leave. I am with the company on this one. Business is business, and it's both manager's and employee's responsibility to grow and keep up with the demands. Learn from errors and move on.
- kiitos 3y agoReads like a fairly junior IC.
- baner2020 3y agoThe flaw in the argument is the assumption that Directors were more adept than Individual contributors. Most dev shops have this problem as folks further away from the IC work , the least they can anticipate the problems
- siliconc0w 3y agoEconomically, if the Amsterdam developer is providing the same value as the bay area developer - you should probably pay the them the same. While it's true that the local market for Amsterdam developers is set lower, you're building a global company and competing with other global companies. Sure you can get a discount now on Amsterdam developers but eventually a competitor will offer them something closer to their value or they will leave to do their own thing. I think we'll see the strongest companies pay location-agnostic prices for talent. Ultimately, it's about value delivered. If I'm Gitlab with a likely large encumbered ruby codebase and I want to sell to large enterprises that care about performance and reliability, I'm probably gonna pay this person more than 120k EUR a year. Most engineers who care about performance, type-safety, and reliability have no interest in Ruby so the market rate of that skillset is definitely higher. Is it bay-area 500k/year high? I dunno, but I imagine that the strongest companies who want top-talent will probably need to tie their comp to value delivered rather than location or they'll lose their talent to the companies that do.
- redwood 3y agoOr you should hire twice as many of them
- siliconc0w 3y agoIf you pay people according to value you get happy value-generating employees. If you pay people the minimum of what you can get them for- you get the minimum - unhappy, less productive employees that eventually leave your company. There is no free lunch. Just because they aren't the type of person to get counteroffers or move to the bay area doesn't mean they don't recognize their value or won't feel taken advantage of.
- belter 3y agoCan somebody help me out here? I went to the GitLab pricing page: https://about.gitlab.com/pricing/ https://about.gitlab.com/pricing/ Where do I select my location so as to get my location based price?
- depereo 3y agoLike with any other business, you talk to the sales team and come to an agreeable middle ground. We don't pay 'list' for gitlab SaaS.
- vmirnv 3y agoSo some reason, I'm chuckling that his site has a favicon similar to yc's.