27 ms·

Why's that company so big? I could do that in a weekend

• olalonde 10y ago
  That $12M/yr figure for indexing 1B documents seems way off to me.
  • svantana 10y ago
    Yes and if you look at the footnote where the arithmetic is laid out, it seems quite confused. It conflates querying with indexing and also flatly assumes that each machine will index 5M pages, which seems low to me.
  • astrieanna 10y ago
    Too high or too low?
    • olalonde 10y ago
      Too high. See http://news.appbase.io/scaling-elasticsearch-writes/ http://news.appbase.io/scaling-elasticsearch-writes/ for example: > In doing these tests, we indexed over 1 billion documents under just two hours. The peak merge rate we saw was 129 MB/s, yes our ElasticSearch cluster on a 24 node setup is gobbling data at this speed. > Most mind boggling of all: At our peak sustained ingestion rate, we would have added over 13 billion documents in a single day and this would have cost us a mere USD 244. Their test documents are much smaller (100 bytes) than a typical web page but if you multiply that figure by 50 to get to 5kb documents (probably a realistic average web page size minus all markup and assets), you get 12200$ for 13 billion documents or 1220$ for 1.3 billion documents (assuming that scales linearly). I'm probably way off in my estimate myself but I'm convinced that a few thousand dollars is much closer than the proposed 12M$.
• mixedCase 10y ago
  I'm sorry for the OT, but I feel I should post this: http://bettermotherfuckingwebsite.com/ http://bettermotherfuckingwebsite.com/ This is unreadable as is.
  • minitech 10y ago
    Resize your browser window?
    • Veen 10y ago
      Yeah, but asking readers to resize windows, resize the font, and tweak the CSS to get decent line heights is a bit of cheek when it's trivially easy to implement at least minimally pleasant-to-read typography and layout. It's a bit shortsighted of a writer to completely ignore the realities of readability. The current layout may conform to some vague idea of purity or minimalism, but that puts form over function in a big way.
      • cpburns2009 10y ago
        Resizing a browser window is hardly as involved as tweaking CSS or browser settings.
  • kakuri 10y ago
    It's readable for some subset of people, but on a 32" monitor it's worthless - I tried to read it, but just abandoned it in frustration.
  • lmm 10y ago
    Au contraire, it's a lot more readable than your link, because it respects my window size.
• dools 10y ago
  To me this post addresses the same naivety that David Graeber demonstrates when he claims that everyone not involved in hammering out a sheet of steel to make a horseshoe or pulling a plough is engaged in a bullshit job[1]. I think it's not a coincidence, therefore that many of the people I run into who advocate a universal basic income, arguing tooth and nail against a job guarantee (the YC guys no different) is young, tech savvy and overly optimistic about the scope of automation to subsume all human labour within months. [1] http://strikemag.org/bullshit-jobs/ http://strikemag.org/bullshit-jobs/
  • Houshalter 10y ago
    Graeber's biggest point is that even when he talks to people and asks them if their job is valuable, and they should know better than anyone else, they often say 'no, not really.' I don't think he thinks "anyone not involved in hammering steel" is unnecessary, that's a bit of a strawman. The argument goes that big corporations have a tendency to hire more people than necessary because middle managers are rewarded based on how many people work under them. Or that people looking to keep their jobs, try to make themselves look more busy and valuable than they really are. People waste a lot of time at offices doing things that aren't work. But our culture demands we be at work 8 hours a day, even if the work only requires 4 hours. Economists in the 1930's thought the world would be so rich by today, that people would only need to work a few hours a week. And the economy has grown vastly since then, and we are much richer. But I don't think most people have the option of working that little, even if they want to. But that's a different subject entirely from automation. Those jobs can really be valuable and still be vulnerable to automation. AI has improved a lot in the last 5 years. I don't think it's implausible at all to imagine most jobs done now being partially or totally automated.
    • dragonwriter 10y ago
      > Graeber's biggest point is that even when he talks to people and asks them if their job is valuable, and they should know better than anyone else, they often say 'no, not really.' I don't think that's really true, especially in a big corporation. Transparency of the value pipeline in such institutions isn't great at any level, and big corps (and similar large institution -- government often has the same problem, for instance) have a tendency toward not making even the concept of the value stream that justifies the position known to the people working in it much of the time. Its quite likely that the people working a position don't understand how its supposed to deliver value (and are in a poor position to see the big picture of whether it does), while the people who do know how it is supposed to deliver value don't see the small picture of what actually goes on (and thus are in a poor position to evaluate whether it does deliver the value it is supposed to.) There's no particular reason to think the people working in the position are in the best position to evaluate value delivery (or that even the people in the best position, whoever they might be, are better at answering the question than a Magic 8-ball.)
• NKCSS 10y ago
  Weird example, where it's assumed that a single machine can only hold 50GB of document data, which results in the assumption that you'd need 200k machines to store 1T documents...
• thallukrish 10y ago
  I feel if you have something better to offer, say a better way to look at information, not just a better page rank algo, but a different way to look at information that is scattered across, one that enables a better insight into existing web pages and less overhead of constant 'searching' by typing a query, something that automatically helps to "see" relevant information with less effort, then it can be an alternative to what a Google search does today and can position itself as a competitor in the long run even if it does not exist today.
• FLUX-YOU 10y ago
  I have worked with one-man companies that did this and often they are shoddy products. Healthcare just seems to suck at software. Full-stack developers are already a very fast evolution from the front/back end developers, which were an evolution from a guy doing just HTML/CSS in the 90s. Now you have to include sales, marketing, negotiation, project management, and business management skills on top of your full-stack development skills. The result is that something is going to suffer unless you are a genius, the product is super-simple, or you have 20-30 years to have learned all of this. There's a much more interesting question in the form of: "Why's that company so big? A tight-knit team could do that in a weekend".
• payne92 10y ago
  Apps aren't just a pile of features, they're also ecosystems. In the early years of Twitter, it would have been relatively easy to clone Twitter, but you won't have had the ecosystem. Poster example: Google+
• shubhamjain 10y ago
  Besides underestimating the scope of the problem, one logical fallacy that programmers fall for is believing that a single developer should be able to juggle lots of tasks easily. If a small team can create a SaaS application, why do large companies need so many engineers for so many specific things? I think, at a certain scale, it becomes necessary for organisations to ensure that there are certain people keeping specific things running. Let's say, a company doesn't have any specific person assigned for server administration and then one day, is confronted with a DDoS attack. Who is supposed to handle it? Whose responsibility is it? You have few people who know the basics of operating servers but no one who can swiftly respond to it. If no clear role is established, then the company is bound to face more attacks and possibly, lose more customers.
• TazeTSchnitzel 10y ago
  Features and scale are huge things. Maybe you could make, say, a Twitter clone, in a weekend. But you couldn't make one that can handle anything near Twitter's scale nor that is as feature-rich as the actual thing, and besides, nobody will be using it.
• notacoward 10y ago
  Everything seems easier when you're not doing it yourself. It's only a minor variation on the grass being greener. Everything gets way more complicated when you're doing it at scale, with high reliability, for real users, with some hope of making money. Practically nobody adds complexity for its own sake. Usually, if you think something is much more complicated than it needs to be, it's because you don't have any idea what the real requirements and constraints are.
• Lxr 10y ago
  The article raises some good points but there is still a dichotomy here to me. Compare a successful startup with 2 founders with an established company with 2000 employees in the same business. The scale of the difference in employees vs the difference in the product is often shocking. BigCo product might be better, but that much better? Obviously the quality of the product won't scale linearly with the number of engineers, but still, startups punch above their weight. Maybe it just highlights how much harder things get when the company reaches a certain size.
  • tmnvix 10y ago
    Those 1998 extra staff don't have to make the product 1000x or even 2x better (however you might quantify that). They just have to make it sufficiently better to maintain or gain market share.
• Hyperized 10y ago
  Always reminds me of: http://i1.wp.com/beckybendylegs.com/wp-content/uploads/2014/01/Modern-Art.jpg http://i1.wp.com/beckybendylegs.com/wp-content/uploads/2014/...
• malux85 10y ago
  I often hear this logic from inexperienced engineers. My answer to them is: Then why don't you do it? <grin> Recently a junior engineer said this to me about Uber, and I helped him to understand - His logic was "it's just a taxi app". There were loads of Taxi apps before Uber and loads of them after Uber. Yet none of them were uber? Sure that a medium complexity CRUD app can do the pickup and scheduling, and spit out some json to the app. But what if you have 100,000 journeys taking place simultaneously, and you must track their positions so that you can render the journey map on the invoice? Can you do live traffic planning and re-route around the busy traffic on the fly? Does your weekend include the full billing system? This is a mandatory requirement for a business. Can you spot busy periods and increase the fares according to the laws of supply and demand? All online in realtime? Can you continuously A/B test this to optimise profits? Can you deal with the fact that maps and roads are always changing? All of these and more are the things that make Uber, Uber, otherwise you're just one of those shitty taxi apps that failed. Still think you can build it in a weekend?
  • dagw 10y ago
    And even that is only a small part of what makes Uber, Uber. If you want to actually compete with Uber, then once you've gotten all those minor details worked out, you're going to have to tackle the real problem which is dealing with 100's of different regulatory frameworks for offering taxi services, being a taxi driver (or a not-taxi driver as the case may be), hiring of private contractors and two dozen or so other entirely non-technical things neither you nor I have considered.
    • nilved 10y ago
      That's what the problem is. You can invent something in a weekend; the red tape takes longer. I think that still counts as "built in a weekend."
      • pixl97 10y ago
        Your comment makes no sense. Invent is a pretty meaningless word in this case. Invent doesn't mean 'make a product capable in working in the real world where it has to follow laws, physical limitations, and deal with stupid people'. If you invent an app in a few hours, then get fined out of existence your app sucks. If you invent an app in a few hours, then it crashes under high load, your app sucks. If you invent an app in a few hours and your code sucks and you can't adapt it later, your app sucks.
• nlawalker 10y ago
  From 2009, still relevant: https://blog.codinghorror.com/code-its-trivial/ https://blog.codinghorror.com/code-its-trivial/
• informatimago 10y ago
  In France, the law (Art. R. 4228-10 du Code du travail) imposes one WC and one urinal per 20 male workers, and two WC per 20 female workers. So basically, it expects that up to 1 worker per each 10 workers will be busy with non-job related stuff. Another datapoint, I heard that an accountant counted that the actual number was 1 for 50. In any case, when you're alone or a small startup, this doesn't seem much, but when you have 100,000 employees, this means that between 2,000 and 10,000 equivalent employees are all the time in the WC (and being paid for that, basically). Of course, there are a lot of other overheard that become very significant when you have a big corporation, but that still exist (and that we easily discount) for small startup and individuals.
  • saryant 10y ago
    In the US, most companies aren't hiring janitors, they outsource it to a facilities management firm or it's handled by building management. I'm also not sure how that's relevant.
  • cauterized 10y ago
    Ever thought that maybe that 1:20 or 1:10 isn't about handling steady load but about handling pre-/post-lunch peak loads without half hour lines? (And I've seen the half hour lines, in an office that had two WCs for 100 people. THEN you end up with people spending 10% of their day doing something other than work.)
  • brohee 10y ago
    Do you stop thinking on the loo? I don't. For knowledge workers, that time isn't completely lost...
• eriknstr 10y ago
  I searched for GTGACCTTGGGCAAGTTACTTAACCTCTCTGTGCCTCAGTTTCCTCATCTGTAAAATGGGGATAATA and found another post by the same author on another website http://bitfunnel.org/strangeloop/ http://bitfunnel.org/strangeloop/ I've seen some other posts by Dan before but didn't know what kind of things he works on because I generally don't check that because if I did then I'd end up just checking people instead of reading what people were writing. Anyhow, from the article linked above -- for which a video is available at https://www.youtube.com/watch?v=80LKF2qph6I https://www.youtube.com/watch?v=80LKF2qph6I -- it turns out that Dan has done quite a bit of work with implementing search related technology, and I think that was interesting. His contributions to BitFunnel can be seen at https://github.com/BitFunnel/BitFunnel/graphs/contributors https://github.com/BitFunnel/BitFunnel/graphs/contributors
  • alva 10y ago
    I thought you had accidentally pasted some of your genetic code from your clipboard. future internet problems
• logicallee 10y ago
  Anyone find the 528 MB C++ file that is referred to in the Twitter screenshot near the end of the article? (With only 57 lines of code but, oh, an array with the first 50M primes hardcoded.) What's that about?
  • omaranto 10y ago
    I'd guess it is probably for Project Euler.
    • logicallee 10y ago
      that "probably" makes me wonder why you would say that? is there something about 50 million primes in there?
      • omaranto 10y ago
        I don't think you really need that many primes for Project Euler, but I can certainly imagine someone tired of including a call to her prime sieve in tons of problems, and of having to waiting for it to run, and thinking, "let's just make a library that has all the primes I could possibly need in a constant array".
• merb 10y ago
  Actually it actually wrong to assume somebody could solve a trivial problem in a weekend, however some companies still have way too many people working at it. Consider twitter with their 3860 people. they could probably do it with half of them. especially the technical people are 40% of the company... way too much. i don't think that they need no people but well...
  • r_smart 10y ago
    Yeah, this post seems to be addressing an annoying conversational experience, but ignores the other side: the reality, which you describe. I understand that just scaling out their platform is a ton of work that justifies having a lot of engineers. But how does Twitter have so many engineers on staff and yet their users constantly complain that they don't improve even seemingly low hanging fruit? Is ~1,500 people not enough? Is there some lingering technical debt they're dealing with? Or is it just people accumulated over time they value too much to let go?
• brachi 10y ago
  Reminds me of this great article about the 'this shit is easy' syndrome: http://steve-yegge.blogspot.com.ar/2009/04/have-you-ever-legalized-marijuana.html http://steve-yegge.blogspot.com.ar/2009/04/have-you-ever-leg...
• informatimago 10y ago
  Otherwise, indeed, it's a basic evaluation error, as described as long ago as in the The Mythical Man-Month; it just seems that nowadays, the multiplier is not 9, but even bigger. https://archive.org/details/mythicalmanmonth00fred https://archive.org/details/mythicalmanmonth00fred Figure 1.1, page 5 To go from a program to a programming system (interfaces, system integration), you need to multiply the effort by 3. To go from a program to a programming product (generalization, testing, documentation, maintainance), you need to multiply the effort by 3. So to get a programming system product, you need to multiply the effort from the program step, by 9. Nowadays, we need to compose indeed multipliers for distribution, reliability, availability, localization, targetting multiple UI platforms, etc. And the original multipliers are also probably too small, when you see the effort required just to integrate testing and CI on some projects.
• GoToRO 10y ago
  "I could do that in a weekend" You could but you didn't so I can't use it...
• wellpast 10y ago
  There's a naïveté in saying "I could do that in a weekend" as a measure for employee bloat, but that's making a straw man of the question. It is nearly always more probable that a large company with several hundred or more plus engineers doesn't need that many. That those engineers are there for other reasons -- political reasons or speculating on new products, valuation reasons, etc. -- than for the reason that those engineers are fundamental to the top or bottom line of the company's core product/s. Uber does not need 2000 engineers. That's a prima facie fact. I've worked inside several large companies and saw this first hand. Hundreds of engineers across teams doing "stuff", no doubt, but nothing that ultimately or ever tied to the top or bottom line. The only ideas that I could come up with for why these people were there were: perhaps the headcount helped company valuation, perhaps we needed that many benchwarmers for when there was company churn, maybe there is some political reasons a manager wants a sizeable headcount on their team, or maybe it's just leadership ignorance as to how things are getting delivered. But in no way could I see a concrete justification for many teams and many engineers. What am I missing?
  • notacoward 10y ago
    Maybe some of those engineers really were superfluous. Maybe some were contributing to the bottom line in ways you don't understand or appreciate. I've seen plenty of engineers take infrastructure for granted, discount the value of research ("speculating on new products" is important), or even dismiss support as a cost with no revenue upside. Most often, as those people gain experience they also gain a broader view, allowing them to distinguish between complementary roles and truly superfluous personnel.
• TempleOS 10y ago
  God made templeos. A nigger does not know what the word "random" means.
• FussyZeus 10y ago
  I personally have never seen this complaint leveled against the mighty Goog, nor really any company that's cash positive. When I see people giggle at bloat, it's more often than not twitter and still more often than not, it's not intrinsically because big company = waste, but because company that should be small is made to be big either by VCs or Wall Street, and then can't turn a profit anymore. No, maybe if twitter was still a tiny company you couldn't use it in Arabic but maybe they also wouldn't constantly be on the edge of going out of business. Edit: I think this is also a regular criticism of Twitter because Twitter has spent the last several years trying to Facebook itself and strayed far from the original idea. You can argue the pros and cons all day since they're pretty much infinite on both ends, but Twitter as a company has spent vast amounts of money on a lot of features that none of it's original userbase really wanted (and drove a lot of people off in the process). Me, I only really ever used it as a time-waster so I'm not awfully attached but I listen to a ton of podcasts and I can't count the number of hosts who disliked more or less every new thing Twitter did.
  • nl 10y ago
    What features has Twitter shipped? Moments??? Pfft. On the contrary I'd argue they've hardly done anything new, and that is why they are struggling. Think of all the things Facebook has done and abandoned in the same time. But I like Twitter - I just wished they fixed some pretty obvious issues. I mean... Tweetstorms are like a request for a feature right there, and Twitter's response is... silence.
  • smcl 10y ago
    Right, I feel like the author has a good angle with which to tackle some of this criticism, but choosing Google as the example company is really unpersuasive for me. I'd agree that Twitter would be a better example (or Uber, 2,000 engineers - doubled in size over the last year)
    • willvarfar 10y ago
      Uber need their 2000 engineers to write their 1000 microservices eh? <snark/>
• willvarfar 10y ago
  As a counter point, on every product I've worked on, on every time we've wanted to make a 'big leap' we've need to get a small expert team to go make it happen.
  • xixixao 10y ago
    Dod that small expert team polish the product and maintained it for the next 2 years? Something about 80%/20% rule...
• lmm 10y ago
  And yet sometimes a small team really could do it. I spent years complaining about MSN/AOL/Skype etc. and talking about how a small team could build a better messenger in a week, but we didn't really believe it. Then Slack actually took that possibility seriously, and look at the results.
  • dingo_bat 10y ago
    Whatsapp is a good example too.
  • dexterdog 10y ago
    They didn't build it in a week.
    • lucasnemeth 10y ago
      Yes, but that's something a lot of people are arguing here and I agree. The "I could build in a week" is ridiculous, but "I could what this 1000 engineers company do that with a small team of 10, in a lean way in 18 months" is very frequent in the market.
  • deleted 10y ago
    [deleted]
  • cocoflunchy 10y ago
    Sure, they were a small team in the beginning. Google too... Now they probably have more than 500 employees. (They had 385 in february 2016: http://uk.businessinsider.com/slack-ceo-worried-about-growing-culture-2016-2 http://uk.businessinsider.com/slack-ceo-worried-about-growin...)
    • bluedino 10y ago
      Slack also jsut recently hit 3 million users. 100 million people were using AOL Messenger at it's peak, and 12 million were using ICQ when AOL bought Mirabillas.
  • vacri 10y ago
• cloudhead 10y ago
  I think a lot of people are willing to "leave cash on the table" if it means a smaller, more effective team and leaner product.
  • Yokohiii 10y ago
    Certainly not if you are VC driven.
• yomly 10y ago
  Yup. Have never gotten the whole "Airbnb/Uber/<startup>" is just a crud app line of thought. If they're crud apps then everything else in between is just glue programming. The author of the post covered some very astute points but one not covered in much detail is the challenge of scale and reliability. Maybe this falls under the bucket of optimization and/or latency but I think this deserves being called out on its own. Once you have Uber-scale number of users requesting taxis at any point, and have a network of drivers constantly communicating their position with the app, suddenly you no longer have a trivial crud app on your hands...
  • ZeroGravitas 10y ago
    I read an article the other day that claimed Uber was beating Lyft because they had a strategy of opening a physical office in each city, having an office for drivers to sign up, and a human being who tailored things for that location. Their rival on the other hand assumed that some programmers in SF could do everything neccessary from a distance. Not sure how true it is, but it seemed an interesting theory.
  • collyw 10y ago
    Most apps are crud at the end of the day. Making them scale is the hard part.
  • problems 10y ago
    Well, that's the thing - they are just a CRUD app in a lot of ways. Even at "uber scale" it really isn't that bad, it'd be a few incremental challenges but nothing a small team couldn't handle from a technical perspective. The catch though is that on top of that CRUD app is a business which involves some serious legal and marketing divisions to make that app work. The tech may be simple, but the business side of it certainly isn't. The "I could hack that up in a weekend" attitude is often true - but it only covers the bare basics of the software it doesn't even attempt to address the business, which if often where the real struggle is these days. Especially if your background is development. I think many engineers still say this knowingly though, just to (perhaps incorrectly) scoff at the value of good business people.
  • new299 10y ago
• systemtest 10y ago
  I recently had a discussion about how hard it would be to have a system inside a bus to announce the next stop. Sounds like a weekend project right? Think a bit further: - Hardware needs to be resistant to harsh diesel engine vibrations - 3G/4G connectivity - Need a mobile data contract with local ISP - Software needs to be able to handle network disconnections - GPS needs to be able to pin-point at which stop the bus is currently stopped, even with bad GPS coverage in larger cities - If the bus skips a stop because of a detour, the software should be able to detect it and announce the next stop - The announcement should be bi-lingual for Airport busses - The announcement should work for people with hearing aids - Server needs to know all bus-stops - Server needs to know different stop types, such as bus terminals, intersection stops, regular stops, hand-over stops, virtual stops - Server needs to be able both work of a yearly bus schedule and real-time update - Server needs to output in an understandable JSON/XML because the government subsidies demand an open-data format - Server needs to publish data to an open-data server because of the subsidies - Because the bus-company is sponsored by European Union money the control interface should be translated to German/French/English/Spanish And now this weekend project takes a team of 10 engineers working for a year.
  • manyxcxi 10y ago
    The GPS and mobile data are overthinking it. The routes are known so simply trigger on door open/close (with some logic for denounce/etc) and cycle through. For maximal accuracy simply have an electronic beacon of any sort broadcasting an ID that corresponds to the stop and you've got it solved. And the entire data set for the route should be able to live offline onboard the bus. Changing bus/numbers routes should include the step for updating route data.
    • oarsinsync 10y ago
      Except that quickly becomes inaccurate if the bus driver decides to re-open the doors for an incoming passenger, or due to traffic conditions, opens the bus doors further from the stop than the beacon reaches. Or skips a stop due to road conditions.
• draw_down 10y ago
  Every variation of this comment/mindset drives me nuts. It's unbelievably thoughtless.
• raverbashing 10y ago
  The question is why do someone needs the first 50Mi primes in a C++ file
  • JoachimSchipper 10y ago
    No idea, but it would be a fairly reasonable way to test a prime-finding algorithm - or to replace a prime-finding algorithm with something less complicated.
  • ddorian43 10y ago
    See making crc 4x faster in redis by keeping precommputed values in a lookup table: https://matt.sh/redis-crcspeed https://matt.sh/redis-crcspeed Maybe something like that ?
  • S4M 10y ago
    crypto?
  • informatimago 10y ago
    Because the alternative: generating the primes at compilation time with a template functional expression would be too slow, and quite possibly would explode the compiler's memory limits. (No Turing Machine here), and doing it at run-time or deliverying it as a "resource file" is obviously out of the question.
• bsandert 10y ago
  To me this is a form of the Dunning-Kruger effect[0], where only somewhat informed people estimate the cost of a "0.1" (or POC) release without considering the famous "unknown unknowns" which may involve scaling, billing, 3rd party integrations, etc. [0]: https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
  • vidarh 10y ago
    I find I've had to bar myself from thinking about many of these factors, after I realised how many projects I stop myself from pursuing because I know about the hard bits, and realise how many of my past projects I'd never have taken a shot at if I knew what was coming up and was thinking like that. It's great to know, but often it's not so great if it leads to inaction... E.g. if you have a "I could do that in a weekend" impulse, sometimes it's worth following that impulse. If you're right, you've spent a weekend building something potentially great. If you're wrong, you've spent a weekend learning about things you clearly didn't know in advance, and the lessons can often be very different from the objections you'd have throw up if you thought about it without doing, and sometimes what you learn may provide the impulses for another new idea. But it's hard. E.g. my first impulse for anything that requires collecting money is how painful it is to deal with credit card processing and reporting and VAT and accounting, because I've done big complex billing systems with heavy reporting requirements, and compliance complexity etc.. It's very hard to put that impulse aside, and "just do it" and think about how to solve the payments afterwards. But that's how I did things he first few times I needed to handle payments.
• interdrift 10y ago
  Yeah right, except that you can't. It takes so much time to nurture something and build it in a way that satisfies the user.
• deleted 10y ago
  [deleted]
• new299 10y ago
  The sad truth is that in the majority of cases the quality or time taken over the implementation isn't particularly important other factors matter much more: * building something people actually want. * advertising * engaging with users * aggressively promoting your product. * customer service It doesn't matter if what you did was hard or not. It matters that you can acquire and retain customers.
• z3t4 10y ago
  If you want to build a "boat" you start with a canoe, you don't build a ferry.
• erichocean 10y ago
  > Businesses should keep adding engineers to work on optimization until the cost of adding an engineering equals the revenue gain plus the cost savings at the margin. This is often many more engineers than people realize. I think this is the key realization, and it's actually a variation on Conway's Law[0]. In a nutshell, some part of BigCorp discovers (or believes) that by hiring a software developer for $X, they can make $Y > $X in return. So they do. As long as some part of BigCorp is able to do that, you get more software engineers. This continues until you either (a) run out of budget, or (b) run out of opportunities. Because it's not done "top down", it less efficient than it could be to get all of the features in total. But due to how these engineers are funded internally, it's not possible to do the "top down" approach at all. And that's okay, all that really matters is that $Y > $X. [0] https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law "Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations." In this case, it's not the communication structure that causes the perceived bloat, it's the funding mechanism. The basic idea that software organizations reflect some non-software, human structure in the business, holds.
• programminggeek 10y ago
  Building an app and building a business aren't the same. Go out and sell the app to people and find out how easy it is.
• chias 10y ago
  Jeff Atwood made a fantastic writeup of a very similar sentiment about StackOverflow / the StackExchange network: https://blog.codinghorror.com/code-its-trivial/ https://blog.codinghorror.com/code-its-trivial/ Until reading that article, I often espoused similar views on applications. It was a really eye-opening read.
  • dlubarov 10y ago
    Fog Creek is something like 50 employees right? That seems like a reasonably sized team for their product, but Twitter/Uber are in a different ballpark.
• ChemicalWarfare 10y ago
  googles and ubers of the world are not just their end-user facing apps + the backend required to support them. these are large businesses with end user-facing "IT" side of the house being just the tip of the iceberg.
• fencepost 10y ago
  I was on the other side of this with a product that I was one of the main people on back in the late '90s - I looked at what we were selling and how much we were selling it for, and thought "Why are people buying this? Anyone with a decent IT department should be able to do this in-house." But really when you sat down and looked at it, what we were selling was a supported and tested product that could be configured and active in less time, with excellent ROI, and at significantly less than the cost of dedicating an in-house developer to do it. What we were selling wasn't cutting-edge technology, it was time and cost savings for a department that was almost certainly already short of both money and staff time.
• wellpast 10y ago
  This is such a straw man conversation. Of course it is a fallacy to think "I could build that in weekend" and translate that to a claim about the inefficiency of a company. I think that's what is being responded to in this thread. But that's not the interesting debate. A better consideration (imho) is this: For a given business/offering/product, what is a likely optimal number of engineers needed to continuously deliver the product's value efficiently? Then: How far is a given company from that optimal number? This is a hard analysis to do. It is virtually impossible to quantify or put this kind of evaluation in a laboratory. There are too many counterfactuals that would be at play. We really just have to consult our experience and consider which of these is the more probable: (1) A business/offering/product team with lots of money is really good at utilizing its hundreds of staff. OR (2) It's not. I've seen a few (2)s from the inside. I have never seen a (1). Therefore I lean more probabilistically toward the belief that a company is needlessly big. What surprises me is that that's not where people on this thread are leaning---but has anyone on this thread actually ever seen a (1)??? If not, where is this belief coming from? I've never seen a ghost, so I don't believe in ghosts....
  • notacoward 10y ago
    Consider, for the sake of argument, the possibility that perfect efficiency is unachievable. Almost physically impossible. As a company gets larger, certain kinds of inefficiency in hiring and utilization become absolutely inevitable. With that premise in mind, how many employees does a company "need"? If it would need fifty in that universe of perfect inelastic spheres with no friction or similar effects, how many times that does it actually need in the real world of such efficiency-robbing factors? In my experience, the answer is two at best and more often, around five. The incremental work product from each new hire diminishes more rapidly than many realize, but the incremental revenue might still make such hires worth it. As Dan points out, a few percent increase in efficiency from optimization can be huge. It might take multiple people quite a while to achieve that, and it will still be worth it. There might be another group of several people trying something else that doesn't pan out, but that doesn't mean they were unproductive dead weight either. Bigger companies have to take risks too. They just take them differently, and the cost is more obviously attributed to them instead of the market. Maybe a single startup could do what Twitter (for example) does, in fairly short order. Even that's unlikely when issues of scale etc. are considered, but that's not even the point. That single startup is not a valid point of comparison, because it doesn't count the cost of failed experiments. Add up all of the startups trying to do what Twitter does, including the failures, and that probably be a better estimate of how big Twitter (Uber, Netflix, whatever) needs to be.
• callesgg 10y ago
  Generally it is Marketing.. And also "Hofstadter's law"
• deleted 10y ago
  [deleted]
• buckbova 10y ago
  Looks like my primary search engine DuckDuckGo tackled this with a relatively small team. https://en.wikipedia.org/wiki/DuckDuckGo https://en.wikipedia.org/wiki/DuckDuckGo https://duckduckgo.com/ https://duckduckgo.com/
  • linkregister 10y ago
    At an Alexa rank of 676, that's pretty impressive. I think the key to DDG's success with a small set of engineers is their small scope. If they ever wanted to expand their product scope, for example, index their own searches instead of aggregation, they would start running into the obstacles discussed in the article. The Lucene comparison in the article doesn't apply to DDG (yet).
• Animats 10y ago
  As of 2008, the core search team at Google was just under 100 people. Then came search customization. Suddenly, no more cached replies.
  • draw_down 10y ago
    100 people?! I could do that myself, in a weekend.
• VestingAxis 10y ago
  >>There’s also a wide body of research that’s found that decreasing latency has a roughly linear effect on revenue over a pretty wide range of latencies and businesses. Anyone have pointers to such research?
  • Dowwie 10y ago
    This was an error by the author of the blog post. There isn't a wide body of research confirming this. There aren't academic papers. There is merely anecdotal evidence/marketing material from industry participants.
  • jedberg 10y ago
    There is no published research I'm aware of. However, you can find an article about every major tech company talking about how reducing their latency increased whatever their core metric was (not necessarily linearly though, sometimes it was and even steeper curve). I know for sure this is true at eBay, Netflix, and reddit because I've seen the numbers personally, and I have friends at Google, Amazon and Facebook who have said the same.
  • jjguy 10y ago
    Came to the comments looking for these links, too. Results of my research are below; everything I found is anecdotal, consistent with the other comments. The most robust is the presentation at Velocity in 2009. [4, 5] My notes: Small ecommerce vendor, 2012. One second delay in page-load causes 7% loss in customer conversions. [1] Marissa Meyer, 2006 Web 2.0 conference. Extra 1/2 second in page load time dropped traffic by 20%. [2] Greg Linden, 2006 (same link as above) "We had similar experience at Amazon.com. We tried delaying page in increments of 100ms and found very small delays result in substantial and costly drops in revenue. [2] Greg Linden, Make Data Useful, 2006 - Every 100ms delay costs 1% of sales. [3] Velocity, 2009. The User and Business Impact of Server Delays, Additional Bytes, and HTTP Chunking in Web Search Presentation. Multiple observations. My net: Delays under 500m measurably impact customer satisfaction metrics, magnitude increases commensurately with delay. (Microsoft, Amazon, Google) [4, 5] 1 - https://info.ensighten.com/rs/ensighten/images/just-one-second-delay-in-page-load-can-cause-7-percent-loss-in-customer-conversions.pdf https://info.ensighten.com/rs/ensighten/images/just-one-seco... 2 - http://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html http://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.... 3 - http://www.gduchamp.com/media/StanfordDataMining.2006-11-28.pdf http://www.gduchamp.com/media/StanfordDataMining.2006-11-28.... 4 - http://radar.oreilly.com/2009/06/bing-and-google-agree-slow-pag.html http://radar.oreilly.com/2009/06/bing-and-google-agree-slow-... 5 - http://conferences.oreilly.com/velocity/velocity2009/public/schedule/detail/8523 http://conferences.oreilly.com/velocity/velocity2009/public/...
• ThomPete 10y ago
  I call this the difference between building a product and a business. It's relatively easy to build a product and even to ship it. The real art in building a business to support it.
• YZF 10y ago
  The other side of the coin is that big companies do move slow. There is a lot of communication overheads, a lot of bureaucracy, backwards and cross compatibility concerns, conservatism, wasted duplicate effort, wasted pointless effort, some people who aren't contributing much etc. etc. This is why tiny startups can sometimes dethrone a large leader. However that's often a multi-man-year effort. So I'd rephrase the statement: "Why's that company so big? I could do that with 20 brilliant engineers and 18 months". I think that order of magnitude can attack pieces of any established software business from a technical perspective. Then there's obviously the business side.
  • bastijn 10y ago
    That would get you your first launch product maybe. Not an actual enterprise product. Sure, people forgive the small startup for their crashes, their unfinished and unpolished app, their weak customer support, their performance hiccups, etc. As long as they bring something that solves an actual new problem that isn't solved by another product easily available. But an enterprise product, no way you will make that in 18 months with your 20 people while keeping up the maintenance burden you introduced in v1 when you took some shortcuts.
• HugoDaniel 10y ago
  Hmm somehow it reminded me of brook's law CEO: "Give me 9 women and I'll give you a baby in 1 month"
• azdavis 10y ago
  Completely unrelated to the article, but I'm curious what people's thoughts are on full-width lines of text characteristic of unstyled HTML. I've seen some[1] bemoan its impacts on readability and others[2] complain about websites which artificially restrict themselves to (what can be) a narrow slice of the viewport. [1]: http://bettermotherfuckingwebsite.com http://bettermotherfuckingwebsite.com [2]: https://plus.google.com/+LinusTorvalds/posts/2Ndtw3ZBRX2 https://plus.google.com/+LinusTorvalds/posts/2Ndtw3ZBRX2
  • rch 10y ago
    On desktop I reflexively zoomed in to %125 and didn't think about it until I read your comment. It also has a reasonable default font size my phone, so it's easy to read quickly. This is pretty rare since browsers gave up on text reflow.
    • hex13 10y ago
      I begin to think that nowadays websites are just designed for "zooming in" in mind (very small fonts, too many space etc.). So maybe we shouldn't complain. These websites are just meant to be zoomed. Although some web developers are cruel and block zoom with meta tag (I never understood why some developers did that, maybe for trolling people with poor sight?).
  • exadeci 10y ago
    Pretty simple, we won't bother reading, I just read the comments instead of trying to read that awful formatting. >7 fucking declarations. Would have been nice
• deleted 10y ago
  [deleted]
• clifanatic 10y ago
  > Businesses that actually care about turning a profit will spend a lot of time (hence, a lot of engineers) working on optimizing systems Hm - every time I suggest optimization (of anything), I'm shot down with "premature optimization is the root of all evil!"
  • tekromancr 10y ago
    There are two reasons to optimize something that don't qualify as premature optimization. 1 it isn't working well enough and is actively causing problems. (Your slow webapp) 2 It could be making more money (Your checkout flow, your lead page, etc.) Everything else is just technical masturbation. Makes you feel better, but otherwise a complete waste of time.
• spectrum1234 10y ago
  What exactly is the point here? At scale, it's not the tech that got the product to the finish line. The endurance part of the company has nothing to do with tech or even scaling or optimizing that tech. Not even close.
• nickbauman 10y ago
  If your problem is small, viable solutions to that problem are vast. If your problem is large (scale), there's usually only one or close to only one way to solve it. So choosing the right path is important. Since you can't predict product market fit (you cannot predict value), you have to figure out how to iterate and test as close to continuously as possible. This last thing is what most people just can't do. Programmers included. So you can see there's a technology side and a culture side and they have to be well-aligned.
• itsyogesh 10y ago
  The question I was wondering about is: When does an organization realize and say, we have hired enough engineers to support our current systems?
• Xyik 10y ago
  Glad to see some sanity for once and people that realize building 'trivial' apps like Twitter and Uber are not feasible 'in a weekend'. Just making sure the app is secure takes months, and security becomes more complex as features are added. Pretty much every feature you add to the app can explode the complexity exponentially. e.g: a sorted timeline if strings is easy, but add the simple feature of 'retweeting' and being able to 'unretweet' for example and things quickly become more complex.
• WalterBright 10y ago
  Even if you're just building an open source app to give away, sure, you can get 99% of it done in a weekend. The last 1% will then consume 99% of your time (if you want anyone to use that app).
• bdrool 10y ago
  I doubt most who say this sort of thing mean it literally (although no doubt some naive sorts do, but they usually are self-identifying through the other outrageous things they say, and thus can be ignored). But one counterpoint to keep in mind is that the alternative view is every company is running as efficiently as possible. I doubt anyone really always believes that, either. For example, when a story about Twitter was posted on HN recently, it was widely discussed that they probably don't need anywhere near as many engineers as they actually have, and could perhaps reduce costs to the point where they are at or near profitability just by reducing staff. There might be some truth to that. Point being, it's not that simple, and there are no absolutes.
• handedness 10y ago
  The technology of a big company is often a necessary condition, but it's seldom (if ever) a sufficient one. We technically-minded people are often pretty terrible at accurately picturing reality as it exists outside of our respective spheres. Consider: "Why is Armani a billion dollar company? I could make a beautiful suit in a weekend," said the talented seamster/seamstress.
• matchagaucho 10y ago
  To this I draw a 2x2 matrix on a whitboard. In each of the 4 quadrants I write: "Product and Engineering" "Ops" "Sales and Marketing" "Legal" Enter tasks and goals for each quadrant. Iterate. This is the minimum effort required to sustain any app. Developers are mostly only aware of the first quadrant.
• eonw 10y ago
  I agree and disagree, I've been with companies and seen how scope can expand as you grow, building internal apps, monitoring, reporting, all kinds of stuff. On the flip side i know a guy that managed a team of 5, the 6 of them total managed the snippet editor in Visual Studio. Six people to manage a snippet editor? Really?
• dantiberian 10y ago
  > Recently, I was curious why an org that’s notorious for producing unreliable services produces so many unreliable services. When I asked around about why, I found that that upper management were afraid of sending out any sort of positive message about reliability because they were afraid that people would use that as an excuse to slip schedules. Upper management changed their message to include reliability about a year ago, but if you talk to individual contributors, they still believe that the message is that features are the #1 priority and slowing down on features to make things more reliable is bad for your career (and based on who’s getting promoted the individual contributors appear to be right). Maybe in another year, the org will have really gotten the message through to the people who hand out promotions, and in another couple of years, enough software will have been written with reliability in mind that they’ll actually have reliable services. Maybe. This sounds like it could be Apple?
• simonw 10y ago
  The code is the easy bit.
• ErikAugust 10y ago
  Same thing happens in software development at the library level: "Why not use [Example OSS Library]?" "Nah, I can do it in 15 minutes." "You know, they have [X] contributors, lots of great documentation, and 6 months of work into it." "Yeah, I know [but whatever.]"
  • jkchu 10y ago
    I get the point that you are making, but if something really only takes 15 minutes to do, I wouldn't necessarily use an open source solution just because one exists. Depending on the situation, the library may have a ton of extra bloat that your application will never use. Also, recklessly deferring to open source libraries can potentially snowball into something like the left-pad fiasco[1]. [1]: http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm
    • ErikAugust 10y ago
      Yeah, and I get your point too. Eight lines of code? Yeah, you don't need a library. But usually when someone says "15 minutes" they mean 1 - 2 days. So things of that complexity is when it's obviously you should choose a good OSS lib.
      • hex13 10y ago
        It depends. In many cases you would be right, in many other cases you wouldn't. It depends on many factors: - what is the problem? (some problems are really hard, some problems are trivial and just seem hard) - is developer experienced enough to build own solution? - are existing solutions good or badly written?(I often refrained from using existing solution when I looked on bloated code in it). - is solution suitable to project? ("too much bundle size", "doesn't support X", "not too configurable", "too slow", "too hard to use"). etc.
• deleted 10y ago
  [deleted]
• candeira 10y ago
  One of my favourite tech jokes used to be "Google? I could build that out of wget, grep and baling wire!". The fact that "I could build that out of Lucene and money!" is not intended as a joke gives me pause.
• kristianp 10y ago
  I really enjoy how this guy's blog refers to research papers. It adds to the level of interest.
• b34r 10y ago
  Those numbers are whacked. We use ElasticSearch with >10B docs ingested and the cost is like a few thousand dollars a month for hosting.
• infinite8s 10y ago
  From the article: > There’s also a wide body of research that’s found that decreasing latency has a roughly linear effect on revenue over a pretty wide range of latencies and businesses. I'd be interested in seeing that citation list, particularly for non-google/amazon businesses.
• sliken 10y ago
  I'm amused the central topic of this post is that google indexes 50M primes in random files in github, which they seem to not. 961748729 is one of the primes just before the 50M'th Your search - site:github.com 961748729 - did not match any documents.
• kriro 10y ago
  Sometimes it takes a lot of time to reach something that is relatively trivial once you know how to do it. I can reproduce E=mc² in a matter of minutes. Doesn't mean I could have ever come up with it in the first place.
• mahranch 10y ago
  > we want to maintain an artisanal search index of 1B documents. Then our cost comes down to $12M/yr. Yeah, no. It doesn't cost that much to maintain an index of that size. Not even remotely close. My roommate coded a search engine back in 2001-02 as part of a class project/hobby and it easily had an index that large, probably larger till he shut it down. The key lies in not crawling it all in a single day. And your index doesn't populate overnight, it takes months of slow crawling. You can maintain an index of practically unlimited size for chump change (well, compared to the millions OP was throwing around). Do people think Excite, Hotbot, Lycos and Yahoo! in the early years had $12 million per year to spend on their indexing? Hell no. Every single one of them would have went bankrupt in a week. Those companies weren't even worth that much back in the mid to late 90s. Your biggest cost is going to be user loads on your servers (CPU/Ram/Bandwidth), not maintaining an index.