23 ms·
How to avoid picking the wrong technology just because it's cool
- pram 9y agoA lot of this just sounds like Resume Driven Development, not people thinking they're Google or Amazon.
- OpenDrapery 9y agoI was thinking the same thing. I wonder how much of it is due to the way we make devs work and the hours we make them keep. For example, I'd be more than happy to use the same old, tried and true, boring tools to just get the job done, if it meant that I could then go play golf or otherwise not be in the office. But if you insist that I be in my seat 8 hours a day regardless of workload, then goddamn let's take this shiny new tool for a spin! Do I want my resume to show that I used the same tool for every job for the last ten years? Or do I want it show some new hotness? The industry and employers are as much to blame for this as the engineers, if not more. When you use middleman firms to find your employees, and all they understand is buzzwords, well then guess what game the devs are gonna play?
- allcentury 9y agoFellow golfer and tinkerer - we are a product of the job markets. Hot employers typically want new and shiny on the resume in addition to fundamentals, seems like everyone is playing the same game...
- collyw 9y agoI get the feeling that my skillset is becoming outdated. Fact is I know Django inside out, and plenty of Python libraries. Its very rarely that I will find something that requires me to learn a new language or tech (I will likely get things done a fair bit faster using the tech that I do know). Anything else feels like resume driven development.
- alecco 9y agoI recently worked at a company with a working batch legacy system and an ongoing update to Python and SQL with newer tools. It processed large data but it fit in a big server. 3x the amount of people required. They replaced it with Spark and Scala. Rewrote the whole thing and bumped into the same previously fixed problems for 8 months. It was not faster and it was a pain to extract the result data for the online website. Oh, and they had only 1 server running Spark. On top of the existing one. They also dumped IRC with very useful plugins for proprietary Slack channels. Yeah, their resumes are very competitive now with the Big Data hype. Worst part is the higher ups liked it because they had more keywords in the bag for investors. We are living in very strange times.
- mparr4 9y agoThis to me is the real problem, especially in a larger company. I mentioned "Resume Driven Development" at work the other day and got the response from a peer of "that's how it should be." Convincing others to not chase the shiny new toy is harder than picking the right technology. The other posts under this parent are right: we are a product of the job market.
- gtirloni 9y agoThis is just a "my technology stack is better than yours" post like countless others we see daily. Sorry to dismiss it so abruptly but it gets tiring.
- gtirloni 9y agoI think people downvoting tend to ignore the fact that the proposed "optimal" solutions for non-Google companies were at some point novelties themselves. If the same logic is applied we'd be using CICS app servers, IMS, and buying terminals. At some point things change, the new normal changes, etc. The shift we are seeing in some areas also contributes to finally accepting the realities of distributed systems.
- icebraining 9y agoThe people are probably downvoting because you have missed the point of the article. Nowhere does it say that one shouldn't change things, or adopt new solutions.
- eropple 9y agoIf you have the mindspace to be sorry, you have the mindspace to not do it. And, since you clearly didn't actually read the article: he never says a word about his own technical preferences except "don't pick ones that don't fit your scale and scope".
- thehardsphere 9y agoWouldn't that actually require the author to promote their own technology stack?
- AznHisoka 9y agoI used to work at Standard and Poors. They do media monitoring for lots of financial companies. When I joined, I thought they using state of the art tools to monitor millions of sources. Turns out they just had a team of thousands in India that manually visited certain websites to check for press releases, transcripts, earnings reports, etc and push them to a database if they found them.
- sharemywin 9y agowonder if that could be crawled and save money.
- bm1362 9y agoWoah there cowboy
- sjg007 9y agoor just set google alerts.
- dismantlethesun 9y agoAccuracy. Every solution I've seen that relies on automatic crawling will eventually have a parsing error when someone changes their sentence structure of a press release. It's not so obvious when you're looking at the breaking releases for a few stocks or companies, but historical records have at least 1 error per stock per year.
- platz 9y agoright, it depends on what the reward for high precision vs high recall is
- greggyb 9y agoSo split your stream: 1. Data matching expectations (you do have a definition of correct, right?) 2. Log for manual review -> manual inserts or correction and placed into queue for (1) Monitor (2). When inserts start trending up, it may be time to update your processing logic.
- komali2 9y agoRegarding "UNPHAT": Is this... serious? Does the author genuinely hope that we will use this acronym as a means to help guide our technology choosing decisions? Is it not their creation and is just something I wasn't aware of yet? Finally, do these forced acronyms ever help anybody else out there? I mean seriously, the "N" standing for "eNumerate?" The "P" standing for "Paper," which barely correlates to the actual meaning "consider a candidate solution." Seems to me just saying "apply a principle of unfattening your technology decisions" would be a hell of a lot easier to remember.
- ng12 9y agoIt seems to me just a complication of YAGNI.
- nine_k 9y agoYAGNI is dismissive. That other approach with a comparably silly acronym is an actually useful checklist.
- paulddraper 9y agoI doubt it. I think it needs to be three or fours letters. (E.g. Always Be Closing. Keep It Simple, Stupid.) --- Trying my hand: - understand the DOMAIN - find the OPTIONS - research a CANDIDATE - know the HISTORY - consider the ADVANTAGES - apply deliberate THOUGHT DOCHAT
- komali2 9y agoHmm. That's a much better acronym, but it makes me come back to my original question - an acronym I guess is a way to make a process (dare I say algorithm) easier to remember, yea? So like, when you're switching to a new technology, the steps are as you listed, which are basically just 1. Understand what your problem is, and the potential solutions to that problem. 2. Use your brain. 3. Make an intelligent decision. Or really just 1. Use your brain I mean I just don't see the need here. Could be arrogance, I guess? Is this the very problem the author was trying to solve? Create a defined process for technology choosing?
- joshribakoff 9y agoI've seen this in action. Using code generators to convert XML configuration to a few API end points. Or using a DSL/rules engine because you don't want to write code. Or having APIs that hit other APIs ad infinitum when the whole thing runs on one server because "micro services are the only right way". The result was we spent time gluing together what was already a monolith disguising as microservices, rather than adding features the customers wanted More recently I had to solve time drift on 1000s of devices. The problem was someone installed puppet to manage those devices which uses NTP. The devices are behind firewalls so if they block the puppet master or mess with SSL puppet doesn't even phone home. Or worse it gets incorrect time from NTP peers on the network. The solution was to throw out the shiny tool "puppet" and just call "date". Puppet and NTP are great in theory for getting time down to the millisecond but totally backfired when some devices were off by over 24 hours. For our purposes as long as all devices were within 5 minutes we were good. The irony was after disabling NTP puppet just started it again. And we couldn't use puppet to fix that since 50% of our users had it blocked. No other choice but to throw out puppet and start over from scratch. The guy who spent months setting up puppet was not happy.
- liveoneggs 9y agothe real issue is why the firewalls were randomly blocking puppetmaster and/or ntp and why the puppet ssl stuff stopped working (apparently randomly?) Everyone involved sounds like they need a lot more experience.
- joshribakoff 9y agoWith all due respect, you don't know the real issue. Your response is the same thing the guy who installed Puppet said to me... just have them unblock it. Our sales pitch is "these devices use plain http and will work behind your corporate firewall". The blockage wasn't an issue that could be solved, it was our whole business model to workaround the blocks by using simple http instead of https, proxying everything through our IP, and things like that. Even the puppet documentation says not to run a puppet master when you have devices that are behind firewalls or limited network. The guy who added puppet apparently didn't read that. I wasn't the one who decided the business model just the guy who fixed it to work as advertised while dealing with the pressure of everything crashing & burning. You're right no one had experience but thats not the point. My point was that the fancier tools sometimes just add new issues without solving your real issue. Despite my lack of experience I solved the time drift using a linux built in "date" to set the date time. It didn't account for network lag like NTP, and an NTP developer would probably laugh at my solution, but now all devices are accurate to within a few minutes & that particular problem was solved. So don't always go for the most complex tool is all I'm saying. For what its worth I do plan to bring back puppet but run it in "puppet agent" (offline) mode. We'll using custom scripts to copy in new puppet configs so puppet does not need to phone home.
- Animats 9y agoThe author's acronym is silly, but it's a real problem. Soylent liked to blither about their "infrastructure", for a product that sells a few times per minute. They could be using CGI scripts on a low-end hosting service and it would work fine. Wikipedia is some MySQL databases with read-only slaves front-ended by Ngnix caches and load balancers. That seems to get the job done. Wikipedia is the fifth busiest web site in the world. Netflix's web site (not the playout system) was originally a bunch of Python programs. The article mentions a PostGres query that required a full table scan. If you're doing many queries that require a full table scan, you're doing something wrong. That's what indices are for.
- godzillabrennus 9y agoQueries that require full table scans? What are we talking about? WooCommerce?
- claudiulodro 9y agoHey! Those all have indexes now. :)
- godzillabrennus 9y agoGood to know! Working on a migration from a site that modified the WooCommerce core to a new ecommerce platform. Couldn't believe WooCommerce ever did that when I started. Congrats on such a widely used platform though.
- squeaky-clean 9y agoI remember reading a scaling-out article from some startup. Some of the things felt a little over-engineered, some were impressive, some seemed wrong. But then they get to the point where they brag about their scale, and the metric they used was that they can handle thousands of requests.... per day.
- opportune 9y ago
- Kiro 9y agoNothing I've built has ever needed anything more than a $5 Digital Ocean droplet and one of my services gets around a thosand requests a second at peak. Purely anecdotal and I'm not doing anything CPU intensive but I really feel startups are overdoing their infrastructure.
- sidlls 9y agoIt isn't just startups. And I agree with another commenter: it seems like resume driven development.
- __jal 9y agoThere's also a weird peer-pressure involved. Overheard a conversation a while back that summarizes it nicely - someone was talking about a scheduling system written they've used for years, and mentioned it was written in Perl. Another participant guffawed, and after the requisite Perl-bashing, the original person allowed that, yes, even though it worked fine, they should rewrite it. No idea what company that was, but I'd love to work in a place where that was the most pressing concern on my plate.
- Piskvorrr 9y agoWell...there's a lot of context in "works fine," from "it is perfect and there's nothing to add to it" through "good enough" to "it sort of works within most expected inputs, but everyone is afraid to touch it, so we've built a bunch of hacks around it."
- sidlls 9y agoI've encountered this sort of thing too many times here in the last few years. A set of Python scripts to ETL a few hundred MB of data once daily is suddenly desperately in need of replacement with a real-time, fault-tolerant MQ backed server (I've seen more than one of these, from custom ZMQ proposals to Kafka). A set of two or three scripts run on a compute server that are kicked off by cron and that do some work in batch on a modest set of data (few hundred GB) suddenly must be replaced with an asynchronous task management service capable of managing a huge set of processes with complicated dependencies (Jenkins, Luigi). A bit of data one team produces (once daily) has another team asking for it: instead of exporting the data from the DB directly or standing up an endpoint on, say, a Flask server there's a drive to slap a Spring microservice architecture in place. And so on. It's really a sight to behold, the rationalizations people go through who, in other contexts, laugh at the inferior logical reasoning abilities of non-CS folks.
- dismantlethesun 9y agoWorking in the D.C. area has given me a high tolerance for acronyms and backronyms (seriously: P.R.O.T.E.C.T. Act stands for "Prosecutorial Remedies and Other Tools to end the Exploitation of Children Today"). U.N.P.H.A.T does raise a smile to my face for trying, but if the author is reading, I'd suggest you change it to a prescriptive paragraph where the first word in each sentence becomes a letter in the acronym (e.g. B.A.M.C.I.S). ==== Here's my best try: UNPHAT: Understand the problem. Nominate multiple solutions. Prepare by reading relevant research papers. Heed the historical context. Appraise advantages versus disadvantages. Think!
- dismantlethesun 9y agoDrat, it's too late to edit my comment and take it out, but "Consider candidate solution" wasn't meant to be included in the acronym. It was part of my brainstorming.
- dang 9y agoOk we took that out for you.
- dismantlethesun 9y agoThanks.
- gaius 9y agoTupac was the master of this GHETTO: Getting Higher Education To Teach Others
- RUG3Y 9y agoBAMCIS: Begin Planning Arrange Recon Make Recon etc...that BAMCIS?
- dismantlethesun 9y agoYep. That one.
- dang 9y agoWe temporarily replaced this article's baity title with the text's more accurate self-description. If someone would care to suggest a good title—i.e. accurate, neutral, and preferably drawn from the language of the article itself—we can change it again.
- stickfigure 9y agoFWIW, I think the original title was pretty good. I have had the unfortunate experience of screaming pretty much exactly those words (albeit replace "Google" with a different company from which one of my CEO's advisors came from). EDIT: how about combine the two? You Are Not Google: Another "Don't Cargo Cult" Article
- nemild 9y agoI like your proposed title, "Another 'Don't Cargo Cult' article" on its own seems dismissive, when the content seems quite useful for many engineers (the acronym need more work, though).
- dang 9y ago"You are not Google" is a linkbait trope ("you") and also one very tired cliché.
- jacques_chester 9y ago
- yumaikas 9y agoI suppose another point worth bringing up is that hardware has made some pretty strong advances in recent, especially with SSDs being widely available. Stuff like that has raised the ceiling on what a single box can do compared to 2000 and earlier, when Google was building MapReduce at first.
- jethro_tell 9y agoPretty sure that's in the article is it not?
- merb 9y agoRule 1: don't use any modern javascript framework.
- marktam264 9y agoI'm typing this impromptu but this article seems to be a qnd informal carnation of the Architectural Tradeoff Analysis Method (https://en.m.wikipedia.org/wiki/Architecture_tradeoff_analysis_method https://en.m.wikipedia.org/wiki/Architecture_tradeoff_analys...).
- wwweston 9y ago> Don’t even start considering solutions until you Understand the problem. Your goal should be to “solve” the problem mostly within the problem domain, not the solution domain. I'd guess that this guideline alone would stop 2/3 adoptions of JS SPA frameworks (and 4/5 Angular adoptions!) if followed.
- BigJono 9y agoAt least the SPA frameworks themselves have a reasonably common legitimate use. The tooling around them is the major problem. Most projects I've worked on, even complex ones, could comfortably trim from 100 dependencies down to 10 and have the developers working on them be an order of magnitude more productive. People wilfully wrestle with thousands of functions worth of APIs every day and don't even notice the immense slowdown it's causing them. It's especially bad in React land, which is ironic seeing as Sebastian Markbage at Facebook has an excellent talk about reducing API surface area.
- lacampbell 9y agoI am fortunate, in that I got a lesson in not over-engineering things very early in my career. My first programming job was a 3 month contract at the maintenance department of an international airport. They had a bunch of information in large, unwieldy ERP system and wanted to automatically generate job sheets for the different maintenance crews. So I did the simplest thing possible - I generated an excel file from the ERP system, then using that file as input, I outputted different excel worksheets for the different crews. It was very plain GUI app that had one or two buttons. I remember being a bit worried that it wasn't nearly fancy enough for 3 months work, but everyone seemed pretty happy with it. Later on I found out that - before me - they had hired an experienced software developer who had worked on the same problem for 6 months, and at the end of that 6 months had apparently not produced a solution. I had done the dumbest, simplest thing - not because I had any insight or wisdom, but because it was really the only thing I had the skills to do. But I delivered. It was a brilliant, accidental first lesson in not over-engineering.
- Can_Not 9y agoSame here, I've watched multiple devs come into my first job with agendas of extreme over-engineering and unnecessary rewrites. I think the company is actually pretty lucky that they haven't gone under from all the damage.
- gaius 9y agoI generated an excel file from the ERP system, then using that file as input, I outputted different excel worksheets for the different crews. As a complete aside, you might be surprised how far you can go with Excel these days. Do you know it has a built-in in-memory columnar database now? You can have millions and millions of rows of data in there that you can use in tables and charts completely independently of the size of the grid. Pull back a huge chunk of data from the DB and slice and dice it to your heart's content locally. I look at people buying expensive "business intelligence solutions" and I think, it's right there on your PC all along and you don't even know it...
- collyw 9y agoThe problem is people using Excel for everything that it shouldn't be used for.
- NTDF9 9y ago"Don't use tensorflow to predict everything"
- dlwdlw 9y agoMany people may not work at Google scale, but name would probably like to work at Google