8 ms·
Let me share a small fun story (of making an application in an hour, not months). At one large insurance company in my role as an architect I have tried to con
by lmilcin 5y ago
Let me share a small fun story (of making an application in an hour, not months).
At one large insurance company in my role as an architect I have tried to convince people we need Elasticsearch to speed up free-form queries to our database containing information about hundreds of millions of people and their contracts. That database was Oracle and so not really amenable to completely arbitrary queries.
I was immediately shot down that the project will take huge amount of time and effort to complete and so is completely off the table.
So I came up with fun plan. I set up an hour-long meeting which was supposed to be my last chance to pitch the project.
Rather than present slides and extol virtues of Elasticsearch, I decided to WRITE THE APPLICATION DURING THE MEETING, FROM SCRATCH.
And that's what I did. I came to the meeting, set my laptop, connected projector, put some music on, and I wrote the entire service from scratch and had time to spare to present how it works.
I cheated a little bit, of course. That 1h meeting was well rehearsed. I spent entire week preparing, finding better ways to do stuff and retrying entire process dozens of times.
I used JHipster to generate 90% of the service and the rest was just tiny bit of well thought glue code.
Unfortunately, this did not sit well with couple of people whom I made complete mockery of. I had to leave soon after.
- digitalsin 5y agoI like this story.
- deanCommie 5y agoYou shouldn't. It's a terrible story in burning bridges, losing trust with leadership, and basically optimizing for being RIGHT over actually fixing any problems or enacting change. If you find yourself considering doing anything of the sort, you should just quit. And you might say what's the difference - he ended up quitting anyway. Yeah, but he also did so by ruining his reputation with a bunch of people who you might say you don't care about their opinion, but they'll talk to other people whose opinion you DO Care about, and they won't tell a charitable version of this incident. They'll just say that you weren't a team player, that you don't know how to collaborate with others, and to not hire you. What he should have done is spent a week writing a document to senior leadership preparing an executive overview explaining the process he would follow, tools to use, and a detailed breakdown of the estimate of building this system (by someone else mind you) - in a timeframe that he understood the leadership would accept as reasonable - but longer than 1 hour. Then he should have presented the document to them.
- lmilcin 5y ago> What he should have done is spent a week writing a document to senior leadership Except this rarely works. For starters, senior leadership also plays games only on higher level. Second, they don't like wild cards they can't control. And even more, they don't like people who skip chain of command because they want to control the information that passes up from their organization. And this particular company is extra political. In my team, there were only people who worked there for more than 10 years or less than a year. There is special type of person that feels well working there -- one that is fine doing nothing and playing cutthroat politics game. Everybody else quits right away. It was my mistake for not learning this before I joined. I have now a protocol to look for that kind of thing.
- d0gsg0w00f 5y agoWhat's the protocol?
- jakelazaroff 5y agoI mean… OP said his stunt didn’t sit well with people and he had to leave soon after. So if you’re looking for an effective strategy, this ain’t it either.
- loopz 5y agoEither it works or it doesn't. But the role of architect is working with leadership on their level, not being a super hyped-up developer. But cool story nonetheless.
- anonymouse008 5y agoEh — that leadership is in title and name only. Anyone who does this is not a dumbshit, those who get in their way probably are. OP: would love to make your acquaintance, email is checked in profile.
- toss1 5y agoNope, gotta disagree. Sure, the ppl you embarrass might talk. But they won't say much, and likely won't have a lot of credibility. At least this shows a grounding in reality, has the only chance of success (a white paper /proposal was already shown to go nowhere), and supports the (likely few) other people who care about reality & getting things done - ir at least shows them the consequences so they can adjust their own path. If you are good, in this economy, you'll be in demand. So don't play the coward's game - be bold, speak (& show) truth to power.
- nerdponx 5y agoThey could have also just built the PoC app and presented it after the fact. In my experience, managers aren't usually malicious, they're just risk-averse. They've seen good pitches turn into failed projects. A working PoC counts for a whole lot more than just a plan.
- arcanon 5y agoHierarchies dont like to be turned upside down on the scale of competence. Managers hate when their managing cant be attributed to a success. The lesson here is if you do something out of band, if you want it to work, find a way to fake the participation of the powers at be. This sucks, but it is the only option when you only improve one facet of something. If on the otherhand you could do the whole business then it is time for you to found.
- fridif 5y agoSee the game theory on this says it wasn't worth it. Should have just laid low and continued to collect the pay checks while plotting your next move.
- teawrecks 5y agoI'm willing to bet they're doing better in life now than if they had stayed.
- fridif 5y agoI think what I've proposed is a short term micro-optimization where you can eek out a few more checks while you search for a place that supports your creative thinking
- ipaddr 5y agoTotally agree: why stop the pay cheques coming in because of pride.
- galaxyLogic 5y agoIt might seem like an easy situation when nobody's demanding you do a good job, on the contrary, as was in this case you did a good job and got fired because of that. But if you agree to their rules then you must play by their rules, politics, and if you then don't put your best effort into playing politics you may find yourself on the losing side soon.
- fridif 5y agoNot sure I follow the logic, he got fired for trying. Simple solution is don't try, no? Ultimately you will lose in either situation, but one solution prolongs the paychecks.
- galaxyLogic 5y ago
- new_stranger 5y agoI feel like I just read this somewhere... https://news.ycombinator.com/item?id=28564507 https://news.ycombinator.com/item?id=28564507
- catsarebetter 5y agoCurious to know your motivation to do this instead of more tactful methods? Were you aware ahead of time how it could come off?
- lmilcin 5y agoI kinda suspected how it will end up. Honestly, I knew immediately when I joined that I made a huge mistake. My patience for bullshit is very short and I am extra infuriated by people who just don't care to do a good job or help the company (that is paying their salary) and are completely fine crippling their own project for some strange power games. I knew that if I was going to do what they (management) wanted of me -- which was to sit quiet and do nothing, I would have to leave anyway. So I decided to give it a shot and it did not work. But it sure was fun to watch these people trying to gather some response before they had a chance to reach their back channels to solve the problem another way.
- noduerme 5y agoThis is why being a solo dev kicks ass, and why it's incredibly painful for me to read the stories of people who care about doing a good job getting crushed by these managerial hierarchies. First of all, a good CTO would probably fire the lot immediately. But secondly, there's this human dynamic - did you ever have to work on a chem project in high school with 3 randomly assigned classmates? Were you the one who made sure nothing exploded, and then found yourself writing the paper and sharing the grade with them? Sure. Because they knew you would pull the weight. And they would go on to be middle managers just as lazy and work-averse as they were as teenagers. They prey on us... the people who think hard enough to do better. I don't know why you'd ever want to join an organization like that, but it's good you found a booming way out of it.
- galaxyLogic 5y agoI much agree with you. I'm curious, does this apply to pair-programming?
- mdoms 5y agoYou indexed hundreds of millions of records in under an hour? Doubt.
- lmilcin 5y agoI have obviously not indexed the records. I just made an application that would index new and modified records and indexing existing ones would have to be performed separately. Also you need to realize it wouldn't be very wise to run your just written code directly on prod.
- orhmeh09 5y agoI could believe it — I’m pretty sure I’ve done it before using Postgres. Benchmarks appear to confirm this possibility —- 100 million towns indexed in about 35 mins using the standard b tree index, and 20 mins for BRIN on a laptop in 2019: https://blog.crunchydata.com/blog/postgresql-brin-indexes-big-data-performance-with-minimal-storage https://blog.crunchydata.com/blog/postgresql-brin-indexes-bi...
- lmilcin 5y agoElasticsearch is famous for slow indexing. This is basically because it is trading off slow indexing for fast search later. And data to be indexed was incredibly complex and rich. So every detail of every car you own, family member information, addresses, phone numbers, emails, identification documents, details of your contracts, and so on.
- orhmeh09 5y agoNeat! I’ve never used Elastic but might have a use for it un the near future. Do you recommend any resources?
- Andys 5y agoIf you insert in batches, elastic is quite fast. Also it shards and scales horizontally really well without the client having to know that. My cluster of 6 instances could index ~20k large, complex docs per second while still serving production query traffic.
- xmprt 5y agoI applaud the guts that it would have taken to do something like that. That one hour meeting uncovered so many issues with the company that everyone probably already knew but didn't want to talk about - bikeshedding; long, pointless meetings; incompetence and apathy; etc. I sometimes dream of doing something like that when there's endless back and forth about the "right" way to do something (in my experience, no one is actually looking for the right way. It's all just politics and laziness which ends up in a subpar compromise). How did you convince people to sit through the entire hour once they caught on to what you were doing?
- DoreenMichele 5y ago"Guts" sometimes means "Something's gotta give or I'm outta here anyway." Other comments here suggest that was likely the case. Always be leery of emulating such actions. No matter how much detail they give, you will never know the full context and how well your situation compares to theirs.
- xmprt 5y agoThat's why it's only a dream ;)
- noduerme 5y agoHang on. Let's assume the story is false. Why would you want to discourage anyone from having the courage of their conviction that they could find a better way to do something their bosses say is impossible?
- DoreenMichele 5y agoSomeone with real conviction isn't going to be talked out of it by a two paragraph comment by a random internet stranger on HN. You are reading way to much motive and way too much power into a casual observation about how so-called courage is sometimes anything but. People who seem "brave" may not feel they are at all brave. They may feel fed up, resigned to their doom, disgusted, defeated, like arguing or cooperating with known liars is utterly pointless, like they are screwed anyway and nothing they do will ever fix it. Sometimes it's nice to get closure and know you weren't imagining things, it really is as bad as you feared and you can stop wasting your time and energy on a situation that simply will never go anyplace acceptable no matter what you do.
- soheil 5y agoSpecially with ElasticSearch there is a huge difference between the time needed to set it up and the time needed to index it, making sure you don't crash your production DB while doing so, maintaining the index as the tiniest incorrect index insertion can mess up the entire index, and finally making sure the nodes in production don't run out of memory or other resources without extensive and property monitoring in production this is a perfect recipe for downtime. So I see their point.
- noduerme 5y agoWell done. And I'd counter the naysayers by asserting that this is the only right way to go over the top of entrenched laziness. If it cost you the job, so much the better, because you'll never run short of clients desperately seeking someone who can think outside the box the IT department creates with every response saying "sorry, that's just not possible."
- melomal 5y agoThe true definition of less talking more doing. I salute you!
- ToJans 5y agoInteresting story. I did something similar for a bespoke monitoring/helpdesk/knowledge base for a big client in energy trading. However it took me an afternoon and I did get permission: I just installed Oracle text on the existing database and replaced the old search by tags with the Oracle text instance. Typically in these large orgs, writing an app is only one of the first steps; they also need to be maintained, updated, monitored which might require extra budget, which typically nobody wants to pay for because "everything works". If you ever end up in a similar situation, you could consider the environment as a part of the challenge before you figure out a solution. Nevertheless, the 1 hour meeting is a hilarious story, although I personally would avoid burning bridges on the way out: it might hit you in the face in another context when you least expect it....
- sdevonoes 5y agoI like this, but somehow I wonder, was this the right thing to do? I would say it wasn't, because both parties lost: lmilcin had to quit, some colleagues felt ashamed (rightly or not), and at the end Elasticsearch wasn't taken into account as solution. Now, my cynic side of me also wonders: perhaps Elasticsearch wasn't the answer and that's why the proposal was shot down? We'll never know. What I do know is this: software engineering is as much about humans as it is about tech, so even if you know you are (technically) right, don't forget to address the issue from a human perspective (i.e., convince your colleagues that your solution is right, but don't make fun of them by highlighting how one individual (you) can make the whole project "in just 1h").
- cutemonster 5y ago> both parties lost: lmilcin had to quit I think lmilcin won when he got to quit :-) Was the right thing, I think (based on the info available here). There's an opportunity cost in staying and wasting years of ones life at such a place (Note that GP wrote: "I was immediately shot down that the project will take huge amount of time and effort to complete and so is completely off the table" -- that was the stated reason for refusing it)
- rhombocombus 5y agoI LOVE jhipster for rapid prototyping, their tool is the best in class.
- vasilia 5y agoI'm working in a company with 20+ years of the legacy code base. We have a custom search engine fully optimized for our purposes. Decades were spent writing this code. It supports all the newest Linux kernel technologies. The codebase is really good. But every year a new inter tries to show that open source solutions are more reliable and faster. They don't try to write code in 1 hr of course. They spend 10 days on average to show proof of concept that is really fast for benchmarking, but it fails when we try to load testing data. I don't know why people think that nobody checked other solutions before them.
- lmilcin 5y agoThe argument wasn't that ES is wrong choice, but that it will take too much effort to integrate and too long to do so. Group of managers wanted to choose Oracle based technology for this that would further entrench their team and they argumented they can do it faster and cheaper than the development team. They have already succeeded getting sizeable part of business logic implemented in PL/SQL supposedly to improve performance. ES would make all their plans obsolete in a hurry. Obviously, they were not interested in improving the product but in grabbing power, budget and control of the development process. Also they needed to score some points to show they accomplished something during calendar year. Of course development team was also slow to deal with problems and overworked with other projects. This only fueled actions of database operations. So my actions angered both managers of database operations as well as development teams.
- vasilia 5y ago> Group of managers wanted to choose Oracle based technology for this that would further entrench their team and they argumented they can do it faster and cheaper than the development team. That's true. Most Open Source solutions don't work on huge workloads. You can read about Hadoop at Facebook or MySQL. They fully rewrote MySQL engine and partially Hadoop. Oracle solutions work well on huge workloads. This company is not a software developer. The software department is just another team inside the company. The first goal of any non-software company is to reduce costs for IT infrastructure and development. It's not Google or Facebook where software development is a key factor of company capitalization. > Obviously, they were not interested in improving the product but in grabbing power, budget and control of the development process. Also they needed to score some points to show they accomplished something during calendar year. Or just do their job: provides a reliable service for storing data. The main goal of the DB/SysAdmin department is to show 99.99% uptime. The software department could not guarantee this uptime. Of course, if all these software developers are good SRE that's another situation. But I'm sure no one has experience of supporting huge systems for years. > So my actions angered both managers of database operations as well as development teams. If you really want to change something it will be better to make a virtual team from both departments and discuss all pitfalls from both teams. Maybe you are solving non-existent problem.