3 ms·

Southern New England, not in a metro area. White male, stubbornly refusing to leave his hometown area for his career. 2009: $25 per hour, contractor, on site a

Southern New England, not in a metro area. White male, stubbornly refusing to leave his hometown area for his career. 2009: $25 per hour, contractor, on site at giant pharmaceutical company. Built a web app using jsp and tomcat 2010: $55k per year, University IT department. Building web apps in python for university staff. Crazy union contract- 35hr work week and 5 weeks PTO 2012: same job, same benefits, applied for a "reclassification" in the union levels. $75k per year. 2015: $85k per year. Onsite, big media and entertainment company. Cut my teeth building highly available distributed systems. 2017: $135k per year. Remote. Jumped ship to do similar kinds of work at a remote-first startup that could pay more. 2018: same company offered me a raise (without my asking!) $150k per year. I've never worked more than 40 hours on a regular basis, but production issues occasionally produce a 60 hour week, maybe twice a year or so.

cpursley 8y ago
What stack / tech did you use for "highly available distributed systems" and do you feel like it helped differentiate your skill set in terms of hireability?
- mac01021 8y ago
  Sorry if this is too much text, but I feel like I have a decent sense about this, since I've been interviewing recently. Of course, we all live in bubbles, and there may be some tech bubbles extremely different from mine. Your tech stack in a distributed system will often be diverse and wide-ranging. In my opinion, no specific combination of tools is very important for hireability, though commonly recurring characters will certainly help your resume. I write or work on lots of small services, mostly in Java, some Scala, occasionally Go or Python. The JVM is probably the most popular programming platform for this domain but some companies do without it entirely (Go, Python, Node.js are all somewhat popular). Most services run in Docker containers on AWS. I feel like almost everyone is in the cloud these days. There are lots of REST APIs, so you'll want a good handle on HTTP. In my work, even more than HTTP, I've dealt with streaming/realtime data coming over Apache Kafka or AWS Kinesis or AWS SQS. Lots of people use RabbitMQ, but I never have. And I interface with lots of different data stores - Postgres, S3, Dynamodb, Elasticsearch, Cassandra, Redis. And Zookeeper, more for coordination than storage. Zookeeper is fun, but very niche. In terms of building a resume to get hired in this domain, these are some things you want: - Be able to work in a few languages and with several data stores. Be able to talk about which kind of DB is appropriate for what use. - Demonstrate proficiency in concurrent programming and asynchrony via multithreading/futures/promises/channels or whatever the relevant constructs on your platform of choice are. - Have used some kind of streaming solution or message queue (kafka is neat and widely used, if don't already have one in mind) - Have experience with provisioning and orchestration on a cloud platform (AWS, GCP, Azure,...) - Be able to talk about tools and techniques for monitoring, alerting, and tracing in systems where a request or a stream element must traverse many services. - Be able to talk about typical distributed-systems challenges at a conceptual level: * Why and how to partition your data, both in a stream and at rest * How to avoid or resolve consistency issues, when and how to trade consistency for availability, coordinations strategies, CRDTs * At-least-once processing, at-most-once, exactly-once, idempotency, why you would want each - how and when you can achieve each. As a side note, I feel like, once you've built a decent distributed system or application, it's easy to make it look good on a resume. Either it's a high-throughput system and you can say you processed X thousands of requests per second with no downtime for a year, or else it's low throughput and you can double down on the fault-tolerance bit "even when we deliberately crashed half of the servers, the service remained available and end-to-end latency was only reduced by X amount". Of course, you need measure some stuff in order to have numbers to brag about. But hopefully you're collecting those metrics anyway.

cpursley 8y ago

What stack / tech did you use for "highly available distributed systems" and do you feel like it helped differentiate your skill set in terms of hireability?

mac01021 8y ago
Sorry if this is too much text, but I feel like I have a decent sense about this, since I've been interviewing recently. Of course, we all live in bubbles, and there may be some tech bubbles extremely different from mine. Your tech stack in a distributed system will often be diverse and wide-ranging. In my opinion, no specific combination of tools is very important for hireability, though commonly recurring characters will certainly help your resume. I write or work on lots of small services, mostly in Java, some Scala, occasionally Go or Python. The JVM is probably the most popular programming platform for this domain but some companies do without it entirely (Go, Python, Node.js are all somewhat popular). Most services run in Docker containers on AWS. I feel like almost everyone is in the cloud these days. There are lots of REST APIs, so you'll want a good handle on HTTP. In my work, even more than HTTP, I've dealt with streaming/realtime data coming over Apache Kafka or AWS Kinesis or AWS SQS. Lots of people use RabbitMQ, but I never have. And I interface with lots of different data stores - Postgres, S3, Dynamodb, Elasticsearch, Cassandra, Redis. And Zookeeper, more for coordination than storage. Zookeeper is fun, but very niche. In terms of building a resume to get hired in this domain, these are some things you want: - Be able to work in a few languages and with several data stores. Be able to talk about which kind of DB is appropriate for what use. - Demonstrate proficiency in concurrent programming and asynchrony via multithreading/futures/promises/channels or whatever the relevant constructs on your platform of choice are. - Have used some kind of streaming solution or message queue (kafka is neat and widely used, if don't already have one in mind) - Have experience with provisioning and orchestration on a cloud platform (AWS, GCP, Azure,...) - Be able to talk about tools and techniques for monitoring, alerting, and tracing in systems where a request or a stream element must traverse many services. - Be able to talk about typical distributed-systems challenges at a conceptual level: * Why and how to partition your data, both in a stream and at rest * How to avoid or resolve consistency issues, when and how to trade consistency for availability, coordinations strategies, CRDTs * At-least-once processing, at-most-once, exactly-once, idempotency, why you would want each - how and when you can achieve each. As a side note, I feel like, once you've built a decent distributed system or application, it's easy to make it look good on a resume. Either it's a high-throughput system and you can say you processed X thousands of requests per second with no downtime for a year, or else it's low throughput and you can double down on the fault-tolerance bit "even when we deliberately crashed half of the servers, the service remained available and end-to-end latency was only reduced by X amount". Of course, you need measure some stuff in order to have numbers to brag about. But hopefully you're collecting those metrics anyway.