3 ms·

Get me in a whiteboard session, and you get responses that I've spent 15 seconds tops thinking about. Send me an email and you'll get responses that I've spent

by chrsig 4y ago

Get me in a whiteboard session, and you get responses that I've spent 15 seconds tops thinking about. Send me an email and you'll get responses that I've spent an hour or two thinking about. There'll also be a written record of the exchange for future reference. The time spent writing the email isn't wasted. It's time saved putting together documentation.

rented_mule 4y ago
I think the huge gap between the parent and grandparent shows that async vs. sync optimality is heavily dependent on the work being done, the people doing it, and the team/company where it's being done. An example on the async side... I worked with a couple of fantastic engineers who would come back with amazing solutions if you let them go away and think about it in depth. That worked particularly well for problems which one person could get their head around with deep focus. If you asked those two for a solution in a meeting or at a whiteboard they'd literally say "I won't answer you in real time." I've seen a lot of solid foundational infrastructure created this way. On the sync side, I worked on a team that needed relatively senior software engineering knowledge (e.g., stream processing of many MBs/second of data before their were decent open source solutions) and PhD+ level math knowledge. The most valuable tradeoffs tended to be across the software engineering / math boundary (e.g., we could use much simpler math if we could process the data faster or process much less data if the math were a little more sophisticated). But it's very rare to find people who are experts at both. Put experts from each side who are good at interactive communication at a whiteboard and tons of value pours out in an hour or two as they leverage each other's knowledge. I saw one such discussion lead directly to 3,500 LOC in one subsystem being replaced with 500 lines of much simpler code and the math being reduced from graduate level to undergraduate level at the same time.
- zmgsabst 4y ago
  Or do both: - have a meeting where you discuss and build a shared intuition for the problem - tomorrow (or next week or…), have a meeting where you solve the problem This is closer to what large data processing does: Synchronize the data, distribute tasks to work, synchronize and compile the results.
- plepbpl 4y ago
  Curious to hear more about the team/problem. (I’m a mathematician in the middle of trying to convince companies that I’m useful)
  - rented_mule 4y ago
    On the off chance you see this... I don't want to get too close to describing things as I'm trying to keep this account relatively anonymous so I feel more comfortable being very honest about things. That being said... The problem was analogous to pricing at Uber or Lyft. The product was very different - not related to transportation in any way. But trying to balance supply and demand through pricing. Supply and demand could vary wildly by the minute and pricing was per customer, so that involved adjusting ~1M prices dozens of times per day. We used control theory, game theory, time series predictions, and lots of other things to understand / control the dynamics of the market and the competition within subsets of the market. I'm convinced the onset of the pandemic would have crushed the company's revenue if this dynamic pricing was not in place. Instead, we merely needed to add some new boundaries to contain what the system was doing (boundaries that were unheard of pre-pandemic like entire sectors of the world economy closing down). As a software engineer with a BS in Math, I did the first version of this out of intuition without knowing anything about the fields I mentioned above. I was accidentally trying to reinvent control theory without knowing it existed (knowing to do a web search for "control theory" would have helped a lot!). As time went by and we hired some people with more applicable math knowledge, they were some combination of impressed and amused at what I'd done. And they were able to take it to the next level. Seeing this, and having become the team's manager, I started to hire people with much more advanced math and stats knowledge. Advanced stats people came last and they should have been earlier. It turned out we had shipped multiple experiments that we shouldn't have because we didn't understand that we weren't measuring what we thought we were. As the system got better and better, measuring the impact of our system on indirect market effects became the hardest part of the problem. For example, a change will favor some competitors in the market - is that positive overall? If you think there's a simple-ish answer to that question (like I did at one point), you probably aren't qualified to design / analyze these experiments. wink Where this eventually led, before I moved on, was the realization that we didn't just need a software architecture, but also a math architecture that helped keep the mathematical dynamics sane and possible to reason about. And it required a lot of effort to help the non-math people in senior management understand this wasn't black magic, but something that needed to be reasoned through by people who understood mathematical rigor (i.e., we needed to continue hiring mathematicians). That's a tough sell given that it's really hard to understand the value of mathematical rigor when you don't have it yourself. We weren't formally proving anything, but when things get complicated, it's really important to understand exactly what assumptions are necessary and when that gets difficult, it's hard to avoid the need for people who know mathematical rigor.
watwut 4y ago
> Send me an email and you'll get responses that I've spent an hour or two thinking about. Realistically, quite a lot of emails are ignored until engineer really really need to answer and then he quick writes something hard to interpret.

by chrsig 4y ago

rented_mule 4y ago

I think the huge gap between the parent and grandparent shows that async vs. sync optimality is heavily dependent on the work being done, the people doing it, and the team/company where it's being done. An example on the async side... I worked with a couple of fantastic engineers who would come back with amazing solutions if you let them go away and think about it in depth. That worked particularly well for problems which one person could get their head around with deep focus. If you asked those two for a solution in a meeting or at a whiteboard they'd literally say "I won't answer you in real time." I've seen a lot of solid foundational infrastructure created this way. On the sync side, I worked on a team that needed relatively senior software engineering knowledge (e.g., stream processing of many MBs/second of data before their were decent open source solutions) and PhD+ level math knowledge. The most valuable tradeoffs tended to be across the software engineering / math boundary (e.g., we could use much simpler math if we could process the data faster or process much less data if the math were a little more sophisticated). But it's very rare to find people who are experts at both. Put experts from each side who are good at interactive communication at a whiteboard and tons of value pours out in an hour or two as they leverage each other's knowledge. I saw one such discussion lead directly to 3,500 LOC in one subsystem being replaced with 500 lines of much simpler code and the math being reduced from graduate level to undergraduate level at the same time.

zmgsabst 4y ago
Or do both: - have a meeting where you discuss and build a shared intuition for the problem - tomorrow (or next week or…), have a meeting where you solve the problem This is closer to what large data processing does: Synchronize the data, distribute tasks to work, synchronize and compile the results.
plepbpl 4y ago
Curious to hear more about the team/problem. (I’m a mathematician in the middle of trying to convince companies that I’m useful)
- rented_mule 4y ago
  On the off chance you see this... I don't want to get too close to describing things as I'm trying to keep this account relatively anonymous so I feel more comfortable being very honest about things. That being said... The problem was analogous to pricing at Uber or Lyft. The product was very different - not related to transportation in any way. But trying to balance supply and demand through pricing. Supply and demand could vary wildly by the minute and pricing was per customer, so that involved adjusting ~1M prices dozens of times per day. We used control theory, game theory, time series predictions, and lots of other things to understand / control the dynamics of the market and the competition within subsets of the market. I'm convinced the onset of the pandemic would have crushed the company's revenue if this dynamic pricing was not in place. Instead, we merely needed to add some new boundaries to contain what the system was doing (boundaries that were unheard of pre-pandemic like entire sectors of the world economy closing down). As a software engineer with a BS in Math, I did the first version of this out of intuition without knowing anything about the fields I mentioned above. I was accidentally trying to reinvent control theory without knowing it existed (knowing to do a web search for "control theory" would have helped a lot!). As time went by and we hired some people with more applicable math knowledge, they were some combination of impressed and amused at what I'd done. And they were able to take it to the next level. Seeing this, and having become the team's manager, I started to hire people with much more advanced math and stats knowledge. Advanced stats people came last and they should have been earlier. It turned out we had shipped multiple experiments that we shouldn't have because we didn't understand that we weren't measuring what we thought we were. As the system got better and better, measuring the impact of our system on indirect market effects became the hardest part of the problem. For example, a change will favor some competitors in the market - is that positive overall? If you think there's a simple-ish answer to that question (like I did at one point), you probably aren't qualified to design / analyze these experiments. wink Where this eventually led, before I moved on, was the realization that we didn't just need a software architecture, but also a math architecture that helped keep the mathematical dynamics sane and possible to reason about. And it required a lot of effort to help the non-math people in senior management understand this wasn't black magic, but something that needed to be reasoned through by people who understood mathematical rigor (i.e., we needed to continue hiring mathematicians). That's a tough sell given that it's really hard to understand the value of mathematical rigor when you don't have it yourself. We weren't formally proving anything, but when things get complicated, it's really important to understand exactly what assumptions are necessary and when that gets difficult, it's hard to avoid the need for people who know mathematical rigor.

watwut 4y ago

> Send me an email and you'll get responses that I've spent an hour or two thinking about. Realistically, quite a lot of emails are ignored until engineer really really need to answer and then he quick writes something hard to interpret.