4 ms·

"Applied ..."? On the Optimization Ph.D. qualifying exam, got a "High Pass" and the best score in the class. In optimization answered a question in the Kuhn-T

"Applied ..."? On the Optimization Ph.D. qualifying exam, got a "High Pass" and the best score in the class. In optimization answered a question in the Kuhn-Tucker constraint qualifications and had the paper accepted quickly in Mathematical Programming. Taught linear programming in a well-known business school for 5 years. NYC had a few users, in a loose group, of linear programming but were not very good at it. Somehow I stumbled into the group and was by far the best, but there was no chance of a career or job there. Net, saw next to nothing in "applied". Believing "applied" would have been homeless living on the street, exactly, no exaggeration. Was not able to help my wife -- she died. Had two cats. They got sick, needed medical care I couldn't afford, and died. Thousands of resume copies posted -- no meaningful responses. The Sheriff showed up with several workers with guns, dumped all my clothes, furniture, music materials, e.g., piano, large professional library on the front -- I had to drive off with a few items. At FedEx I saw a good application, got a memo from the Founder, COB, CEO to pursue it, pleased two crucial members of the BOD, saved FedEx twice, but one manager tried to fire me, another said there was "no money in the budget" for me, the stock promised in 3 weeks was WAY late, and I'd been commuting between Memphis and Maryland. Could have done better with a lawn mowing company, LITERALLY. Actually, big picture: Heavily long, likely still, the main US customer, apparently 99+%, for optimization of any kind was US national security. I was manipulated, fooled, and very seriously hurt by the claim "applied". A fact of life: Nearly always people with money, power, and optimization problems don't understand optimization, fear and resent those who do, and choose just to avoid the subject. So, no wife, no cats, starting a business, no boss, sole-solo founder, do have some original applied math deep in the technology, and face just US Internet startup reality. Optimization? Get a 2 hour overview and otherwise f'get about it.

whiterknight 2y ago
You’re attributing these life challenges to your STEM PhD speciality? I am sensitive to your hardship, but optimization has nothing to do with it. People are succeeding with bachelor’s degrees in Latin.
- graycat 2y ago
  The point of the post is that in my long experience there was not much about optimization that in any significant practical career sense was "applied", i.e., no jobs even to keep one from living on the streets, far from a career to buy a house and support a family. The point about my Ph.D. with a lot in optimization is that I was quite well qualified in the field, but even with all those qualifications "applied" was not real, i.e., there just was nothing like a career in optimization applications. Can suspect that, yes, "Ph.D." did seriously damage all career prospects, optimization, math, even computing. The successes I did have were in computing. At the time the math was a small aid. Did get paid for some work in applied optimization on some military problems. Looking back, there was some stranger in the office eager to discuss the weather, etc. with me. Maybe I said the wrong things about some of the US foreign wars, and then the stranger was gone. Might have been some high end military job interview that needed only gung ho attitudes toward foreign policy. Delicate political situation, and I was oblivious about politics. Like, "stick to two subjects, the weather and everyone's health" and avoid "sex, politics, and religion". Early on while I was in a grad program teaching math, some recruiters came from DC desperate for anyone with some math/physics education. I interviewed: Got a offer and took a job right away. Soon bought a new car and got married. In those days, around DC, to get a job, just look in the WaPo, apply, go on the interview, show some knowledge of some of computing, get an offer, compare a few offers, with, say, a 15% raise, and accept one -- worked great. For a while, at GE time sharing national HQ, was the main guy for the applied math library, e.g., the FFT, regression analysis. Later a good background in "applied" optimization, worthless.
  - richrichie 2y ago
    You should have tried wall street. At this point they are the real supporters of mathematicians. We had optimization problems everywhere and had physics PhDs reinventing mathematical algorithms and keeping things “proprietary”. Right now i work in a startup that essentially writes optimization routines for portfolio problems. I will blame your phd advisor.
    - graycat 2y ago
itsthecourier 2y ago
Bro, you spend a really long time getting great at this and from the bottom of my heart I feel your pain and believe we need to improve a lot as a society to take advantage of talents like you better. Thanks for investing so much time to it, I share your sadness for your wife and hope you get ao much good as you tried to bring to the world, man
- graycat 2y ago
  Simpler than that: For "applied" optimization, much of a career, many applications, f'get about it. I was naive, manipulated, and fooled. Now on the OP, there is a book title with "Applied Optimization", and I'm outraged. So: "Always look for the hidden agenda."
frutiger 2y ago
> A fact of life: Nearly always people with money, power, and optimization problems don't understand optimization, fear and resent those who do, and choose just to avoid the subject. Worth considering this hard question: who optimized their life better? Them or us?
- graycat 2y ago
  Yup. Applied Optimization 101: For decisions in life and career, avoid the applied math approach to optimization. In particular, except maybe for some work in US national security, don't try to have the applied math of applied optimization for a career. Don't spend a lot of time studying optimization. Later might be able to be the founder, COB, CEO of a startup where some math is a big advantage.
  - chipdart 2y ago
    > Yup. Applied Optimization 101: For decisions in life and career, avoid the applied math approach to optimization. It pains me to say this, but might it ever crossed your mind that the problem does not lie in applied mathematics of even optimization? We need soft skills to push the output from hard skills, and interacting with decision-makers requires people skills, not hard skills. If you can't communicate with someone, it doesn't really matter what the numbers say. Do you get what I mean?
    - graycat 2y ago
      Fine. But I'm still not finding the private planes, yachts, mansions, or even houses with wife and children of people with careers in "applied" optimization. Computing, startups, venture capital, lots of careers. "Applied" optimization? In the years after my Ph.D., I didn't hear of them. As I was teaching optimization in a business school, the phone didn't ring. The phone did ring for some statistics, one from a company with sales districts and one from a law firm.
chipdart 2y ago
> A fact of life: Nearly always people with money, power, and optimization problems don't understand optimization, fear and resent those who do, and choose just to avoid the subject. Food for thought: the solution to real-world optimization problems is often dictated by constraints instead of optimal values. This means that if you fail to understand the constraints, or even fail to identify them, then whatever your solution to the problem is, it will be wrong. And it will be obvious to those who are aware of the constraints. Now, you're complaining that those presenting you with problems "don't understand optimization". From your anecdotes you were the one tasked with clarifying things to them. From the sound of it, you didn't accomplished that, and it was unclear to stakeholders whether your output even provided any value worth keeping. Have you ever considered the possibility that you failed to understand the actual problems presented to you and even failed to clarify why your output was aligned with anyone's best interests?
- graycat 2y ago
  Naw, you list some mistakes, but I didn't make any of those. > From your anecdotes you were the one tasked with clarifying things to them. From the sound of it, you didn't accomplished that, and it was unclear to stakeholders whether your output even provided any value worth keeping. First, for any application, there has to be some practical interest. My view, there isn't much. The schools of math, engineering, and business have given optimization a big push, back at least to Dantzig, but from my long experience the interest was and is still just too low for "applied" optimization to have much in applications. Cases: Sure, there has been a professor at Princeton who applied optimization to oil refining: What mixture of crude oil to mix into the refinery and what mixture of refined products to take out. Maybe a few, large livestock operations actually do run some diet problem solutions. And can use 0-1 optimization for Sudoku problems? What path for picking orders in a big warehouse, for Amazon or Walmart? A simple traveling salesman problem, and for a good enough solution build a minimum spanning tree and walk around that -- maybe they are doing that already. Assembly line balancing: Assign workers to positions to maximize the speed of the slowest worker assignment. Is anyone actually doing that? Even if they are, the solution is quite simple. Yes, a start on P vs NP was at Bell Labs designing networks. So, maybe with the Internet there are still valuable applications? Considered that. Got an interview at a company trying that. They were impressed by what I'd done at FedEx, but they were nearly dead and, I suspect, soon died. Maybe with big logistics, ocean, rail, trucks, warehouses, there are some big logistics problems where optimization could save a lot -- applications enough for careers? Better than grass mowing? When I got my Ph.D., the Chair of my dissertation orals committee was a big name in logistics -- saw no evidence of significant interest in applications. No ones in the halls. Phone not ringing. No suggestions of contacts for me. Look, when there is a big need, ESPECIALLY when there is big money involved, it soon gets obvious, and the US economy gets to it right away. In that, "applied" optimization is not hot, warm, or much above freezing. Right, you are mentioning formulation: (1) They had already formulated a 0-1 optimization problem. It had 40,000 constraints and 600,000 variables. They had tried the then popular simulated annealing, ran for days, and quit. So, the formulation was done and not mine. I worked hard, with the IBM OSL (optimization subroutine library), did 900 primal-dual iterations, Lagrangian relaxation, got a feasible solution within 0.025% of optimality, within two weeks, for free, a free sample, and never heard from them again. They resented and were afraid of my success. (2) Another company was working a little more generally in optimization. Had a crude heuristic running. On some of their problems, 0-1, linear, again was successful with the OSL, and got only insults and resentment. Continued on, gave them a nice formulation, better than their heuristic, and path through optimization, and got fired. They'd hired me and wanted to fire me before 6 months was up. They were not very good with linear programming at all, and I was a LOT better at what they were doing in the formulation, math, and computing, and their reaction was they didn't want me for competition. (3) In a military group, did well with some non-linear optimization (their formulation). Then they had a challenging strategic problem. I did a formulation of a Monte-Carlo solution and wrote and ran the code (used an Oak Ridge random number generator I'd programmed in assembler). They called in a famous probability professor for a review. His remark was that there was no way the Monte-Carlo could "fathom" the tree. He was right; the tree was huge. But each trial of my Monte-Carlo yielded at each point in time a random variable on 0-15, and the law of large numbers applied right away. It wasn't D-day, but suppose it was: The tree of possibilities was enormous, but the, say, number of Allied soldiers killed was, what, 0-200,000. So, each trial give a random variable value at, say, each second, for, say, 48 hours -- the law of large numbers applies and could tell Ike the distribution of number of deaths, the expected values, the median, the variance for each second of the 48 hours. Passed the review. One guy there used my random number generator on one of his old problems, got significantly different results, was afraid, said "I don't want you in the center of all my projects", and I got ignored on the way to being fired. (4) At FedEx, had written a program that showed the BOD that the program made the fleet scheduling easy enough and saved the company. So, to do better, formulated a set covering direction. Savings? 1% would have been $millions a year. The founder, COB, CEO wrote a memo making that my project, but my boss, a Senior VP, said that there was no money in the budget for me; I'd been commuting between Memphis and Maryland where my wife was in her Ph.D. program; the stock promised in three weeks was very late; and I went for a Ph.D. Actually another student at another school ran with my set covering formulation for his dissertation. The high level, overview, simple fact of life, is as I described: There just is no real career in "applied" optimization. That horse is nearly dead and should not be further flogged. Millions of US families have a house, stable marriage, and healthy children, and I'd believe that fewer than 20 of those families are supported by careers in "applied" optimization -- maybe 0 families.
epalarepa 2y ago
Hey I'm just replying because I currently have a BSc with Honors in math and a double major with econ but I'm having trouble in the current market. I also am interested in pursuing a master's in combinatorics or optimization. Is there anything you recommend besides not getting the PhD?
- mjburgess 2y ago
  What is your ideal career in 10 years: topics, day-in-the-life, talens you're using, etc.? Work backwards, check those jobs exist, look at what qualifications they'd benefit from
  - epalarepa 2y ago
    I know it goes against the thread but I am considering a career in academia. I'd like to become a Prof if possible. If that doesn't work out then a job that allows me to solve interesting math problems everyday. I was thinking optimization research but maybe not from how this thread is going. Besides that working for Renaissance tech or a similar company also interests me.
    - mjburgess 2y ago
      I'd strongly encourage you to leave academia for at least 2 years, preferably a little more -- and go intern, or work at, at least 2 or 3 different companies. This will benefit your mental health, working practices, and sense of perspective. Academia is full of superstitions and misunderstandings about 'industry', and is an ideologically closed system designed to present itself and its practices as of the highest value. Including, as mentioned here, hijacking 'industrial values' (such as utility, applicability, etc.) and presenting itself with them. This is in part a straight deception: academics need to lie about this to governments and other funding agencies in order to get funding. And in part naivety, their sense of 'application' is so far removed from any, that they are poor judges of it. Enabling these deceptions is, of course, that academic output does occasionally have an impact over longer time horizons. But it is more the case that something written ~30 years ago will today find some use by a small team somewhere -- which might have a lot of impact, but isnt on the scale or time lines you'd need for your research today to give you an ROI. Indeed, as you'll see if you're in industry long enough, there is often something very suspicious and deceptive about the values of various research communities. Even when researchers are told: no one cares about this, this does not have the value you say it does, this isn't a real problem etc.-- they continue. Why? Because they aren't interested in solving problems people have, they are interested in whatever puzzles are current within their community. If you want to solve real problems using research, it is better to go out and undertand what problems people actually have, then go back into research aiming to help them.

whiterknight 2y ago

You’re attributing these life challenges to your STEM PhD speciality? I am sensitive to your hardship, but optimization has nothing to do with it. People are succeeding with bachelor’s degrees in Latin.

graycat 2y ago
The point of the post is that in my long experience there was not much about optimization that in any significant practical career sense was "applied", i.e., no jobs even to keep one from living on the streets, far from a career to buy a house and support a family. The point about my Ph.D. with a lot in optimization is that I was quite well qualified in the field, but even with all those qualifications "applied" was not real, i.e., there just was nothing like a career in optimization applications. Can suspect that, yes, "Ph.D." did seriously damage all career prospects, optimization, math, even computing. The successes I did have were in computing. At the time the math was a small aid. Did get paid for some work in applied optimization on some military problems. Looking back, there was some stranger in the office eager to discuss the weather, etc. with me. Maybe I said the wrong things about some of the US foreign wars, and then the stranger was gone. Might have been some high end military job interview that needed only gung ho attitudes toward foreign policy. Delicate political situation, and I was oblivious about politics. Like, "stick to two subjects, the weather and everyone's health" and avoid "sex, politics, and religion". Early on while I was in a grad program teaching math, some recruiters came from DC desperate for anyone with some math/physics education. I interviewed: Got a offer and took a job right away. Soon bought a new car and got married. In those days, around DC, to get a job, just look in the WaPo, apply, go on the interview, show some knowledge of some of computing, get an offer, compare a few offers, with, say, a 15% raise, and accept one -- worked great. For a while, at GE time sharing national HQ, was the main guy for the applied math library, e.g., the FFT, regression analysis. Later a good background in "applied" optimization, worthless.
- richrichie 2y ago
  You should have tried wall street. At this point they are the real supporters of mathematicians. We had optimization problems everywhere and had physics PhDs reinventing mathematical algorithms and keeping things “proprietary”. Right now i work in a startup that essentially writes optimization routines for portfolio problems. I will blame your phd advisor.
  - graycat 2y ago

itsthecourier 2y ago

Bro, you spend a really long time getting great at this and from the bottom of my heart I feel your pain and believe we need to improve a lot as a society to take advantage of talents like you better. Thanks for investing so much time to it, I share your sadness for your wife and hope you get ao much good as you tried to bring to the world, man

graycat 2y ago
Simpler than that: For "applied" optimization, much of a career, many applications, f'get about it. I was naive, manipulated, and fooled. Now on the OP, there is a book title with "Applied Optimization", and I'm outraged. So: "Always look for the hidden agenda."

frutiger 2y ago

graycat 2y ago
Yup. Applied Optimization 101: For decisions in life and career, avoid the applied math approach to optimization. In particular, except maybe for some work in US national security, don't try to have the applied math of applied optimization for a career. Don't spend a lot of time studying optimization. Later might be able to be the founder, COB, CEO of a startup where some math is a big advantage.
- chipdart 2y ago
  > Yup. Applied Optimization 101: For decisions in life and career, avoid the applied math approach to optimization. It pains me to say this, but might it ever crossed your mind that the problem does not lie in applied mathematics of even optimization? We need soft skills to push the output from hard skills, and interacting with decision-makers requires people skills, not hard skills. If you can't communicate with someone, it doesn't really matter what the numbers say. Do you get what I mean?
  - graycat 2y ago
    Fine. But I'm still not finding the private planes, yachts, mansions, or even houses with wife and children of people with careers in "applied" optimization. Computing, startups, venture capital, lots of careers. "Applied" optimization? In the years after my Ph.D., I didn't hear of them. As I was teaching optimization in a business school, the phone didn't ring. The phone did ring for some statistics, one from a company with sales districts and one from a law firm.

chipdart 2y ago

> A fact of life: Nearly always people with money, power, and optimization problems don't understand optimization, fear and resent those who do, and choose just to avoid the subject. Food for thought: the solution to real-world optimization problems is often dictated by constraints instead of optimal values. This means that if you fail to understand the constraints, or even fail to identify them, then whatever your solution to the problem is, it will be wrong. And it will be obvious to those who are aware of the constraints. Now, you're complaining that those presenting you with problems "don't understand optimization". From your anecdotes you were the one tasked with clarifying things to them. From the sound of it, you didn't accomplished that, and it was unclear to stakeholders whether your output even provided any value worth keeping. Have you ever considered the possibility that you failed to understand the actual problems presented to you and even failed to clarify why your output was aligned with anyone's best interests?

graycat 2y ago
Naw, you list some mistakes, but I didn't make any of those. > From your anecdotes you were the one tasked with clarifying things to them. From the sound of it, you didn't accomplished that, and it was unclear to stakeholders whether your output even provided any value worth keeping. First, for any application, there has to be some practical interest. My view, there isn't much. The schools of math, engineering, and business have given optimization a big push, back at least to Dantzig, but from my long experience the interest was and is still just too low for "applied" optimization to have much in applications. Cases: Sure, there has been a professor at Princeton who applied optimization to oil refining: What mixture of crude oil to mix into the refinery and what mixture of refined products to take out. Maybe a few, large livestock operations actually do run some diet problem solutions. And can use 0-1 optimization for Sudoku problems? What path for picking orders in a big warehouse, for Amazon or Walmart? A simple traveling salesman problem, and for a good enough solution build a minimum spanning tree and walk around that -- maybe they are doing that already. Assembly line balancing: Assign workers to positions to maximize the speed of the slowest worker assignment. Is anyone actually doing that? Even if they are, the solution is quite simple. Yes, a start on P vs NP was at Bell Labs designing networks. So, maybe with the Internet there are still valuable applications? Considered that. Got an interview at a company trying that. They were impressed by what I'd done at FedEx, but they were nearly dead and, I suspect, soon died. Maybe with big logistics, ocean, rail, trucks, warehouses, there are some big logistics problems where optimization could save a lot -- applications enough for careers? Better than grass mowing? When I got my Ph.D., the Chair of my dissertation orals committee was a big name in logistics -- saw no evidence of significant interest in applications. No ones in the halls. Phone not ringing. No suggestions of contacts for me. Look, when there is a big need, ESPECIALLY when there is big money involved, it soon gets obvious, and the US economy gets to it right away. In that, "applied" optimization is not hot, warm, or much above freezing. Right, you are mentioning formulation: (1) They had already formulated a 0-1 optimization problem. It had 40,000 constraints and 600,000 variables. They had tried the then popular simulated annealing, ran for days, and quit. So, the formulation was done and not mine. I worked hard, with the IBM OSL (optimization subroutine library), did 900 primal-dual iterations, Lagrangian relaxation, got a feasible solution within 0.025% of optimality, within two weeks, for free, a free sample, and never heard from them again. They resented and were afraid of my success. (2) Another company was working a little more generally in optimization. Had a crude heuristic running. On some of their problems, 0-1, linear, again was successful with the OSL, and got only insults and resentment. Continued on, gave them a nice formulation, better than their heuristic, and path through optimization, and got fired. They'd hired me and wanted to fire me before 6 months was up. They were not very good with linear programming at all, and I was a LOT better at what they were doing in the formulation, math, and computing, and their reaction was they didn't want me for competition. (3) In a military group, did well with some non-linear optimization (their formulation). Then they had a challenging strategic problem. I did a formulation of a Monte-Carlo solution and wrote and ran the code (used an Oak Ridge random number generator I'd programmed in assembler). They called in a famous probability professor for a review. His remark was that there was no way the Monte-Carlo could "fathom" the tree. He was right; the tree was huge. But each trial of my Monte-Carlo yielded at each point in time a random variable on 0-15, and the law of large numbers applied right away. It wasn't D-day, but suppose it was: The tree of possibilities was enormous, but the, say, number of Allied soldiers killed was, what, 0-200,000. So, each trial give a random variable value at, say, each second, for, say, 48 hours -- the law of large numbers applies and could tell Ike the distribution of number of deaths, the expected values, the median, the variance for each second of the 48 hours. Passed the review. One guy there used my random number generator on one of his old problems, got significantly different results, was afraid, said "I don't want you in the center of all my projects", and I got ignored on the way to being fired. (4) At FedEx, had written a program that showed the BOD that the program made the fleet scheduling easy enough and saved the company. So, to do better, formulated a set covering direction. Savings? 1% would have been $millions a year. The founder, COB, CEO wrote a memo making that my project, but my boss, a Senior VP, said that there was no money in the budget for me; I'd been commuting between Memphis and Maryland where my wife was in her Ph.D. program; the stock promised in three weeks was very late; and I went for a Ph.D. Actually another student at another school ran with my set covering formulation for his dissertation. The high level, overview, simple fact of life, is as I described: There just is no real career in "applied" optimization. That horse is nearly dead and should not be further flogged. Millions of US families have a house, stable marriage, and healthy children, and I'd believe that fewer than 20 of those families are supported by careers in "applied" optimization -- maybe 0 families.

epalarepa 2y ago

Hey I'm just replying because I currently have a BSc with Honors in math and a double major with econ but I'm having trouble in the current market. I also am interested in pursuing a master's in combinatorics or optimization. Is there anything you recommend besides not getting the PhD?

mjburgess 2y ago
What is your ideal career in 10 years: topics, day-in-the-life, talens you're using, etc.? Work backwards, check those jobs exist, look at what qualifications they'd benefit from
- epalarepa 2y ago
  I know it goes against the thread but I am considering a career in academia. I'd like to become a Prof if possible. If that doesn't work out then a job that allows me to solve interesting math problems everyday. I was thinking optimization research but maybe not from how this thread is going. Besides that working for Renaissance tech or a similar company also interests me.
  - mjburgess 2y ago
    I'd strongly encourage you to leave academia for at least 2 years, preferably a little more -- and go intern, or work at, at least 2 or 3 different companies. This will benefit your mental health, working practices, and sense of perspective. Academia is full of superstitions and misunderstandings about 'industry', and is an ideologically closed system designed to present itself and its practices as of the highest value. Including, as mentioned here, hijacking 'industrial values' (such as utility, applicability, etc.) and presenting itself with them. This is in part a straight deception: academics need to lie about this to governments and other funding agencies in order to get funding. And in part naivety, their sense of 'application' is so far removed from any, that they are poor judges of it. Enabling these deceptions is, of course, that academic output does occasionally have an impact over longer time horizons. But it is more the case that something written ~30 years ago will today find some use by a small team somewhere -- which might have a lot of impact, but isnt on the scale or time lines you'd need for your research today to give you an ROI. Indeed, as you'll see if you're in industry long enough, there is often something very suspicious and deceptive about the values of various research communities. Even when researchers are told: no one cares about this, this does not have the value you say it does, this isn't a real problem etc.-- they continue. Why? Because they aren't interested in solving problems people have, they are interested in whatever puzzles are current within their community. If you want to solve real problems using research, it is better to go out and undertand what problems people actually have, then go back into research aiming to help them.