4 ms·

Part II For what companies to go to to get going on a career in such applied math, I can't think of any. Mostly people and their organizations would rather wa

by graycat 12y ago

Part II For what companies to go to to get going on a career in such applied math, I can't think of any. Mostly people and their organizations would rather waste money than get involved with mathematicians doing work only mathematicians understand. Again, if want to make math pay, just sell the results, usually as a vendor outside the customer. If are inside the company, then likely no good work will go unpunished. Basically to be successful, you will need the strong personal support of the CEO and hopefully the Board -- literally. And the CEO will routinely have to go around and break 2 x 4s over the heads of recalcitrant middle managers to see if they are just asleep or really dead -- otherwise those middle managers can sabotage you with high determination and creativity. E.g., twice I saved FedEx from going out of business and had much more that would have saved FedEx maybe tens of millions of dollars a year, but I got flack and push back and contempt and none of the promised stock. Such are some of the challenges in getting stuff done. But successful people have been overcoming challenges, some much worse than I faced at FedEx, for many centuries. Now that you know about the possibilities of some of the absurd challenges, don't be surprised. Really, one of the best rules is just to keep your good technical work essentially a secret until you have a solution that is just too darned good to refuse, and then pull back the curtain, let people see the results and also see that it's too late to fight with you. Better still, just provide such results as a business and own the business. E.g., in some cases, just do the work for free, show the results, and then announce "You can have this for your real operations for a fee of only 15% of the savings." Once there was an in-house research group that was successful. It was for WestVaco paper -- think specialty papers, e.g., coated papers for milk cartons. So, they had Ph.D. chem eng guys running their plants in jungles, etc., that is, where the trees were, but it was a family owned business with HQ in NYC. The research shop was in Maryland, and I got an explanation from them: Each year Research went to NYC for their budget. Always the CEO, family guy, offered much more money for the Research shop than the shop was willing to accept (first-cut, good budget situation!). Why? The rules: Research picks their own projects. Projects tend to run from a few months to a few years. One project in 10 gets to operations. When Research has a project ready for operations, they approach the relevant operational managers. In the first three years, savings are allocated half to Research and half to the operational unit. After three years, all the savings are allocated to the operational unit. With this accounting, Research was returning $3 to the company for each $1 in their budget. So, sure, HQ wanted to increase the budget, but Research didn't want to have to find ways to return $3 for each extra budget $1 so didn't want a bigger budget! It is just crucial that the CEO is supporting such in-house innovation. The rules are also important. Ah, while I was in grad school, to make enough money to support my wife and I while we both finished our Ph.D.s, I took a job at a DC area think tank working for the US Navy. At one point our problem sponsor group wanted an analysis of a special, worrisome scenario of global nuclear war but limited to sea -- they wanted to know how long the US SSBN fleet could survive if it was just held in reserve and didn't shoot its missiles with nuke warheads. And they wanted the results in two weeks. Gee, two weeks to model global nuclear war limited to sea! How generous! Good that they were not in a real hurry! Besides it was good timing since the day after the due date my wife had already gotten us reservations for a vacation at Shenandoah! Well, there was a guy B. Koopman who in WWII had written a report OEG-56 on finding things at sea. So, given A and B wandering around at sea, when might they have an encounter? Well, Koopman took the area of the sea, the two velocities, and the detection radius and came up with an arrival rate for a Poisson process. Hmm .... So, list the Red forces and Blue forces (the US SSBNs were part of the Blue forces), that is, list airplanes, helicopters, destroyers, battleships, attack submarines, etc., the speed of each, and the detection radius. Then have a table with a row for each Blue weapon type and a column for each Red weapon type and the detection radius and, if there was an encounter, the probability that one, the other, or both died. Now tap lightly with an independence assumption (maybe not wildly unjustified), realize that a sum of independent Poisson processes is again a Poisson process with arrival rate the sum of the contributing arrival rates, and get for the state of the war a continuous time, discrete state space (very large from a combinatorial explosion) Markov process subordinated to a Poisson process. So, yes, there is a closed form solution as a matrix exponential for the transition probabilities from state to state, but that matrix exponential is absurdly large. Instead using Monte Carlo to run off, say, 500 sample paths was easy. I typed in the code. Prof J. Keilson did a technical review. He said, "There's no way you can fathom that enormous state space.". I said, "At each point in time, say, 10 days into the war, the number of SSBNs remaining is a random variable. Moreover it is bounded so has finite expectation and finite variance. So, the law of large numbers applies. So, run off 500 independent, identically distributed samples, add them up, divide by 500, and get the expectation within a gnat's ass nearly all the time. Really the Monte Carlo puts the effort where the action is.". He agreed with me and passed my work. The Navy got their results on time, and my wife got her vacation on time. Later I was told that my work had been sold to a leading US intelligence agency. I could tell you which one, but then I'd have to ...! I thought that the whole effort, which ignored so much maybe crucially important detail, was close to a joke, but, surprisingly, the output did look reasonable. Also, the two weeks was also so short it made the whole exercise next to absurd, another joke. The group I was working for didn't trust me to get results on time so had another person working independently. Well, at the end of the two weeks, the other person had nothing at all. So, I'd won. Office politics: One guy said "You write nice computer programs" and otherwise I got no thanks but just some jealousy, contempt, and silence. So, soon afterward when I'd gotten my Ph.D., I wanted to leave and did. Again, no good deed need go unpunished. Again, own the business, use math as an internal secret sauce and technological advantage, and sell just the results.

hgibbs 12y ago
Thanks. I really appreciate you taking the time to write so much. Honestly, this seems to be the best description of potential careers that I have ever gotten and I feel so much more assured about my choice to study Maths now. It is also really interesting to hear about how colourful such a career could be! You sound like, despite the politics, you have had a really interesting career and have done the kind of things that I would exactly like to do! Again, thanks so much. One last question, what are you doing with your life right now? It seems like you have had a pretty broad career and I am interested in the end game. If you are retired, what is your lifestyle like and in particular how has your career impacted your life as a retiree.
- graycat 12y ago
  Part II For now, sure, I know some math and can stir up some more, and I know some computing and can learn more. So, I've got a 1.8 GHz AMD single core processor, Windows XP Professional SP3 (I have an official copy of Windows 7 Professional on DVD but have seen no great reason to go to all the trouble to install it and rebuild all my software environment yet), three hard disks, 100 million files, .NET Framework 4.0, a good text editor (KEdit), Visual Basic .NET (comes with the .NET Framework), ASP.NET, ADO.NET, IIS (low level Web server), SQL Server Express, etc. I thought, why not go after about 2/3rds of Internet search, the safe for work part served at best poorly by looking for keywords/phrases? So, how to do that? Not with just routine software! And not with anything commonly talked about for search! I stirred up some math, typed the theorems and proofs into Knuth's TeX, worked up a scalable architecture, and typed in the software. Maybe I should have used Redis, but instead I thought that writing my own session state server would be faster than even understanding Redis. So, my session state server is single threaded ("Look, Ma, no concurrency problems"), for faster lookup rates is trivial to run as several instances as in sharding, and is just some TCP/IP sockets, some de/serialization of instances of my session state class, and uses two instances of a .NET collection class. Simple. It's fast! A server for less than $1500 should be able to keep session state for an hour of inactivity for each user and do the session state work for sending 5000+ Web pages a second. Two standard racks should be able to handle session state for the world. The rest of the software is also readily scalable also from just simple sharding. Currently I have one bug in one Web page -- it's not handling session state just right! But I have the fix in code in another Web page and should copy it over today! My interactive development environment is just KEdit with about 100 macros and some careful use of file system directories. I'm using SQL Server only to record the data from the users and for the results of some batch computations; at one point I actually do make use of a transaction; the data for the searches is drawn from SQL Server with a batch program (run it maybe once a day); some solid state drives (write rarely, read thousands of times a second) should do wonders for the data for the searches. I was about to fix the bug in the Web page but took some opportunities to gather some good initial data. The site will start focused and only slowly grow to be comprehensive -- right, at first do some things that don't scale and please some niche group of users a lot instead of trying to please 2+ billion users a little. Currently the database has only some meaningless data I put in for first testing of the software. It's about time to load in some of the good initial data I have. Then give a critical review, go live, etc. It's getting there. All the work uniquely mine has been fast, fun, and easy, but the whole project has taken far, far, far too long. Why? I worked through about a cubic foot of books and 6000+ Web pages of documentation of Windows, .NET, Visual Basic .NET, ASP.NET, ADO.NET, SQL Server, etc. The main problems: (1) Badly written, obscure documentation (worst bottleneck in the future of computing); (2) computer viruses; (3) SQL Server installation bugs destroying my boot partition requiring rebuilding starting with the XP DVD (barbed wire enema with an unanesthetized upper molar root canal procedure); (4) SQL Server management and administration (e.g., a week of throwing stuff against a wall to see what sticks just to get a SQL Server connection string that will let code for a server side Web page connect with SQL Server); (5) clean, smooth means of system backup and recovery (including for both user data and bootable partitions); (6) Sony DVD drives that quit for no good reason (and inability to buy more IDE DVD drives). Good stuff: (1) KEdit and its macro language; (2) the scripting language Rexx (Microsoft's PowerShell may also be terrific but have yet to move to it); (3) NTFS (fantastic); (4) Visual Basic .NET design, functionality, speed of compilation, compiler error messages, minimal bugs (sweetheart language); (5) what ASP.NET does when it compiles a Visual Basic program (enough for a really nice IDE); (6) NTBACKUP (once understand how to use it, e.g., do have to ask to save "system state", whatever the heck that is, or the saved copy, restored, won't boot -- learn this and how to get around it the hard way, weeks of work); (7) XCOPY; (8) the tools to have server side Web page code write to a log file; (9) Firefox (except for virus vulnerabilities); (10) the classes in the .NET Framework (once learn how to learn about them and use them); (11) Adobe Acrobat (except for virus vulnerabilities); (12) the ability of XP to find device drivers and recognize new devices; (13) Microsoft's anti-virus Safety Scanner (if only from CP67/CMS and Multics, there should be no virus vulnerabilities, but since there are the safety scanner is terrific to have); (14) Knuth's TeX; (15) the Western Digital Passport Ultra 2 TB USB drive! I'm not retired or retiring! Likely I'm still 100% unemployable at anything that would pay enough to let me keep a car going to commute to the job. Y Combinator and VCs want nothing to do with me. But if I can get my Web site up to a search a second, on a server for about $2000, I will be in decent shape financially and on the way to organic growth for my business and much more. Then I'll get a nice house, a building for some cars, at least one Corvette, visit the rest of the family still alive, take off two weeks to pig out on lobster in Maine, get some good grape juice from between Beaune and Dijon, do some cooking, give some dinner parties, go to concerts and operas, continue with the business, go to seminars on mathematical physics, etc. I'll implement and deploy my server farm monitoring techniques and maybe spin it off as a separate business. I have some guesses for some approaches to real AI and might try to implement those. And I will get a kitty cat! We'll see! Good luck on your work with math. Maybe what I typed in here will help; I wish I'd known all that when I started.

by graycat 12y ago

hgibbs 12y ago

Thanks. I really appreciate you taking the time to write so much. Honestly, this seems to be the best description of potential careers that I have ever gotten and I feel so much more assured about my choice to study Maths now. It is also really interesting to hear about how colourful such a career could be! You sound like, despite the politics, you have had a really interesting career and have done the kind of things that I would exactly like to do! Again, thanks so much. One last question, what are you doing with your life right now? It seems like you have had a pretty broad career and I am interested in the end game. If you are retired, what is your lifestyle like and in particular how has your career impacted your life as a retiree.

graycat 12y ago
Part II For now, sure, I know some math and can stir up some more, and I know some computing and can learn more. So, I've got a 1.8 GHz AMD single core processor, Windows XP Professional SP3 (I have an official copy of Windows 7 Professional on DVD but have seen no great reason to go to all the trouble to install it and rebuild all my software environment yet), three hard disks, 100 million files, .NET Framework 4.0, a good text editor (KEdit), Visual Basic .NET (comes with the .NET Framework), ASP.NET, ADO.NET, IIS (low level Web server), SQL Server Express, etc. I thought, why not go after about 2/3rds of Internet search, the safe for work part served at best poorly by looking for keywords/phrases? So, how to do that? Not with just routine software! And not with anything commonly talked about for search! I stirred up some math, typed the theorems and proofs into Knuth's TeX, worked up a scalable architecture, and typed in the software. Maybe I should have used Redis, but instead I thought that writing my own session state server would be faster than even understanding Redis. So, my session state server is single threaded ("Look, Ma, no concurrency problems"), for faster lookup rates is trivial to run as several instances as in sharding, and is just some TCP/IP sockets, some de/serialization of instances of my session state class, and uses two instances of a .NET collection class. Simple. It's fast! A server for less than $1500 should be able to keep session state for an hour of inactivity for each user and do the session state work for sending 5000+ Web pages a second. Two standard racks should be able to handle session state for the world. The rest of the software is also readily scalable also from just simple sharding. Currently I have one bug in one Web page -- it's not handling session state just right! But I have the fix in code in another Web page and should copy it over today! My interactive development environment is just KEdit with about 100 macros and some careful use of file system directories. I'm using SQL Server only to record the data from the users and for the results of some batch computations; at one point I actually do make use of a transaction; the data for the searches is drawn from SQL Server with a batch program (run it maybe once a day); some solid state drives (write rarely, read thousands of times a second) should do wonders for the data for the searches. I was about to fix the bug in the Web page but took some opportunities to gather some good initial data. The site will start focused and only slowly grow to be comprehensive -- right, at first do some things that don't scale and please some niche group of users a lot instead of trying to please 2+ billion users a little. Currently the database has only some meaningless data I put in for first testing of the software. It's about time to load in some of the good initial data I have. Then give a critical review, go live, etc. It's getting there. All the work uniquely mine has been fast, fun, and easy, but the whole project has taken far, far, far too long. Why? I worked through about a cubic foot of books and 6000+ Web pages of documentation of Windows, .NET, Visual Basic .NET, ASP.NET, ADO.NET, SQL Server, etc. The main problems: (1) Badly written, obscure documentation (worst bottleneck in the future of computing); (2) computer viruses; (3) SQL Server installation bugs destroying my boot partition requiring rebuilding starting with the XP DVD (barbed wire enema with an unanesthetized upper molar root canal procedure); (4) SQL Server management and administration (e.g., a week of throwing stuff against a wall to see what sticks just to get a SQL Server connection string that will let code for a server side Web page connect with SQL Server); (5) clean, smooth means of system backup and recovery (including for both user data and bootable partitions); (6) Sony DVD drives that quit for no good reason (and inability to buy more IDE DVD drives). Good stuff: (1) KEdit and its macro language; (2) the scripting language Rexx (Microsoft's PowerShell may also be terrific but have yet to move to it); (3) NTFS (fantastic); (4) Visual Basic .NET design, functionality, speed of compilation, compiler error messages, minimal bugs (sweetheart language); (5) what ASP.NET does when it compiles a Visual Basic program (enough for a really nice IDE); (6) NTBACKUP (once understand how to use it, e.g., do have to ask to save "system state", whatever the heck that is, or the saved copy, restored, won't boot -- learn this and how to get around it the hard way, weeks of work); (7) XCOPY; (8) the tools to have server side Web page code write to a log file; (9) Firefox (except for virus vulnerabilities); (10) the classes in the .NET Framework (once learn how to learn about them and use them); (11) Adobe Acrobat (except for virus vulnerabilities); (12) the ability of XP to find device drivers and recognize new devices; (13) Microsoft's anti-virus Safety Scanner (if only from CP67/CMS and Multics, there should be no virus vulnerabilities, but since there are the safety scanner is terrific to have); (14) Knuth's TeX; (15) the Western Digital Passport Ultra 2 TB USB drive! I'm not retired or retiring! Likely I'm still 100% unemployable at anything that would pay enough to let me keep a car going to commute to the job. Y Combinator and VCs want nothing to do with me. But if I can get my Web site up to a search a second, on a server for about $2000, I will be in decent shape financially and on the way to organic growth for my business and much more. Then I'll get a nice house, a building for some cars, at least one Corvette, visit the rest of the family still alive, take off two weeks to pig out on lobster in Maine, get some good grape juice from between Beaune and Dijon, do some cooking, give some dinner parties, go to concerts and operas, continue with the business, go to seminars on mathematical physics, etc. I'll implement and deploy my server farm monitoring techniques and maybe spin it off as a separate business. I have some guesses for some approaches to real AI and might try to implement those. And I will get a kitty cat! We'll see! Good luck on your work with math. Maybe what I typed in here will help; I wish I'd known all that when I started.