3 ms·

> Python, R, Matlab and Excel Mentioning Matlab and Excel immediately puts the product in the category "they know what they are doing" as opposed to "another g

by _pmf_ 9y ago

> Python, R, Matlab and Excel Mentioning Matlab and Excel immediately puts the product in the category "they know what they are doing" as opposed to "another group of sophomores trying to reinvent data science". I'm still waiting for a raw data dump with "avoid copies at all cost" access for very raw, very verbose vehicle and manufacturing data that is in 99.9 per cent never accessed, but must be analyzed when errors are detected late in the process. I.e. there's practically no transformation to apply during storage, but upon access, transformation must be done. If TileDB is kind of like a more structured, more low-level struct oriented Redis, it is a very welcome addition.

srean 9y ago
Apart from a diminishing population of old farts and a clutch of national labs and universities that Mathworks keeps well greased, is MATLAB even relevant anymore ?
- _pmf_ 9y ago
  Pervasive in the German automotive industry.
- jakebol 9y ago
  Jake from TileDB, Inc. Engineering as a discipline is conservative (for good reason). The tools and the processes they use change slowly. Matlab is still entirely relevant both in the sciences and in industry. There are people with huge amounts of domain knowledge (who may only know Matlab or Excel) that are increasingly called upon to analyze and interpret larger amounts of data. These "old farts" are the people engineering, designing, and debugging our modern world. Empowering people with domain knowledge to answer data driven questions is what the democratization of data science is all about. There is tremendous value in building bridges across communities and across generations here. I say this as someone who helped in small ways to develop open source alternatives to Matlab.
  - srean 9y ago
    > These "old farts" are the people engineering and designing our modern world. Empowering people with domain knowledge to answer data driven questions is what the democratization of data science is all about. BTW I don't begrudge that TileDB has MATLAB support. I see plenty of modern scientific/engineering workload in my day job. From what I see around myself it is usually a handful of people set in their ways that are holding others back by keeping a ridiculously overpriced tool alive that has comparable if not better alternatives. I gather from comments that it is different in Europe.
  - curious12345 9y ago
    So Jake, how are the Julia bindings coming along?
  - StanSeltser 9y ago
    Stanislav, Seltser, Petacube agree with Jake. Matlab still has some unique features to it which many open source projects (eg python) don't have - eg FPGA integration, System Modeling and Simulation. Combine that with decent language, nice debugger and many strong industry-specific solutions its good bet its will be around for a while. The reasons people dump matlab is not that its overpriced but due to the lack of integration with big data systems and the fact that matlab license cost at large scale becomes untenable.Plus number of industries using these unique matlab features is relatively small.
- jeff_science 9y ago
  Yeah, those deep-pocketed government lab and university researchers are clearly the dominant revenue source for MathWorks. Everybody in the corporate world is super nimble and moved on to Numpy as soon as they could.
- bigger_cheese 9y ago
  It is used pretty extensively in my industry (industrial manufacturing). There has been some talk about R but not sure it will be widely adopted. You are right that engineers are conservative which no doubt plays some role. But I'd say technical debt and legacy code plays much more of a role. Most Engineers (Chem, Mech, Matls etc) are not exposed to code during university. Maybe it is changing now but it is slow process. Often first exposure comes when the engineer enters industry and is asked to work on existing model - usually under supervision of a senior engineer. You learn whatever the language the senior engineer knows and that is typically what they learnt from similar mentorship - it is often Fortran or C/C++. Our Industrial Process has not changed dramatically in last 30 years. So once efficiently written and accurate simulation code exists very little reason to rewrite (if given the choice to rewrite an existing model in whatever the current language du jour is or continue to hack on something that already exists most engineers - at least the ones I know, would chose to hack on existing code). For pretty much this reason there is a heap of Fortan that is still alive in my org (with roots that can be traced back to the '80's). I think it's probably worth mentioning a lot of engineers don't approach code thinking about algorithms - it's equations i.e. I need to write something to solve Bernoulli's equation - or Ergun's equation or similar. Linear programming (i.e solving simultaneous equations) is the other main reason to write code and hey MATLAB does this pretty well... The other reason for being "gunshy" about new technology at least in my org is that a lot of the senior engineering people still remember when we got bitten by investing in Microsoft "Stack" in the late 90's. There were a lot of Modelling done with VB6 and Access which tuned out to be a technology dead end. Access databases in particular have plagued our org in many cases multiyear effort to migrate something out of Access. Open source probably protects against this but everyone wants to be sure any new technology will still be around in 15 years time. I think that is why people in my org are finally starting to look at R it has passed that initial hurdle and there is some confidence it is not just a fad...
StanSeltser 9y ago
Stanislav Seltser, Petacube _pmf: think of distributed persistent numpy arrays as opposed to key-value pairs in Redis. The idea you will not pull data to do computation, you will push your code in. pretty good for archiving due to compression

by _pmf_ 9y ago

srean 9y ago

Apart from a diminishing population of old farts and a clutch of national labs and universities that Mathworks keeps well greased, is MATLAB even relevant anymore ?

_pmf_ 9y ago
Pervasive in the German automotive industry.
jakebol 9y ago
Jake from TileDB, Inc. Engineering as a discipline is conservative (for good reason). The tools and the processes they use change slowly. Matlab is still entirely relevant both in the sciences and in industry. There are people with huge amounts of domain knowledge (who may only know Matlab or Excel) that are increasingly called upon to analyze and interpret larger amounts of data. These "old farts" are the people engineering, designing, and debugging our modern world. Empowering people with domain knowledge to answer data driven questions is what the democratization of data science is all about. There is tremendous value in building bridges across communities and across generations here. I say this as someone who helped in small ways to develop open source alternatives to Matlab.
- srean 9y ago
  > These "old farts" are the people engineering and designing our modern world. Empowering people with domain knowledge to answer data driven questions is what the democratization of data science is all about. BTW I don't begrudge that TileDB has MATLAB support. I see plenty of modern scientific/engineering workload in my day job. From what I see around myself it is usually a handful of people set in their ways that are holding others back by keeping a ridiculously overpriced tool alive that has comparable if not better alternatives. I gather from comments that it is different in Europe.
- curious12345 9y ago
  So Jake, how are the Julia bindings coming along?
- StanSeltser 9y ago
  Stanislav, Seltser, Petacube agree with Jake. Matlab still has some unique features to it which many open source projects (eg python) don't have - eg FPGA integration, System Modeling and Simulation. Combine that with decent language, nice debugger and many strong industry-specific solutions its good bet its will be around for a while. The reasons people dump matlab is not that its overpriced but due to the lack of integration with big data systems and the fact that matlab license cost at large scale becomes untenable.Plus number of industries using these unique matlab features is relatively small.
jeff_science 9y ago
Yeah, those deep-pocketed government lab and university researchers are clearly the dominant revenue source for MathWorks. Everybody in the corporate world is super nimble and moved on to Numpy as soon as they could.
bigger_cheese 9y ago
It is used pretty extensively in my industry (industrial manufacturing). There has been some talk about R but not sure it will be widely adopted. You are right that engineers are conservative which no doubt plays some role. But I'd say technical debt and legacy code plays much more of a role. Most Engineers (Chem, Mech, Matls etc) are not exposed to code during university. Maybe it is changing now but it is slow process. Often first exposure comes when the engineer enters industry and is asked to work on existing model - usually under supervision of a senior engineer. You learn whatever the language the senior engineer knows and that is typically what they learnt from similar mentorship - it is often Fortran or C/C++. Our Industrial Process has not changed dramatically in last 30 years. So once efficiently written and accurate simulation code exists very little reason to rewrite (if given the choice to rewrite an existing model in whatever the current language du jour is or continue to hack on something that already exists most engineers - at least the ones I know, would chose to hack on existing code). For pretty much this reason there is a heap of Fortan that is still alive in my org (with roots that can be traced back to the '80's). I think it's probably worth mentioning a lot of engineers don't approach code thinking about algorithms - it's equations i.e. I need to write something to solve Bernoulli's equation - or Ergun's equation or similar. Linear programming (i.e solving simultaneous equations) is the other main reason to write code and hey MATLAB does this pretty well... The other reason for being "gunshy" about new technology at least in my org is that a lot of the senior engineering people still remember when we got bitten by investing in Microsoft "Stack" in the late 90's. There were a lot of Modelling done with VB6 and Access which tuned out to be a technology dead end. Access databases in particular have plagued our org in many cases multiyear effort to migrate something out of Access. Open source probably protects against this but everyone wants to be sure any new technology will still be around in 15 years time. I think that is why people in my org are finally starting to look at R it has passed that initial hurdle and there is some confidence it is not just a fad...

StanSeltser 9y ago

Stanislav Seltser, Petacube _pmf: think of distributed persistent numpy arrays as opposed to key-value pairs in Redis. The idea you will not pull data to do computation, you will push your code in. pretty good for archiving due to compression