3 ms·

Thank you for your work. I used PDL early 2000 when working in bioinformatics area. I did not know at the time any of the specialized languages, so intially ap

by audit 5y ago

Thank you for your work. I used PDL early 2000 when working in bioinformatics area. I did not know at the time any of the specialized languages, so intially approaching the project -- I was very concerned on how to deal with matrices, but as I got to understand the PDL better -- i was getting better and better at it. If I may suggest someting (this is based on the old experience though) -- a) some 'built-in' way to seamlessly distribute work across processes and machines. b) some seamless excel and libreoffice calc integration. Meaning that I should be able to 'release' my programs as Excel/Libre Office files. Where I code in PDL but leverage Spreadsheet as a 'UI' + calc runtime. So that when I run my 'make' I get out a Excel/Libre office file that I can version and distribute into user or subsequent compute environments. Where the PDL code is translated into the runtime understood by the spreadsheet engine. I know this is a lot to ask, and may be not in the direction you are going, but wanted to mention still.

sivoais 5y ago
Good ideas! a) A built-in way would be good. There is some work being explored in using OpenMP with Perl/PDL to get some of that. In the mean time, there is MCE which does distribute across processes and there are examples of using this with PDL <https://github.com/marioroy/mce-cookbook#sharing-perl-data-language-pdl https://github.com/marioroy/mce-cookbook#sharing-perl-data-l...>, but I have not had an opportunity to use it. b) Output for a spreadsheet would be difficult if I understand the problem correctly. This would more about creating a mapping of PDL function names to spreadsheet function names --- not all PDL functions exist in spreadsheet languages. It might be possible to embed or do IPC with a Perl interpreter like <https://www.pyxll.com/ https://www.pyxll.com/>, but I don't know about how easy that would be to deploy when distributing to users. Am I understanding correctly? Interestingly enough, creating a mapping of PDL functions would be useful for other reasons, so the first part might be possible, but the code might need to be written in a certain way that makes writing the dataflow between cells easier.
- audit 5y ago
  thank you for follow up ! on b) Yes, you understood correctly. Perhaps I did not think of one-to-one mapping between PDL and Excel calleable functions, as such. But a sort of a PDL-to-Spreadsheet transplier that looks at the PDL as an input syntax and transplies it into something that runs on excel/libre calc runtime. While at the same -- if the transplier's build target is Perl -- it produces something that runs under 'perl'. This kind of multi-target 'transplantation' would allow (in my non-expert thinking) the same computation 'source' to run in backends (eg under Perl, distributed across multiple machines (as in (a)), while portions that require interactivity, or allow 'one-workstation' kind of runs would be executed within a Spreadsheet run time, on a workstation. The dissonance between spreadsheet flow-oriented language with a well-understood interactivity model but a small dataset, and 'batch/program' oriented expressive language like PDL for large datasets-- perhaps could be solved, with something like the above. In most settings that I was in, things that run on a single workstation are used for 'modeling' or testing or troubleshooting. So in that setting a spreadsheet program (Excel/Libre Calc) is used more like a 'repl'. While things that run in productions deal with large data sets and quicker. So this duality of usecases created, and lack of a unifying technology -- created whole organization/people profession shifts. I know that Julia is trying to solve it by creating a language that is as interactive as a spreadsheet, while as performant as a backend language. But another approach, may be, is to look at PDL and make a transplier that can facilitate this duality of use case. Apologies in advance for the long reply, and may be taking the thread of onto a tangent!

by audit 5y ago

sivoais 5y ago

Good ideas! a) A built-in way would be good. There is some work being explored in using OpenMP with Perl/PDL to get some of that. In the mean time, there is MCE which does distribute across processes and there are examples of using this with PDL <https://github.com/marioroy/mce-cookbook#sharing-perl-data-language-pdl https://github.com/marioroy/mce-cookbook#sharing-perl-data-l...>, but I have not had an opportunity to use it. b) Output for a spreadsheet would be difficult if I understand the problem correctly. This would more about creating a mapping of PDL function names to spreadsheet function names --- not all PDL functions exist in spreadsheet languages. It might be possible to embed or do IPC with a Perl interpreter like <https://www.pyxll.com/ https://www.pyxll.com/>, but I don't know about how easy that would be to deploy when distributing to users. Am I understanding correctly? Interestingly enough, creating a mapping of PDL functions would be useful for other reasons, so the first part might be possible, but the code might need to be written in a certain way that makes writing the dataflow between cells easier.

audit 5y ago
thank you for follow up ! on b) Yes, you understood correctly. Perhaps I did not think of one-to-one mapping between PDL and Excel calleable functions, as such. But a sort of a PDL-to-Spreadsheet transplier that looks at the PDL as an input syntax and transplies it into something that runs on excel/libre calc runtime. While at the same -- if the transplier's build target is Perl -- it produces something that runs under 'perl'. This kind of multi-target 'transplantation' would allow (in my non-expert thinking) the same computation 'source' to run in backends (eg under Perl, distributed across multiple machines (as in (a)), while portions that require interactivity, or allow 'one-workstation' kind of runs would be executed within a Spreadsheet run time, on a workstation. The dissonance between spreadsheet flow-oriented language with a well-understood interactivity model but a small dataset, and 'batch/program' oriented expressive language like PDL for large datasets-- perhaps could be solved, with something like the above. In most settings that I was in, things that run on a single workstation are used for 'modeling' or testing or troubleshooting. So in that setting a spreadsheet program (Excel/Libre Calc) is used more like a 'repl'. While things that run in productions deal with large data sets and quicker. So this duality of usecases created, and lack of a unifying technology -- created whole organization/people profession shifts. I know that Julia is trying to solve it by creating a language that is as interactive as a spreadsheet, while as performant as a backend language. But another approach, may be, is to look at PDL and make a transplier that can facilitate this duality of use case. Apologies in advance for the long reply, and may be taking the thread of onto a tangent!