12 ms·
Perl Data Language: Scientific Computing with Perl
- anonu 5y agoI think Perl missed the boat... too little way way too late. Also - they can't even get their SSL certs straight. I loved Perl - it was certainly my gateway to coding. Regex was beautiful in perl. But I shot myself in the foot so many times with that language.
- redis_mlc 5y ago> I think Perl missed the boat... too little way way too late. Perl PDL (1990s) was available a decade before anything similar in Python like NumPy (~2005.) > Also - they can't even get their SSL certs straight. It's not an encrypted site: http://pdl.perl.org/ http://pdl.perl.org/ Tool.
- hatsuseno 5y agoSpecifically how can't Perl get their SSL certs straight?
- sega_sai 5y agoI'm afraid this is ~ 10-15 years too late. I've used PDL somewhat in ~ 2005-2010 when python didn't have that many packages/numpy, and it did the job and could substitute IDL to some degree. But realistically, right now I don't see any reason to use PDL instead of Python.
- bjornjajayaja 5y agoMeh… Python zealots abound. Folks often say the same thing about Fortran but frankly, those are the engines that power ALL scripting languages math features. You’re running on C/C++/FORTRAN. If one needs better text-parsing features on top of that: use Perl; if you use a Python environment: use Python. Perl is superior in many ways and I think using it for data-exploration still has tons of merit. Folks “shoot themselves in the foot” when they don’t understand the language. In Perls case: list/scalar context is usually the culprit, but it is quite easy to understand. It’s more flexible and concise which in many ways makes Perl better at exploratory programming.
- 7thaccount 5y agoThe question isn't about Perl vs Python (which I still think is a clear winner for Python, although there are some uses where I could perfectly understand reaching for Perl first). The question is whether Perl's PDL is better than ScyPy/Numpy/Pandas/Cython, the Spyder IDE, bunch of plotting libraries...etc.
- analog31 5y agoIndeed, I'm a scientific programmer, and I've noticed that when most scientists talk about "Python," they don't make a clear distinction between where the language ends and the libraries begin. If they even know. Most users, myself included, download some big installer like Anaconda or WinPython, and off we go.
- 7thaccount 5y agoSame. Obviously a OO scripting language by itself isn't fast enough for most scientific coding, but when the C/Fortran libraries are tightly integrated with lots of stats/plots and other libraries with great IDE/database...etc it makes for a great and free/open scientific modeling platform. It's really the whole package and the language is just 1 component of that.
- analog31 5y agoIndeed, and it's my understanding that supporting that kind of integration is a strength of Python. Since I use Python in the lab, I've also found that the whole ctypes thing is a life saver when dealing with hardware drivers that are only furnished with a C API.
- mangecoeur 5y agoPeople like other people using python because it makes it easier to share and exchange and built a community of knowledge and libraries. Science is a lot about ease of collaboration.
- deleted 5y ago[deleted]
- dolodoot 5y ago> right now I don't see any reason to use PDL instead of Python. I like python a lot and actually i wish less scientists used it. Perl/PDL is in my opinion much better suited to this (understandable) just getting the job done approach i often find in sciene or other areas in which writing software is not the primary goal.
- leephillips 5y agoHow interesting. I’m surprised that I’d never heard of this before.
- the_only_law 5y agoI remember many years ago, outside programming I had a moderate interest in biology. I recall looking into bioinformatics, and reading about Python, but also about how popular Perl was in field. This was 5-10 years ago, though the sources I was reading very likely could have been 15 years old or more.
- leephillips 5y agoThe first thing that occurs to me, and this may be superficial, is that bioinformatics has a linguistic character to it, with pattern matching of sequences and so forth, and this is something where Perl is more adept than most other languages.
- etangent 5y agoyour hunch is correct
- f6v 5y agoBioinformatics today is much more about complex statistical models applied to sequencing data. There’s some pattern matching when doing low-level FASTQ processing for your fancy combinatorial barcoding. But other than that you need tools capable of processing huge datasets. Python ecosystem fits better in modern era.
- warlog 5y agoWhich .env fits better, exactly?
- MontyCarloHall 5y agoYup. I’ll occasionally still break out Perl for quickly parsing FASTQs/SAM alignments/VCFs on the command line when sed/awk/grep won’t cut it, but for everything else it’s Pandas, NumPy/SciPy, Torch, and TensorFlow.
- psychometry 5y agoPlease don't. Most of the Python/R/C code I've seen from scientists was terrible. 100% of the Perl code was, though.
- adenozine 5y agoAs if research scientists' code wasn't unfun enough to read already!
- codezero 5y agoWow what a throwback. I used IDL when I worked in astrophysics, and I loved it. It had a lot of fun capabilities and was super easy to build C libraries for. Back then there was also an attempt at making GDL (gnu data language), but the tooling for IDL was so deep I doubt anything can directly replace it. Then again I've been out of this for twelve years.
- wumpus 5y agoI just got back into astrophysics after having been gone for 25 years, and no one uses IDL anymore.
- codezero 5y agoI’d love to know what it looks like now. I knew a few folks doing a lot of GPU work, but so many old folks still making charts and figures in IDL, myself included.
- wumpus 5y agoIt's mostly Python most of the time, from my vantage point. Several of the GPU projects I'm watching have GPU kernels contained inside of python frameworks. So the typical user only has to change the python part of the code.
- codezero 5y agoHah, that's fantastic. This reminds me of a talk I attended about how most of science/academia has a lot of fast-moving glue to stitch together all the old/hard bits. https://www.easterbrook.ca/steve/2010/12/agu-session-on-software-engineering-for-climate-modeling/ https://www.easterbrook.ca/steve/2010/12/agu-session-on-soft...
- flobosg 5y agoSomewhat related, “How Perl Saved the Human Genome Project (1996)” : https://www.foo.be/docs/tpj/issues/vol1_2/tpj0102-0001.html https://www.foo.be/docs/tpj/issues/vol1_2/tpj0102-0001.html
- denimboy 5y agoSee also integration between PDL and MXNet deep learning framework: https://mxnet.apache.org/versions/1.8.0/api/perl
- f6v 5y agoBioinformatics just finished transitioning from Perl to R and Python. The last thing I need is a Perl renaissance. Hope it never happens.
- hoytech 5y agoI really like PDL, even though I don't get to use it much anymore. Here's a fun application, FM radio demodulation in a page or so of pretty straightforward code: https://hoytech.com/talks/fm-demodulation https://hoytech.com/talks/fm-demodulation
- aduitsis 5y agoI can see in the page that last PDL release was on February, okay. Probably the presence of activity indicates that there are people using PDL, and more than that, maintaining it. Since PDL existed for many years, I would speculate that these people presumably didn't jump into PDL yesterday. There can very well be large codebases using it for a long time. All the comments suggesting that this somehow shouldn't be and that people should move away from PDL, are depressing in the way that, in a split second, years of effort and thousands lines of code that are probably doing good work are dismissed just like that. If a bank is still using Cobol, it is interesting and a testament to how a Cobol programmer can still make a good living on it. But if a scientist is using Perl to carry out calculations, this is somehow bad?
- MontyCarloHall 5y ago> If a bank is still using Cobol, it is interesting and a testament to how a Cobol programmer can still make a good living on it. But if a scientist is using Perl to carry out calculations, this is somehow bad? Yes. The negative externalities of doing scientific analyses in Perl are much greater than a bank having a legacy COBOL codebase. Only a handful of engineers within the bank will ever see that COBOL codebase. Science is globally collaborative; many people at many different institutions across the world would have to deal with some idiosyncratic scientist’s decision to write their analysis in Perl. Also, the bank only has a COBOL codebase because it’s reluctant to make major changes to an extremely important system that’s been working flawlessly since the early 60s. There’s absolutely no reason to start a totally brand new project in Perl (or COBOL, for that matter), when far superior alternatives exist.
- caslon 5y agoPerl is shipped in virtually every Linux distribution; it's much closer to standard than, say, Node.
- MontyCarloHall 5y agoSo is ed. That doesn’t mean it’s a better coding environment than modern IDEs, even though it’s far more “standard.”
- 0xbadcafebee 5y agoPlease read this page first (and note the date): http://pdl.perl.org/?docs=Philosophy&title=PDL::Philosophy http://pdl.perl.org/?docs=Philosophy&title=PDL::Philosophy
- anthk 5y agoWho cares about the date. AWK is old as heck, and with The C Programming Language 2e over "The Unix Programming Environment" (Overriding pre ANSI-C) you can do magic with very few resources. Oh, you need to understand the underlying maths well in order to code your functions right? That's the biggest issue in data science management. Too much relying on specialized " biggies" like Numpy/CUDA with atrocious codebases where the calculations lasts months compared to a 1 hour chore with C or even Perl with Gnuplot.
- FabHK 5y agoNext in this series: Embedded Systems programming with HTML/CSS, Computer Graphics with SQL, and Text Processing with Matlab.
- anthk 5y agoOr scientific data processing with Python.
- anthk 5y agoOn science and energy saving: http://phroxy.z3bra.org/bitreich.org:70/0/con/2020/rec/energy-efficient-programming.md http://phroxy.z3bra.org/bitreich.org:70/0/con/2020/rec/energ... It's C, well, but C, Perl and Unix are related cousins and prototyping in Perl it's really fast.
- sivoais 5y agoHi, PDL core dev here. Feel free to ask me anything about it. The last release wasn't in February, it was just last week! <https://metacpan.org/release/ETJ/PDL-2.050 https://metacpan.org/release/ETJ/PDL-2.050>. I agree with many of the commenters here that Python has a lot of great libraries and is a major player for scientific computing these days. I also code in Python from time to time, but I prefer the OO modelling and language flexibility features of Perl. Speaking for myself and not the other PDL devs, I don't think this is an issue for Perl-using scientists as Perl can actually call Python code quite easily using Inline::Python. In the future I will be working on interoperability between the two better specifically for NumPy / Pandas. This is also the path being taken by Julia and R.
- 1996 5y agoVery interesting, thank you! Do you have a tutorial and some examples? If not, could you write one? I sometimes deploy perl code at large scale for financial computing where only performance matters: with XS the overhead is low while gaining language flexibility. Even in 2021, this is usually faster than alternatives by orders of magnitude. PDL could be a good addition to our toolset for specific workloads.
- sivoais 5y agoHere is a link to the PDL book <http://pdl.perl.org/content/pdl-book-toc.html http://pdl.perl.org/content/pdl-book-toc.html>. I can share some examples of using PDL: - Demos of basic usage <https://metacpan.org/release/ETJ/PDL-2.050/source/Demos/General.pm https://metacpan.org/release/ETJ/PDL-2.050/source/Demos/Gene...> - Image analysis <https://nbviewer.ipython.org/github/zmughal/zmughal-iperl-notebooks/blob/master/science/chemistry/chromatography/thin-layer-blob-demo/Thin-layer-chromatography-analysis.ipynb https://nbviewer.ipython.org/github/zmughal/zmughal-iperl-no...> (I am also the author of IPerl, so if you have questions about it, let me know. My top priority with IPerl right now is to make it easy to install.) - Physics calculations <https://github.com/wlmb/Photonic https://github.com/wlmb/Photonic> - Access to GSL functions for integration and statistics (with comparisons to SciPy and R): <https://gist.github.com/zmughal/fd79961a166d653a7316aef2f010a1a2 https://gist.github.com/zmughal/fd79961a166d653a7316aef2f010...>. Note how PDL can take an array of values as input (which gets promoted into a PDL of type double) and then returns a PDL of type double of the same size. The values of that original array are processed entirely in C once they get converted to a PDL. - Example of using Gnuplot <https://github.com/PDLPorters/PDL-Graphics-Gnuplot/blob/master/examples/pause-until-close.pl https://github.com/PDLPorters/PDL-Graphics-Gnuplot/blob/mast...>. --- Just to give a summary of how PDL works relative to XS: PDL allows for creating numeric ndarrays of any number of dimension of a specific type (e.g., byte, float, double, complex double) that can be operated on by generalized functions. These functions are compiled using a DSL called PP that generates multiple XS functions by taking a signature that defines the number of dimensions that the function operates over for each input/output variable and adding loops around it. These loops are quite flexible and can be made to work in-place so that no temporary arrays are created (also allows for doing pre-allocation). The loops will run multiple times over that same piece of memory --- this is still fast unless you have many small computations. And if you do have many small computations, the PP DSL is available for the user to use as well so if they need to take a specific PDL computation written in Perl, they can translate the innermost loop into C and then it can do the whole computation in one loop (a faster data access pattern). There is a book for that as well called "Practical Magick with C, PDL, and PDL::PP -- a guide to compiled add-ons for PDL" <https://arxiv.org/abs/1702.07753 https://arxiv.org/abs/1702.07753>. --- I'm also active on the `#pdl` IRC channel on <https://www.irc.perl.org/ https://www.irc.perl.org/>, so feel free to drop by.
- dima55 5y agoI used perl and PDL heavily, before moving to Python and numpy. Both have annoying issues, and oddly, their warts are complementary. Particularly, the core API in PDL is miles better than numpy's. Before I could tolerate actually using numpy, I had to write a library to patch away numpy's warts, by effectively writing a PDL compatibility layer. Check it out: https://github.com/dkogan/numpysane/ https://github.com/dkogan/numpysane/ Now the core numpy has usable broadcasting, concatenation and basic linear algebra. Kudos to the PDL team for the excellent core design.