5 ms·

I've not tried it yet, but we have to do a particular distributed crawler app for something and it's been recommended to look at it. For me if a language is am

by inovica 18y ago

I've not tried it yet, but we have to do a particular distributed crawler app for something and it's been recommended to look at it. For me if a language is amazing at something then it would be prudent to focus on that. I often use more than one language in my projects - the best language for the particular job.

simonw 18y ago
I'd be a bit scared of using Erlang for crawling since the actual text processing capabilities of the language are pretty weak. Crawling seems like the kind of thing where you'd want a message queue style system (potentially written in Erlang) that co-ordinates plenty of individual crawling processes written in a language better suited for text handling.
- inovica 18y ago
  Thanks for the tip. I'll bear that in mind.
jfarmer 18y ago
Erlang is best for software that works like telco switches, i.e., you have the need to route thousands of messages in real-time. Chat, IM, SMS, message queues, etc. are all well-suited for Erlang. Tasks that involves lots of I/O and parsing, not so much.
- inovica 18y ago
  Thanks
- KirinDave 18y ago
  So you're suggesting that IM, Chat, SMS and message queues aren't about Parsing and I/O? Erlang has weaknesses, but certainly those are not it. Basically, Erlang excells at writing servers. Take a look at YAWS (http://yaws.hyber.org/ http://yaws.hyber.org/) for a web server written in Erlang that can actually outperform more traditional web servers in very realistic scenarios. If I had to name a weakness or two of erlang offhand: 1. Integrating foreign code with Erlang is very different from traditional FFI and requires some initial legwork (libraries like Erlectricity http://github.com/KirinDave/erlectricity/tree/master http://github.com/KirinDave/erlectricity/tree/master , for example). 2. String manipulation. Erlang doesn't have a real string type. Its string libraries and core string type assume using linked lists for strings, which is spectacularly inefficient. However, there is a movement and a proposal to change the string type to Erlang binaries and provide an alternative string library. Hopefully that will address the problem. Other than that, it's a reasonably fast and rock-solid implementation of an FP language with a mountain of solid IPC library code integrated at the language level.
  - yariv 18y ago
    Representing strings as lists can be very efficient depends on what you're trying to do. Incidentally, Haskell does the same thing. I think the "Erlang sucks for strings" argument is overblown, and it's rarely if ever backed by real world experience and/or data. Erlang is fine for most string handling tasks you'll find in a typical (web) application.
    - KirinDave 18y ago
      My personal experience has had erlang very poor at handling and manipulating large strings. By poor, let me be clear: slow and spending far too much memory. We have to be honest identifying our tool's weaknesses, Yariv, even if we like them. EEP-009 (http://www.erlang.org/eeps/eep-0009.html http://www.erlang.org/eeps/eep-0009.html) and EEP-010 (http://www.erlang.org/eeps/eep-0010.html http://www.erlang.org/eeps/eep-0010.html) are extremely important for the continued growth and adoption of Erlang.
      - yariv 18y ago
        I'm curious -- what kind of string processing tasks did you try to use it for? Did you compare its performance to other languages? Also, did you try to use binaries? Erlang may not be optimal for applications that do heavy string processing, but the significance of that statement is overblown. Take typical webapps for example. What kind of string processing do they do? Very light stuff. They spend of their cycles slurping binary data from the database and sending it down a socket. Erlang is excellent for this usage. They occasionally take form inputs from a user, in which case they do some simple processing to ensure the input is properly escaped. Erlang handles that just fine. Most of the "Erlang sucks for strings" arguments remind me of the "Ruby/Java is slow" arguments. They're sort-of true, but they just don't matter for most applications.
        KirinDave 18y ago
        Why not contact me personally if you'd like more details via Twitter or AIM (this name). The short story: An attempt at a distributed web crawler showed that Erlang's default string type was not suitable for large strings. Please don't confuse this sentiment with any anti-erlangism or any assertion that Erlang isn't fantastic. I just want to be realistic, and I'd love to have faster string processing to make faster routing code for Fuzed (http://github.com/KirinDave/fuzed/tree/master http://github.com/KirinDave/fuzed/tree/master).
- simpleenigma 18y ago
  I have done quite a bit of work in Erlang over the past 4 or so years. I even wrote an anti-spam engine hat was heavy on parsing. The best thing Erlang has going in the parsing area is the binary pattern matching, which if you convert a string to a binary works wonders and is much faster. IN fact switching a string to a binary, doing your parsing with binary pattern matching and switching it back to a string CAN be faster in Erlang than using Erlang's string functions. I won't go into comparisons to other languages, mostly because my personal opinion is that Erlang has so many other great features that not being good at string processing is a minor issue.

by inovica 18y ago

simonw 18y ago

I'd be a bit scared of using Erlang for crawling since the actual text processing capabilities of the language are pretty weak. Crawling seems like the kind of thing where you'd want a message queue style system (potentially written in Erlang) that co-ordinates plenty of individual crawling processes written in a language better suited for text handling.

inovica 18y ago
Thanks for the tip. I'll bear that in mind.

jfarmer 18y ago

Erlang is best for software that works like telco switches, i.e., you have the need to route thousands of messages in real-time. Chat, IM, SMS, message queues, etc. are all well-suited for Erlang. Tasks that involves lots of I/O and parsing, not so much.

inovica 18y ago
Thanks
KirinDave 18y ago
So you're suggesting that IM, Chat, SMS and message queues aren't about Parsing and I/O? Erlang has weaknesses, but certainly those are not it. Basically, Erlang excells at writing servers. Take a look at YAWS (http://yaws.hyber.org/ http://yaws.hyber.org/) for a web server written in Erlang that can actually outperform more traditional web servers in very realistic scenarios. If I had to name a weakness or two of erlang offhand: 1. Integrating foreign code with Erlang is very different from traditional FFI and requires some initial legwork (libraries like Erlectricity http://github.com/KirinDave/erlectricity/tree/master http://github.com/KirinDave/erlectricity/tree/master , for example). 2. String manipulation. Erlang doesn't have a real string type. Its string libraries and core string type assume using linked lists for strings, which is spectacularly inefficient. However, there is a movement and a proposal to change the string type to Erlang binaries and provide an alternative string library. Hopefully that will address the problem. Other than that, it's a reasonably fast and rock-solid implementation of an FP language with a mountain of solid IPC library code integrated at the language level.
- yariv 18y ago
  Representing strings as lists can be very efficient depends on what you're trying to do. Incidentally, Haskell does the same thing. I think the "Erlang sucks for strings" argument is overblown, and it's rarely if ever backed by real world experience and/or data. Erlang is fine for most string handling tasks you'll find in a typical (web) application.
  - KirinDave 18y ago
    My personal experience has had erlang very poor at handling and manipulating large strings. By poor, let me be clear: slow and spending far too much memory. We have to be honest identifying our tool's weaknesses, Yariv, even if we like them. EEP-009 (http://www.erlang.org/eeps/eep-0009.html http://www.erlang.org/eeps/eep-0009.html) and EEP-010 (http://www.erlang.org/eeps/eep-0010.html http://www.erlang.org/eeps/eep-0010.html) are extremely important for the continued growth and adoption of Erlang.
    - yariv 18y ago
      I'm curious -- what kind of string processing tasks did you try to use it for? Did you compare its performance to other languages? Also, did you try to use binaries? Erlang may not be optimal for applications that do heavy string processing, but the significance of that statement is overblown. Take typical webapps for example. What kind of string processing do they do? Very light stuff. They spend of their cycles slurping binary data from the database and sending it down a socket. Erlang is excellent for this usage. They occasionally take form inputs from a user, in which case they do some simple processing to ensure the input is properly escaped. Erlang handles that just fine. Most of the "Erlang sucks for strings" arguments remind me of the "Ruby/Java is slow" arguments. They're sort-of true, but they just don't matter for most applications.
      - KirinDave 18y ago
        Why not contact me personally if you'd like more details via Twitter or AIM (this name). The short story: An attempt at a distributed web crawler showed that Erlang's default string type was not suitable for large strings. Please don't confuse this sentiment with any anti-erlangism or any assertion that Erlang isn't fantastic. I just want to be realistic, and I'd love to have faster string processing to make faster routing code for Fuzed (http://github.com/KirinDave/fuzed/tree/master http://github.com/KirinDave/fuzed/tree/master).
simpleenigma 18y ago
I have done quite a bit of work in Erlang over the past 4 or so years. I even wrote an anti-spam engine hat was heavy on parsing. The best thing Erlang has going in the parsing area is the binary pattern matching, which if you convert a string to a binary works wonders and is much faster. IN fact switching a string to a binary, doing your parsing with binary pattern matching and switching it back to a string CAN be faster in Erlang than using Erlang's string functions. I won't go into comparisons to other languages, mostly because my personal opinion is that Erlang has so many other great features that not being good at string processing is a minor issue.