4 ms·
While we're on the topic of calling fork and overloading servers to the point that they can't establish database connections or serve web pages, I'd like to poi
by DonHopkins 8y ago
While we're on the topic of calling fork and overloading servers to the point that they can't establish database connections or serve web pages, I'd like to point out that forking another process is an astronomically EXPENSIVE and WASTEFUL way of performing simple arithmetic and string manipulation.
Please stop wasting electricity and time by writing shell scripts that fork processes to do ridiculously simple things like calling "bc" to add two numbers or "sed" to perform simple string manipulation, since your computer has a "mul" instruction that doesn't require millions of instructions and bytes of memory and disk accesses to perform.
It's like using Bitcoin to pay for a stick of gum.
https://www.quora.com/How-much-energy-does-it-take-to-make-a-Bitcoin-transaction https://www.quora.com/How-much-energy-does-it-take-to-make-a...
>According to Digiconmist, each bitcoin transaction took 215 kWh. Whereas an average Indian household consume : 90 kWh per month. It sums up that, each bitcoin consumption can power a average indian household for 2.5 months probably. The average electricity cost in India is 5 rupees per kwh.
It's like rolling coal because you want to trigger the libs.
https://www.youtube.com/watch?v=fkmLf5OXxz0 https://www.youtube.com/watch?v=fkmLf5OXxz0
Stop and think before you write that shell script. It's not going to take you any more time to write it in Python, and then you won't be so shocked and surprised and totally fucked when you realize later that you need to add more features and make it more complicated than you originally though it would need to be in the first place.
It just boggles my mind that some people have such a hard time understanding how spectacularly inefficient shell scripts are, on top of how horrible to write, read, understand and maintain they are.
https://news.ycombinator.com/item?id=19886164 https://news.ycombinator.com/item?id=19886164
- saagarjha 8y ago> Stop and think before you write that shell script. It's not going to take you any more time to write it in Python Have you actually measured the amount of energy each takes? I would not be surprised if the Python VM's startup makes it more inefficient.
- DonHopkins 8y agoYou don't need to start a new Python VM every time you need to multiply two numbers or split two strings, like you need to do with a bash script that calls dc and sed. That is because Python actually has a built-in way to multiply numbers and perform string manipulation, right out of the box, without forking another process! The overhead of the Python interpreter required to get to that "mul" instruction is nothing compared to the overhead of forking another process to run "dc". This isn't rocket science, it's not even close. You really should check it out some time. So do a lot of other languages. Just not bash. I can't believe I need to explain this. This isn't about Python being amazingly well designed or efficient. It's about bash being extremely badly designed and inefficient, and people having absolutely no clue how expensive it is to fork processes like "bc" to multiply numbers and "sed" to split strings.
- saagarjha 8y ago> That is because Python actually has a built-in way to multiply numbers and perform string manipulation, right out of the box! You really should check it out some time. So do a lot of other languages. Just not bash. $ echo $((1*2)) 2 $ string="hello, world" $ echo ${string/hello/goodbye} goodbye, world
- DonHopkins 8y agoNow show me how to add 1.5 and 0.5. bash-3.2$ echo $((1.5+0.5)) bash: 1.5+0.5: syntax error: invalid arithmetic operator (error token is ".5+0.5") https://unix.stackexchange.com/questions/40786/how-to-do-integer-float-calculations-in-bash-or-other-languages-frameworks https://unix.stackexchange.com/questions/40786/how-to-do-int... I gave up INTEGER BASIC for APPLESOFT several decades ago, and never looked back. (LOADING INTEGER INTO LANGUAGE CARD) ]INT >PRINT 1.5 + 0.5 *** SYNTAX ERROR >FP ]PRINT 1.5 + 0.5 2 It's 2019, and you're struggling to do in bash what you could have done on an Apple ][ in 1979, but the syntax you need to use is even WORSE than BASIC, so your code looks like line noise from when your mom picks up the phone while you're using a 300 baud modem. Is that really progress? https://en.wikipedia.org/wiki/Integer_BASIC https://en.wikipedia.org/wiki/Integer_BASIC >Integer BASIC was phased out in favor of Applesoft BASIC starting with the Apple II Plus in 1979. This was a licensed but modified version of Microsoft BASIC, which included the floating point support missing in Integer BASIC. https://en.wikipedia.org/wiki/Applesoft_BASIC https://en.wikipedia.org/wiki/Applesoft_BASIC >Applesoft BASIC is a dialect of Microsoft BASIC, developed by Marc McDonald and Ric Weiland, supplied with the Apple II series of computers. It supersedes Integer BASIC and is the BASIC in ROM in all Apple II series computers after the original Apple II model. It is also referred to as FP BASIC (from "floating point") because of the Apple DOS command used to invoke it, instead of INT for Integer BASIC. Notice that there are even machine language instructions for cos, atan, tan, sin, ln, etc. How do you call those instructions in bash? https://docs.oracle.com/cd/E18752_01/html/817-5477/eoizy.html https://docs.oracle.com/cd/E18752_01/html/817-5477/eoizy.htm... >x86 Assembly Language Reference Manual: Floating-Point Instructions >The floating point instructions operate on floating-point, integer, and binary coded decimal (BCD) operands.
- narnianal 8y agoFor many things shell scripts are the easiest and most expressive way to write them. And probably you don't know that human resources are more expensive than cpu cycles. Sure, I love python, but if I want to find and filter some files nothing is easier to write than bash, so it's possible to have a quite big bash script even doing a good job and being well readable. I'm honestly a little shocked that this doesn't seem to be common knowledge anymore and your post got upvoted so high. My main language is Python btw.
- meddlepal 8y agoEasiest and most expressive way for you maybe. And yea human resources are expensive... doubly so when you need to throw more of them at this thing written by a former developer that thought he or she was saving money for the org by not spending an extra 30 minutes writing the program in Python or Go. > but if I want to find and filter some files nothing is easier to write than bash That's like 10 lines in Python and much easier to read and test.
- F-0X 8y ago> That's like 10 lines in Python and much easier to read and test. It's a single command in bash, invoking a program far more efficient than anything you can put together in python. It is literally a tool written for this exact purpose. I think it is simply ridiculous to not use a pre-existing tool that frankly you cannot out-do with a scripting language. Even if you are writing a python script, you should _still_ use find to search for files if there is _any_ complexity in the search requirements.
- narnianal 8y ago> Easiest and most expressive way for you maybe. Look. We are working an professional field with a strong scientific foundation. There is no "this bridge is harder to drive over for me therefore I think it's bad architecture". The easiness and expressiveness depend on the problem you try to solve, not on the people, their tastes or their experience. Shell tools are highly optimized to work in pipes, to work on files and to work on line separated entries like lines of code or lines of CSV. So if you have a problem where you need to pipe file handling into some filtering and into some line based transformations then you will have a hard time to find anything as easy or expressive as a bash script. And most projects on a unix-like operating system at least start like this: Understand the problem based on database dumps, logs, csv data measurements, previously written code etc. If your regular expressions are not able to cover all the edge cases anymore and/or become unreadably complex then of course you should start an actual program. That only happens a few weeks into most topics though. > doubly so when you need to throw more of them at this thing written by a former developer that thought he or she was saving money for the org by not spending an extra 30 minutes writing the program in Python or Go. Yes, there are people who lack the skill or ambition to develop good code. We all hate when we need to deal with their stuff. But honestly, would they have written better code if they had used Python or Java or C++ for that problem? Unlikely. A good coder would have some feeling for when he outsources some functionality to a python script. And if he develops a code base for a longer time most of his code will be in a real programming language. But even then it might be wrapped into a thin layer of bash to use find or grep or something that cleans up the input a little.
- overgard 8y agoSometimes I think complicated shell scripting is just some weird form of vanity. By making something simple look utterly incomprehensible some people think they're looking really clever to their colleagues (when in reality they're just annoying the hell out of them.) It's the same urge that spawns bizarre C++ template meta-programming and macros that are more complicated than the problem they're solving. I don't mind those things when people just admit they were bored or doing it for laughs, but the number of people that do those things with no sense of irony is way more than I'd care to really think about.
- meribold 8y agoI think you can actually start a Bash process and quite many bc processes in the time the CPython interpreter needs for startup.
- pdkl95 8y agoAs others have already menti9oned, running 'bc' isn't actually that expensive. Comparing it to Bitcoin is ridiculous. Let's investigate the actual cost of that sum: $ cat foo.sh #!/bin/bash a="1.5" b="0.5" sum="$(echo "${a} + ${b}" | bc)" echo "The sum of ${a} and ${b} is ${sum}" $ /usr/bin/strace -q -D -w -c ./foo.sh The sum of 1.5 and 0.5 is 2.0 % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 55.53 0.001225 123 10 read 9.75 0.000215 215 1 execve 5.53 0.000122 7 18 mmap 4.03 0.000089 8 11 3 open 3.04 0.000067 7 10 mprotect 2.99 0.000066 5 13 rt_sigprocmask 2.54 0.000056 56 1 clone 2.45 0.000054 5 10 close 2.40 0.000053 5 11 rt_sigaction 1.81 0.000040 5 8 fstat 1.41 0.000031 8 4 lseek 1.22 0.000027 5 5 3 stat 0.91 0.000020 20 1 write That was 56us to clone(2) the process and 215us to execve(2) 'bc'. All together just 12.74% of the time spent running syscalls, and an even smaller portion of the total time to launch and execute this simple example. Most of the time is sent launching bash itself at the beginning. Using a language such as python or launching a custom binary will have similar costs to to launch the program. If we're talking about a single persistent process (e.g. an interactive shell), then we're still talking about paying just 0.25ms of CPU time to perform a useful calculation. Compared to intentionally wasting time by design maximizing hash prefix guesses to "mine" Bitcoin "ain't the same fuckin' ballpark, it ain't the same league, it ain't even the same fuckin' sport"[1]. However, the actual cost to launch an external program like 'bc' isn't particularly important. It could be multiple orders of magnitude slower and it would still be acceptable because we're talking ab9ut a cost that only needs to be paid a few times. Yes, compared to "full" programming languages like python, running an external program to perform a simple calculation might seem like a wasteful, half-assed solution. As Gary Bernhardt explained[2], "Half-assed is OK when you only need half of an ass." I don't care about the cost of running 'bc' to do simple floating point calculations because I usually don't need to do floating point calculations (which are a lot harder to use safely[3] compared to simple integer math) for the type of task I would implement in a shell script. If the task required more than a few calculations (or other complex features such like a GUI), using something other language like python/ruby/C/rust would be a better tool than shell script. Learning how to choose the appropriate tools to use when solving a problem is an important skills for any engineer. Shell scripting languages like Bourne (sh) are designed to be a glue language that you use for simple (often interactive) tasks. They are the user interface (or "IDE") of Unix. Asking why shell d9oesn't have built-in support for efficient floating point math or complex GUI features is kind of like asking why Eclipse or Visual Studio don't have a built-in spreadsheet or email client. Yes, you could probably make that happen as a half-assed hack, but it's obviously not the type of task those tools were designed for. As for the syntax, if you find it confusing or ugly and prefer to use other tools, that's fine. There are a lot of great languages available, so use the tool that is best for you. If it had #!/bin/sh art the top, you can probably substitute any language you prefer without and problems. However, remember that for some of us Bourne shell is the easy, reasonably efficient tool we prefer to use for some problems. [1] https://en.wikiquote.org/wiki/Pulp_Fiction#Dialogue https://en.wikiquote.org/wiki/Pulp_Fiction#Dialogue [2] https://www.youtube.com/watch?v=sCZJblyT_XM https://www.youtube.com/watch?v=sCZJblyT_XM [3] https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.h...