6 ms·
Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport whe
by 0xbadcafebee 2y ago
Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.
- AkBKukU 2y agoI have a project that uses a proprietary SDK for decoding raw video. I output the decoded data as pure RGBA in a way FFMpeg can read through a pipe to re-encode the video to a standard codec. FFMpeg can't include the Non-Free SDK in their source, and it would be wildly impracticable to store the pure RGBA in a file. So pipes are the only way to do it, there are valid reasons to use high throughput pipes.
- whartung 2y agoWhat about domain sockets? It's clumsier, to be sure, but if performance is your goal, the socket should be faster.
- AkBKukU 2y agoIt looks like FFmpeg does support reading from sockets natively[1], I didn't know that. That might be a better solution in this case, I'll have to look into some C code for writing my output to a socket to try that some time. [1] https://ffmpeg.org/ffmpeg-protocols.html#unix https://ffmpeg.org/ffmpeg-protocols.html#unix
- ptx 2y agoWhy should sockets be faster?
- uncanneyvalley 2y agoSockets remap pages without moving any data while pipes have to copy the data between fds.
- CyberDildonics 2y agoSo pipes are the only way to do it Lets not get carried away. You can use ffmpeg as a library and encode buffers in a few dozen lines of C++.
- Almondsetat 2y agoffmpeg's library is notorious for being a complete and utter mess
- CyberDildonics 2y agoIt worked extremely well when I did something almost exactly like this. I gave it buffers of pixels in memory and it spit out compressed video.
- quietbritishjim 2y agoThe parent comment mentioned license incompatibility, which I guess would still apply if they used ffmpeg as a library.
- seba_dos1 2y agoIf the license is incompatible, it would still be incompatible regardless of whether you use library API calls or pipes.
- whiterknight 2y agoAnd you go from having a well defined modular interface that’s flexible at runtime to a binary dependency.
- CyberDildonics 2y agoYou have the dependency either way, but if you use the library you can have one big executable with no external dependencies and it can actually be fast. If there wasn't a problem to solve they wouldn't have said anything. If you want something different you have to do something different.
- jcelerier 2y agoWhy not just store the output of the proprietary codec in an AVFrame that you'd pass to libavcodec in your own code?
- ploxiln 2y agoWhat percentage of CPU time is used by the pipe in this scenario? If pipes were 10x faster, would you really notice any difference in wall-clock-time or overall-cpu-usage, while this decoding SDK is generating the raw data and ffmpeg is processing it? Are these video processing steps anywhere near memory copy speeds?
- Sesse__ 2y agoAt some point, I had a similar issue (though not related to licensing), and it turned out it was faster to do a high-bitrate H.264-encode of the stream before sending it over the FFmpeg socket than sending the raw RGBA data, even over localhost… (There was some minimal quality loss, of course, but it was completely irrelevant in the big picture.)
- jraph 2y ago> There was some minimal quality loss, of course, but it was completely irrelevant in the big picture But then the solutions are not comparable anymore, are they? Would a lossless codec instead have improved speed?
- chupasaurus 2y agoH.264 has lossless mode.
- Sesse__ 2y agoNo, because I had hardware H.264 encoder support. :-) (The decoding in FFmpeg on the other side was still software. But it was seemingly much cheaper to do a H.264 software decode.)
- Someone 2y agoThis isn’t code in some project that will run only a few billion times in its lifetime; it is used frequently on millions, if not billions, of computers. Because of that, it is economical to spend lots of time optimizing it, even if it only makes the code marginally more efficient.
- samastur 2y agoThat's not how economics works. If 100 million people each save 1 cent because of your work, you saved 1 million in total, but in practice nobody is observably better off.
- deleted 2y ago[deleted]
- sebstefan 2y agoSo? It doesn't need to be visible to be worth optimizing?
- samastur 2y agoIf you are making an economic (financial) argument for change like the original comment did, then yes, it should be visible positive effect. Obviously not if you are doing for your own fun or just improving the state of art.
- sebstefan 2y agoEventually everything ends up pumping the same resources from the same earth. The billion devices pool their 1 cents saved into the same pool of fossil fuels and the same power plants. You don't need the effect to be observable on an individual level It's something that is worth an engineer's time
- h0p3 2y agoThere are people whose lives are improved by having an extra cent to spend. Seriously. It is measurable, observable, and real. It might not have a serious impact on the vast majority of people, but there are people who have very, very little money or have found themselves on a tipping point that small; pinching pennies alters their utility outcomes.
- Ultimatt 2y agoA better analogy is its like a society that uses steam trains attempting to industrially compete with a society that uses bullet trains (literally similar by factor of improvement). The UK built its last steam train for national use in 1960, four years later the Shinkansen was in use in Japan. Which of those two nations has a strong international industrial base in 2024?
- ploxiln 2y agoIndeed. In the author's case, the slow pipe is moving data at 17 GB/s which is over 130 gbps. I've used pipes for a lot of stuff over 10+ years, and never noticed being limited by the speed of the pipe, I'm almost certain to be limited by tar, gzip, find, grep, nc ... (even though these also tend to be pretty fast for what they do).
- crabbone 2y agoI had two cases in my practice where pipes were slow. Both related to developing a filesystem. 1. Logging. At first our tools for reading the logs from a filesystem management program were using pipes, but they would be overwhelmed quickly (even before it would overwhelm pagers and further down the line). We had to write our own pager and give up on using pipes. 2. Storage again, but a different problem: we had a setup where we deployed SPDK to manage the iSCSI frontend duties, and our component to manage the actual storage process. It was very important that the communication between these two components be as fast and as memory-efficient as possible. The slowness of pipes comes also from the fact that they have to copy memory. We had to extend SPDK to make it communicate with our component through shared memory instead. So, yeah, pipes are unlikely to be the bottleneck of many applications, but definitely not all.
- jiehong 2y agoReplace “Linux pipes” by “Electron apps”, and people would not agree. Also, why leave performance on the table by default? Just because “it should be enough for most people I can think of”? Add Tesla motors to a Toyota Corolla and now you’ve got a sportier car by default.
- azulster 2y agoelectron apps are an optimization all by itself. it's not optimizing footprint or speed of application. it's optimizing the resources and speed of development and deployment
- paulannesley 2y agoSometimes the best answer really is a faster Corolla! https://www.toyota.com/grcorolla/ https://www.toyota.com/grcorolla/ (These machines have amazing engineering and performance, and their entire existence is a hack to work around rules making it unviable to bring the intended GR Yaris to the US market.. Maybe just enough eng/perf/hack/market relevance to HN folk to warrant my lighthearted reply. Also, the company president is still on the tools.
- 2OEH8eoCRo0 2y agoThere's no replacement for displacement.
- Sohcahtoa82 2y agoApparently there is, because that car only has a 1.6L 3-cylinder engine and yet produces a whopping 300 horsepower.
- 2OEH8eoCRo0 2y agoWhen? In the RPM sweet spot after waiting an eternity for the turbos to spool? There's always a catch.
- zackmorris 2y agoI didn't expect to be writing this comment on this article hah, but apparently there is such a thing called a surge tank for storing boost pressure to mostly eliminate turbo lag: https://www.highpowermedia.com/Archive/the-surge-tank https://www.highpowermedia.com/Archive/the-surge-tank https://forums.tdiclub.com/index.php?threads/air-tank-or-compressor-to-fix-turbo-lag.146578/ https://forums.tdiclub.com/index.php?threads/air-tank-or-com... It's such an obvious idea that I'm kind of shocked it took them until 2003 to do it. Surely someone thought of this in like the 60s. I would probably do it differently with a separate supercharger to intermittently maintain another 1-2+ bar of boost to make the tank less than half as large, but that would add complexity, and what do I know.
- mort96 2y agoI mean why waste CPU time moving data between buffers when you could get the same semantics and programming model without wasting that CPU time?
- qsantos 2y agoTo be frank, this is more of a pretext to understand what pipes and vmsplice do exactly.
- bastawhiz 2y agoI'm not sure that logic makes sense. Making a thing that's used ubiquitously a few percent faster it's absolutely a worthwhile investment of effort. Individual operations might but be very much faster but it's (in aggregate) a ton of electricity and time globally.
- 0xbadcafebee 2y agoThat's what's called premature optimization. Everywhere in our lives we do inefficient things. Despite the inefficiency we gain us something else: ease of use or access, simplicity, lower cost, more time, etc. The world and life as we know it is just a series of tradeoffs. Often optimization before it's necessary actually creates more drawbacks than benefits. When it's easy and has a huge benefit, or is necessary, then definitely optimize. It may be hard to accept this as a general principle, but in practice (mostly in hindsight) it becomes very apparent. Donald Knuth thinks the same: https://en.wikipedia.org/wiki/Program_optimization#When_to_optimize https://en.wikipedia.org/wiki/Program_optimization#When_to_o...
- bastawhiz 2y agoIt's definitionally not premature optimization. Pipes exist (and have existed for decades). This is just "optimization". "Premature" means it's too soon to optimize. When is it no longer too soon? In another few decades? When Linux takes another half of Windows usage? It would be premature if they were working on optimizations before there were any users or a working implementation. But it's not: they're a fundamental primitive of the OS used by tens of millions of applications. The tradeoffs you're discussing are considerations. Is it worth making a ubiquitous thing faster at the expense of some complexity? At some point that answer is "yes", but that is absolutely not "When it's easy and has a huge benefit". The most important optimizations you personally benefit from were not easy OR had a huge benefit. They were hard won and generally small, but they compound on other optimizations. I'll also note that the Knuth quote you reference says exactly this: > Yet we should not pass up our opportunities in that critical 3%
- 0xbadcafebee 2y ago
- tacone 2y agoWait, it depends on what you're doing. Pipes also create a subshell so they are a big nono when used inside a loop. Suppose you're cycling on the lines of stdout and need to use sed, cut and so on, using pipes will slow down things considerably (and sed, cut startup time will make things worse). Using bash/zsh string interpolation would be much faster.