5 ms·
A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventu
by brentroose 7mo ago
A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more.
That's why I built a performance challenge for the PHP community
The goal of this challenge is to parse 100 million rows of data with PHP, as efficiently as possible. The challenge will run for about two weeks, and at the end there are some prizes for the best entries (amongst the prize is the very sought-after PhpStorm Elephpant, of which we only have a handful left).
I hope people will have fun with it :)
- gib444 7mo ago> A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds That's a huge improvement! How much was low hanging fruit unrelated to the PHP interpreter itself, out of curiosity? (E.g. parallelism, faster SQL queries etc)
- brentroose 7mo agoAlmost all, actually. I wrote about it here: https://stitcher.io/blog/11-million-rows-in-seconds https://stitcher.io/blog/11-million-rows-in-seconds A couple of things I did: - Cursor based pagination - Combining insert statements - Using database transactions to prevent fsync calls - Moving calculations from the database to PHP - Avoiding serialization where possible
- tiffanyh 7mo agoAren’t these optimizations less about PHP, and more about optimizing how your using the database.
- hu3 7mo agoIt's still valid as as example to the language community of how to apply these optimizations.
- swasheck 7mo agoin all my years doing database tuning/admin/reliability/etc, performance have overwhelmingly been in the bad query/bad data pattern categories. the data platform is rarely the issue
- tosti 7mo agoThe worst offenders I've seen were looping over a shitty ORM
- toast0 7mo agoPHP is kind of like C. It can be very fast if you do things right, and it gives you more than enough rope to tie yourself in knots. Making your application fast is less about tuning your runtime and more about carefully selecting what you do at runtime. Runtime choice does still matter, an environment where you can reasonably separate sending database queries and receiving the result (async communication) or otherwise lets you pipeline requests will tend to have higher throughput, if used appropriately, batching queries can narrow the gap though. Languages with easy parallelism can make individual requests faster at least while you have available resources. Etc. A lot of popular PHP programs and frameworks start by spending lots of time assembling a beautiful sculpture of objects that will be thrown away at the end of the request. Almost everything is going to be thrown away at the end of the request; making your garbage beautiful doesn't usually help performance.
- Joel_Mckay 7mo agoIn general, it is bad practice to touch transaction datasets in php script space. Like all foot-guns it leads to Read-modify-write bugs eventually. Depending on the SQL engine, there are many PHP Cursor optimizations that save moving around large chunks of data. Clean cached PHP can be fast for REST transactional data parsing, but it is also often used as a bodge language by amateurs. PHP is not slow by default or meant to run persistently (low memory use is nice), but it still gets a lot of justified criticism. Erlang and Elixir are much better for clients/host budgets, but less intuitive than PHP =3
- user3939382 7mo agoexec(‘c program that does the parsing’); Where do I get my prize? ;)
- brentroose 7mo agoThe FAQ states that solutions like FFI are not allowed because the goal is to solve it with PHP :)
- Tade0 7mo agoPitch this to whoever is in charge of performance at Wordpress. A Wordpress instance will happily take over 20 seconds to fully load if you disable cache.
- embedding-shape 7mo agoMicrobenchmarks are very different from optimizing performance in real applications in wide use though, they could do great on this specific benchmark but still have no clue about how to actually make something large like Wordpress to perform OK out of the box.
- monkey_monkey 7mo agoThat's often a skill issue.
- almosthere 7mo agoskill issue being they only know php
- rkozik1989 7mo agoMuch like anything else your performance is going to vary a lot based on architecture of implementation. You really shouldn't deploying anything into production without some kind of caching. Whether that's done in the application itself or with memcached/redis or varnish or OPcache.
- slopinthebag 7mo agoEither you use a slow language and deal with caching or you use a fast language and just put Cloudflare/Bunny/etc in front.
- paulryanrogers 7mo agoAren't CF, Bunny, etc CDNs and therefore cache?
- 7mo ago
- onion2k 7mo agoA month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. When people say leetcode interviews are pointless I might share a link to this post. If that sort of optimization is possible there is a structures and algorithms problem in the background somewhere.
- nicoburns 7mo agoI find that these kind of optimizations are usually more about technical architecture than leetcode. Last time I got speedups this crazy the biggest win was reducing the number of network/database calls. There were also optimisations around reducing allocations and pulling expensive work out of hot loops. But leetcode interview questions don't tend to cover any of that. They tend to be about the implementation details of specific algorithms and data structures. Whereas the important skill in most real-world scenarios would be to understand the trade-offs between different algorithms and data structures so that you pick an appropriate off-the-shelf implementation to use.
- LollipopYakuza 7mo agoI agree. The "advanced" leetcode is about those last % of optimization. But when network latency is involved in a flow, it is usually the most obvious low hanging fruit.
- tuetuopay 7mo agoWell leetcode asks you to implement the data structure, not how and when to use which data structure. I don’t need to know how to implement a bloom filter on a whiteboard off the top of my head to know when to use it.
- Twirrim 7mo agoHell, the number of times I've used a lot of the data structures that come up in leetcode exercises without at least looking at some reference material is pretty small. I usually assume I'm going to misremember it, and go double check before I write it so I don't waste ages debugging later.
- CyberDildonics 7mo agoUsing a language that is 100x slower than naive native programs to do a "speed challenge" is like spending your entire day speed walking to run errands when you can just learn how to drive a car.
- ge96 7mo ago5 days to 30 seconds? What kind of factor/order of magnitude is that damn What takes 5 days to run
- hosteur 7mo agoPoorly made analytics/datawarehouse stuff.
- slopinthebag 7mo agoOne query per column per row
- contingencies 7mo agoHehe. Optimization ... it's a good way to learn. Earlier in my career I did a lot of PHP. Usually close to bare. Other than the obvious point that writing an enormous JSON file is a dubious goal in the first place (really), while PHP can be very fast this is probably faster to implement in shell with sed/grep, or ... almost certainly better ... by loading to sqlite then dumping out from there. Your optimization path then likely becomes index specification and processing, and after the initial load potentially query or instance parallelization. The page confirms sqlite is available. If the judges whinge and shell_exec() is unavailable as a path, as a more acceptable path that's whinge-tolerant, use PHP's sqlite feature then dump to JSON. If I wanted to achieve this for some reason in reality, I'd have the file on a memory-backed blockstore before processing, which would yield further gains. Frankly, this is not much of a programming problem, it's more a system problem, but it's not being specced as such. This shows, in my view, immaturity of conception of the real problem domain (likely IO bound). Right tool for the job.
- lofaszvanitt 7mo agoDo not update the leaderboard.... at all.
- NorwegianDude 7mo agoFun challenge, but running the benchmark on Apple hardware is a weird decision as Apple doesn't even have server hardware. Would make much more sense to run it on a dedicated Linux box as that is more accessible and more realistic.