23 ms·

Tracking developer build times to decide if the M3 MacBook is worth upgrading

• kabes 3y ago
  Must be fun having gathered enough investor money to waste time on this sort of shenanigans.
• lawrjone 3y ago
  Author here, thanks for posting! Lots of stuff in this from profiling Go compilations, building a hot-reloader, using AI to analyse the build dataset, etc. We concluded that it was worth upgrading the M1s to an M3 Pro (the max didn’t make much of a difference in our tests) but the M2s are pretty close to the M3s, so not (for us) worth upgrading. Happy to answer any questions if people have them.
  • BlueToth 3y ago
    Hi, Thanks for the interesting comparison. What I would like to see added would be a build on a 8GB memory machine (if you have one available).
  • aranke 3y ago
    Hi, Thanks for the detailed analysis. I’m wondering if you factored in the cost of engineering time invested in this analysis, and how that affects the payback time (if at all). Thanks!
    • smabie 3y ago
      But then they'd have to factor in the engineering time invested in the analysis of the analysis?
      • zarzavat 3y ago
        Zeno’s nihilism: nothing is worth doing because the mandatory analysis into whether something is worth doing or not takes infinite time.
    • bee_rider 3y ago
      They could also add in the advertising benefit of showing off some fun data on this site :)
    • lawrjone 3y ago
      Author here: this probably took a 2.5 days to put together, all in. The first day was spent hacking together a new hot reloaded but this also fixed a lot of issues we’d had with the previous loader such as restarting into stale code, which was really harming people’s productivity. That was well worth even several days of effort really! The second day I was just messing around with OpenAI to figure out how I’d do this analysis. We’re right now building an AI assistant for our actual product so you can ask it “how many times did I get paged last year? How many were out-of-hours? Is my incident workload increasing?” Etc and I wanted an excuse to learn the tech so I could better understand that feature. So for me, well worth investing a day to learn. Then the article itself took about 4hrs to write up. That’s worth it for us given exposure for our brand and the way it benefits us for hiring/etc. We trust the team to make good use of their team and allowing people to do this type of work if they think it’s valuable is just an example of that. Assuming I have a £1k/day rate (I do not) we’re still only in for £2.5k, so less than a single MacBook to turn this around.
  • afro88 3y ago
    Great analysis! Thanks for writing it up and sharing. Logistical question: did management move some deliverables out of the way to give you room to do this? Or was it extra curricular?
  • firecall 3y ago
    So can we assume the M3 Max offered little benefit because the workloads couldn’t use the cores? Or the tasks maybe finished so fast that it didn’t make a difference in real world usage?
  • hedgehog 3y ago
    I'm curious how you came to the conclusion the Max SKUs aren't much faster, the distributions in the charts make them look faster but the text below just says they look the same.
  • deleted 3y ago
    [deleted]
• LanzVonL 3y ago
  We've found that distributed building has pretty much eliminated the need to upgrading developer workstations. Super easy to set up, too.
  • packetlost 3y ago
    Distributed building of what? Because for every language the answer of whether it's easy or not is probably different.
    • LanzVonL 3y ago
      [flagged]
      • packetlost 3y ago
        Sooooo... C? You didn't answer the question and I would be surprised to hear that freaking Java had support for distributed compilation, and that has been around since the 90s. Take your pretentious non-answers elsewhere please.
        • LanzVonL 3y ago
          [flagged]
          • deleted 3y ago
            [deleted]
  • lawrjone 3y ago
    I’m not sure this would work well for our use case. The distributed build systems only really benefit from aggressively caching the modules that are built, right? But the majority of the builds we do are almost fully cached, having changed just one module that needs recompiling then the linker sticks everything back together, which the machines would then need to download from the distributed builder and at 300MB a binary that’s gonna take a while. I may have this totally wrong though. Would distributed builds actually get us a new binary faster to the local machine? I suspect we wouldn’t want this anyway (lots of our company work on the go, train WiFi wouldn’t cut it for this!) but interested nonetheless.
• vaxman 3y ago
  1. If, and only if, you are doing ML or multimedia, get a 128GB system and because of the cost of that RAM, it would be foolish not to go M3 Max SoC (notwithstanding the 192GB M2 Ultra SoC). Full Stop. (Note: This is also a good option for people with more money than brains.) 2. If you are doing traditional heavyweight software development, or are concerned with perception in an interview, promotional context or just impressing others at a coffee shop, get a 32GB 16” MBP system with as large a built-in SSD as you can afford (it gets cheaper per GB as you buy more) and go for an M2 Pro SoC, which is faster in many respects than an M3 Pro due to core count and memory bandwidth. Full Stop. (You could instead go 64GB on an M1 Max if you keep several VMs open, which isn’t really a thing anymore (use VPS), or if you are keeping a 7-15B parameter LLM open (locally) for some reason, but again, if you are doing much with local LLMs, as opposed to being always connectable to the 1.3T+ parameter hosted ChatGPT, then you should have stopped at #1.) 3. If you are nursing mature apps along, maybe even adding ML, adjusting UX, creating forks to test new features, etc.. then your concern is with INCREMENTAL COMPILATION and the much bigger systems like M3 Max will be slower (bc they need time to ramp up multiple cores and that’s not happening with bursty incremental builds), so might as well go for a 16GB M1 MBA (add stickers or whatever if you’re ashamed of looking like a school kid) and maybe invest the savings in a nice monitor like the 28” LG DualUp (bearing in mind you can only use a single native-speed external monitor on non-Pro/Max SoCs at a time). (In many cases, can even use an 8GB M1 MBA for incremental builds because, after loading the project, the MacOS memory compressor is really good and the SSD is really fast and you can use a real device instead of a Simulator. But do you want any M2 MBA? No, it has inferior thermals, is heavier, larger, fingerprints easy, lack’s respect and the price performance doesn’t make sense given the other options. Same goes for 13” M1/M2 Pro and all M3 Pro.) Also, make sure you keep hourly (or better) backups on all Apple laptops WHILE CODING. There is a common failure scenario where the buck converter that drops voltage for the SSD fails, sending 13VDC into the SSD for long enough to permanently destroy the data on it. https://youtu.be/F6d58HIe01A https://youtu.be/F6d58HIe01A
  • whatshisface 3y ago
    Good to know I have commercial options for overcoming my laptop shame at interviews. /s
• vessenes 3y ago
  The upshot: M3 Pro is slightly better than M2 and significantly better than M1 Pro is what I've experienced with running local LLMs on my Macs; currently M3 memory bandwidth options are lower than for M2, and that may be hampering the total performance. Performance per watt and rendering performance are both better in the M3, but I ultimately decided to wait for an M3 Ultra with more memory bandwidth before upgrading my daily driver M1 Max.
  • lawrjone 3y ago
    This is pretty much aligned with our findings (am the author of this post). I came away feeling that: - M1 is a solid baseline - M2 improves performance by about 60% - M3 Pro is marginal on the M2, more like 10% - M3 Max (for our use case) didn’t seem that much different on the M3 Pro, though we had less data on this than other models I suspect Apple saw the M3 Pro as “maintain performance and improve efficiency” which is consistent with the reduction in P-cores from the M2. The bit I’m interested about is that you say the M3 Pro is only a bit better than the M2 at LLM work, as I’d assumed there were improvements in the AI processing hardware between the M2 and M3. Not that we tested that, but I would’ve guessed it.
    • vessenes 3y ago
      Yeah, agreed. I'll say I do use the M3 Max for Baldur's gate :). On LLMs, the issue is largely that memory bandwidth: M2 Ultra is 800GB/s, M3 Max is 400GB/s. Inference on larger models are simple math on what's in memory, so the performance is roughly double. Probably perf / watt suffers a little, but when you're trying to chew through 128GB of RAM and do math on all of it, you're generally maxing your thermal budget. Also, note that it's absolutely incredible how cheap it is to run a model on an M2 Ultra vs an H100 -- Apple's integrated system memory makes a lot possible at much lower price points.
      • lawrjone 3y ago
        Ahh right, I'd seen a few comments about the memory bandwidth when it was posted on LinkedIn, specifically that the M2 was much more powerful. This makes a load of sense, thanks for explaining.
• Erratic6576 3y ago
  Importing a couple thousand RAW pictures into a Capture One library would take 2 h on my 2017 iMac. 5 min on my m3 mbp pro. Geekbench score differences were quite remarkable. I am still wondering if I should return it, though
  • lawrjone 3y ago
    Go on, I’ll bite: why?
    • ac2u 3y ago
      They miss the 2 hours procrastination time. It's a version of "code's compiling" :)
      • Erratic6576 3y ago
        Ha ha ha. You can leave it overnight and importing files is a 1 time process so not much to win
      • teaearlgraycold 3y ago
        The foam swords are collecting dust.
    • Erratic6576 3y ago
      2,356 € is way over my budget. The machine is amazing but the specs are stingy. Returning it and getting a cheaper one would give me a lot of disposable money to spend in restaurants
      • tomaskafka 3y ago
        Get a 10-core M1 Pro then - I got mine for about 1200 eur used (basically undistinguishable from new), and the difference (except GPU) is very small. https://news.ycombinator.com/item?id=38810228 https://news.ycombinator.com/item?id=38810228
      • maerF0x0 3y ago
• kingTug 3y ago
  Does anyone have any anecdoctal evidence around the snappiness of VsCode with Apple Silicon? I very begrudgingly switched over from SublimeText this year (after using it as my daily driver for ~10yrs). I have a beefy 2018 MBP but VScode just drags. This is the only thing pushing me to upgrade my machine right now but I'd be bummed if there's still not a significant improvement with an m3 pro.
  • deleted 3y ago
    [deleted]
  • lawrjone 3y ago
    If you're using an Intel Mac at this point, you should 100% upgrade. The performance of the MX chips blows away the Intel chips and there's almost no friction with the arm architecture at this point. I don't use VSCode but most of my team do and I frequently pair with them. Never noticed it to be anything other than very snappy. They all have M1s or up (I am the author of this post, so the detail about their hardware is in the link).
    • hsbauauvhabzb 3y ago
      There can be plenty of friction depending on your use case.
      • grecy 3y ago
        It would be very helpful if you told us the use cases that have friction so we know if they apply to our use cases
      • hmottestad 3y ago
        What software are you worried about? Hackernews is a great place to share and hear if anyone has similar experiences or maybe managed to find a fix!
        • hsbauauvhabzb 3y ago
• throwaway892238 3y ago
  MacBooks are a waste of money. You can be just as productive with a machine just as fast for 1/2 the price that doesn't include the Apple Tax. Moreover, if your whole stack (plus your test suite) doesn't fit in memory, what's the point of buying an extremely expensive laptop? Not to mention constantly replacing them just because a newer, shinier model is released? If you're just going to test one small service, that shouldn't require the fastest MacBook. To test an entire product suite - especially one that has high demands on CPU and RAM, and a large test suite - it's much more efficient and cost effective to have a small set of remote servers to run everything on. It's also great for keeping dev and prod in parity. Businesses buy MacBooks not because they're necessary, but because developers just want shiny toys. They're status symbols.
  • deleted 3y ago
    [deleted]
  • cedws 3y ago
    It's OK to just not like Apple. You don't have to justify your own feelings with pejoratives towards other peoples' choice of laptop.
    • boringuser2 3y ago
      You really need to learn what a"pejorative" is before using the term publicly.
      • edwinjm 3y ago
        "waste of money" is a pejorative. Go away.
• mgaunard 3y ago
  [flagged]
  • Larrikin 3y ago
    WFH is effectively work from anywhere. I can take a month long working vacation and save a vacation day by working on the airplane. I'd be very annoyed if my work chained me to my home desk.
    • mgaunard 3y ago
      Most companies are cracking down on this kind of behaviour nowadays, in case you didn't get the memo.
    • Aurornis 3y ago
      > WFH is effectively work from anywhere. Great if your company allows it, but the “working vacation” thing has been either discourage or disallowed at every remote job I’ve had due to past abuse. Some jobs can handle truly asynchronous work and some people can be responsible about getting their work done away from home, but for every 1 employee who was pulling it off well we had 3-5 more who were just writing short responses in Slack and to emails to pretend they weren’t actually on vacation. Ruined it quickly for everyone. Lately I try to look for jobs at smaller companies where everyone can be trusted and there isn’t room for people to get away with those games. Seems much more likely to find more WFH freedom in those environments.
    • jnaina 3y ago
      Can't agree more. My M2 MacBook Air has been a major game changer. No battery anxiety, no throttling, no heat issues, etc. Allows me to work from anywhere. The productivity boost has been excellent.
  • t-writescode 3y ago
    I can’t use a desktop on a ferry or plane or at my Aunt’s over Thanksgiving
    • 3y ago
• swader999 3y ago
  My main metrics are 1) does the fan turn on, 2) does it respond faster than I think and move? Can't be any happier with the M2 at top end specs. It's an amazing silent beast.
• LispSporks22 3y ago
  I wish I needed a fast computer. It’s the CI/CD that’s killing me. All this cloud stuff we use - can’t test anything locally anymore. Can’t use the debugger. I’m back to glorified fmt.Printf statements that hopefully have enough context that the 40 min build/deploy time was worth it. At least it’s differential ¯\_(ツ)_/¯ All I can say is “I compiles… I think?” The unit tests are mostly worthless and the setup for sending something to a lambda feels like JCL boiler plate masturbation from that z/OS course I took out of curiosity last year. I only typing this out because I just restarted CI/CD to redeploy what I already pushed because even that’s janky. Huh, it’s an M3 they gave me.
  • lawrjone 3y ago
    Yeah everything you just said is exactly why we care so much about a great local environment. I've not seen remote tools approach the speed/ease/flexibility you can get from a fast local machine yet, and it makes a huge difference when developing.
    • LispSporks22 3y ago
      In the back of my mind I’m worried that our competitors have a faster software development cycle.
• orenlindsey 3y ago
  This is pretty cool, also I love how you can use AI to read the data. Would take minutes if not hours to do it even just a year ago.
  • lawrjone 3y ago
    Yeah, I thought it was really cool! (am author) It's pretty cool how it works, too: the OpenAI Assistant uses the LLM to take your human instructions like "how many builds is in the dataset?" and translate that into Python code which is run in a sandbox on OpenAI compute with access to the dataset you've uploaded. Under the hood everything is just numpy, pandas and gnuplot, you're just using a human interface to a Python interpreter. We've been building an AI feature into our product recently that behaves like this and it's crazy how good it can get. I've done a lot of data analysis in my past and using these tools blew me away, it's so much easier to jump into complex analysis without tedious setup. And a tip I figured out halfway through: if you want to, you can ask the chat for an iPython notebook of it's calculations. So you can 'disable autopilot' and jump into manual if you ever want finer control over the analysis it runs. Pretty wild.
    • guax 3y ago
      I also got surprised about using it for this kind of work. I don't have access to copilot and gpt-4 at work but my first instinct is to ask, did you double check its numbers? Knowing how it works now makes more sense that it would make less mistakes but I'm still skeptical :P
• tomaskafka 3y ago
  My personal research for iOS development, taking the cost into consideration, concluded: - M2 Pro is nice, but the improvement over 10 core (8 perf cores) M1 Pro is not that large (136 vs 120 s in Xcode benchmark: https://github.com/devMEremenko/XcodeBenchmark https://github.com/devMEremenko/XcodeBenchmark) - M3 Pro is nerfed (only 6 perf cores) to better distinguish and sell M3 Max, basically on par with M2 Pro So, in the end, I got a slightly used 10 core M1 Pro and am very happy, having spent less than half of what the base M3 Pro would cost, and got 85% of its power (and also, considering that you generally need to have at least 33 to 50 % faster CPU to even notice the difference :)).
  • geniium 3y ago
    Basically the Pareto effect in choosing the right cpu vs cost
  • mgrandl 3y ago
    The M3 Pro being nerfed has been parroted on the Internet since the announcement. Practically it’s a great choice. It’s much more efficient than the M2 Pro at slightly better performance. That’s what I am looking for in a laptop. I don’t really have a usecase for the memory bandwidth…
    • tomaskafka 3y ago
      Everyone has a different needs - for me, even M1 Pro has more battery life than I use or need, so further efficiency differences bring little value.
    • dgdosen 3y ago
      I picked up an M3Pro/11/14/36GB/1TB to 'test' over the long holiday return period to see if I need an M3 Max. For my workflow (similar to blog post) - I don't! I'm very happy with this machine. Die shots show the CPU cores take up so little space compared to GPUs on both the Pro and Max... I wonder why.
    • wlesieutre 3y ago
      I don’t really have a usecase for even more battery life, so I’d rather have it run faster
• aschla 3y ago
  Side note, I like the casual technical writing style used here, with the main points summarized along the way. Easily digestible and I can go back and get the details in the main text at any point if I want.
  • lawrjone 3y ago
    Thank you, really appreciate this!
• isthisreallife9 3y ago
  [flagged]
  • lawrjone 3y ago
    Fwiw I've spent my whole career doing data analysis but the ease at which I was able to use OpenAI to help me for this post (am author) blew me away. The fact that I can do this type of analysis is why I appreciate it so much. It's one of the reasons I'm convinced AI engineering find its way into the average software engineer's remit (https://blog.lawrencejones.dev/2023/#ai https://blog.lawrencejones.dev/2023/#ai) because it makes this analysis far more accessible than it was before. I still don't think it'll make devs redundant, though. Things the model can't help you with (yet, I guess): - Providing it with clean data => I had to figure out what data to collect, write software to collect it, ship it to a data warehouse, clean it, then upload it into the model. - Knowing what you want to achieve => it can help suggest questions to ask, but people who don't know what they want will still struggle to get results even from a very helpful assistant. These tools are great though, and one of the main reasons I wrote this article was to convince other developers to start experimenting with them like this.
    • gray_-_wolf 3y ago
      > it makes this analysis far more accessible than it was before How does the average engineer verify if the result is correct? You claim (and I believe you) to be able to do this "by hand", if required. Great, but that likely means you are able to catch when LLM makes an mistake. Any ideas on how average engineer, without much experience in this area, should validate the results?
      • lawrjone 3y ago
        I mentioned this in a separate comment but it may be worth bearing in mind how the AI pipeline works, in that you’re not pushing all this data into an LLM and asking it to produce graphs, which would be prone to some terrible errors. Instead, you’re using the LLM to generate Python code that runs using normal libraries like Pandas and gnuplot. When it makes errors it’s usually generating totally the wrong graphs rather than inaccurate data, and you can quickly ask it “how many X Y Z” and use that to spot check the graphs before you proceed. My initial version of this began in a spreadsheet so it’s not like you need sophisticated analysis to check this stuff. Hope that explains it!
• Aurornis 3y ago
  This is a great write-up and I love all the different ways they collected and analyzed data. That said, it would have been much easier and more accurate to simply put each laptop side by side and run some timed compilations on the exact same scenarios: A full build, incremental build of a recent change set, incremental build impacting a module that must be rebuilt, and a couple more scenarios. Or write a script that steps through the last 100 git commits, applies them incrementally, and does a timed incremental build to get a representation of incremental build times for actual code. It could be done in a day. Collecting company-wide stats leaves the door open to significant biases. The first that comes to mind is that newer employees will have M3 laptops while the oldest employees will be on M1 laptops. While not a strict ordering, newer employees (with their new M3 laptops) are more likely to be working on smaller changes while the more tenured employees might be deeper in the code or working in more complicated areas, doing things that require longer build times. This is just one example of how the sampling isn’t truly as random and representative as it may seem. So cool analysis and fun to see the way they’ve used various tools to analyze the data, but due to inherent biases in the sample set (older employees have older laptops, notably) I think anyone looking to answer these questions should start with the simpler method of benchmarking recent commits on each laptop before they spend a lot of time architecting company-wide data collection
  • lawrjone 3y ago
    I totally agree with your suggestion, and we (I am the author of this post) did spot-check the performance for a few common tasks first. We ended up collecting all this data partly to compare machine-to-machine, but also because we want historical data on developer build times and a continual measure of how the builds are performing so we can catch regressions. We quite frequently tweak the architecture of our codebase to make builds more performant when we see the build times go up. Glad you enjoyed the post, though!
    • FunnyLookinHat 3y ago
      I think there's something to be said for the fact that the engineering organization grew through this exercise - experimenting with using telemetry data in new ways that, when presented to other devs in the org, likely helped them to all level up and think differently about solving problems. Sometimes these wandering paths to the solution have multiple knock-on effects in individual contributor growth that are hard to measure but are (subjectively, in my experience) valuable in moving the overall ability of the org forward.
• dash2 3y ago
  As a scientist, I'm interested how computer programmers work with data. * They drew beautiful graphs! * They used chatgpt to automate their analysis super-fast! * ChatGPT punched out a reasonably sensible t test! But: * They had variation across memory and chip type, but they never thought of using a linear regression. * They drew histograms, which are hard to compare. They could have supplemented them with simple means and error bars. (Or used cumulative distribution functions, where you can see if they overlap or one is shifted.)
  • mnming 3y ago
    I think it's partly because the audiences are often not familiar with those statistics details either. Most people hates nuances when reading data report.
  • jxcl 3y ago
    Yeah, I was looking at the histograms too, having trouble comparing them and thinking they were a strange choice for showing differences.
  • Herring 3y ago
    >They drew histograms, which are hard to compare. Note that in some places they used boxplots, which offer clearer comparisons. It would have been more effective to present all the data using boxplots.
  • tmoertel 3y ago
    > They drew histograms, which are hard to compare. Like you, I'd suggest empirical CDF plots for comparisons like these. Each distribution results in a curve, and the curves can be plotted together on the same graph for easy comparison. As an example, see the final plot on this page: https://ggplot2.tidyverse.org/reference/stat_ecdf.html https://ggplot2.tidyverse.org/reference/stat_ecdf.html
  • dgacmu 3y ago
    I'm glad you noted programmers; as a computer science researcher, my reaction was the same as yours. I don't think I ever used a CDF for data analysis until grad school (even with having had stats as a dual bio/cs undergrad).
• fsckboy 3y ago
  I feel like there is a correlation between fast-twitch programming muscles and technical debt. Some coding styles that are rewarded by fast compile times can be more akin to "throw it at the wall, see if it sticks" style development. Have you ever been summoned to help a junior colleague who is having a problem, and you immediately see some grievous errors, errors that give you pause. You point the first couple out, and the young buck is ready to send you away and confidently forge ahead, with no sense of "those errors hint that this thing is really broken". but we were all young once, I remember thinking the only thing holding me back was 4.77MHz
  • wtetzner 3y ago
    There's a lot of value in a short iteration loop when debugging unexpected behavior. Often you end up needing to keep trying different variations until you understand what's going on.
    • lawrjone 3y ago
      Yeah there’s a large body of research that shows faster feedback cycles help developers be more productive. There’s nothing that says you can’t have fast feedback loops _and_ think carefully about your code and next debugging loop, but you frequently need to run and observe code to understand the next step. In those cases even the best programmer can’t overcome a much slower build.
    • fsckboy 3y ago
      thank you for quickly writing the fast twitch response! :)
  • clintonb 3y ago
    > I feel like there is a correlation between fast-twitch programming muscles and technical debt. Being fast doesn't mean you do everything at maximum speed. Usain Bolt doesn't sprint everywhere he goes, but he can sprint quite fast when he needs to do so.
    • fsckboy 3y ago
• LASR 3y ago
  Solid analysis. A word of warning from personal experience: I am part of a medium-sized software company (2k employees). A few years ago, we wanted to improve dev productivity. Instead of going with new laptops, we decided to explore offloading the dev stack over to AWS boxes. This turned out to be a multi-year project with a whole team of devs (~4) working on it full-time. In hindsight, the tradeoff wasn't worth it. It's still way too difficult to remap a fully-local dev experience with one that's running in the cloud. So yeah, upgrade your laptops instead.
  • jiggawatts 3y ago
    https://xkcd.com/1205/ https://xkcd.com/1205/
    • mdbauman 3y ago
      This xkcd seems relevant also: https://xkcd.com/303/ https://xkcd.com/303/ One thing that jumps out at me is the assumption that compile time implies wasted time. The linked Martin Fowler article provides justification for this, saying that longer feedback loops provide an opportunity to get distracted or leave a flow state while ex. checking email or getting coffee. The thing is, you don't have to go work on a completely unrelated task. The code is still in front of you and you can still be thinking about it, realizing there's yet another corner case you need to write a test for. Maybe you're not getting instant gratification, but surely a 2-minute compile time doesn't imply 2 whole minutes of wasted time.
      • chiefalchemist 3y ago
        Spot on. The mind often needs time and space to breathe, especially after it's been focused and bearing down on something. We're humans, not machines. Creativity (i.e., problem solving) needs to be nurtured. It can't be force fed. More time working doesn't translate to being more effective and more productive. If that were the case then why are a disproportionate percentage of my "Oh shit! I know what to do to solve that..." in the shower, on my morning run, etc.?
• boringuser2 3y ago
  I get a bit of a toxic vibe from a couple comments in that article. Chiefly, I think the problem is that the CTO solved the wrong problem: the right problem to solve includes a combination of assessing why company public opinion is generating mass movements of people wanting a new MacBook literally every year, if this is even worth responding to at all (it isn't), and keeping employees happy. Most employees are reasonsble enough to not be bothered if they don't get a new MacBook every year. Employers should already be addressing outdated equipment concerns. Wasting developer time on a problem that is easily solvable in one minute isn't worthwhile. You upgrade the people 2-3 real generations behind. That should already have been in the pipeline, resources notwhistanding. I just dislike this whole exercise because it feels like a perfect storm of technocratic performativity, short sighted "metric" based management, rash consumerism, etc.
  • BlueToth 3y ago
    It's really worth the money if it keeps employees happy! Besides that the conclusion was updating M1 to M3, but not every year.
  • lawrjone 3y ago
    Sorry you read it like this! If it’s useful: Pete wasn’t really being combative with me on this. I suggested we should check if the M3 really was faster so we could upgrade if it was, we agreed and then I did the analysis. The game aspect of this was more for a bit of fun in the article than how things actually work. And in terms of why we didn’t have a process for this: the company itself is about two years old, so this was the first hardware refresh we’d ever needed to schedule. So we haven’t a formal process in place yet and probably won’t until the next one either!
  • FireBeyond 3y ago
    Hah, yes. There's a balancing act, and investing in tools is important. I also worked at a company whose CIO put out an email noting how "amazing" a coincidence it was just how many company iPhones got broken/water damaged/lost or stolen in the 2 months following a new iPhone announcement event.
  • varispeed 3y ago
• joshspankit 3y ago
  Since RAM was a major metric, there should have been more focus on IO Wait to catch cases where OSX was being hindered by swapping to disk. (Yes, the drives are fast but you don’t know until you measure)
  • cced 3y ago
    This. I’ve routinely got a 10-15GB page file on an M2 pro and need to justify bumping the memory up a notch or two. I’m consistently in the yellow memory and in the red while building. How can I tell how much I would benefit from a memory bump?
    • anileated 3y ago
      How much RAM do you have?
      • cced 3y ago
        16 :/
    • joshspankit 3y ago
      Mainly you’d want to watch for 1. disk usage of the page file and 2. IO wait of the overall system. Here are some commands to get you started: https://stackoverflow.com/questions/15786618/per-process-disk-read-write-statistics-in-mac-os-x https://stackoverflow.com/questions/15786618/per-process-dis...
• mixmastamyk 3y ago
  A lot of the graphs near the end comparing side-to-side had different scales on the Y axis. Take results with a grain of salt. https://incident.io/_next/image?url=https%3A%2F%2Fcdn.sanity.io%2Fimages%2Foqy5aexb%2Fproduction%2Fc8d95c71304e90f555b6402cbacb67b0ef10b314-1608x834.png%3Fq%3D75%26fit%3Dclip%26auto%3Dformat&w=3840&q=75 https://incident.io/_next/image?url=https%3A%2F%2Fcdn.sanity...
  • lawrjone 3y ago
    They’re normalised histograms so the y axis is deliberately adjusted so you can compare the shape of the distribution, as the absolute number of builds in each bucket means little when there are a different count of builds for each platform.
• hk1337 3y ago
  I wonder why they didn't include Linux since the project they're building is Go? Most CI tools, I believe, are going to be Linux. Sure, you can explicitly select macOS in Github CI but Linux seems like it would be the better generic option? *EDIT* I guess if you needed a macOS specific build with Go you would us macOS but I would have thought you'd use Linux too. Can you build a Go project in Linux and have it run on macOS? I suppose architecture would be an issue building on Linux x86 would not run on macOS Apple Silicon but the reverse is true too a build on Apple Silicon would not work on Linux x86 maybe not even Linux Arm.
  • xp84 3y ago
    I know nothing about Go, but if it's like other platforms, builds intended for production or staging environments are indeed nearly always for x86_64, but those are done somewhere besides laptops, as part of the CI process. The builds done on the laptops are to run each developer's local instance of their server-side application and its front-end components, That instance is always being updated to whatever is in-progress at the time. Then they check that code in, and eventually it gets built for prod on an Intel, Linux system elsewhere.
  • varispeed 3y ago
    Cross compilation is probably easiest in Go. If I recall you can just give it a different arch parameter and it will produce a working build for any supported platform.
• SSLy 3y ago
  > Application error: a client-side exception has occurred (see the browser console for more information). When I open the page.
• rendaw 3y ago
  > People with the M1 laptops are frequently waiting almost 2m for their builds to complete. I don't see this at all... the peak for all 3 is at right under 20s. The long tail (i.e. infrequently) goes up to 2m, but for all 3. M2 looks slightly better than M1, but it's not clear to me there's an improvement from M2 to M3 at all from this data.
• varispeed 3y ago
  M1 is such a big difference. I bought mine two years ago and it still feels like and performs like new. Never had even remotely similar experience to Intel laptops. I am hoping that companies that are getting rid of their M1 stock, put them on a second hand market rather than through disposal service (where they get physically destroyed).
• mcny 3y ago
  > All developers work with a fully fledged incident.io environment locally on their laptops: it allows for a <30s feedback loop between changing code and running it, which is a key factor in how productively you can work with our codebase. This to me is the biggest accomplishment. I've never worked at a company (besides brief time helping out with some startups) where I have been able to run a dev/local instance of the whole company on a single machine. There's always this thing, or that, or the other that is not accessible. There's always a gotcha.
  • teaearlgraycold 3y ago
    I’m currently doing my best to make this possible with an app I’m building. I had to convince the CEO the M2 Max would come in handy for this (we run object detection models and stable diffusion). So far it’s working out!
  • 8n4vidtmkvmk 3y ago
    I never couldn't run the damn app locally until my latest job. Drives me bonkers. I don't understand how people aren't more upset and this atrocious devex. Damn college kids don't know what they're missing.
  • kamikaz1k 3y ago
    I used to work in a company like that, and since leaving it I’ve missed that so much. People who haven’t lived in that world just cannot understand how much better it is, and will come up with all kinds of cope.
  • physicles 3y ago
    I can’t imagine not having this. We use k3s to run everything locally and it works great. But we (un)fortunately added snowflake in the last year — it solves some very real problems for us, but it’s also a pain to iterate on that stuff.
  • daxfohl 3y ago
    We used to have that, but it's hard to support as you scale. The level of effort is somewhat quadratic to company size: linear in the number of services you support and in the number of engineers you have to support. Also divergent use cases come up that don't quite fit, and suddenly the infra team is the bottleneck to feature delivery, and people just start doing their own thing. Once that Pandora's Box is opened, it's essentially impossible to claw your way back. I've heard of largeish companies that still manage to do this well, but I'd love to learn how. That said, yeah I agree this is the biggest accomplishment. Getting dev cycles down from hours or days to minutes is more important than getting them down from minutes to 25% fewer minutes.
• brailsafe 3y ago
  Fun read, I like how overkill it is. When I was still employed, I was building our django/postgres thing locally in Docker, with 32gb of ram, and it was a wild improvement in terms of feedback loop latency over my shitty 13" intel mbp, and I think it's seriously underappreciated how important it is to keep that pretty low, or as low as is cost effective. Now that I'm not employed and don't hope to be for a while, I do think the greatest bottleneck in my overall productivity is my own discipline since I'm not compiling anything huge or using Docker. The few times I do really notice how slow it is, it's in big I/O or ram operations like indexing, or maybe the occasional xcode build, but it's still low in absolute terms and the lack of some stressful deadline doesn't have me worrying so much about it. That makes me happy in some ways, because I used to feel like I could just throw new hardware at my overall productivity and solve any issues, but I think that's true only for things that are extremely computationally expensive. Normally, I'd just spend the cash, even as a contractor, because it's investment in my tools and that's good, but the up-charge for ram and ssd is so ludicrously high that I have no idea when that upgrade will come, and the refurb older models of M1 and M2 just aren't that much lower. My battery life also sucks, but it's not $5k sucky yet. Also worth joting I'm just a measly frontend developer, but there have been scenarios in which I've been doing frontend inside either a big Docker container or a massive Tomcat java app, and for those I'd probably just go for it.
  • karolist 3y ago
    I don't know where in the world you are, but B&H in US still sells new 16" M1 Max machines with 64GB memory, 2TB SSD for 2499-2599 depending on the current deal. This is around the price of base M3 Pro with 18/512 configuration, I figure you'll still get 5+ years of use with such machine and never worry about storage or memory.
    • brailsafe 3y ago
      Good point, although it would feel odd to spend what would amount to about $3500 after tax in CAD on a 3 y.o laptop, albeit a new 3 y.o laptop. For now I'll just stick it out with my Intel thing, since that $3500 is a more expensive $3500 than ever anyway, until my requirements get more demanding.
• berkut 3y ago
  It's interesting to me that being on A/C and Battery power showed a performance difference: when I benchmarked a M1 Pro MBPro against an Intel 2015 MBPro two years ago, I didn't notice any performance differences between being on battery or using mains power in either laptop, which surprised me at the time: https://pixelesque.net/blog/2022/08/basic-apple-m1-pro-cpu-benchmarks/ https://pixelesque.net/blog/2022/08/basic-apple-m1-pro-cpu-b... I wonder what the difference is... Maybe my tests were too short to show the difference between mains and battery?
  • alecthomas 3y ago
    My M2 defaults to using a "low power mode" when on battery, which I noticed initially because some graphics code I was writing was getting significantly lower frame rates when on battery. Edit: 310fps vs 240fps
• uncanny 3y ago
  I just upgraded to the top M3 Max CPU from the top M2 Max CPU. I benchmarked several test suites to compare the machines. The test suites for two different Rails applications were around 10% faster on the M3 Max. A test suite for a Go web application that does significant integration testing in parallel (i.e. multiple complete stacks of database, application server, and browser) improved by around 20%. The extra P-cores help. Miscellaneous other tests and benchmarks were around 7-10% faster. Didn't notice any difference in normal usage. So not likely worth the upgrade unless your workload can really make use of the extra cores.
• postalrat 3y ago
  If you are working at home why bother with a laptop? You can get a much more powerful desktop for the same price.
  • rapfaria 3y ago
    Threadripper away, most folks are sent a machine by their company tho
  • faeriechangling 3y ago
    What if you don't need a much more powerful desktop? What if you get more utility from being able to move the computer around at least occasionally?
    • FpUser 3y ago
      For "occasional move" I have small factor desktop. It is relatively tiny but still has 16 cores AMD and 128GB RAM.
  • ewoodrich 3y ago
    I ditched my desktop once "plug in a single USB-C cable and get dual 4k monitors + peripherals all ready to go" became a thing. I love the flexibility of a laptop combined with a dedicated workstation setup.
    • postalrat 3y ago
      If flexibility is your #1 priority then usb-c never mattered. You could get mobility long before usb-c. I'm saying if you want a fast machine then you can't beat a desktop for value or potential.
      • ewoodrich 3y ago
        It matters in the sense that my 2015 Macbook Pro required multiple cables to unplug/replug in at my desk which seems like a small thing but significantly reduced the convenience factor for me. Plus I can now dock either my Lenovo or my MacBook with the same multi-monitor dock making it even more versatile. I don't game or do anything more intensive than web development so I recognize it wouldn't work as well for everyone.
• pmarreck 3y ago
  One note, the extra memory in the 128GB model is directly usable for locally-running LLM models
• relativeadv 3y ago
  anecdata: my hobby project went from a 1m 24s build time (flutter) on an M1 Pro to an 8s build time on M3 Pro.
• djha-skin 3y ago
  I was curious about something the OP said. He said that they were going dockerless and that they do it in part to reclaim some memory on Mac laptops. Is anyone else doing this or why would anyone else do this?
  • prsutherland 3y ago
    I'm guessing it is because docker runs a linux VM when it is running on macos.
  • lawrjone 3y ago
    Am OP: the Docker for Mac daemon can claim a lot of system resources and the FS can have a lot of latency. This means devs have less system memory available for their host because it’s in the VM, battery is often harmed because the VM box is constantly churning, and as we run our Postgres database in docker the FS latency makes for a noticeably slower dev environment. Docker is great for standardising the dev env but honestly, running Postgres locally isn’t hard, and we don’t have many other deps. If the quality of experience is better at the cost of the occasional “let me help you out with this pg_upgrade” then I’d consider that a win.
• imran-iq 3y ago
  Is that screenshot at the end real? 27 thousand pounds for a machine with only a 1 TB ssd? Edit: Nvm didnt see the quantity 10
  • smilespray 3y ago
    10 machines, not 1.
• justinclift 3y ago
  Wonder who ended up with the M3 Max 36GB machine? ;)
  • lawrjone 3y ago
    Hahaha yep, Milly won the lottery on this one. We chose the person who was doing most builds that week to give it to, so it’s well deserved!
• actinium226 3y ago
  Honestly, I found the analysis a little difficult to read. Some of the histograms weren't normalized which made the comparison a bit difficult. One thing that really stood out to me was: > People with the M1 laptops are frequently waiting almost 2m for their builds to complete. And yet looking at the graph, 120s is off the scale on the right side, so that suggests almost literally no one is waiting 2m for a build, and most builds are happening within 40s with a long tail out to 1m50s.
  • _teyd 3y ago
    I think the main point was justifying getting new M3s
• prvc 3y ago
  How about: incentivize efficient use of resources by making devs use slower computers!
  • yieldcrv 3y ago
    there is a perpetual ESG argument that always wins, older computers which are slower use more energy than the new one thats fast. older computers that were top of the line use waaay more energy than the new one that’s fast.
• drgo 3y ago
  To people who are thinking about using AI for data analyses like the one described in the article: - I think it is much easier to just load the data into R, Stata etc and interrogate the data that way. The commands to do that will be shorter and more precise and most importantly more reproducible. - the most difficult task in data analysis is understanding the data and the mechanisms that have generated it. For that you will need a causal model of the problem domain. Not sure that AI is capable of building useful causal models unless they were somehow first trained using other data from the domain. - it is impossible to reasonably interpret the data without reference to that model. I wonder if current AI models are capable of doing that, e.g., can they detect confounding or oversized influence of outliers or interesting effect modifiers. Perhaps someone who knows more than I do on the state of current technology can provide a better assessment of where we are in this effort
  • balls187 3y ago
    That is effectively what the GPT4 based AI Assistant is doing. Except when I did it, it was python and pandas. You can ask it to show you the code it used to do it's analysis. So you can load the data into R/Python and google "how do I do xyzzzy" and write the code yourself, or use ChatGPT.
    • drgo 3y ago
      so ChatGPT can build a causal model for a problem domain? How does it communicate that (using a DAG?)? It would be important for the data users to understand that model.
• balls187 3y ago
  > a chat interface to your ... data Generally, the process includes: > Exporting your data to a CSV > Create an ‘Assistant’ with a prompt explaining your purpose, and provide it the CSV file with your data. Once MSFT build's the aforementioned process into Excel, it's going to be a major game changer.
• pmarreck 3y ago
  Is the M3 Ultra laptop coming anytime soon?
• Tarball10 3y ago
  This is only tangentially related, but I'm curious how other companies typically balance their endpoint management and security software with developer productivity. The company I work for is now running 5+ background services on their developer laptops, both Mac and Windows. Endpoint management, priviledge escalation interception, TLS interception and inspection, anti-malware, and VPN clients. This combination heavily impacts performance. You can see these services chewing up CPU and I/O performance while doing anything on the machines, and developers have complained about random lockups and hitches. I understand security is necessary, especially with the increase in things like ransomware and IP theft, but have other companies found better ways to provide this security without impacting developer productivity as much?
  • andrekandre 3y ago
    > have other companies found better ways to provide this security without impacting developer productivity as much? only way i've seen is if things get bad, report it to it/support and tell them what folder/files to exclude from inspection so your build temp files and stuff don't clog and slow up everything
    • suchar 3y ago
      Same here, but IMO, if company believes that such software is useful (and they wouldn't be using it if company believed otherwise), then why do they often (always?) include node_modules in exclusion rules? After all, node_modules usually contains a lot of untrusted code/executables
• euos 3y ago
  I am ex-core contributor Chromium and Node.js and current core contributor to gRPC Core/C++. I am never bothered with build times. There is "interactive build" (incremental builds I use to rerun related unit tests as I work on code) and non-interactive build (one I launch and go get coffee/read email). I have never seen hardware refresh toggle non-interactive into interactive. My personal hardware (that I use now and then to do some quick fix/code review) is 5+ year old Intel i7 with 16Gb of memory (had to add 16Gb when realized linking Node.js in WSL requires more memory). My work laptop is Intel MacBook Pro with a touch bar. I do not think it has any impact on my productivity. What matters is the screen size and quality (e.g. resolution, contrast and sharpness) and storage speed. Build system (e.g. speed of incremental builds and support for distributed builds) has more impact than any CPU advances. I use Bazel for my personal projects.
  • gizmo 3y ago
    Somehow programmers have come to accept that a minuscule change in a single function that only result in a few bytes changing in a binary takes forever to compile and link. Compilation and linking should be basically instantaneous. So fast that you don't even realize there is a compilation step at all. Sure, release builds with whole program optimization and other fancy compiler techniques can take longer. That's fine. But the regular compile/debug/test loop can still be instant. For legacy reasons compilation in systems languages is unbelievably slow but it doesn't have to be this way.
    • cole-k 3y ago
      I mean yeah I've come to accept it because I don't know any different. If you can share some examples of large-scale projects that you can compile to test locally near-instantly - or how we might change existing projects/languages to allow for this - then you will have my attention instead of skepticism.
      • euos 3y ago
        That’s why I write test first. I don’t want to build everything.
• eutropia 3y ago
  To play devil’s advocate: did you see any improvement in ticket velocity, features shipped, bugs dispatched, prod events resolved - i.e. is this improvement in productivity measurable by what you judge the business by, or does it just feel better and make devs happy?
  • lawrjone 3y ago
    The machines haven’t arrived yet but I wouldn’t expect we’d have the data to prove this out. As you’ll know, number of tickets complete is a weak proxy for value delivered, and can be so wrong as to imply a team that you know is 2x as productive is half as much as another, or worse. The best way (imo) to measure this is setting a threshold that we agree is bad (in our case, this is 1m+ builds) and continually measuring it so we catch when that starts to happen more frequently. And listening to devs when they say things have got slow/have become irritating.
  • jes 3y ago
    Eli Goldratt has entered the chat...
  • lagadu 3y ago
    Happy devs will at the very least reduce turnover rate, which will have a net positive effect in all the metrics you mentioned.
• jorgemf 3y ago
  My problem with this analysis is ignoring the fact of who is using which computer. So far new people in the company get the M3, while old people have M2, and the people who has been the longest time in the company have an M1. Who is going to work on more critical tasks with more changes in the code? who is going to work mostly in easy bugs until they get some experience with the code in the company? I bet you if you give both populations the same computer the compiling times are going to be faster for the new people. For me the analysis doesn't have enough dimensions, it should take into account the time since the person was hired in the company and the seniority. I would also have added more type of graphs (boxplots seems a better way to compare the information), and also I would have measure the total % of CPU usage. The battery/AC analysis gave me the impression that M3 might be underutilized and that it is going to be impossible to get lower compiling times without faster single core speeds (which might be a relevant information for the future).
• eole666 3y ago
  Maybe before buying M3 MacBook for everyone you should consider splitting your 1M line code base to avoid recompiling the whole thing for every change.. I don't know well the go ecosystem at all but it seems to me you should really consider optimizing your code base for compilation speed before creating fancy build time chart per cpu with AI and buying hundreds of super powerfull and expensive machines to compile it a bit faster...
  • lawrjone 3y ago
    We do frequently adjust the codebase to improve build times, but the developer productivity hit we’d take by splitting our app into several microservices would be much more substantial than much slower build times than we currently see. We’re super happy with 30s as an average build time right now. Assume we spent about £50k for the entire teams laptops, how much time would an engineer need to break our monolith into microservices? How much extra investment would we need to make it work as well as our monolith, or debug production issues that are now much more complex? I’d wager it’s much more than half an engineers time continually, at which point you’ve blown the money you saved on a solution that works far less well for dev productivity and debugging simplicity in production.
    • eole666 3y ago
      Maybe you could split the code base without going full micro service. Keep the monolith base, and split some parts in independent module/libraries that won't be rebuild every time, something like that.
• maerF0x0 3y ago
  PSA: Dont get a mac with apple silicon if you need/want Virtual Box https://www.virtualbox.org/wiki/Downloads https://www.virtualbox.org/wiki/Downloads
  • neurostimulant 3y ago
    There is UTM which is good enough: https://mac.getutm.app/ https://mac.getutm.app/
• ekianjo 3y ago
  [flagged]
• zwischenzug 3y ago
  How much did the analysis cost vs the laptop upgrade?
  • lawrjone 3y ago
    Hahah good question: 2.5 days all in, including building the new hot-reloader which fixed a number of outstanding bugs. - 1 day for hot-reload (I don’t think this should count tbh) - 1 day for messing with data - 0.5 day for write-up Assuming my day rate is £1k/day (it is not) we land at £1.5k, less than a laptop and less than I’d pay for the impressions I got from my post sharing this on LinkedIn, so feels worthwhile!
• 1-6 3y ago
  >People with the M1 laptops are frequently waiting almost 2m for their builds to complete. I guess M1 laptops are here to stay until M5 laptops come out.
• mdswanson 3y ago
  Just to add: https://github.com/devMEremenko/XcodeBenchmark https://github.com/devMEremenko/XcodeBenchmark
• marsven_422 3y ago
  [dead]
• ksec 3y ago
  To me, that is just another Data Set to prove how most of the MacBook are used in tech industry. And their upgrade cycle. 37 Million Registered Developers ( Many Overlap ). And many Devs using it for Web development. It seems Apple could have 40M of software developers using their platform. Out of ~125M active Mac users.
• JonChesterfield 3y ago
  If dev machine speed is important, why would you develop on a laptop? I really like my laptop. Spend a lot of time typing into it. It's limited to a 30W or similar power budget on thermal and battery constraints. Some of that is spent on a network chip which grants access to machines with much higher power and thermal budgets. Current employer has really scary hardware behind a VPN to run code on. Previous one ran a machine room with lots of servers. Both expected engineer laptops to be mostly thin clients. That seems obviously the right answer to me. Thus marginally faster dev laptops don't seem very exciting.
  • solatic 3y ago
    > Current employer has really scary hardware behind a VPN to run code on. Previous one ran a machine room with lots of servers. Both expected engineer laptops to be mostly thin clients. That seems obviously the right answer to me. It's quite expensive to set up, regardless of whether we're talking about on-prem or cloud hardware. Your employer is already going to buy you a laptop; why not try to eke out what's possible from the laptop first? The typical progression, I would think, is (a) laptop only, (b) compilation times get longer -> invest in a couple build cache servers (e.g. Bazel) to support dozens/hundreds of developers, (c) expand the build cache server installation to provide developer environments as well
• mattlondon 3y ago
  This is cool, but so you really trust the LLM to have not just hallucinated things during analysis?
• waffleiron 3y ago
  For new laptops this might hold, but for early upgrades you need to also calculate in the overhead of setting up the new laptops. This could be substantial depending on how it’s done.
• DeathArrow 3y ago
  >M2 is a significant upgrade on the M1 machines. M3 is an incremental improvement on M2. Cinebench and Geekbench tell a different story. M2 is a small upgrade over M1 while M3 is a bigger upgrade over M2.
  • brandall10 3y ago
    This quoted statement is true for the Pro chips. For the Max, yes the M3 series is a sizable upgrade. It's worth noting they're using the binned Max and quite frankly are undercutting the actual results the Max produces, it appears to be substantial.
• ribit 3y ago
  Interesting idea, but the quality of data analysis is rather poor IMO and I'm not sure that they are actually learning what they think they are learning. Most importantly, I don't understand why they would see such a dramatic increase of sub 20s build times going from M1 Pro to M2 Pro. The real-world performance delta between the two on code compilation workloads is around 20-25%. It also makes little sense to me that M3 machines have fewer sub-20s builds than M2 machines. Or that M3 Pro with half the cores has more sub-20s builds than M3 Max. I suspect there might be considerable difference in developer behavior which results in these differences. Such as people with different types of laptops typically working on different things. And a few random observations after a very cursory reading (I might be missing something): - Go compiler seems to take little advantage from additional cores - They are pooling data in ways that makes me fundamentally uncomfortable - They are not consistent in their comparisons, sometimes they use histograms, sometimes they use binned density plots (with different y axis ranges), it's real unclear what is going on here... - Macs do not throttle CPU performance on battery. If the builds are really slower on battery (which I am not convinced about btw looking at graphs), it will be because of "low power" setting activated
  • epolanski 3y ago
    > also makes little sense to me that M3 machines have fewer sub-20s builds than M2 machines. M3s have a smaller memory bandwidth, they are effectively a downgrade for some use cases.
    • ribit 3y ago
      You are not going to saturate a 150GB/s memory interface building some code on a six-core CPU... these CPUs are fast, but not that fast.
      • inkyoto 3y ago
        Oh, yes, you are. The optimising steps, linking and the link-time optimisation (LTO) are very heavily memory bound, especially on large codebases. The Rust compiler routinely hits the 45-50Gb/sec ballpark of the intra-memory transfer speed on compiling a medium sized project, more if the code base large. Haskell (granted, a fringe yet revealing case) case just as routinely hits the 60-70 Gb/sec memory transfer speed at the compile time, and large to very large C++ codebases add a lot of stress on the memory at the optimisation step. If I am not mistaken, Go is also very memory bound. Then there comes the linking and particularly the LTO that want all the memory bandwidth they can get to get the job done quickly, and the memory speed becomes a major bottleneck. Loading the entire codebase into memory, in fact, the major optimisation technique used in mold[0] that can vastly benefit from a) faster memory, b) a wider memory bus. [0] https://github.com/rui314/mold https://github.com/rui314/mold
• neya 3y ago
  Last month I upgraded from an M1 Pro to an M3 Pro - 16GB RAM, 512 GB. Here is an honest review of my experience. The chassis of both the machines are exactly the same. The bad thing about that is the newer MacBooks come with a matte finish for the keyboard and the finish wears off like in just 3 weeks even with proper care. As a matter of fact, in the newer machine, I used ISP 70% with water to wipe it off after each use thinking it was due to grease/oil from the fingers. However the finish still wore off, suggesting that it has more to do with the finish itself. With that out of the way, I maintain a very large site with over 50,000 posts in a static site setup. It is built on a Phoenix/Elixir setup I wrote from scratch and has been serving my client well without hiccups for the last 5 years. From time to time, some of the writers may mess up something and they would want us to run a sweep occasionally - back track to the last X number of posts and re-publish them. Usually about 100-500 posts which covers a few weeks/months worth of posts depending on the run. So, for a 100 post sweep, the M3 pro was slightly faster, by an order of magnitude of over 1.5x. But, for other everyday tasks, like say opening Affinity Designer/Photo and editing files, I didn't notice much improvements. And Apple's website is notoriously deceptive about the improvements. Their graphs always showcase comparisons with a top specc'ed M3 machine with a base M1 variant or M2 which makes no sense to me as a developer. Disk (SSD) speeds were slightly slower on the M3 than on my old M1 variant. But, I am unable to attribute it to the processor as it could be a bad driver or software version. For those interested, it is one of those internal Samsung SSDs connected via a SATA-USBC converter. Long story short, if you are on M2 - Not worth upgrading. If you are on an M1 or M1 Pro, you can still hold on to it a bit longer, the only reason you would want to upgrade is if you got a good deal like me. I sold my 21' M1 Pro for almost $1800 US (it still had Apple care) and got a student discount (I'm in the middle of some certification programs) on the new M3 Pro, so I got it. If I didn't have these financial incentives, probably wouldn't have been worth upgrading. Hope this helps someone.
• coreyh14444 3y ago
  Companies spend billions of dollars building BI tools that can do analysis against a data warehouse and they just... exported a CSV and asked ChatGPT to analyze it. One of the biggest shifts you can imagine in terms of the value of pre-AI and post worlds.
• bluelightning2k 3y ago
  Great write up. Would personally be interested to see something similar for TypeScript/Vite as that's what I personally use (less likely to actually improve with beefier machine). Made me very curious for my own stack
• wouldbecouldbe 3y ago
  What I've found is that low-power mode is much slower, even if this article claims it's not. In my experience the difference of low power mode is bigger when doing something else CPU intensive such as video calling.
• db48x 3y ago
  This is bad science. You compared the thing you had to the thing you wanted, and found a reason to pick the thing you wanted. Honesty should have compelled you to at least compare against a desktop–class machine, or even a workstation with a Threadripper CPU. Since you know that at least part of your workload is concurrent, and 14 CPUs are better than 10, why not check to see if 16, 32, or 64 is better still? And the linker is memory bound, so it is worth considering not just the quantity of memory but the actual memory bandwidth and latency as well.
  • danhau 3y ago
    I don‘t think this is bad science at all. From the article: > All incident.io developers are given a MacBook which they use for their development work. Non-MacBook machines are apparently not an option, for whatever reason. Comparing against other machines would be interesting, but irrelevant.
    • db48x 3y ago
      So it’s ok for science to be limited by politics?
      • p_j_w 3y ago
        They’re looking at their particular use case. That may limit the applicability of this to other people or companies, but that doesn’t make it politics.
        • db48x 3y ago
          When your CEO decides that your company will only buy Apple laptops, that is definitely a political football. Your CEO likes shiny things, and is willing to purchase the shiniest of them without regard to effectiveness or cost.
  • minimaul 3y ago
    Being Mac only can be an advantage - I’ve been on both sides of trying to maintain & use non-trivial dev environments and the more OSes you bring in for people to work on, the harder it gets. Bringing in Windows or Linux has a set up cost and a maintenance cost that may exclude it from even being considered. Edit: plus, Macs are ARM, other options are inevitably x86. So it’s also two CPU architectures to maintain support for, on top of OS specific quirks - and even if you use eg Docker, you still have a lot of OS specific quirks in play :/
• coldtea 3y ago
  How does the build time average time being around 400 (seconds?) for M1 and M2 and around 200 for M3 make the M2 a substantial upgrade over M1 but M3 "an incremental improvement on M2"? Also would it hurt the author if they had kept the same max y-value on all plots?
• fastaguy88 3y ago
  Lots of interesting data, but for me, clear examples of why histograms are not particularly useful for comparing performance. Box plots would have been much more informative, particularly for comparisons of M2/M3 and memory effects, where the histograms look a little different, but it is unclear how different.
• lulznews 3y ago
  Bro if you have time to do this you have too many developers.
• Liebnitz 3y ago
  Network full builds might be faster, but would incremental builds be? Would developers still be able to use their favourite IDE and OS? Would developers be able to work without waiting in a queue? Would developers be able to work offline? If you have a massive, monolithic, single-executable-producing codebase that can't be built on a developer machine, then you need network builds. But if you aren't Google, building on laptops gives developers better experience, even if it's slower.
• nggkkg 3y ago
  [flagged]