7 ms·

I'm ex-sun, kernel group. I'm retired, early at 55, but after reading Brendan's write up, I'd work for that guy. Holy smokes, he is all over it. Reminds me o

I'm ex-sun, kernel group. I'm retired, early at 55, but after reading Brendan's write up, I'd work for that guy. Holy smokes, he is all over it. Reminds me of me when I was in the groove. Brendan if you read this, I'm old school, not good at all the ruby on rails etc, but systems, yeah, pretty good. Not looking for money, looking for working with smart people, I can be paid in stock. If it works out, great, if it doesn't still great because I like smart people. Sorry for making it about me, you all should read his post, it's someone who is completely in the groove, has breadth and depth, knows about systems. These people are rare, but hugely valuable when you need to scale stuff up.

jamesmishra 9y ago
If you are indeed interested in coming out of retirement, Brendan Gregg does performance engineering at Netflix. It might be worth reaching out. http://www.brendangregg.com/blog/2017-05-16/working-at-netflix-2017.html http://www.brendangregg.com/blog/2017-05-16/working-at-netfl...
- luckydude 9y ago
  I'm kinda old and burned out but thanks for the link. I'll reach out. It could be a boatload of fun.
  - rurban 9y ago
    It's even much more fun than described in this article, because Netflix mostly works with FreeBSD not Linux. So no NIH syndrom, proper dtrace (no eBFS hacks), proper ZFS, proper tooling, easy kernel maintenance.
    - gribbly 9y ago
      As I've seen it described here before (by Brendan Gregg ?) Netflix uses FreeBSD for the CDN servers (streaming the video), and Linux for everything else, browsing Netflix, encoding, etc.
      - filomeno 9y ago
        That's true, although he was right about the Linux counterparts. Reading this migration guide, at least to me, it seems that all Linux replacements for things like dtrace, ZFS, SMF, zones... are just subpar.
        luckydude 9y ago
        So I got a different vibe. I can't speak to how good all that stuff is in Linux vs Solaris (or BSD) so it's just a vibe. The feeling I got was it was someone who was extracting the best value out of the system he was using. Which is refreshing, the constant whining about Linux vs $SOMEONES_FAVORITE_OS gets old. Very pleasant to have a "just the facts" pile of info. And I learned a few tools that I didn't know about.
        filomeno 9y ago
        Of course it is valuable, and being myself a Linux user, I'm glad somebody did the effort to get all those tools available for us. However, I feel sad seeing the best tech from Sun going down the drain despite being open source.
        luckydude 9y ago
        I've been here before. I fought like crazy to prevent SunOS 4.x from being tossed on the trash pile. If you think Solaris going away sucks, it sucked harder (for me) to see SunOS go away. It was a far more pleasant environment than Solaris ever was. As an example of that, because it's fair to go "what??!", I installed some open source solaris in the last few years to play with it. The default install was miserable, the tools were crap. I was having lunch with Bryan Cantrill and some of his team and I mentioned my negative reaction. They laughed and said you have to install the GNU tools and put that in your path first. Say what? SunOS 4.x came with good tools. In /usr/bin. The default install was useful. Solaris was "standard" in that it came with all the System V stuff in /usr/bin. And Shannon and crew "protected" those standard tools and refused to evolve them to make them useful. They started sucky and stayed sucky all in the name of being "standard". Except nobody else used those tools. There was no other System V based Unix that had any significant volume. *BSD certainly didn't move to System V, Linux wasn't System V, the only System V Unix with any volume was Solaris. So they were standard for no good reason and all these years later Solaris still has crappy stuff in /usr/bin, you want /opt/GNU/bin or something. Sorry for the rant, and it's off topic. Well maybe it's off topic, maybe not. I sort of wonder if Sun had shipped the GNU stuff and installed it by default as the real tools, would it have made any difference? Probably not but boy, do the default tools make a bad first impression.
- brendangregg 9y ago
  we have an opening on my team right now, for a Senior Performance Engineer (https://jobs.netflix.com/jobs/865018 https://jobs.netflix.com/jobs/865018). My team is kinda small, so we can't hire everyone (much as that would be great), there's a lot of hiring elsewhere in the company, and I regularly work with different teams.
  - luckydude 9y ago
    Man, I'd be interested but all the web stuff is not my wheelhouse. Maybe the better path is I tinker with lmbench for you just for fun. I'm not really looking for work so much as some interactions with sharp people. Since I retired I've been doing some for hire tractor/excavator work (I live above you in the Santa Cruz mountains, have 3 Kubotas) and a lot of wrenching with my mechanic. Who is a decent mechanic but because he's in so much pain he frequently self medicates which makes him not a member of the sharp people team. So I'm missing the conversations that engineers have. Normal people are fine and all but not as fun as poking at a hard problem with someone smarter than me. I kinda think the job that might be good for me is helping out a VC firm that funds sort of system stuff, like cloud stuff, I/O stuff, etc. I'm a dinosaur, I like C, I like kernels, I like thinking about I/O and how to scale it. Systems stuff is where I like to be and there don't seem to be too many places that want that anymore. Or I'm just not aware of them. Brendan if you ever want to go to lunch and yap about lmbench or tell me what you are working on, hey, lunch is on me. We can go to crappy chinese (I used to have offices in the water tower plaza and that's what we called the chinese place that's just up the street from you. It's the best chinese we could find but it's nowhere near as good as San Francisco chinese food, hence the name) or whereever is good, I'd love to find a new good place for food down there.
kev009 9y ago
If you're open to other opportunities and interested in TCP, Ethernet drivers, file system work, jails stuff on FreeBSD for one of the biggest CDNs you can contact me at kbowling at llnw dot com. I am the hiring manager.
nullnilvoid 9y ago
Brendan is one of the few guys who try to squeeze the last drop of performance from kernels. He is brilliant and has done some great work.
deleted 9y ago
[deleted]
setheron 9y ago
I share the thought. Happiness from work comes from the work you are doing and whom you are working with. If offered the chance to work with someone like Brendan I would jump at the opportunity.
- Annatar 9y ago
  My sentiment exactly. I'd give almost anything for the opportunity to work on SmartOS with Bryan Cantrill.
  - luckydude 9y ago
    If you have some skills that he would value, he's a friend of mine, we aren't like best buds or anything but we share the same passion for operating systems so we get together once in a while. I certainly know him well enough to introduce you. I'd want to have some idea that it would be a good fit, so if you are really interested please contact me via email. Look at my profile, I think I stuck my email there, if not you can find it easily enough.
duncan_bayne 9y ago
I really enjoyed using a Sun box around 17 years ago for C dev - my first professional programming job. Thanks for your work :)
- luckydude 9y ago
  So was that some random Sun machine or are you one of the few people who had what was code named sunbox? It was a clustered NFS server and it was my baby. Pretty sure you just mean a random sun machine, there weren't a lot of sunboxen sold.
  - duncan_bayne 9y ago
    Sadly, the former. It was a pizza box shape, and had (compared to the beige box systems in use in the company) a gloriously large and clear CRT. Would compile our code base as fast as one of the aforementioned beige boxes with twice the clock speed.
weavie 9y ago
His books are awesome too. I keep them by my side pretty much constantly.
brendangregg 9y ago
Kind words, thanks! ... so I was using lmbench last Friday to profile memory latency (lat_mem_rd), is that one of yours?
- luckydude 9y ago
  Ah, lat_mem_rd, so awesomely named :) Yeah, that's mine and I think I was the first to do it that way though I saw the idea described in Hennessy & Patterson and they didn't give me credit so they must have dreamed it up too. It's clever but pretty obvious once you think about it. Very cool that you are still using it. lmbench has aged pretty well, mhz.c still works. BTW, lat_mem_rd has changed processors all over the place. They all have code to detect sequential strides and prefetch. I think Carl and I switched it to go backwards to try and defeat their prefetch. Dunno if they prefetch that way as well. The prefetch is sort of cool and sort of annoying because it hides how the hardware would perform if the access was random. Without the prefetch you can determine cache line size, L1,L2,L3 size and latency, and TLB size, and main mem latency. With the prefetch you think you are getting all that stuff but it looks different. If you have ideas on how to defeat the prefetch and still get the info I'd love to hear them. Another funny thing about lmbench: I was recently approached by a company that's got it working on phones. They want to feed back their stuff and have me do another release that makes it easy to run it on phones. I might do that, still waiting their dump of code. Kind of a neat idea. I still tinker with lmbench, if you have something you want measured and it fits with the idea of measuring bandwidth and latency of everything (and fit's with what Linus and I wanted out of lmbench, if you tune for those metrics you are making the hardware/OS better, so no silly show off your whatever, has to be generically useful), let me know what it is and I'll see if I can code up a benchmark. Making new measurements using the lmbench framework is pretty easy for me, I know that code. BTW, I have an internal version of lmbench that adds stdio support to lmdd (if you haven't played with lmdd, man you've missed out. It can simulate many I/O benchmarks and give you results really quickly). So why would I want stdio support? Because BitKeeper has an enhanced stdio library that can stack filters on a FILE*. And it has some interesting filters, like gzip, lz4. And some much more interesting filters that do CRCs on blocks and an XOR block at the end. And another one that is so complicated I would want to describe it in person. You can combine lz4 with the CRC stuff. Once I had stdio support, I linked with BK's stdio instead and added options to push all those filters and I could see what the overhead was. I did all this when Wayne Scott was putting that crud in BK so I could make sure we were not slowing BK down (we weren't, I think all that crud runs around 1 GB/sec which was fast enough for me). Oh and I also added support to do I/O backwards because we did that in BK as well so we had an append only file format for the ChangeSet file. Lots of funs stuff in there, I need to package it up and release it. If that I/O stuff is of any interest to you, it's open source under the Apache v2 license and if the license bugs you tell me what one you want and I'll rerelease it under that one. We picked that license because we thought it was the one that was easiest for everyone to use, if we're wrong we'll change it. Nice writeup on the migration stuff, sounds like you are having fun. Getting paid to have fun is sweet, that's what it was like for me at Sun in the SunOS 4.x days. I got there and spent the first 3 years telling people "man, I love this job so much. I'd work here for free if I had the money." Someone took pity on me and said "Do you never want a raise? Because that's how you never get a raise" :) I shut up and started getting raises. Enjoy that job. Not every job is that fun, you are at a special part of your career, enjoy the heck out of it.

jamesmishra 9y ago

If you are indeed interested in coming out of retirement, Brendan Gregg does performance engineering at Netflix. It might be worth reaching out. http://www.brendangregg.com/blog/2017-05-16/working-at-netflix-2017.html http://www.brendangregg.com/blog/2017-05-16/working-at-netfl...

luckydude 9y ago
I'm kinda old and burned out but thanks for the link. I'll reach out. It could be a boatload of fun.
- rurban 9y ago
  It's even much more fun than described in this article, because Netflix mostly works with FreeBSD not Linux. So no NIH syndrom, proper dtrace (no eBFS hacks), proper ZFS, proper tooling, easy kernel maintenance.
  - gribbly 9y ago
    As I've seen it described here before (by Brendan Gregg ?) Netflix uses FreeBSD for the CDN servers (streaming the video), and Linux for everything else, browsing Netflix, encoding, etc.
    - filomeno 9y ago
      That's true, although he was right about the Linux counterparts. Reading this migration guide, at least to me, it seems that all Linux replacements for things like dtrace, ZFS, SMF, zones... are just subpar.
      - luckydude 9y ago
        So I got a different vibe. I can't speak to how good all that stuff is in Linux vs Solaris (or BSD) so it's just a vibe. The feeling I got was it was someone who was extracting the best value out of the system he was using. Which is refreshing, the constant whining about Linux vs $SOMEONES_FAVORITE_OS gets old. Very pleasant to have a "just the facts" pile of info. And I learned a few tools that I didn't know about.
        filomeno 9y ago
        Of course it is valuable, and being myself a Linux user, I'm glad somebody did the effort to get all those tools available for us. However, I feel sad seeing the best tech from Sun going down the drain despite being open source.
        luckydude 9y ago
        I've been here before. I fought like crazy to prevent SunOS 4.x from being tossed on the trash pile. If you think Solaris going away sucks, it sucked harder (for me) to see SunOS go away. It was a far more pleasant environment than Solaris ever was. As an example of that, because it's fair to go "what??!", I installed some open source solaris in the last few years to play with it. The default install was miserable, the tools were crap. I was having lunch with Bryan Cantrill and some of his team and I mentioned my negative reaction. They laughed and said you have to install the GNU tools and put that in your path first. Say what? SunOS 4.x came with good tools. In /usr/bin. The default install was useful. Solaris was "standard" in that it came with all the System V stuff in /usr/bin. And Shannon and crew "protected" those standard tools and refused to evolve them to make them useful. They started sucky and stayed sucky all in the name of being "standard". Except nobody else used those tools. There was no other System V based Unix that had any significant volume. *BSD certainly didn't move to System V, Linux wasn't System V, the only System V Unix with any volume was Solaris. So they were standard for no good reason and all these years later Solaris still has crappy stuff in /usr/bin, you want /opt/GNU/bin or something. Sorry for the rant, and it's off topic. Well maybe it's off topic, maybe not. I sort of wonder if Sun had shipped the GNU stuff and installed it by default as the real tools, would it have made any difference? Probably not but boy, do the default tools make a bad first impression.
brendangregg 9y ago
we have an opening on my team right now, for a Senior Performance Engineer (https://jobs.netflix.com/jobs/865018 https://jobs.netflix.com/jobs/865018). My team is kinda small, so we can't hire everyone (much as that would be great), there's a lot of hiring elsewhere in the company, and I regularly work with different teams.
- luckydude 9y ago
  Man, I'd be interested but all the web stuff is not my wheelhouse. Maybe the better path is I tinker with lmbench for you just for fun. I'm not really looking for work so much as some interactions with sharp people. Since I retired I've been doing some for hire tractor/excavator work (I live above you in the Santa Cruz mountains, have 3 Kubotas) and a lot of wrenching with my mechanic. Who is a decent mechanic but because he's in so much pain he frequently self medicates which makes him not a member of the sharp people team. So I'm missing the conversations that engineers have. Normal people are fine and all but not as fun as poking at a hard problem with someone smarter than me. I kinda think the job that might be good for me is helping out a VC firm that funds sort of system stuff, like cloud stuff, I/O stuff, etc. I'm a dinosaur, I like C, I like kernels, I like thinking about I/O and how to scale it. Systems stuff is where I like to be and there don't seem to be too many places that want that anymore. Or I'm just not aware of them. Brendan if you ever want to go to lunch and yap about lmbench or tell me what you are working on, hey, lunch is on me. We can go to crappy chinese (I used to have offices in the water tower plaza and that's what we called the chinese place that's just up the street from you. It's the best chinese we could find but it's nowhere near as good as San Francisco chinese food, hence the name) or whereever is good, I'd love to find a new good place for food down there.

kev009 9y ago

If you're open to other opportunities and interested in TCP, Ethernet drivers, file system work, jails stuff on FreeBSD for one of the biggest CDNs you can contact me at kbowling at llnw dot com. I am the hiring manager.

nullnilvoid 9y ago

Brendan is one of the few guys who try to squeeze the last drop of performance from kernels. He is brilliant and has done some great work.

deleted 9y ago

[deleted]

setheron 9y ago

I share the thought. Happiness from work comes from the work you are doing and whom you are working with. If offered the chance to work with someone like Brendan I would jump at the opportunity.

Annatar 9y ago
My sentiment exactly. I'd give almost anything for the opportunity to work on SmartOS with Bryan Cantrill.
- luckydude 9y ago
  If you have some skills that he would value, he's a friend of mine, we aren't like best buds or anything but we share the same passion for operating systems so we get together once in a while. I certainly know him well enough to introduce you. I'd want to have some idea that it would be a good fit, so if you are really interested please contact me via email. Look at my profile, I think I stuck my email there, if not you can find it easily enough.

duncan_bayne 9y ago

I really enjoyed using a Sun box around 17 years ago for C dev - my first professional programming job. Thanks for your work :)

luckydude 9y ago
So was that some random Sun machine or are you one of the few people who had what was code named sunbox? It was a clustered NFS server and it was my baby. Pretty sure you just mean a random sun machine, there weren't a lot of sunboxen sold.
- duncan_bayne 9y ago
  Sadly, the former. It was a pizza box shape, and had (compared to the beige box systems in use in the company) a gloriously large and clear CRT. Would compile our code base as fast as one of the aforementioned beige boxes with twice the clock speed.

weavie 9y ago

His books are awesome too. I keep them by my side pretty much constantly.

brendangregg 9y ago

Kind words, thanks! ... so I was using lmbench last Friday to profile memory latency (lat_mem_rd), is that one of yours?

luckydude 9y ago
Ah, lat_mem_rd, so awesomely named :) Yeah, that's mine and I think I was the first to do it that way though I saw the idea described in Hennessy & Patterson and they didn't give me credit so they must have dreamed it up too. It's clever but pretty obvious once you think about it. Very cool that you are still using it. lmbench has aged pretty well, mhz.c still works. BTW, lat_mem_rd has changed processors all over the place. They all have code to detect sequential strides and prefetch. I think Carl and I switched it to go backwards to try and defeat their prefetch. Dunno if they prefetch that way as well. The prefetch is sort of cool and sort of annoying because it hides how the hardware would perform if the access was random. Without the prefetch you can determine cache line size, L1,L2,L3 size and latency, and TLB size, and main mem latency. With the prefetch you think you are getting all that stuff but it looks different. If you have ideas on how to defeat the prefetch and still get the info I'd love to hear them. Another funny thing about lmbench: I was recently approached by a company that's got it working on phones. They want to feed back their stuff and have me do another release that makes it easy to run it on phones. I might do that, still waiting their dump of code. Kind of a neat idea. I still tinker with lmbench, if you have something you want measured and it fits with the idea of measuring bandwidth and latency of everything (and fit's with what Linus and I wanted out of lmbench, if you tune for those metrics you are making the hardware/OS better, so no silly show off your whatever, has to be generically useful), let me know what it is and I'll see if I can code up a benchmark. Making new measurements using the lmbench framework is pretty easy for me, I know that code. BTW, I have an internal version of lmbench that adds stdio support to lmdd (if you haven't played with lmdd, man you've missed out. It can simulate many I/O benchmarks and give you results really quickly). So why would I want stdio support? Because BitKeeper has an enhanced stdio library that can stack filters on a FILE*. And it has some interesting filters, like gzip, lz4. And some much more interesting filters that do CRCs on blocks and an XOR block at the end. And another one that is so complicated I would want to describe it in person. You can combine lz4 with the CRC stuff. Once I had stdio support, I linked with BK's stdio instead and added options to push all those filters and I could see what the overhead was. I did all this when Wayne Scott was putting that crud in BK so I could make sure we were not slowing BK down (we weren't, I think all that crud runs around 1 GB/sec which was fast enough for me). Oh and I also added support to do I/O backwards because we did that in BK as well so we had an append only file format for the ChangeSet file. Lots of funs stuff in there, I need to package it up and release it. If that I/O stuff is of any interest to you, it's open source under the Apache v2 license and if the license bugs you tell me what one you want and I'll rerelease it under that one. We picked that license because we thought it was the one that was easiest for everyone to use, if we're wrong we'll change it. Nice writeup on the migration stuff, sounds like you are having fun. Getting paid to have fun is sweet, that's what it was like for me at Sun in the SunOS 4.x days. I got there and spent the first 3 years telling people "man, I love this job so much. I'd work here for free if I had the money." Someone took pity on me and said "Do you never want a raise? Because that's how you never get a raise" :) I shut up and started getting raises. Enjoy that job. Not every job is that fun, you are at a special part of your career, enjoy the heck out of it.