6 ms·

Building a Graphics Card For the Internet: A tour of the newest Imgix datacenter

82 points by zacman85 12y ago

KaiserPro 12y ago
What do you guys use the mac minis for?(I mean whats the compelling function, they must do something amazing to justify the cost and lack of features) I also notice that you don't appear to have any physical isolation between the hot and cold aisles, is that just for the photos?
- nulltype 12y ago
  "Apple‘s operating system offers numerous advantages over other platforms when it comes to image processing, and it is also favored by many designers for the same reasons. imgix has written tools which allow us to leverage these strengths with our service, using OS X for image operations where it provides superior quality and performance. " Looks like they use them for OS X.
- gonzo 12y ago
  From the article: "OS X Apple‘s operating system offers numerous advantages over other platforms when it comes to image processing, and it is also favored by many designers for the same reasons. imgix has written tools which allow us to leverage these strengths with our service, using OS X for image operations where it provides superior quality and performance. Deploying this type of hardware in a datacenter environment is a bit of a challenge, and one that we worked on extensively with our systems integrator. Our custom cabinet design allows us to operate these servers the same as any other in our production environment, and ensures they are a reliable component of the imgix infrastructure."
  - pan69 12y ago
    I would say that's a vague description at best. "Numerous advantages" and "favored by many designers".
  - izacus 12y ago
    You just copy-pased marketing spew. I think the author of original post wanted to know more about exact technologies/image processing at which OS X excels so much that it's worth deploying Mac Minis in such numbers. I did enough work professionally with image/video/CV processing that I have a rough idea, but I'd still like to hear some non-marketing speak about why the company decided to invest in the huge hassle that is having rack-mounted Apple gear which misses all redundancy, connectivity and management equipment of standard rackmount servers.
    - skuhn 12y ago
      We debated whether or not this Exposure article was the right place to dive into specifics about our infrastructure and the choices we made (including our decision to use OS X), but ultimately decided that it wasn't a good fit. Exposure is more about showing than telling, with some light descriptive text for context. imgix's founder and CEO (zacman85) replied in another comment with a few more details about why we use OS X for some things, but there's a lot more detail we could go into. We're thinking about exploring these kind of topics in more depth in future articles, and I hope you'll check them out.
- microcolonel 12y ago
  They say “Apple‘s operating system offers numerous advantages over other platforms when it comes to image processing, and it is also favored by many designers for the same reasons.” Although I have no idea what those advantages could be.... AppleScript+Photoshop? Who knows... On any platform you have essentially the same choice in performance and flexibility of imaging libraries, so it's anyone's guess for now what they think they need them for. It's especially funny because of just how much they talk up fibre connectivity, and how the copper is just used as a fallback management fabric.... but these Mac Minis are obviously connected with copper.
  - defen 12y ago
    If you use CoreImage you get a ton of GPU-optimized filters for free, plus you can compose them at very little cost. For example if you wanted to crop, scale, rotate, and guassian blur an image, you can do that with one pass through the GPU instead 4.
    - Keyframe 12y ago
      I think ImageMagick also supports GPU/OpenCL if you compile it in.
      - shiven 12y ago
        As kogir mentioned elsewhere in the thread, ImageMagick is not designed to be used as a long running process, it will destroy RAM and likely lead to stability issues. I think with OSX they used the right tool for the right job, because, as always, there is never a perfect solution to any problem on the trilateral axis (good/fast/cheap).
        Keyframe 12y ago
        Destroy RAM? What?
        shiven 12y ago
- theandrewbailey 12y ago
  My guess is that they have some custom Photoshop remote tools.
- zacman85 12y ago
  Having OS X in our stack allows us to tackle a lot of use cases that come out of the prototypical design process. Format support, color profiles, color space conversion, typography, etc. are all mastered in OS X, but lacking in other operating systems. Apple has had the best imaging scientists in the world working for the last 30 years on getting these features right. We want to be able to leverage that expertise, whenever it makes sense, to produce the highest quality image. In this case, the consequence of that decision means racking Macs.
  - angersock 12y ago
    So, what does that all have that isn't covered by, say, ImageMagick, OpenCV, or whatever?
    - kogir 12y ago
      ImageMagick clearly grew up in environments with short lived processes - the command line, php, etc. Run it in a long lived process at your peril, and watch all your memory leak away. I also don't think it makes very good use of available GPUs at all.
  - RowanH 12y ago
    So hazarding a guess here - you've written your own services on top of CoreImage, Core Text etc? https://developer.apple.com/library/mac/documentation/macosx/conceptual/osx_technology_overview/MediaLayer/MediaLayer.html#//apple_ref/doc/uid/TP40001067-CH273-SW30 https://developer.apple.com/library/mac/documentation/macosx...
  - Sephr 12y ago
    If it's just OS X you need, and not the overpriced Apple hardware, why not just buy a bunch of OS X licenses and install it on your normal servers (a la Hackintosh)? It would cost much much less and you would be able to have completely uniform server hardware with faster networking.
    - voltagex_ 12y ago
- deleted 12y ago
  [deleted]
- skuhn 12y ago
  We're developing a cold aisle containment solution, but it wasn't ready to be photographed. It's the one major component that I decided to wait on deploying, since there isn't an urgent need to contain a space that is (at first) mostly empty. We're aiming to have it fully contained by the time 1/4 of the cabinet positions are occupied.
radicaledward 12y ago
You seem to have some missing content on your homepage: https://imgix.exposure.co/ https://imgix.exposure.co/ Two white boxes on the right. I'm on Chrome 36 on OS X if it helps? It looks like the content is just plain missing. I disabled adblock but it didn't help. The ascii art logo in your web site's source code is amusing! I had no idea what the "Enjoyed" button at the bottom of the page was for. When I clicked it, the counter incremented. Is this some new stand alone like button that people are doing?
- taylorhughes 12y ago
  That's imgix's profile on exposure.co, not their website. Exposure is a separate product/company. imgix website: http://www.imgix.com/ http://www.imgix.com/
brokentone 12y ago
This is one of the most beautiful photostory layouts I've ever seen. But I couldn't focus on the text at all.
- jayd16 12y ago
  When you click a photo it loads the fragment url and gets added to the back history but when you hit back it doesn't close the photo takeover. That's pretty annoying but otherwise it works well.
coops 12y ago
What's it like to administer a cluster of Mac Minis? Do any of the popular open-source orchestration tools work?
- nacs 12y ago
  It's a BSD-based OS so I would expect most OSS orchestration tools to work (via SSH and such).
- skuhn 12y ago
  We mostly operate the OS X and Linux hosts using the same or similar tools; Ansible for automation, ssh / rsync / git for getting files onto hosts, and an automated network installer that fills the same niche as the one we use with Linux. All of the core infrastructure is shared between OS X and Linux systems -- same DNS caches, NTP servers, etc. -- so there's no additional work there. There are some things that are a little tougher to do on OS X. The userland utilities are different (and anything GPL is trapped on GPLv2 versions for all eternity, such as bash and rsync), and sometimes we have to work around that or build local packages. Ansible doesn't really know about things like launchd processes or cron jobs, and OS X has nothing analogous to useradd, so we had to write some modules and implement branching in common playbooks to handle tasks in different ways on Linux and OS X. Ultimately they run Unix, and once our environment is deployed onto a server you don't really notice the operational differences on a day to day basis.
lelandbatey 12y ago
Can I cross post some of the images from the blog post over to a community based on images of excellent cable management[0]? Also, I have to say I really like the API(?) for manipulating images via url parameters, and even more, I really like that you used it on this blog post. When I went to pull the images from your site, I was able to get the full resolution versions of those images just by making some minor edits to the urls. This made making an album really easy[1]. [0] - http://www.reddit.com/r/cableporn http://www.reddit.com/r/cableporn [1] - http://imgur.com/a/OFVqX http://imgur.com/a/OFVqX
- skuhn 12y ago
  Sure, as long as it references imgix. It would be nice if it linked back to the Exposure post too.
hughw 12y ago
Apropos. I'm reading this article, killing time waiting for my EC2 high compute spot instance to finish converting 33,000 TIFF images to PNG and resizing them.
billconan 12y ago
Apple‘s operating system offers numerous advantages over other platforms when it comes to image processing. what? like only support opengl 3.0?
lobster_johnson 12y ago
How does CoreImage etc. compare to Intel's Image Processing library (IPP)?
pyvpx 12y ago
who did your photography and how difficult was it to get permission to shoot inside the datacenter you colocate at? the shots are excellently done.
- skuhn 12y ago
  Our datacenter provider was great about it, we got approval in a matter of hours. We never photograph anything that doesn't belong to imgix or provide details that would identify the building, so that makes the approval process much simpler.   The photography was all done by imgix's lead designer Miguel Cardona, and I helped out a bit by finding things for him to shoot. I think he did a great job -- it can be a real challenge to get visually interesting shots in a datacenter environment.

82 points by zacman85 12y ago

What do you guys use the mac minis for?(I mean whats the compelling function, they must do something amazing to justify the cost and lack of features) I also notice that you don't appear to have any physical isolation between the hot and cold aisles, is that just for the photos?

nulltype 12y ago
"Apple‘s operating system offers numerous advantages over other platforms when it comes to image processing, and it is also favored by many designers for the same reasons. imgix has written tools which allow us to leverage these strengths with our service, using OS X for image operations where it provides superior quality and performance. " Looks like they use them for OS X.
gonzo 12y ago
From the article: "OS X Apple‘s operating system offers numerous advantages over other platforms when it comes to image processing, and it is also favored by many designers for the same reasons. imgix has written tools which allow us to leverage these strengths with our service, using OS X for image operations where it provides superior quality and performance. Deploying this type of hardware in a datacenter environment is a bit of a challenge, and one that we worked on extensively with our systems integrator. Our custom cabinet design allows us to operate these servers the same as any other in our production environment, and ensures they are a reliable component of the imgix infrastructure."
- pan69 12y ago
  I would say that's a vague description at best. "Numerous advantages" and "favored by many designers".
- izacus 12y ago
  You just copy-pased marketing spew. I think the author of original post wanted to know more about exact technologies/image processing at which OS X excels so much that it's worth deploying Mac Minis in such numbers. I did enough work professionally with image/video/CV processing that I have a rough idea, but I'd still like to hear some non-marketing speak about why the company decided to invest in the huge hassle that is having rack-mounted Apple gear which misses all redundancy, connectivity and management equipment of standard rackmount servers.
  - skuhn 12y ago
    We debated whether or not this Exposure article was the right place to dive into specifics about our infrastructure and the choices we made (including our decision to use OS X), but ultimately decided that it wasn't a good fit. Exposure is more about showing than telling, with some light descriptive text for context. imgix's founder and CEO (zacman85) replied in another comment with a few more details about why we use OS X for some things, but there's a lot more detail we could go into. We're thinking about exploring these kind of topics in more depth in future articles, and I hope you'll check them out.
microcolonel 12y ago
They say “Apple‘s operating system offers numerous advantages over other platforms when it comes to image processing, and it is also favored by many designers for the same reasons.” Although I have no idea what those advantages could be.... AppleScript+Photoshop? Who knows... On any platform you have essentially the same choice in performance and flexibility of imaging libraries, so it's anyone's guess for now what they think they need them for. It's especially funny because of just how much they talk up fibre connectivity, and how the copper is just used as a fallback management fabric.... but these Mac Minis are obviously connected with copper.
- defen 12y ago
  If you use CoreImage you get a ton of GPU-optimized filters for free, plus you can compose them at very little cost. For example if you wanted to crop, scale, rotate, and guassian blur an image, you can do that with one pass through the GPU instead 4.
  - Keyframe 12y ago
    I think ImageMagick also supports GPU/OpenCL if you compile it in.
    - shiven 12y ago
      As kogir mentioned elsewhere in the thread, ImageMagick is not designed to be used as a long running process, it will destroy RAM and likely lead to stability issues. I think with OSX they used the right tool for the right job, because, as always, there is never a perfect solution to any problem on the trilateral axis (good/fast/cheap).
      - Keyframe 12y ago
        Destroy RAM? What?
        shiven 12y ago
theandrewbailey 12y ago
My guess is that they have some custom Photoshop remote tools.
zacman85 12y ago
Having OS X in our stack allows us to tackle a lot of use cases that come out of the prototypical design process. Format support, color profiles, color space conversion, typography, etc. are all mastered in OS X, but lacking in other operating systems. Apple has had the best imaging scientists in the world working for the last 30 years on getting these features right. We want to be able to leverage that expertise, whenever it makes sense, to produce the highest quality image. In this case, the consequence of that decision means racking Macs.
- angersock 12y ago
  So, what does that all have that isn't covered by, say, ImageMagick, OpenCV, or whatever?
  - kogir 12y ago
    ImageMagick clearly grew up in environments with short lived processes - the command line, php, etc. Run it in a long lived process at your peril, and watch all your memory leak away. I also don't think it makes very good use of available GPUs at all.
- RowanH 12y ago
  So hazarding a guess here - you've written your own services on top of CoreImage, Core Text etc? https://developer.apple.com/library/mac/documentation/macosx/conceptual/osx_technology_overview/MediaLayer/MediaLayer.html#//apple_ref/doc/uid/TP40001067-CH273-SW30 https://developer.apple.com/library/mac/documentation/macosx...
- Sephr 12y ago
  If it's just OS X you need, and not the overpriced Apple hardware, why not just buy a bunch of OS X licenses and install it on your normal servers (a la Hackintosh)? It would cost much much less and you would be able to have completely uniform server hardware with faster networking.
  - voltagex_ 12y ago
deleted 12y ago
[deleted]
skuhn 12y ago
We're developing a cold aisle containment solution, but it wasn't ready to be photographed. It's the one major component that I decided to wait on deploying, since there isn't an urgent need to contain a space that is (at first) mostly empty. We're aiming to have it fully contained by the time 1/4 of the cabinet positions are occupied.

radicaledward 12y ago

You seem to have some missing content on your homepage: https://imgix.exposure.co/ https://imgix.exposure.co/ Two white boxes on the right. I'm on Chrome 36 on OS X if it helps? It looks like the content is just plain missing. I disabled adblock but it didn't help. The ascii art logo in your web site's source code is amusing! I had no idea what the "Enjoyed" button at the bottom of the page was for. When I clicked it, the counter incremented. Is this some new stand alone like button that people are doing?

taylorhughes 12y ago
That's imgix's profile on exposure.co, not their website. Exposure is a separate product/company. imgix website: http://www.imgix.com/ http://www.imgix.com/

brokentone 12y ago

This is one of the most beautiful photostory layouts I've ever seen. But I couldn't focus on the text at all.

jayd16 12y ago
When you click a photo it loads the fragment url and gets added to the back history but when you hit back it doesn't close the photo takeover. That's pretty annoying but otherwise it works well.

coops 12y ago

What's it like to administer a cluster of Mac Minis? Do any of the popular open-source orchestration tools work?

nacs 12y ago
It's a BSD-based OS so I would expect most OSS orchestration tools to work (via SSH and such).
skuhn 12y ago
We mostly operate the OS X and Linux hosts using the same or similar tools; Ansible for automation, ssh / rsync / git for getting files onto hosts, and an automated network installer that fills the same niche as the one we use with Linux. All of the core infrastructure is shared between OS X and Linux systems -- same DNS caches, NTP servers, etc. -- so there's no additional work there. There are some things that are a little tougher to do on OS X. The userland utilities are different (and anything GPL is trapped on GPLv2 versions for all eternity, such as bash and rsync), and sometimes we have to work around that or build local packages. Ansible doesn't really know about things like launchd processes or cron jobs, and OS X has nothing analogous to useradd, so we had to write some modules and implement branching in common playbooks to handle tasks in different ways on Linux and OS X. Ultimately they run Unix, and once our environment is deployed onto a server you don't really notice the operational differences on a day to day basis.

lelandbatey 12y ago

Can I cross post some of the images from the blog post over to a community based on images of excellent cable management[0]? Also, I have to say I really like the API(?) for manipulating images via url parameters, and even more, I really like that you used it on this blog post. When I went to pull the images from your site, I was able to get the full resolution versions of those images just by making some minor edits to the urls. This made making an album really easy[1]. [0] - http://www.reddit.com/r/cableporn http://www.reddit.com/r/cableporn [1] - http://imgur.com/a/OFVqX http://imgur.com/a/OFVqX

skuhn 12y ago
Sure, as long as it references imgix. It would be nice if it linked back to the Exposure post too.

hughw 12y ago

Apropos. I'm reading this article, killing time waiting for my EC2 high compute spot instance to finish converting 33,000 TIFF images to PNG and resizing them.

billconan 12y ago

Apple‘s operating system offers numerous advantages over other platforms when it comes to image processing. what? like only support opengl 3.0?

lobster_johnson 12y ago

How does CoreImage etc. compare to Intel's Image Processing library (IPP)?

pyvpx 12y ago

who did your photography and how difficult was it to get permission to shoot inside the datacenter you colocate at? the shots are excellently done.

skuhn 12y ago
Our datacenter provider was great about it, we got approval in a matter of hours. We never photograph anything that doesn't belong to imgix or provide details that would identify the building, so that makes the approval process much simpler.   The photography was all done by imgix's lead designer Miguel Cardona, and I helped out a bit by finding things for him to shoot. I think he did a great job -- it can be a real challenge to get visually interesting shots in a datacenter environment.