6 ms·

FreeBSD on EC2 FAQ

49 points by l0stman 16y ago

jblphx 16y ago
I've been a FreeBSD fan since 1997. It is nice to see FreeBSD running on EC2. However, for die hard performance junkies, I have built something faster and more efficient. For people that love all the BSD variants out there, my startup - tegataiphoenix.com - offers FreeBSD, NetBSD, and OpenBSD virtual machines (in addition to all the other usual suspects). The performance we are getting is around 500%-600% faster than Xen based BSD virtual machines. Mention HN, and get a discount. :)
- cperciva 16y ago
  Hmm. User for 30 days; karma 1; one submission, promoting your startup; one comment, promoting your startup. Welcome to Hacker News! I hope you'll stick around for a while and contribute.
  - jblphx 16y ago
    I've been a fan of your work for ages Collin, but your attitude needs adjustment.
    - jcr 16y ago
      Please don't feel too offended. It's totally normal for people to check out profiles when someone posts something interesting. Your know-how on BSD virtualization IS interesting to a lot of folks.
- cperciva 16y ago
  The performance we are getting is around 500%-600% faster than Xen based BSD virtual machines I don't know where you're getting your numbers from, but Xen doesn't have a 5x performance penalty. I'd believe you if you said that QEMU or Bochs had a 5x performance penalty, but Xen is more along the lines of 5%. (Of course, if you run an instance and give it a small fraction of a CPU, you get what you pay for.)
  - jblphx 16y ago
    I'll go ahead and publish the benches shortly.
Andys 16y ago
I see virtualisation as a promising way forward for FreeBSD. Hardware support has always lagged Linux (and still does). The limited resources of BSD hackers could be more appropriately concentrated on needed areas instead of constantly playing hardware catch-up.
- cperciva 16y ago
  Hardware support has always lagged Linux (and still does). Not everywhere. Last I checked we do considerably better for 10GbE, for instance. We certainly consistently lag behind Linux for desktop hardware support, but I think we hold our own pretty well in the server space. As a BSD hacker who hates anything hardware-related, however, I do love the idea of outsourcing such irritants, though.
  - Andys 16y ago
    I held this view ("hold own in server hardware") until I actually tried to run FreeBSD on modern server hardware. The Intel 10G NICs worked, but slower in benchmarks than Linux and Opensolaris. SCSI/SAS support has been rotting slowly, with the now industry-standard LSI cards I tested, working, but without interrupt coalesce and thus using more CPU and reaching only 60% of the speed of the same card in Linux or Opensolaris. Infiniband support is nearly all happening in Linux. Then there was the problems with the new Core i7 CPUs and Turbo Mode. NUMA support has been missing until 9.0(unstable), and even now its only for the allocator and not the scheduler. The final straw was when I had USB issues (modern server motherboards have mostly dropped PS2 ports) which meant keyboard wasn't working at all on a server. ZFS is less stable and slower than in OpenSolaris, and for all these years FreeBSD has lacked a journalling filesystem. Which I consider basic stuff, after all, Linux has had XFS for 10 years now. Linux has several virtualisation options, and while VirtualBox has been ported to BSD, it isn't really suited for server use. FreeBSD Jails have stagnated for years and still cannot handle running multiple copies of PostgreSQL because of SYSV support. I believe FreeBSD is the best and most enjoyable OS for hacking on, but this was 2 years ago, and things have further stagnated since then, so like many others, I have been forced to give up and move on.
ez77 16y ago
I don't know much about virtualization, but could anyone explain how sites such as [1] and [2] are able to virtualize OpenBSD? Do you think they develop the Xen support internally? [1] http://bsdvm.com/ http://bsdvm.com/ [2] http://www.rootbsd.net/blog/2010/openbsd-4-8/ http://www.rootbsd.net/blog/2010/openbsd-4-8/
- cperciva 16y ago
  Short answer: There's two types of virtualization. With full virtualization, you can run unmodified operating systems (in theory, at least). VMWare is an example of a fully virtualized system. Recently Intel and AMD added features to their CPUs which make this much easier. With para virtualization, the guest operating system is aware of the hypervisor and cooperates with it -- for example, rather than modifying page tables directly, it sends a message to the hypervisor saying "please map this page for me". (There's also a hybrid 75%-virtualized model, where the system is fully virtualized except that some devices are exposed via hypervisor calls -- this is seen most commonly as Xen "HVM with PV drivers".)
  - ez77 16y ago
    Great, thanks for the explanation.

49 points by l0stman 16y ago

I've been a FreeBSD fan since 1997. It is nice to see FreeBSD running on EC2. However, for die hard performance junkies, I have built something faster and more efficient. For people that love all the BSD variants out there, my startup - tegataiphoenix.com - offers FreeBSD, NetBSD, and OpenBSD virtual machines (in addition to all the other usual suspects). The performance we are getting is around 500%-600% faster than Xen based BSD virtual machines. Mention HN, and get a discount. :)

cperciva 16y ago
Hmm. User for 30 days; karma 1; one submission, promoting your startup; one comment, promoting your startup. Welcome to Hacker News! I hope you'll stick around for a while and contribute.
- jblphx 16y ago
  I've been a fan of your work for ages Collin, but your attitude needs adjustment.
  - jcr 16y ago
    Please don't feel too offended. It's totally normal for people to check out profiles when someone posts something interesting. Your know-how on BSD virtualization IS interesting to a lot of folks.
cperciva 16y ago
The performance we are getting is around 500%-600% faster than Xen based BSD virtual machines I don't know where you're getting your numbers from, but Xen doesn't have a 5x performance penalty. I'd believe you if you said that QEMU or Bochs had a 5x performance penalty, but Xen is more along the lines of 5%. (Of course, if you run an instance and give it a small fraction of a CPU, you get what you pay for.)
- jblphx 16y ago
  I'll go ahead and publish the benches shortly.

Andys 16y ago

I see virtualisation as a promising way forward for FreeBSD. Hardware support has always lagged Linux (and still does). The limited resources of BSD hackers could be more appropriately concentrated on needed areas instead of constantly playing hardware catch-up.

cperciva 16y ago
Hardware support has always lagged Linux (and still does). Not everywhere. Last I checked we do considerably better for 10GbE, for instance. We certainly consistently lag behind Linux for desktop hardware support, but I think we hold our own pretty well in the server space. As a BSD hacker who hates anything hardware-related, however, I do love the idea of outsourcing such irritants, though.
- Andys 16y ago
  I held this view ("hold own in server hardware") until I actually tried to run FreeBSD on modern server hardware. The Intel 10G NICs worked, but slower in benchmarks than Linux and Opensolaris. SCSI/SAS support has been rotting slowly, with the now industry-standard LSI cards I tested, working, but without interrupt coalesce and thus using more CPU and reaching only 60% of the speed of the same card in Linux or Opensolaris. Infiniband support is nearly all happening in Linux. Then there was the problems with the new Core i7 CPUs and Turbo Mode. NUMA support has been missing until 9.0(unstable), and even now its only for the allocator and not the scheduler. The final straw was when I had USB issues (modern server motherboards have mostly dropped PS2 ports) which meant keyboard wasn't working at all on a server. ZFS is less stable and slower than in OpenSolaris, and for all these years FreeBSD has lacked a journalling filesystem. Which I consider basic stuff, after all, Linux has had XFS for 10 years now. Linux has several virtualisation options, and while VirtualBox has been ported to BSD, it isn't really suited for server use. FreeBSD Jails have stagnated for years and still cannot handle running multiple copies of PostgreSQL because of SYSV support. I believe FreeBSD is the best and most enjoyable OS for hacking on, but this was 2 years ago, and things have further stagnated since then, so like many others, I have been forced to give up and move on.

ez77 16y ago

I don't know much about virtualization, but could anyone explain how sites such as [1] and [2] are able to virtualize OpenBSD? Do you think they develop the Xen support internally? [1] http://bsdvm.com/ http://bsdvm.com/ [2] http://www.rootbsd.net/blog/2010/openbsd-4-8/ http://www.rootbsd.net/blog/2010/openbsd-4-8/

cperciva 16y ago
Short answer: There's two types of virtualization. With full virtualization, you can run unmodified operating systems (in theory, at least). VMWare is an example of a fully virtualized system. Recently Intel and AMD added features to their CPUs which make this much easier. With para virtualization, the guest operating system is aware of the hypervisor and cooperates with it -- for example, rather than modifying page tables directly, it sends a message to the hypervisor saying "please map this page for me". (There's also a hybrid 75%-virtualized model, where the system is fully virtualized except that some devices are exposed via hypervisor calls -- this is seen most commonly as Xen "HVM with PV drivers".)
- ez77 16y ago
  Great, thanks for the explanation.