6 ms·

> With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. I have next to zero experience with server administration, but 384

by habosa 12y ago

> With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. I have next to zero experience with server administration, but 384GB seems like a lot to me. Is that common for production servers for popular web services? Do you need a customized OS to address that much memory? Seems like you'd really need to beef up the cache hierarchy make 0.38TB of RAM fast.

x1798DE 12y ago
It's not a server, but I've seen machines used in physics simulations have something on the order of 256 GB RAM in them. As far as I know they cost maybe $20k-$30k, so not absurdly expensive. I don't think addressing that memory is a problem on any x64 architecture. Windows Server 2012 has a 4TB upper limit on RAM, apparently [1], and as far as I can tell there's no reason for this other than product differentiation between different "levels" of the operating system. 1. http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778%28v=vs.85%29.aspx#physical_memory_limits_windows_server_2012 http://msdn.microsoft.com/en-us/library/windows/desktop/aa36...
- mrb 12y ago
  $4k is all you need to build a whitebox 256GB machine: dual-socket G34 motherboard with 16 DDR3 slots lke the ASUS KGPE-D16 ($500) + two Opteron 6320 (2 * $300) + sixteen 16GB registered ECC DDR3-1600 (16 * $160) + chassis/PSU ($500) = $4160. Doing this on Amazon r3.8xlarge ($2.8/hr) would cost you $8100+ over 4 months. A DIY server can make sense in some cases. Eg. you are a very early-stage startup with almost no funding, and you are good at assembling and troubleshooting computers, and are ok with mere warranty on parts as opposed to full vendor support from HP/Oracle/Dell, go for it. That's how many startups started (see Google and their makeshift half-donated half-loaned boxes in 1998).
  - ZachPruckowski 12y ago
    Don't forget to factor in the costs of running, cooling, and maintaining that much hardware (also bandwidth costs). I'm not saying it's $2k/month, I'm just saying don't compare the physical box and AWS without factoring everything in. At the end of the day, if you don't need the support, you're probably right that a dedicated box makes sense there.
    - adventured 12y ago
      AWS has among the highest bandwidth costs you're going to run into, if we're talking co-location, dedicated, or self-hosting. Their bandwidth costs are the sole reason I won't go anywhere near AWS yet, I consider their prices outrageously high. 10tb = roughly $1,000 per month They're between 5 and 50 times more expensive on bandwidth than the options in dedicated / colo / self hosting.
      - dekhn 12y ago
        You can negotiate with AWS.
        liotier 12y ago
        I don't believe that startups in their garage days have anywhere near the necessary leverage to negotiate anything with Amazon...
        jaequery 12y ago
        Agree, not many people actually realize just how slow and expensive AWS actually is. Kudos to SO for going with their own hardware.
        dekhn 12y ago
        they negotiate, still. I know several startups which went to amazon, said, "hey, your competitor is cheaper, what is your value prop" and suddenly had 50% discounts, etc. It's amazing how much you get simply asking!
claudius 12y ago
Linux (SuSE enterprise something, possibly with a custom kernel, I’m just a user, not admin) runs just fine on 5.2TB machine, though I can easily imagine you’d have problems if you tried it with, say, DOS :) The system obviously behaves a bit different from a standard desktop machines, e.g. different areas of RAM are differently fast, depending on the core on which your current process runs, you need to disable individual CPU lines displayed in top etc., but apart from these, mostly everything seems “normal” to me.
- tekacs 12y ago
  > e.g. different areas of RAM are differently fast, depending on the core on which your current process runs, you need to disable individual CPU lines displayed in top etc. this is NUMA, in case anyone 'new to sysadmin' or to architecture is trying to Google this
Sanddancer 12y ago
384 gigs is pretty common these days. With Windows, you need at least the server edition to address that much, but if you're running a server, you're not gonna be using the desktop versions of the OS anyways. Pretty much everything else, it's just plug and play. The RAM is quite a bit if you're doing a typical cloud deployment, but SO seems to be more in the model of making sure all your eggs are in a really strong basket. Besides, at the end of the day, RAM tends to be a lot cheaper than lag.
- ddorian43 12y ago
  *everything else is plug and pay
- abritishguy 12y ago
  Not true. Windows 8.1 64 bit would be happy with that much.
junto 12y ago
Their SQL Server is Microsoft SQL Server running on (shock horror) Windows. Says in the article: Windows 2012 is used in New York but are upgrading to 2012 R2 (Oregon is already on it).
- Nick-Craver 12y ago
  To be clear, most servers are moving to 2012 R2 (most already have) but the machines our SQL clusters run on, specifically, will not. This is because you have to effectively (or literally) rebuild the Windows cluster from scratch and we just don't get that level of benefit from the 2012 to 2012 R2 upgrade. There are quite a few improvements we care about: native NVMe, better dynamic quorum, better DSC support, better SMB, and such...but not enough to make the upgrade worth it. Nick Craver - Stack Exchange Sysadmin & Developer
  - Just-A-Guest 12y ago
    Do I get this right? You can not upgrade a server cluster to the next version of the operating system without complete new installation??? Could you please write something about these kind of things, this is very interesting!
    - Nick-Craver 12y ago
      I'm not sure a full writeup would have much more detail, but yes. With Windows clustering (on top of which SQL 2012/2014 Availability Groups are based), you can only have a homogenous OS version across the board. Since they all must have the same version, you can't have something like a single 2012 R2 instance in an otherwise 2012 cluster...and since you can't do that, you can't upgrade one at a time. Yay! New cluster time instead! Trust me, we're bitching about this as are most people and I think changes must be coming there. It doesn't matter how many fancy features you add to the OS if we can't upgrade to it, so they'll have to stop and address that problem.
octo_t 12y ago
You can get servers with 384GB of RAM now from a vendor like Dell, without needing to call them for just under $14k.
bradyd 12y ago
I don't believe that much RAM is uncommon for large scale database servers. 384GB RAM is only about $5000 from Dell. They also have a new server model coming out that supports up to 6TB of RAM [0]. [0] http://www.dell.com/us/business/p/poweredge-r920/pd http://www.dell.com/us/business/p/poweredge-r920/pd
- valarauca1 12y ago
  Most people over estimate hardware cost. Buying a 300-500GB ram server maybe very expensive. But buying a distributed data base expert likely costs at least an order of magnitude higher per year.
  - jaequery 12y ago
    Same server in AWS, it'll cost $50k+/month for the same thing. SO just bought it one time fee for probably around $30k or so, I'll say they are doing something right!
    - kondro 12y ago
      The don't have servers with that much RAM, but a 244GB RAM server with 8x800GB SSDs and 32 cores costs <$5k a month ad-hoc and about $1300/month with a 3 year commitment.
      - valarauca1 12y ago
        $1300/month w/ a 3 year commitment is $46.8k over the lifespan. Which is more then 4x more then the actual hardware cost. Your ROI on a physical server with the same stats at that price tag would be 8 months.
        habosa 12y ago
        Not saying its totally worth it, but the upside is that they take care of power costs, physical space costs, networking, and if the hardware fails they'll put your application on a new server. Basically you take all of the risk out the equation.
snowwrestler 12y ago
I believe the idea is to hold as much of the DB in memory as possible, for speed, so huge RAM totals on DB servers are not unusual. 384 GB should be well within modern OS reach. Once you're computing at 64 bits, the next theoretical limit on addresses would be somewhere around 16 million terabytes. edit: wrong number
EvanAnderson 12y ago
Windows Server handles that much RAM fine, and, so long as it fits, SQL Server will end up caching the entire database in RAM. If I remember correctly they're using Dell R720's for their SQL Server machines. At current Dell retail pricing 384GB of RAM would be about $5,900.00. At that price I'd vote for throwing money at raw hardware and reaping the benefits of caching the database in RAM, versus trying to pay programmers to come up with clever techniques to deal with having less RAM.
callesgg 12y ago
Not that much, we have 128 gb ram on our hardware. That is pretty standard for upper midle class rackmounted servers. 64 bit computers has a memmory limit so large that it is not understandeble. The ram is still lightning fast compared to other stuff.
daigoba66 12y ago
384 GB is fairly easy to achieve today with 16 GB DIMMs x 24 slots (many general purpose servers have up to 24 DIMM slots). Beyond that you need to jump to 24 GB DIMMs which are fairly expensive.
AaronFriel 12y ago
There are a few erroneous replies to your comment, re: Windows support for that much memory. Memory support for versions of Windows: User-mode process Kernel-mode address space address space (process bit-ness) 32-bit 64-bit all Server Operating Systems: Windows Server 2008 R2 4 GB* 2048 GB 2048 GB Windows Server 2012 4 GB* 4096 GB 4096 GB Windows Server 2012 R2 4 GB* 131072 GB 131072 GB Client Operating Systems: Windows 7 (x86) 3 GB* n/a 2 GB Windows 7 (64-bit) 4 GB* 192 GB 192 GB Windows 8 (64-bit) 4 GB* 512 GB 512 GB Windows 8.1 (64-bit) 4 GB* 131072 GB 131072 GB * requires IMAGE_FILE_LARGE_ADDRESS_AWARE flag, otherwise 2GB ** All units use binary exponents (GB = 2^30 bytes)
- yuhong 12y ago
  process address space != physical memory
lnanek2 12y ago
Even before OS'es supported that much people were sticking that much RAM on expansion bus cards and plugging them in and treating them like fast disks. So it is going to happen either way. The performance boost for the DB is just too much compared to it thrashing disks constantly and there are quite a few benefits to having fewer machines compared to many DB servers as well.
buckbova 12y ago
SQL Server only reads data from memory. When something is requested it is first looked for in RAM (or virtual) and if not found is copied from disk to RAM. The more RAM you have the better without exception here.
dekhn 12y ago
The simplest rule of computing is, if you have a problem that can be solved economically by just throwing more RAM into a single machine at it, you should do it (instead of any other dimension, and assuming you don't have other constraints). No, that's no common for production servers for popular web services, though- they tend to shard across other dimensions because the web services are often CPU or network-bound.

by habosa 12y ago

x1798DE 12y ago

It's not a server, but I've seen machines used in physics simulations have something on the order of 256 GB RAM in them. As far as I know they cost maybe $20k-$30k, so not absurdly expensive. I don't think addressing that memory is a problem on any x64 architecture. Windows Server 2012 has a 4TB upper limit on RAM, apparently [1], and as far as I can tell there's no reason for this other than product differentiation between different "levels" of the operating system. 1. http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778%28v=vs.85%29.aspx#physical_memory_limits_windows_server_2012 http://msdn.microsoft.com/en-us/library/windows/desktop/aa36...

mrb 12y ago
$4k is all you need to build a whitebox 256GB machine: dual-socket G34 motherboard with 16 DDR3 slots lke the ASUS KGPE-D16 ($500) + two Opteron 6320 (2 * $300) + sixteen 16GB registered ECC DDR3-1600 (16 * $160) + chassis/PSU ($500) = $4160. Doing this on Amazon r3.8xlarge ($2.8/hr) would cost you $8100+ over 4 months. A DIY server can make sense in some cases. Eg. you are a very early-stage startup with almost no funding, and you are good at assembling and troubleshooting computers, and are ok with mere warranty on parts as opposed to full vendor support from HP/Oracle/Dell, go for it. That's how many startups started (see Google and their makeshift half-donated half-loaned boxes in 1998).
- ZachPruckowski 12y ago
  Don't forget to factor in the costs of running, cooling, and maintaining that much hardware (also bandwidth costs). I'm not saying it's $2k/month, I'm just saying don't compare the physical box and AWS without factoring everything in. At the end of the day, if you don't need the support, you're probably right that a dedicated box makes sense there.
  - adventured 12y ago
    AWS has among the highest bandwidth costs you're going to run into, if we're talking co-location, dedicated, or self-hosting. Their bandwidth costs are the sole reason I won't go anywhere near AWS yet, I consider their prices outrageously high. 10tb = roughly $1,000 per month They're between 5 and 50 times more expensive on bandwidth than the options in dedicated / colo / self hosting.
    - dekhn 12y ago
      You can negotiate with AWS.
      - liotier 12y ago
        I don't believe that startups in their garage days have anywhere near the necessary leverage to negotiate anything with Amazon...
        jaequery 12y ago
        Agree, not many people actually realize just how slow and expensive AWS actually is. Kudos to SO for going with their own hardware.
        dekhn 12y ago
        they negotiate, still. I know several startups which went to amazon, said, "hey, your competitor is cheaper, what is your value prop" and suddenly had 50% discounts, etc. It's amazing how much you get simply asking!

claudius 12y ago

Linux (SuSE enterprise something, possibly with a custom kernel, I’m just a user, not admin) runs just fine on 5.2TB machine, though I can easily imagine you’d have problems if you tried it with, say, DOS :) The system obviously behaves a bit different from a standard desktop machines, e.g. different areas of RAM are differently fast, depending on the core on which your current process runs, you need to disable individual CPU lines displayed in top etc., but apart from these, mostly everything seems “normal” to me.

tekacs 12y ago
> e.g. different areas of RAM are differently fast, depending on the core on which your current process runs, you need to disable individual CPU lines displayed in top etc. this is NUMA, in case anyone 'new to sysadmin' or to architecture is trying to Google this

Sanddancer 12y ago

384 gigs is pretty common these days. With Windows, you need at least the server edition to address that much, but if you're running a server, you're not gonna be using the desktop versions of the OS anyways. Pretty much everything else, it's just plug and play. The RAM is quite a bit if you're doing a typical cloud deployment, but SO seems to be more in the model of making sure all your eggs are in a really strong basket. Besides, at the end of the day, RAM tends to be a lot cheaper than lag.

ddorian43 12y ago
*everything else is plug and pay
abritishguy 12y ago
Not true. Windows 8.1 64 bit would be happy with that much.

junto 12y ago

Their SQL Server is Microsoft SQL Server running on (shock horror) Windows. Says in the article: Windows 2012 is used in New York but are upgrading to 2012 R2 (Oregon is already on it).

Nick-Craver 12y ago
To be clear, most servers are moving to 2012 R2 (most already have) but the machines our SQL clusters run on, specifically, will not. This is because you have to effectively (or literally) rebuild the Windows cluster from scratch and we just don't get that level of benefit from the 2012 to 2012 R2 upgrade. There are quite a few improvements we care about: native NVMe, better dynamic quorum, better DSC support, better SMB, and such...but not enough to make the upgrade worth it. Nick Craver - Stack Exchange Sysadmin & Developer
- Just-A-Guest 12y ago
  Do I get this right? You can not upgrade a server cluster to the next version of the operating system without complete new installation??? Could you please write something about these kind of things, this is very interesting!
  - Nick-Craver 12y ago
    I'm not sure a full writeup would have much more detail, but yes. With Windows clustering (on top of which SQL 2012/2014 Availability Groups are based), you can only have a homogenous OS version across the board. Since they all must have the same version, you can't have something like a single 2012 R2 instance in an otherwise 2012 cluster...and since you can't do that, you can't upgrade one at a time. Yay! New cluster time instead! Trust me, we're bitching about this as are most people and I think changes must be coming there. It doesn't matter how many fancy features you add to the OS if we can't upgrade to it, so they'll have to stop and address that problem.

octo_t 12y ago

You can get servers with 384GB of RAM now from a vendor like Dell, without needing to call them for just under $14k.

bradyd 12y ago

I don't believe that much RAM is uncommon for large scale database servers. 384GB RAM is only about $5000 from Dell. They also have a new server model coming out that supports up to 6TB of RAM [0]. [0] http://www.dell.com/us/business/p/poweredge-r920/pd http://www.dell.com/us/business/p/poweredge-r920/pd

valarauca1 12y ago
Most people over estimate hardware cost. Buying a 300-500GB ram server maybe very expensive. But buying a distributed data base expert likely costs at least an order of magnitude higher per year.
- jaequery 12y ago
  Same server in AWS, it'll cost $50k+/month for the same thing. SO just bought it one time fee for probably around $30k or so, I'll say they are doing something right!
  - kondro 12y ago
    The don't have servers with that much RAM, but a 244GB RAM server with 8x800GB SSDs and 32 cores costs <$5k a month ad-hoc and about $1300/month with a 3 year commitment.
    - valarauca1 12y ago
      $1300/month w/ a 3 year commitment is $46.8k over the lifespan. Which is more then 4x more then the actual hardware cost. Your ROI on a physical server with the same stats at that price tag would be 8 months.
      - habosa 12y ago
        Not saying its totally worth it, but the upside is that they take care of power costs, physical space costs, networking, and if the hardware fails they'll put your application on a new server. Basically you take all of the risk out the equation.

snowwrestler 12y ago

I believe the idea is to hold as much of the DB in memory as possible, for speed, so huge RAM totals on DB servers are not unusual. 384 GB should be well within modern OS reach. Once you're computing at 64 bits, the next theoretical limit on addresses would be somewhere around 16 million terabytes. edit: wrong number

EvanAnderson 12y ago

Windows Server handles that much RAM fine, and, so long as it fits, SQL Server will end up caching the entire database in RAM. If I remember correctly they're using Dell R720's for their SQL Server machines. At current Dell retail pricing 384GB of RAM would be about $5,900.00. At that price I'd vote for throwing money at raw hardware and reaping the benefits of caching the database in RAM, versus trying to pay programmers to come up with clever techniques to deal with having less RAM.

callesgg 12y ago

Not that much, we have 128 gb ram on our hardware. That is pretty standard for upper midle class rackmounted servers. 64 bit computers has a memmory limit so large that it is not understandeble. The ram is still lightning fast compared to other stuff.

daigoba66 12y ago

384 GB is fairly easy to achieve today with 16 GB DIMMs x 24 slots (many general purpose servers have up to 24 DIMM slots). Beyond that you need to jump to 24 GB DIMMs which are fairly expensive.

AaronFriel 12y ago

There are a few erroneous replies to your comment, re: Windows support for that much memory. Memory support for versions of Windows: User-mode process Kernel-mode address space address space (process bit-ness) 32-bit 64-bit all Server Operating Systems: Windows Server 2008 R2 4 GB* 2048 GB 2048 GB Windows Server 2012 4 GB* 4096 GB 4096 GB Windows Server 2012 R2 4 GB* 131072 GB 131072 GB Client Operating Systems: Windows 7 (x86) 3 GB* n/a 2 GB Windows 7 (64-bit) 4 GB* 192 GB 192 GB Windows 8 (64-bit) 4 GB* 512 GB 512 GB Windows 8.1 (64-bit) 4 GB* 131072 GB 131072 GB * requires IMAGE_FILE_LARGE_ADDRESS_AWARE flag, otherwise 2GB ** All units use binary exponents (GB = 2^30 bytes)

yuhong 12y ago
process address space != physical memory

lnanek2 12y ago

Even before OS'es supported that much people were sticking that much RAM on expansion bus cards and plugging them in and treating them like fast disks. So it is going to happen either way. The performance boost for the DB is just too much compared to it thrashing disks constantly and there are quite a few benefits to having fewer machines compared to many DB servers as well.

buckbova 12y ago

SQL Server only reads data from memory. When something is requested it is first looked for in RAM (or virtual) and if not found is copied from disk to RAM. The more RAM you have the better without exception here.

dekhn 12y ago

The simplest rule of computing is, if you have a problem that can be solved economically by just throwing more RAM into a single machine at it, you should do it (instead of any other dimension, and assuming you don't have other constraints). No, that's no common for production servers for popular web services, though- they tend to shard across other dimensions because the web services are often CPU or network-bound.