Estimated reading time: 19 minutes

Thank you for reading this post, don't forget to subscribe! Happy New Year 2024!

As always, I try to blog on questions I get from the field and recently, I helped some colleagues and customers build a DAG on Nutanix 2 times last week ;).  The process is pretty straightforward, but there are some best practices that everyone should follow to ensure a healthy DAG environment.

Creating a two node Microsoft Exchange 2013 Database Availability Group

Introduction

Ok, let me tell you how to make your Exchange 2013 Mailbox role deployment highly available (HA) by using a Database Availability Group (DAG).. Let’s get rocking’.

Before we get too deep into it, let’s first make sure you’re on the right plane. This flight will take you through configuring a DAG in Exchange 2013 with two nodes in the same site to make your Mailbox role highly available. In a later blog post I will be covering how to make your Exchange 2013 Mailbox role deployment site resilient by adding additional nodes in a remote location. If you’re still with me, buckle your seatbelt and keep your hands inside the cabin at all times.

Server Safety Check

Before we can take off, we need to do a pre-check routine on all our servers because the last thing we want is to get cruising at 20,000 users and then have it crash. Let’s discuss the operating system requirements.

Operating System

DAGs utilize Windows Failover Clustering so you’re going to need a Windows OS that supports that and since Exchange 2013 can only be installed on Server 2012 and Server 2008 R2 SP1, we’ll focus on those. If you’re running 2008 R2, you’ll want Enterprise or Datacenter. If you’re running Server 2012, you can use Standard or Datacenter. There’s nothing worse than installing Exchange only to find that you put the wrong OS on there when you go to configure the DAG. I have seen this happen many times at customers. Just in case you’re wondering, Microsoft does not support upgrading the OS once Exchange has been installed so your only option is to uninstall Exchange, install the correct OS, and then reinstall Exchange or build a new server with the correct OS. After you’ve got the correct OS, you’ll want to install the most current updates.

The Network

DAGs need to replicate and clients need to connect, so it’s only natural that you’d need two NICs to create a DAG. Microsoft supports a single NIC but it’s recommended you use two so you can separate the replication traffic from the client traffic. You don’t want the reseeding of a database to affect your users’ experience. Let’s talk about the two NICs real quick. The first thing I like to do is name them appropriately. I usually name my replication NIC REPL and my client NIC MAPI, that way there’s no confusion in this next step I take, and that’s to put them in the correct binding order.

The MAPI NIC needs to be listed first. This is how you do that.

1. From the command prompt, type in NCPA.CPL to open up the Network Connections screen.
2. Press the Alt key and that will display the File menu, then click on Advanced and select Advanced Settings…
3. On the Adapters and Bindings tab, order the adapters so that MAPI is first and REPL is second.

Note: If you’re using network teaming, ensure the MAPI team is listed first and the REPL team is listed second.

Quick note about Network Teaming. I’m a fan of it because I don’t want a NIC failure to cause a database failover. NIC teaming is supported on client and replication networks but you have to configure each team so that only one NIC in the team is up and the other NIC is in standby, which means Active/Passive.

Last thing I want to point out is the IPv6 check box on each NIC. Everywhere I go I see admins – this box because they don’t “support” IPv6. First point is, unchecking that box doesn’t truly disable IPv6. Second point is, if you’re having issues with Exchange connectivity, chances are having the check box checked is not your issue. Lastly, Microsoft does not test Exchange with that box unchecked, so technically you’re in an unsupported configuration by unchecking it. All that was said to say, leave it alone even though you really, really want to uncheck it.

Client Network

This is the king of NICs because it gets it all, default gateways, DNS settings, and priority, and there can only be one! You’ll configure this just like you would any other NIC card for a server.

Replication Network

The replication network needs to be on a different subnet than the client network or the DAG won’t see them as two separate networks, obviously. For this post, I used 10.1.1.0/24 for my MAPI network and 192.168.1.0/24 for my replication network. If you have a separate network for backups or administration, I would use that. I wouldn’t create a dedicated network just for Exchange replication unless you have to. DAG replication networks can share. It learned how to in kindergarten. If you want to use more than one replication network, then each replication network will need to be on different subnets as well. There are a few things you need to do with this network. Let’s go through that because you get to uncheck stuff, just not the IPv6 box

1. From a command prompt, type in NCPA.CPL to open the Network Connections window
2. Right-Click the Replication NIC and go to Properties
3. Uncheck Client for Microsoft Networks and File and Print Sharing for Microsoft Networks
4. Select IPv4 and click Properties
5. Input your IP Address and Subnet mask. Do NOT put in a Default Gateway or DNS servers.
6. Click on Advanced…
7. Click on the DNS tab and deselect Register this connection’s addresses in DNS
8. Click on the WINS tab and select Disable NetBIOS over TCP/IP
   Wow that was a lot of steps. Ensure you click Ok to save all that stuff!

Nutanix Best Practices for Exchange Storage

• Separate database and log files into separate containers
• Databases should be on a container with in-line compression and EC-X enabled
• The Windows NTFS Allocation unit size should be set to 64KB for Database Drives
• Log Files should be on a container with no attributes and located on a volume with 4k block allocation size.
• Archive Databases should be on a separate container from active databases with In-Line Compression and EC-X enabled

As per Microsoft Best Practices, Nutanix recommends to disable deduplication for the container hosting active Exchange Databases workloads. Deduplication does not provide significant benefits due to frequency of change in the Exchange Databases.
Alright, we’ve got our servers ready for the DAG, now it’s time to install Exchange. If you need help doing that, I’ve will be covering that in a future blog post. If you have Exchange already install, then move on forward.

Creating the DAG

Now the real fun begins! We get to create stuff. Let’s open up the Exchange Admin Center (EAC) and click on Servers –> database availability groups.

As you can see I already have a DAG but don’t worry, I didn’t get started without you. I’m going to create another one cleverly named DAG02. Click on the ‘+’ sign so we can create a new DAG.

In the new database availability group windows, You’ll give this a name, pick a witness server and give it an IP. Let’s talk about each of those real quick.

DAG Name

This name needs to be unique in the environment because once you join the first server to the DAG, a Cluster Name Object (CNO) will be created in Active Directory with this name. This CNO can be pre-staged, meaning you can create it manually before Exchange does. You might want to do that in scenarios where you don’t have rights to create Computer objects in Active Directory. At the time I’m writing this, Technet states that in Exchange 2013, if you’re DAG members are running 2012 that you have to pre-stage the CNO, but my testing has proven otherwise. Regardless, stay on the safe side and pre-stage the CNO if you’re OS is 2012.

Witness Server

You have to pick a server to host a share for the DAG. This share is only needed when there is an even number of nodes in the DAG, and it’s not utilized until you need it to maintain quorum. I’ll explain that in more detail in my Nerd Knowledge section below. Skip it if you don’t care. If you plan to have an odd number of nodes in your DAG you have to configure it regardless because Exchange will use it to adjust the quorum configuration automatically as you add and remove nodes from your DAG.

The next question is, “Rob, who should I give the honor of being the witness to?” Good question! We’ll, if you are separating your CAS and Mailbox roles out, pick a CAS server. In fact, if you leave the witness server blank and there is a CAS server in that site, Exchange will automatically pick it. If you’re not separating out the roles, then you have to pick a server. A good candidate would be a server that’s not going to be rebooted a lot but I wouldn’t have a dedicated witness server, it’s a waste of resources, so pick an existing file server or a server that’s within your area of control. Before you decide to pick a Domain Controller, review the witness bullet points below.

• Witness Servers must be in the same forest and not be a member of the DAG.
• Witness Server must have the Exchange Trusted Subsystem (ETS) group added to the local administrators group. If this is a Domain Controller, it needs to be added to DomainAdministrators. This is why you might want to not pick a domain controller. This gives the ETS administrative rights to the entire domain, not just that server. Granted ETS has crazy rights already, but putting the ETS in the Administrators group for the domain might make your security people nervous.
• Witness Server must be running Server 2012, Server 2008 R2, Server 2008, Server 2003 R2, or Server 2003.

Few more notes about the Witness server. It does not need to be highly available, meaning don’t put it on a cluster, and a server can serve as the witness server for multiple DAGs but each DAG needs it’s own witness share. Also, you don’t need to specify a path when creating the DAG. Leave it blank and Exchange will create the proper folder structure. Lastly, I’m assuming you’re installing two or more nodes in one physical site. If you plan to put nodes in other sites, then you’ll want to pick a Witness server in the site that holds the majority of your users.

Nerd Knowledge: I mentioned that the Witness server is only used when there is an even number of nodes and only utilized when it’s needed to maintain quorum, and those of you that are nerds like me, I wanted to explain that a little more. Like I mentioned before, a Witness server is only used when there is an even number of nodes in the DAG, but Exchange will want to configure one so that it can automatically adjust the quorum configuration as you add and remove nodes. For example, if you have two nodes, under Failover Cluster Manager you’ll see the quorum configuration set to Node and File Share Majority. If you add a third node, the quorum configuration will change to Node Majority and the Witness server is not used at all.

Now to address the other statement of the Witness server only being utilized to maintain quorum. A DAG must “have quorum” to mount databases and if it loses quorum, you’re environment will go down. To maintain quorum there must be enough votes in the DAG and each server is a voting member. For example, if you have a two node DAG the number of votes to stay up is 2. That can be found by taking the number of nodes in the DAG, dividing it by 2, and then adding 1. In a two node DAG that looks like this (2 nodes / 2) +1 = 2. Again, that means in a two node DAG we need two votes to maintain quorum and keep our databases mounted.

I know what you’re thinking, “But, Rob, we only have two members in our DAG and we need two votes. Doesn’t that mean we can’t lose a server without dismounting all of our databases?” Fair question, young padawan so let me tell you. This is where the Witness server comes into play. In a two node DAG, in normal operation, a Witness server is not used because we don’t need it. We have our two voting members up, but if we lose one of those nodes, the surviving node will try to lock the witness server. If it successfully locks the witness server, it becomes special and gets a second vote, which gives us our two required votes to maintain quorum and life is good. However, if you reboot the witness server in this scenario, you’ll lose quorum and your mailboxes will dismount. Even worse, they won’t mount again until you either “shrink” your DAG, or get both nodes back up. That might sound confusing and I can cover that in another post because the length of this is getting out of control, but just remember that if you have one of your two DAG members down, don’t reboot your witness server or your surviving node.

Wow, that’s a lot, but I need to explain one more example to ensure we fully understand this. Let’s assume we have six nodes in our DAG. To maintain quorum we need four votes (6/2) + 1 = 4. We have an even number of nodes so we need a witness server, but we won’t use it until we need it to maintain quorum. If we lose one or two of the six nodes, we still have enough votes, so right now we don’t need the witness server. If we lose a third node, one of the surviving nodes will try and lock the witness server.

Whoever grabs it first wins and that surviving node gets an extra vote making it 2 +1 +1 = 4. Make sense? I hope so, cause I lost my own attention two paragraphs ago! LOL Let’s move on.

DAG IP Address

This is the IP address used by the DAG. It uses DHCP by leaving it blank or you can specify a static IP address here.

Now that you’ve put in all the information, click Ok. Finally!!
You can now see the DAG in the EAC, and if wanted to be a true nerd, you can open ADSIEDIT and go to

CN=Database Availability Groups,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=<Org Name>,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=<Domain>,DC=<Com>. You should see the msExchMBDatabaseAvailabilityGroup object.

You should also check the witness server to ensure the file share was created.

Adding Servers to the DAG

Now that we have an empty DAG and a witness server, we need to add servers to it. This is done by highlighting the DAG in the EAC and clicking the Managed DAG membership.

On the manage database availability group membership screen, click the ‘+’ sign to add servers to the DAG.

Select the servers you want to add to the DAG. You can select multiple servers here. I’m adding two servers at the same time. I’m skipping RED-15EXCH02 because it’s running 2008 R2 and the other two servers I’m adding are running 2012.

When you add the first Exchange server to the cluster, that server creates the CNO in Active Directory if you didn’t pre-stage it. Microsoft has reported that if you add Exchange servers to the DAG too quickly and Active Directory does not have time to replicate, the second Exchange server might not see the CNO and then create it’s own. This might be another reason to pre-stage the CNO before adding a node to the DAG or at least make sure you force replication after you add the first node to the DAG to replicate around the CNO.

Once the servers have been added to the DAG, you should see something like this.

Now that we’ve created our DAG, we can take a look at it by opening up the Exchange Management Shell (EMS) and running Get-DatabaseAvailabilityGroup DAG02 -Status | FL. That should give you something like this. You can notice a few things from this screenshot. You can see that Compression and Encryption are enabled between DAG members in other sites if we decided to extend this DAG to another location.

Note that it’s a DAG property not a network property so that means it’s either enabled for all networks in the DAG or none of them. Another thing worth noting is the port that the DAG replicates on, 64327. This is helpful when there is a firewall between your Exchange servers.

Now your screenshot might look a little different than mine because I ran Set-DatabaseAvailabilityGroup DAG02 -DatacenterActivationMode DagOnly. This prevents a split brain cluster from forming, so that makes this setting a must in all DAGs. I highly recommend you set it too.

Lastly, you can see that our witness server is configured and what server is hosting the Primary Active Manager.

Let’s take a quick look at the Failover Cluster Manager to see what happened there.

From above screenshot, you can see that we have a cluster named after the DAG and our two servers are added as nodes. You can also see that the cluster configuration is Node and File Share Majority. Remember, it’s because we have an even nodes in the cluster. If we added a third node, it would configure the cluster to use Node Majority.

Quick Tip: If you’re worried about your cluster configuration, witness server, or your DAG settings in general, try running Set-DatabaseAvailabilityGroup DAG02 from EMS. That’s right, no parameters. It will make sure you didn’t do anything stupid. I use this like Roy uses, “Have you tried turning it off and on again?” If you don’t know who Roy is and have never heard that phrase, you need to stop reading this post, open a Netflix account, if you don’t already have one, look up “IT Crowd”, watch the “The Speech” episode on Season 3, and come back here and post about how funny it was and how much you want to thank me for telling you about it.

One last thing I’d like to point out here are the DAG networks. If you open up the EMS, and run Get-DatabaseAvailabilityGroupNetwork, you’ll see your DAG networks. Remember that I have two DAGs, but you should see something similar to DAG02’s networks DAG02MapiDagNetwork and DAG02ReplicationDagNetwork01. Why did it put a ’01’ at the end of the replication network and not the MAPI network? Because MAPI is the highlander and there can only be one! You can also see which networks are enabled for replication. I’ve noticed that my MAPI network for DAG02 is configured for replication, but I’m going to fix that below.

In Exchange 2010, you had to configure your DAG networks manually, such as disabling replication on the MAPI network. If you run Get-DatabaseAvailabilityGroupNetwork, you can see that Exchange 2013 auto configured it for us. In fact, if you want to configure the DAG networks manually in Exchange 2013, you have to run Set-DatabaseAvailabilityGroup dag02 -ManualDagNetworkConfiguration $True. I’m going to run this command to disable replication on my MAPI network because the DAG enabled it for me and I want that disabled on that network to force everything through the replication network. This doesn’t meant that the DAG will never use the MAPI network to replicate because in an emergency situation when all replication networks are down, DAG02 will use the MAPI network if it has to so that replication continues. You can run the following command to see which network your DAG is using for replication. Get-MailboxDatabaseCopyStatus -Server red-15exch01 | fl name,incoming,outgoing. Running this from both servers in our DAG will show which network the log copying is coming in on.

Adding Database Copies

Well done so far! You’ve created a DAG and hopefully learned some stuff. Now that we do have this fancy DAG, we need to use it by creating Database copies. Let’s jump back over to the EAC and click on servers, then databases. One quick thing here. Am I the only one that hates it when menu items aren’t capitalized??? That drives me nuts that the menu options in the EAC are not capitalized, but I’m open to change. We’ll see how that goes. Ok, back on track.

Click on the database hosted by one of the member in our DAG and click on the “…” and select Add database copy. This will “seed” the database, meaning copy over the EDB file over to the server you pick and then copy over the log files. If you’re database is large, it could take some time depending on your bandwidth. If you’re doing this in a lab with an empty database, it should go pretty quick. You’ll want to do this for each database in the DAG that you want protected.
And we’re done!!!
In this post you learned how to create and get started on configuring a DAG. I talked about Nutanix Best Practices around DAG’s.  I talked way too much about the Witness Server.. Man there was a lot in this post, but I truly hoped you learned something even if it’s never to read one of my posts again. :).

As always, I encourage your feedback and any questions you might have..

Until next time, Rob