Exchange 2016 Back Pressure TroubleshootingIn most cases, first time people realize this feature ever exists is when the senders receive this error, and mail delivery becomes slow or stops at all: 452 4.3.1 Insufficient system resources

It means, that the server is overloaded, and lacks system resources, that might be:

Memory

Free space on the hard disk

Uncommitted transactions in queue

A reboot will temporarily solve the problem, but you definitely don’t want to do this. So, what is the Back Pressure, and how solve this problem? Read on.

Exchange 2016 Back Pressure Troubleshooting – Resource monitoring

Back pressure is a helpful feature, eventually. It monitors consumption of system resources by the Exchange Transport service. If any of the resources becomes overloaded, Back Pressure takes actions to prevent server to be completely overloaded. And the main action of this feature is to delay message receiving. If that doesn’t solve the problem, Back Pressure stops Exchange Transport service from receiving new messages to process already queued messages. When consumption of system resources comes back to normal, Back Pressure resumes normal operation of the Exchange transport service.

Here are resources that are monitored with Back Pressure:

  • DatabaseUsedSpace[%ExchangeInstallPath%TransportRoles\data\Queue]   Hard drive utilization for the drive that holds the message queue database.
  • PrivateBytes   The memory that’s used by the EdgeTransport.exe process.
  • QueueLength[SubmissionQueue]   The number of messages in the Submission queue.
  • SystemMemory   The memory that’s used by all other processes.
  • UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data\Queue]   Hard drive utilization for the drive that holds the message queue database transaction logs.
  • UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data]   Hard drive utilization for the drive that’s used for content conversion.
  • UsedVersionBuckets[%ExchangeInstallPath%TransportRoles\data\Queue\mail.que]   The number of uncommitted message queue database transactions that exist in memory.

Each of these metrics is measured individually, and for each of these metrics three levels of load (or Pressure) are defined:

  • Low or Normal   The resource isn’t overused. The server accepts new connections and messages.
  • Medium   The resource is slightly overused. Back pressure is applied to the server in a limited manner. Mail from senders in the organization’s authoritative domains can flow. However, depending on the specific resource under pressure, the server uses tarpitting to delay server response or rejects incoming MAIL FROM commands from other sources.
  • High   The resource is severely overused. Full back pressure is applied. All message flow stops, and the server rejects all new incoming MAIL FROM commands.

For disk space metrics the back pressure condition causes messages to be rejected. However for memory utilization metrics, before rejecting connections the server will first take actions to attempt to relieve the conditions. Full list of actions is described in this wonderful arcicle.

There are default thresholds for each of these metrics, and of course, that can be changed.

Exchange 2016 Back Pressure Troubleshooting – Changing defaults

For managing this feature, you need to to edit it’s configuration file. Remember to Stop the Microsoft Exchange Transport service before editing, then save the config and start it again. By default, it is located in:

C:\Program Files\Microsoft\Exchange Server\V15\Bin\EdgeTransport.exe.config

If you installed Exchange Server to a different location then you will find the config file in the \bin folder of your installation directory.

The first two values control whether resource monitoring is enabled, and at what interval resource monitoring is performed. Though the option exists it is not recommended to actually disable resource monitoring. Instead the recommended approach is to identify the cause of resource over-utilization and correct it.

Another way to view thresholds is by using this command in Exchange Management Shell:

Exchange 2016 Back Pressure Troubleshooting – Detecting the problem

Back Pressure conditions generate following Event ID’s in application log:

  • Event ID 15004: Increase in the utilization level for any resource (eg from Normal to Medium)
  • Event ID 15005: Decrease in the utilization level for any resource (eg from High to Medium)
  • Event ID 15006: High utilization for disk space (ie critically low free disk space)
  • Event ID 15007: High utilization for memory (ie critically low available memory)

Exchange 2016 Back Pressure Troubleshooting – Solving the most common problem

And here is the most common one: Event ID 15006.

Microsoft Exchange Transport is rejecting message submissions because the available disk space has dropped below the configured threshold.The following resources are under pressure:
Used disk space ("C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue")

Option 1 is to fine-grain the thresholds, but there is another possibility to solve the issue: you may want to transfer Queue files to another location. This process describes steps for transferring the queue files:

  1. Create a folder for queue files (for example, D:\Queue)
  2. Stop the Microsoft Exchange Transport service
  3. Go to the original queue folder (C:\Program Files\Microsoft\Exchange Server\V15\Bin)
  4. Open the configuration file with Notepad (you need EdgeTransport.exe.config)
  5. Edit the following strings, entering the paths inside your newly created folder, for example:
  6. <add key="QueueDatabasePath" value="D:\Queue\QueueDB" />
    <add key="QueueDatabaseLoggingPath" value="D:\Queue\QueueLogs" />
  7. Start the Microsoft Exchange Transport service

After starting the service, you can check everything is ok by opening folders from your config file (in the example, D:\Queue\QueueDB and D:\Queue\QueueLogs) and check the existence of:

  •  Mail.que and Trn.chk in the D:\Queue\QueueDB folder
  • Transaction log files Trn.log, Trntmp.log, Trnres00001.jrs, Trnres00002.jrs, and Temp.edb in the D:\Queue\QueueLogs

After that, messaging should return to normal state

 

Resources:

Change the location of the queue database

Understanding back pressure