Hi,
from some of our users we got complains that the SMTP send process on port 587 is sometimes quite slow, taking about 5 seconds before the message is sent.
After analyzing the SMTP receive logs on the Exchange 2010 CAS/HTS server with latest Rollup and updates on up-to-date Windows Server 2008 R2 Enterprise, I noticed that on many client connections the initial "[Set Session Permissions]" operation takes 5 seconds while on other connections it takes 0 seconds. This is even before the server sends the first 220 to the client. For example:
18:53:24.659 None [Set Session Permissions]
18:53:29.667 220 <FQDN omitted> Microsoft ESMTP MAIL Service ready at Wed, 28 Oct 2015 18:53:24 +0100
18:53:29.713 EHLO <client omitted>
18:53:29.713 250-MSX-CAS1.w2k.biochem.mpg.de Hello [141.61.32.80]
18:53:29.713 250-SIZE 41943040
18:53:29.713 250-PIPELINING
18:53:29.713 250-DSN
18:53:29.713 250-ENHANCEDSTATUSCODES
18:53:29.713 250-STARTTLS
18:53:29.713 250-AUTH GSSAPI NTLM
18:53:29.713 250-8BITMIME
18:53:29.713 250-BINARYMIME
18:53:29.713 250 CHUNKING
18:53:29.760 STARTTLS
18:53:29.760 220 2.0.0 SMTP server ready
18:53:29.760 [Sending certificate]
The receive connectors are configured as follows:
RunspaceId : <omitted> AuthMechanism : Tls, Integrated, BasicAuth, BasicAuthRequireTLS Banner : BinaryMimeEnabled : True Bindings : {0.0.0.0:587} ChunkingEnabled : True DefaultDomain : DeliveryStatusNotificationEnabled : True EightBitMimeEnabled : True BareLinefeedRejectionEnabled : False DomainSecureEnabled : False EnhancedStatusCodesEnabled : True LongAddressesEnabled : False OrarEnabled : False SuppressXAnonymousTls : False AdvertiseClientSettings : False Fqdn : <omitted> Comment : Enabled : True ConnectionTimeout : 00:10:00 ConnectionInactivityTimeout : 00:05:00 MessageRateLimit : 20 MessageRateSource : User MaxInboundConnection : 5000 MaxInboundConnectionPerSource : 50 MaxInboundConnectionPercentagePerSource : 2 MaxHeaderSize : 64 KB (65,536 bytes) MaxHopCount : 60 MaxLocalHopCount : 12 MaxLogonFailures : 3 MaxMessageSize : 40 MB (41,943,040 bytes) MaxProtocolErrors : 5 MaxRecipientsPerMessage : 200 PermissionGroups : ExchangeUsers, Custom PipeliningEnabled : True ProtocolLoggingLevel : Verbose RemoteIPRanges : {0.0.0.0-255.255.255.255} RequireEHLODomain : False RequireTLS : False EnableAuthGSSAPI : True ExtendedProtectionPolicy : None LiveCredentialEnabled : False TlsDomainCapabilities : {} Server : <omitted> SizeEnabled : Enabled TarpitInterval : 00:00:05 MaxAcknowledgementDelay : 00:00:30 AdminDisplayName : ExchangeVersion : 0.1 (8.0.535.0) Name : Client DistinguishedName : <omitted> Identity : <omitted> Guid : <omitted>
ObjectCategory : <omitted>/Configuration/Schema/ms-Exch-Smtp-Receive-Connector ObjectClass : {top, msExchSmtpReceiveConnector} WhenChanged : 10.12.2014 14:51:25 WhenCreated : 13.05.2013 14:00:11 WhenChangedUTC : 10.12.2014 13:51:25 WhenCreatedUTC : 13.05.2013 12:00:11 OrganizationId : OriginatingServer : <omitted> IsValid : True
For the network setup: the SMTP access is routed through a web application firewall and KEMP load balancer to the CAS/HTS servers, so all client connections come from the same remote IP (the WAF). The delay is however definitely on the Exchange servers, as visible from the SMTP receive logs.
From the SMTP logs I can rule out the tarpit. Although configured with default 5 seconds, the tarpit is not invoked on affected connections.
There are custom permissions set on the receive connector and custom permissions inherited, but the operation does not take so long on every connection.
Is it possible that "MaxInboundConnectionPerSource" of 50 can cause this? How would the server react, if the threshold is exceeded?
Is there some way I can identify what makes the system take so long on setting the session permissions for about 30 to 50% of the connections? It's always a delay of nearly exactly 5 seconds.
Thanks in advance for any hints on how to further debug this issue.
Gregor