Hi TechNet Experts,
Out Mail Server is using Exchange Server 2013 and we currently have 2 of Exchange (somehow not setup as DAG, as it is configured by unprofessional vendor previously).
Recently we confront an issue that certain incoming mail from various customer & certain outgoing mail from our internal to external user, unable to receive or send.
Outbound email usually receive error message below after 2 days delay:
The server has tried to deliver this message, without success, and has stopped trying. Please try sending this message again. If the problem continues, contact your helpdesk.
#550 4.4.7 QUEUE.Expired; message expired ##
X-MS-Exchange-Transport-Rules-Defer-Count: 24After investigation I realized that over 40+ emails stuck in Queue of old server's shadow redundancy with "Ready" status, and caused message expired after 2 days.
anyone here with Exchange Server knowledge could advice?
as we have no very professional knowledge on Exchange Server, bottleneck on technical part...
Attached below is Transcript for our Exchange Server properties (domain name replaced by "domain"), and queue information for better troubleshooting.
Appreciate.
**********************
Windows PowerShell transcript start
Start time: 20200708091115
Username: domain\administrator
RunAs User: domain\administrator
Machine: EXHG (Microsoft Windows NT 6.3.9600.0)
Host Application: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -noexit -command . 'C:\Program Files\Microsoft\Exchange Server\V15\bin\RemoteExchange.ps1'; Connect-ExchangeServer -auto -ClientApplication:ManagementShell
Process ID: 26212
**********************
Transcript started, output file is log1.txt
E:\Diagnosis>
PS>Get-OabVirtualDirectory | fl server, Name, *URL*, *auth*
Server : EXHG
Name : OAB (Default Web Site)
InternalUrl : https://exhg.domain.local/OAB
ExternalUrl :
BasicAuthentication : False
WindowsAuthentication : True
OAuthAuthentication : False
InternalAuthenticationMethods : {WindowsIntegrated}
ExternalAuthenticationMethods : {WindowsIntegrated}
Server : PANDORA
Name : OAB (Default Web Site)
InternalUrl : https://pandora.domain.local/OAB
ExternalUrl :
BasicAuthentication : False
WindowsAuthentication : True
OAuthAuthentication : True
InternalAuthenticationMethods : {OAuth, WindowsIntegrated}
ExternalAuthenticationMethods : {OAuth, WindowsIntegrated}
E:\Diagnosis>
PS>Get-WebServicesVirtualDirectory | fl server, name, *URL*, *auth*, *mrs*
Server : EXHG
Name : EWS (Default Web Site)
InternalNLBBypassUrl :
InternalUrl : https://exhg.domain.local/EWS/Exchange.asmx
ExternalUrl :
CertificateAuthentication :
InternalAuthenticationMethods : {Ntlm, WindowsIntegrated, WSSecurity, OAuth}
ExternalAuthenticationMethods : {Ntlm, WindowsIntegrated, WSSecurity, OAuth}
LiveIdNegotiateAuthentication :
WSSecurityAuthentication : True
LiveIdBasicAuthentication : False
BasicAuthentication : False
DigestAuthentication : False
WindowsAuthentication : True
OAuthAuthentication : True
AdfsAuthentication : False
MRSProxyEnabled : False
Server : PANDORA
Name : EWS (Default Web Site)
InternalNLBBypassUrl :
InternalUrl : https://pandora.domain.local/EWS/Exchange.asmx
ExternalUrl :
CertificateAuthentication :
InternalAuthenticationMethods : {Ntlm, WindowsIntegrated, WSSecurity, OAuth}
ExternalAuthenticationMethods : {Ntlm, WindowsIntegrated, WSSecurity, OAuth}
LiveIdNegotiateAuthentication :
WSSecurityAuthentication : True
LiveIdBasicAuthentication : False
BasicAuthentication : False
DigestAuthentication : False
WindowsAuthentication : True
OAuthAuthentication : True
AdfsAuthentication : False
MRSProxyEnabled : False
E:\Diagnosis>
PS>Get-EcpVirtualDirectory | fl server, Name, *URL*, *auth*
Server : EXHG
Name : ecp (Default Web Site)
InternalUrl : https://exhg.domain.local/ecp
ExternalUrl : https://mail.domain.com/ecp
InternalAuthenticationMethods : {Basic, Fba}
BasicAuthentication : True
WindowsAuthentication : False
DigestAuthentication : False
FormsAuthentication : True
LiveIdAuthentication : False
AdfsAuthentication : False
OAuthAuthentication : False
ExternalAuthenticationMethods : {Fba}
Server : PANDORA
Name : ecp (Default Web Site)
InternalUrl : https://mail.domain.com/ecp
ExternalUrl : https://mail.domain.com/ecp
InternalAuthenticationMethods : {Basic, Fba}
BasicAuthentication : True
WindowsAuthentication : False
DigestAuthentication : False
FormsAuthentication : True
LiveIdAuthentication : False
AdfsAuthentication : False
OAuthAuthentication : False
ExternalAuthenticationMethods : {Fba}
E:\Diagnosis>
PS>Get-ActiveSyncVirtualDirectory | fl server, Name, *URL*, *auth*
Server : EXHG
Name : Microsoft-Server-ActiveSync (Default Web Site)
MobileClientCertificateAuthorityURL :
InternalUrl : https://exhg.domain.local/Microsoft-Server-ActiveSync
ExternalUrl :
MobileClientCertificateAuthorityURL :
BasicAuthEnabled : True
WindowsAuthEnabled : False
ClientCertAuth : Ignore
InternalAuthenticationMethods : {}
ExternalAuthenticationMethods : {}
Server : PANDORA
Name : Microsoft-Server-ActiveSync (Default Web Site)
MobileClientCertificateAuthorityURL :
InternalUrl : https://pandora.domain.local/Microsoft-Server-ActiveSync
ExternalUrl :
MobileClientCertificateAuthorityURL :
BasicAuthEnabled : True
WindowsAuthEnabled : False
ClientCertAuth : Ignore
InternalAuthenticationMethods : {}
ExternalAuthenticationMethods : {}
E:\Diagnosis>
PS>Get-OutlookAnywhere | fl server, Name, *URL*, *hostname*, *auth*
Server : EXHG
Name : Rpc (Default Web Site)
XropUrl :
ExternalHostname : mail.domain.com
InternalHostname : mail.domain.com
ExternalClientAuthenticationMethod : Basic
InternalClientAuthenticationMethod : Basic
IISAuthenticationMethods : {Basic}
Server : PANDORA
Name : Rpc (Default Web Site)
XropUrl :
ExternalHostname : mail.domain.com
InternalHostname : mail.domain.com
ExternalClientAuthenticationMethod : Basic
InternalClientAuthenticationMethod : Basic
IISAuthenticationMethods : {Basic, Ntlm, Negotiate}
E:\Diagnosis>
PS>Get-OwaVirtualDirectory | fl server, Name, *URL*, *auth*
Server : EXHG
Name : owa (Default Web Site)
Url : {}
SetPhotoURL :
Exchange2003Url :
FailbackUrl :
InternalUrl : https://exhg.domain.local/owa
ExternalUrl : https://mail.domain.com/owa
ClientAuthCleanupLevel : High
InternalAuthenticationMethods : {Basic, Fba}
BasicAuthentication : True
WindowsAuthentication : False
DigestAuthentication : False
FormsAuthentication : True
LiveIdAuthentication : False
AdfsAuthentication : False
OAuthAuthentication : False
ExternalAuthenticationMethods : {Fba}
Server : PANDORA
Name : owa (Default Web Site)
Url : {}
SetPhotoURL :
Exchange2003Url :
FailbackUrl :
InternalUrl : https://mail.domain.com/owa
ExternalUrl : https://mail.domain.com/owa
ClientAuthCleanupLevel : High
InternalAuthenticationMethods : {Basic, Fba}
BasicAuthentication : True
WindowsAuthentication : False
DigestAuthentication : False
FormsAuthentication : True
LiveIdAuthentication : False
AdfsAuthentication : False
OAuthAuthentication : False
ExternalAuthenticationMethods : {Fba}
PS>Get-ClientAccessServer | fl Name, OutlookAnywhereEnabled, AutodiscoverServiceInternalUri
Name : EXHG
OutlookAnywhereEnabled : True
AutoDiscoverServiceInternalUri : https://exhg.domain.local/Autodiscover/Autodiscover.xml
Name : PANDORA
OutlookAnywhereEnabled : True
AutoDiscoverServiceInternalUri : https://pandora.domain.local/Autodiscover/Autodiscover.xml
E:\Diagnosis>
PS>Get-ExchangeCertificate | fl FriendlyName, Subject, CertificateDomains, Thumbprint, Services, Issuer, *not*
FriendlyName : Veeam Agent Certificate
Subject : CN=cc4ce6c1-f3f4-40b2-a6ad-xxx
CertificateDomains : {cc4ce6c1-f3f4-40b2-a6ad-xxx}
Thumbprint : 20940DE7BF92BE06E4F0A1F8AAxxx
Services : None
Issuer : CN=Veeam Backup Server Certificate
NotAfter : 6/7/2030 2:08:04 AM
NotBefore : 6/6/2020 2:08:04 AM
FriendlyName : *.domain.com
Subject : CN=*.domain.com, OU=Domain Control Validated
CertificateDomains : {*.domain.com, domain.com}
Thumbprint : 8E0E0E81765825B0182DBE64635xxx
Services : IIS, SMTP
Issuer : CN=Go Daddy Secure Certificate Authority - G2, OU=http://certs.godaddy.com/repository/,
O="GoDaddy.com, Inc.", L=Scottsdale, S=Arizona, C=US
NotAfter : 3/30/2022 10:46:38 AM
NotBefore : 3/17/2020 7:00:23 PM
FriendlyName : Exchange Delegation Federation
Subject : CN=Federation
CertificateDomains : {Federation}
Thumbprint : 4C87B75C354C7601763E25EFD904E42xxxx
Services : SMTP, Federation
Issuer : CN=Federation
NotAfter : 11/7/2024 12:13:11 PM
NotBefore : 11/7/2019 12:13:11 PM
FriendlyName : Microsoft Exchange Server Auth Certificate
Subject : CN=Microsoft Exchange Server Auth Certificate
CertificateDomains : {}
Thumbprint : E8ABF7AC6024FB5906F5D43A87E73D2Exxxx
Services : SMTP
Issuer : CN=Microsoft Exchange Server Auth Certificate
NotAfter : 2/8/2024 10:55:49 AM
NotBefore : 2/8/2019 10:55:49 AM
FriendlyName : Microsoft Exchange
Subject : CN=pandora
CertificateDomains : {pandora, pandora.domain.local}
Thumbprint : E74BADCDD7C7E3FDDF03CCA77BB3AC9Fxxxx
Services : IMAP, POP, IIS, SMTP
Issuer : CN=pandora
NotAfter : 7/31/2023 11:22:28 PM
NotBefore : 7/31/2018 11:22:28 PM
FriendlyName : WMSVC
Subject : CN=WMSvc-PANDORA
CertificateDomains : {WMSvc-PANDORA}
Thumbprint : CBA7A95F414420D91DD916AC69D4E0xxxx
Services : None
Issuer : CN=WMSvc-PANDORA
NotAfter : 7/28/2028 9:18:47 PM
NotBefore : 7/31/2018 9:18:47 PM
E:\Diagnosis>
PS>Get-MapiVirtualDirectory | fl server, Name, *URL*, *auth*
Server : PANDORA
Name : mapi (Default Web Site)
InternalUrl : https://pandora.domain.local/mapi
ExternalUrl :
IISAuthenticationMethods : {Ntlm, OAuth, Negotiate}
InternalAuthenticationMethods : {Ntlm, OAuth, Negotiate}
ExternalAuthenticationMethods : {Ntlm, OAuth, Negotiate}
E:\Diagnosis>
PS>Get-ClientAccessArray | fl
E:\Diagnosis>
PS>Get-OutlookProvider
Name Server CertPrincipalName TTL
---- ------ ----------------- ---
EXCH 1
EXPR 1
WEB 1
E:\Diagnosis>
PS>Get-ExchangeServer | fl *version*
Name: EXHG
AdminDisplayVersion : Version 15.0 (Build 516.32)
ExchangeVersion : 0.1 (8.0.535.0)
Name:: PANDORA
AdminDisplayVersion : Version 15.0 (Build 1367.3)
ExchangeVersion : 0.1 (8.0.535.0)
E:\Diagnosis>
PS>Get-OrganizationConfig | fl *mapi*
MapiHttpEnabled : False
E:\Diagnosis>
PS>Stop-Transcript
**********************
Windows PowerShell transcript end
End time: 20200708091457
**********************
**********************
Windows PowerShell transcript start
Start time: 20200708230441
Username: domain\administrator
RunAs User: domain\administrator
Machine: PANDORA (Microsoft Windows NT 6.3.9600.0)
Host Application: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -noexit -command . 'C:\Program Files\Microsoft\Exchange Server\V15\bin\RemoteExchange.ps1'; Connect-ExchangeServer -auto -ClientApplication:ManagementShell
Process ID: 11320
**********************
Transcript started, output file is queuePANDORA8July2020-11pm.txt
C:\Windows\system32>
PS>get-queue
Identity DeliveryType Status MessageCount Velocity RiskLevel OutboundIPPool NextHopDomain
-------- ------------ ------ ------------ -------- --------- -------------- -------------
pandora\Submission Undefined Ready 0 0 Normal 0 Submission
pandora\Shadow\163 ShadowRed... Ready 41 0 Normal 0 exhg.domain.local
C:\Windows\system32>
PS>Get-Queue -Identity "pandora\shadow\163" | fl
RunspaceId : 1c303a68-6bbb-496a-a66a-4431e86662d5
DeliveryType : ShadowRedundancy
NextHopDomain : exhg.domain.local
TlsDomain :
NextHopConnector : 00000000-0000-0000-0000-000000000000
Status : Ready
MessageCount : 42
LastError :
RetryCount : 0
LastRetryTime :
NextRetryTime :
FirstRetryTime :
DeferredMessageCount : 0
LockedMessageCount : 0
MessageCountsPerPriority :
DeferredMessageCountsPerPriority :
RiskLevel : Normal
OutboundIPPool : 0
NextHopCategory : Internal
IncomingRate : 0
OutgoingRate : 0
Velocity : 0
OverrideSource :
QueueIdentity : pandora\Shadow\163
PriorityDescriptions : {High, Normal, Low, None}
Identity : pandora\Shadow\163
IsValid : True
ObjectState : New
C:\Windows\system32>
PS>Get-ExchangeServer -Identity exhg.domain.local | Format-List
RunspaceId : 1c303a68-6bbb-496a-a66a-4431e86662d5
Name : EXHG
DataPath : C:\Program Files\Microsoft\Exchange Server\V15\Mailbox
Domain : domain.local
Edition : Standard
ExchangeLegacyDN : /o=domain/ou=Exchange Administrative Group
(FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=EXHG
ExchangeLegacyServerRole : 0
Fqdn : exhg.domain.local
CustomerFeedbackEnabled : True
InternetWebProxy :
IsHubTransportServer : True
IsClientAccessServer : True
IsExchange2007OrLater : True
IsEdgeServer : False
IsMailboxServer : True
IsE14OrLater : True
IsE15OrLater : True
IsProvisionedServer : False
IsUnifiedMessagingServer : True
IsFrontendTransportServer : True
NetworkAddress : {ncacn_vns_spp:EXHG, netbios:EXHG, ncacn_np:EXHG, ncacn_spx:EXHG,
ncacn_ip_tcp:exhg.domain.local, ncalrpc:EXHG}
OrganizationalUnit : domain.local/EXHG
AdminDisplayVersion : Version 15.0 (Build 516.32)
Site : domain.local/Configuration/Sites/Default-First-Site-Name
ServerRole : Mailbox, ClientAccess
ErrorReportingEnabled :
StaticDomainControllers : {}
StaticGlobalCatalogs : {}
StaticConfigDomainController :
StaticExcludedDomainControllers : {}
MonitoringGroup :
CurrentDomainControllers : {}
CurrentGlobalCatalogs : {}
CurrentConfigDomainController :
ProductID : 02064-004-0074037-02498
IsExchangeTrialEdition : False
IsExpiredExchangeTrialEdition : False
MailboxProvisioningAttributes :
RemainingTrialPeriod : 00:00:00
Identity : EXHG
IsValid : True
ExchangeVersion : 0.1 (8.0.535.0)
DistinguishedName : CN=EXHG,CN=Servers,CN=Exchange Administrative Group
(FYDIBOHF23SPDLT),CN=Administrative Groups,CN=domain,CN=Microsoft
Exchange,CN=Services,CN=Configuration,DC=domain,DC=local
Guid : b4997e56-2290-4904-8247-7f6f31f339d3
ObjectCategory : domain.local/Configuration/Schema/ms-Exch-Exchange-Server
ObjectClass : {top, server, msExchExchangeServer}
WhenChanged : 11/4/2019 10:37:21 AM
WhenCreated : 3/25/2014 2:42:55 PM
WhenChangedUTC : 11/4/2019 2:37:21 AM
WhenCreatedUTC : 3/25/2014 6:42:55 AM
OrganizationId :
Id : EXHG
OriginatingServer : pandora.domain.local
ObjectState : Unchanged
C:\Windows\system32>
PS>Stop-transcript
**********************
Windows PowerShell transcript end
End time: 20200708230611
**********************