So far I have been unable to track down the cause of this problem.
The problem that is happening is the Exchange queues are building on the Exchange servers. I have 2 Exchange 2013 SP1 servers that are clustered together using DAGs, and Windows Clustering. I'm not really sure when the problem started, or if any changes had been made to the environment.
I have a client who is using Google Apps as a Smart host, but in the Hub->Send protocol logs I keep receiving this error over and over:
"Failed to connect. Winsock error code: 10051, Win32 error code: 10051, Error Message: A socket operation was attempted to an unreachable network [2607:f8b0:400e:c01::1c]:25"
From what I can tell the problem is that it is resolving the Smart host (smtp-relay.gmail.com) to it's IPv6 address rather than it's IPv4 address which is generating a socket error.
On the Exchange server itself if I do a nslookup or a ping it always resolves to the IPv4 address, but for some reason the queue keeps trying IPv6. At one point I unchecked IPv6 on the network cards (yes I know this is a bad thing, but I was testing) and I still had the exact same problem. I have not gone as far as trying to disable IPv6 in the registry yet.
On the send connector itself I have tried both the "Use the external DNS lookup settings" and using the Internal DNS. On the server itself in ECP I configured the external DNS to 8.8.8.8 and 8.8.4.4. But I still have the exact same issue.
The one lone send connector is setup to "Route mail through smart hosts" with smtp-relay.gmail.com as the Smart host and using "None" as the authentication. The scoping is set as SMTP * 1
I monitored the Firewall (Watchguard) and verified that SMTP traffic is flowing out using the IPv4 addresses, so the Firewall isn't blocking IPv4 SMTP traffic, but there is a chance that it is blocking IPv6 addresses. But then again I don't have any external IPv6 addresses that I could NAT to externally anyways.
I have not tried to disable the Smart host yet to see if mail will flow to the Internet without issue because all of my SPF records are pointing to Google.
I am unsure at this point what to try next. I have been debating about hard coding an IP address in the LMHosts for smtp-realy.gmail.com to see if that resolves the problem, but that would not be a permanent fix.
Any help would be greatly appreciated.