Exchange Server 2013 CU1, two mailbox servers (DAG, only one mail database), two CAS servers (NLB cluster). Two Sendmail/CentOS-based SMTP relays route mail between Exchange servers and Internet (incoming mail is routed to NLB cluster name).
Usually mail flow is OK: messages are sent and received normally. However, I've got several complains from different users. They state that occasionally they don't receive messages that were sent to them from outside. Tomorrow morning I've decided to investigate the issue.
So, we have a message sent from Gmail account that was not delivered in user mailbox. It was received and routed to Exchange by Sendmail host:
Jun 27 22:06:37 MX01 sendmail[12169]: r5RG6HC8012166: to=<USER_ADDRESS>, delay=00:00:19, xdelay=00:00:19, mailer=smtp, pri=124702, relay=NLB-FQDN [NLB-IP], dsn=2.0.0, stat=Sent (<CAHgK_sjEkj41htuVsutMP3XhgR5RRDs8-EgSC-UfHoaiHWFj=A@mail.gmail.com> [InternalId=13069585481819] Queued mail for delivery)
Running Get-MessageTrackingLog applet on the mailbox servers displayed the followed log entries for the message:
DB Server #1:
22:11:18 HAREDIRECT
22:11:18 RECEIVE
22:11:18 AGENTINFO
DB Server #2:
22:11:18 HARECEIVE
22:12:44 HADISCARD
So, we can see that the message was placed into shadow queues, but was NOT delivered into mailbox (no DELIVER status entries). Also pay attention to the timing. Time on all the servers is in sync, but Sendmail timestamp is 22:06, and Exchange timestamps are 5 minutes behind it.
Today a message sent to the same address from the same Google mailbox was received by Exchange and placed into user mailbox with no glitches.
We use Exchange antispam system activated on Exchange mailbox servers. Spam is never dropped silently. It is either returned to sender (SCL 9) or placed into quarantine mailbox (SCL 6 and above). I'm pretty sure that the lost message was not in the quarantine mailbox this morning, and the sender didn't receive any NDRs.
Any ideas what to investigate next?