Receive connector - Allow only one account

I need to create a recieve connector which only allows one domain account the ability to relay externally. The account will authenticate from many different IP's

I created the connector. If I allow "Exchange Users" in the Permissions Group I am able to relay as that account, including other domain accounts (as expected).

I tried setting security on the connector Get-ReceiveConnector "EX13CA\External-Relay" | Add-ADPermission -User "DOMAIN\AllowRelayAccount" -ExtenddRights "ms-Exch-SMTP-Accept-Any-Recipient". I then removed the group permission for "Exchange Users". I tried sending mail as AllowRelayAccount and I get  "The Server Response was not available".

How can I restrict this connector to just a single account?

Thanks