Read this first and see if it is applicable: http://support.microsoft.com/kb/q196271
What mail server do you use? We use the free hMailServer for SMTP and find it very stable. Can you also also enable logging in your mail server to see what that says? It it doesn't have logging, consider changing mail server as logging is essential to diagnose email-related issues.
I don't think it is the CF Admin settings, as they are minimal. Can the CF Admin connect to the server? There is a test checkbox in the SMTP settings of the CF Admin. Also check the mail queue and do a test with a single email first.