Thanks to those folks who have been in touch today and last Friday, letting me know that Gmail was flagging my daily “GCHQ” newsletter as suspicious.
Not everyone appears to have experienced the problem, but some subscribers have reported that Gmail put up a big red banner at the top of the email, warning that “it contains content that’s typically used to steal personal information”.
That may have resulted in some of you believing that my mailing list had been compromised, or I had sent out a dodgy link.
Fortunately neither of those scenarios are the truth. :)
Instead, what I believe is happening is that Google’s email filters are false alarming on a link to a story I wrote – for the Bitdefender Hot for Security blog – about the conviction of a man who hacked a lottery in an attempt to win $14.3 million.
I suspect in Gmail’s eyes the combination of words “lottery”, “win $14.3 million”, etc.. were too much for it to take, and it got its knickers in a twist and decided to err on the side of caution, warning readers that there might be bad stuff in my newsletter.
If you received the warning from Gmail, please help it learn that my newsletter is okay by clicking the “Ignore, I trust this message” link. Thanks!
Interestingly, I’ve received similar reports that the newsletter of my former stomping ground Sophos Naked Security was similarly mislabelled by Gmail – presumably because they were also talking about the lottery story.
In Naked Security’s case it may have been worse than I experienced, as it looks like the emails may actually have been tagged as spam – which isn’t (as far as I know) a fate which befell my newsletter.
I guess we’ve all learnt to be more careful when we write headlines about lottery security now…
Oh, and if you haven’t already done so and feel like you’re missing out on all the fun, sign up for my free GCHQ Newsletter today. ;-)
I've had this happen with another company from which I regularly receive genuine mails. I'm beta testing some hardware for them and I've had to set up a bunch of Gmail rules to make sure I don't miss out on communication from them and from others on the beta mail list. I've also seen this happen with one or two other people I know personally but who regularly send out group mails. I don't know whether Gmail anti-spam is just getting over sensitive, or whether it's just annoyed recipients flagging up mails as spam, as I sometimes do, as a last resort (or lazy alternative) after failing to unsubscribe.
My gripe with Gmail;
I host my own mail server. With SPF, DKIM, DMARC implemented and configured correctly, Gmail still holds prejudice against my host.
Not because it's a spam bot, mis-configured or compromised, no, Google simply doesn't like the static IP range provided by my reputable, well respected ISP here in Australia.
Privately hosted mail servers accepts my emails without any problems. I don't sell, promote anything nor do I use profanity or compose suspicious emails either.
Once a user moves my initial correspondence from Spam to Inbox and adds me as a contact, it's fine for only that single recipient from there on.
Opening two support cases with Google resulted a zero help or acknowledgement. At least my ISP made an attempt to assist.
Thanks for nothing Google.
Surely google checks against RBLs ? You could find out which they check and see if there is a problem with them.
Alternatively, and I think google does this (I don't have the problem though, because it doesn't apply to my configuration), it is due to invalid or no PTR records in DNS. I've seen this happen before, and some mail servers do indeed check for matching PTR records to A/AAAA records.
Then again, if I understand you right, the mail arrives, only that it is flagged as potential spam. If that is the case, perhaps it is something else entirely and if anything will help, full headers will. But that is google, for you. There isn't much positive that can be said about them.
Hi Coyote, thanks for your reply.
When investigating the issue at the time, I didn't appear to be listed on the top 100 RBL's.
One of the first things I asked my ISP was to create a PTR to the mail host. My primary/reverse zones have also been configured correctly. I don't permit recursion or transfers of my zones either.
IIRC, initially I was using my ISP as a Smart Host and wasn't experiencing this problem. The ISP's SMTP uses a different IP range.
Mind you, my host was using/propagating a dynamic IP (yep, you read right). Routing via the SMTP relay, emails were delivered to the users Inbox without any issue. No SPF, DKIM or DMARC was in-place at the time…
Granted, the IP block that my provider uses for static addresses have probably ran a-muck/abused by previous subscribers and Google/Hotmail simply hold prejudice because of this perhaps…
Interestingly though, Yahoo accepted emails with/without the use of a Smart Host.
To resolve the issue, I could;
– Roll the dice and request another static IP, re-propagate my DNS records and see if it makes things better/worse
– Go back to using my ISP as a Smart Host, or
– See whether I can subscribe for a totally different static IP (from APNIC) and have my ISP host it.
I'm not sure if the latter is conducive and/or whether it'll resolve the issue either… The 2nd option is the easiest to implement.
Ah the joys of hosting your own mail.
Without seeing your zones I of course can't offer too much on DNS, and I wouldn't dream of asking you to show it (and I see you understand this but if you were to offer it I would say absolutely not). But you might consider one of the many websites that do DNS checks (including PTR<->A/AAAA records, MX records, whether the hosts can be contacted, etc.). http://mxtoolbox.com is one of them. I know some also check DNSSEC. Actually, I'm not sure that mxtoolbox does check PTR<->A/AAAA records but it does check several things anyway.
Whether a new block would help – who knows. Probably not worth the effort given that you do have to reconfigure DNS (and other services, even firewalls and the like).
I think the last option would be pricey (in addition to the above). Maybe I'm remembering wrong but I think that costs a lot, to obtain your own block (if you can even get a block ?). Do you have the problem with IPv6 (or do you not have IPv6) ?
The question is ultimately this, I think: the mail arrives at gmail, only that it is marked as spam, is that right ? If it is, then it obviously isn't a black list, and so something phishy is going on there. You might consider debug logging in your MTA, though, to get as much information as you can on the SMTP session (and check all headers at the recipient; also might consider telnet to your mail server to send mail to gmail so you can specify every last detail you want). I guess google wouldn't be nice enough to you to show you more than that, which does make it difficult to diagnose, but those are the immediate thoughts I have. I don't have any problems on my end but I consider myself lucky when it comes to Google.
Thanks again for your reply Coyote.
Not putting anything to chance, I had an independent inspect my DNS, no fault was found.
mxtoolbox is great I've used in the past. Running it before and now gave me all green ticks again.
DNSSEC isn't something I've done yet. I guess I could look into it, though you'd assume SPF, DKIM coupled with DMARC would be enough?
Yep, buying a IP block isn't something I'd wish to pursue. I'd only need one IP, I'm not sure whether you can buy just a single IP. I don't use IPv6 and my WAN gear is only set up to use IPv4.
New recipients of Gmail/Hotmail, my email is delivered to their Spam/Junk folder. . Gmail users must add me as a contact and also move the email into Inbox. Hotmail users just need to move it out of Junk. Once this is done, ongoing correspondences are delivered to the Inbox.
When lodging two support requests to Google, they asked for a list of telemetry, ie: Email headers, telnet verbose, mail logs, etc… as you've mentioned. Twice I submitted these. I got no reply/acknowledgement that someone was looking into the matter.
Microsoft asked a series of questions and also wanted headers, but constantly palmed off the problem to their SmartScreen filter, which (they said) works on logic/self analytics and cannot be manually overwritten to make exceptions or whitelist MTA's, IP's, etc…
For a sanity check, I had two others cluey on the subject inspect my set up. This anomaly also had them stumped. The consensus was to just route via a Smart Host or use an alternative IP.
As my contact list isn't that dynamic, this issue is not a major problem. It's only an issue when I have to email someone that has their email hosted with Gmail, Google or Hotmail. I then have a 'face palm' moment, get onto the phone to explain what to do or use an alternative email account…
It's no problem (responding). Configuring MTAs to have no problems can be difficult exactly because of all the problems and variables encountered (and we have spammers, scammers and the like, to blame for this). I'm guilty of aggressive filtering but I'm not a provider, so I don't have the same problem as others.
DNSSEC isn't really relevant to this; I was only mentioning it because there are some websites that also do checks specific to DNSSEC.
I would say you couldn't buy a single IP from such an organisation but whether you can from your ISP is another matter entirely. I used to have a single static IP (different service) but nowadays I have a /29 (and a /60 for IPv6). If reconfiguring services for a different IP isn't a problem, I suppose you could try a new one, but whether it will solve the problem is hard to know (see below). When you refer to a 'Smart Host' I assume you're talking about a mail relay ? And similar for alternative IP (or are they suggesting getting a different IP)? It does seem odd though, that it would work that way (adds to the number of possible sources of the problem).
The fact the recipients do get the mail, only in spam, makes me think it isn't your end. Without knowing the content of your mails, I of course cannot judge there, but clearly it isn't an RBL at play here. I do know there have been times where google has made changes that flag certain types of mail (maybe by host or content; not sure) as spam, but then once it is accepted by the user, all is OK.
At times, using a gmail account of my own (for those who don't need to know about the mailboxes under my domains), I have had the mail vanish. I don't think it was discarded but I can't be certain: it was to a mailman list, and the list itself had the mails in the archive, but the members with gmail accounts did not receive the mails (and they weren't in inbox, weren't in spam, they weren't there at all). Then they suddenly started arriving (additional mails; the lost ones were lost). The best I could figure it out, is that I was sending mail from a normal computer (not an MTA, in other words) but sending it through my dynamic IP. As soon as I made some slight changes (ending up in sending mail via one of my static IPs with a proper PTR RR), it seemed OK. But I'm not convinced that was the problem because how do others without static IPs manage it?
I have a suspicion (rather strong one) that the problem isn't at your end, but happens to be that Google (and the others you mention) have certain filters or rules of some kind, that find the mail (whether headers or body) as phishy (perhaps not in the sense of phishing but phishy as in fishy). What can be done for that, who knows. I'm not surprised with MS's response. I'm also not surprised with the lack of response from Google. Not being able to solve the problem can be infuriating but if it is out of your control you can go mad (if you aren't already, like I am) trying to solve [it].
"It's only an issue when I have to email someone that has their email hosted with Gmail, Google or Hotmail."
By hosted you mean they have a gmail or hotmail account, right ? Or are these domains that google (say) hosts (mail included) ? (Not sure how that would work with hotmail but maybe they have something like that, too ?)
As for telnet, I was thinking more like if you could experiment to see if you can find out what is triggering the flagging of (false) spam (try different combinations of headers and so on). Might not be useful though. Does your MTA logs show anything ? Those are the thoughts that come to mind at this time.
I noticed the same thing with your newsletter and I told Gmail to ignore the warning. Only time that has happened. However, I cannot remember if it was this e-mail or one from Sophos that did end up in my spam folder. Since you mentioned that it was happening to them maybe that's what I experienced too. I quickly reminded Gmail that the message was indeed not spam and since then I have been good.
Your theory is sound, Graham. I see similar in Thunderbird from time to time, with logwatch (daily logs in email will suffice for here, even though it isn't as simple as dumping logs in to a mail). Some times it is spam, sometimes it is something else. At times clamav has been triggered and the mail goes in quarantine (therefore not being in the mail queue). This happens despite the fact the sender address (in amavis) is set to a negative score (which means it should be ham).
All of this because of some idiot(s) attempting an (or multiple) exploit(s) or perhaps scanning and therefore being logged. Then, because the logs include this, it is in the mail, which triggers the warnings.
I'm running into the same problem. I see some of you contacted Google. How do you go about this? Running into a wall on how/where to contact. I've only posted on the forum (without response).
What I did, and I'm still doing some tests – add SPF record for MailChimp and verified my domain (authenticated with them) at MailChimp. I thought it would be fixed, but my stats for Open Rate are still down which indicates.. nope, it's still happening.
Thanks!
hank
I'll see if I can shed some light on SPF, albeit only in that it isn't the answer (though really, the best would be to check the RFCs; rfc 7208 and it is updated by RFC 7372: http://tools.ietf.org/html/rfc7208 and http://tools.ietf.org/html/rfc7372 ).
SPF is – as the name suggests – a policy. But that means it can be ignored by those who check (if it is checked at all). Google, afair, does check SPF but I don't think they reject mail if the the DNS server suggests it is fake (or invalid) [but I'm not going to update any of my zones to find out]. And there is the problem: it isn't always checked, and it isn't always relevant. By it being irrelevant, I mean that some admins have a neutral policy (for one example) and so there isn't much to do with it; it fails, but how to act? If there is no policy then it is also irrelevant. There is also soft-failure which allows for it to fail but ignore the failure (this might actually be 'neutral' – I can't recall specifically because I use hard fail).
(The above is simplified, maybe too much so, but hopefully you get the idea)
In the end, the problem is in this case, Google. Email is unfortunately very complicated these days, all because of spammers (and other vermin). It used to be much simpler and I wish it was still that way (for many reasons) but that is unrealistic. In some ways it is better (more servers check for authentication before accepting mail to send from anyone, for example) but spam filters and anything similar is complicated and there is nothing that can solve all of the problems of mail server administrators or even users of email.
Hi again Coyote,
To clarify, when I said Smart Host I was referring to a Mail Relay. IIRC mail was delivered fine when my host was traversing through my local ISP's, rather than my host being the sole MTA.
I agree with your point about emails delivered to Spam. If their was something afoul, methinks they wouldn't be accepted to begin with.
It has to be their (Microsoft/Google) mail filters holding prejudice towards the subnet I'm on that is the crux of the issue. I've done all that was advised (by Microsoft), SPF (which I had initialled), DKIM, DMARC, even composed a legitimate business-like signature block with a unsubscribe link, contact us, etc… in my thorough testing… It made no difference.
Interesting that you've had emails in your Gmail account just vanish… If they've been delivered and no user interaction is to blame, I'd question the account activity or perhaps a view /sort issue as silly as that may sound. Probably enable TFA also to rule out unauthorized access…
Thus far, the problematic recipients are those who use have their mail hosted by Microsoft (Hotmail, Live, etc…) and Google (Gmail and private domains).
From memory, both the telnet verbose and MTA logs didn't seem to flag anything unusual messages or show any alarming.
Having said that, I'm not specialist in this space, but a few independent eyes claim I've done what's required to build a satisfactory/trustworthy MTA…
If the issue gets the better of me, I'm just route though my ISP's MTA, impeding the growth of my grey hair…
Cheers
I completely forgot about this until now.
Right. I thought that is what you meant – a mail relay. It is interesting that that works because it raises the question of if there is a problem with your IP (whether you can control it or not). Maybe some lists think it is dynamic (or something else that isn't quite right) and so it is marked as spam? (though I have only seen reject sending in the first place, at least without the recipient mail server intervening). It is hard to know whether another IP would help because it is hard to know whether it is Google and Microsoft (I cannot really answer that without bias so I won't even try). If you could find out how they configure their servers, and how they are similar, it might help. But then again it might not.
As for gmail, it was actually sending from one of my gmail accounts (that I use when dealing with others that also use gmail; I really do not trust google. at all). I wouldn't have mail servers configured with dynamic IPs unless I had no choice (and would have to deal with the risk of mail being rejected). Verbose logs.. but what about debugging logs (maybe same thing in your mind or your mail server) ? It probably won't show much, though, at least as far as it revealing what is wrong.
If only spam could be eradicated… we wouldn't have so many complications. But unfortunately they are ruthless vermin that spread like wildfires do around here (desert). Good luck with the issue. I know you'll likely need it.