Content management system Drupal has issued a chilling public service announcement to website admins and internet users who might visit the hundreds of thousands of sites running its software.
According to the company, “automated attacks” started to hit websites running Drupal version 7 within a matter of hours of it disclosing a highly critical SQL injection vulnerability on October 15th.
Automated attacks began compromising Drupal 7 websites that were not patched or updated to Drupal 7.32 within hours of the announcement of SA-CORE-2014-005 – Drupal core – SQL injection. You should proceed under the assumption that every Drupal 7 website was compromised unless updated or patched before Oct 15th, 11pm UTC, that is 7 hours after the announcement.
If a site using a vulnerable version of the Drupal CMS is attacked, hackers could steal information from the site or open backdoors to allow them continued remote access to the system.
This is a big problem. Because if you *now* update your website to Drupal 7.32 (which doesn’t suffer from the vulnerability) that won’t get rid of any backdoor that the hackers may have already implanted into your system.
Furthermore (and you may feel like banging your head against a wall at this point), Drupal says that some attackers may have actually applied the patch for website admins to keep out rival hackers, and potentially throw IT teams off the scent that their site has been compromised.
Updating to version 7.32 or applying the patch fixes the vulnerability but does not fix an already compromised website. If you find that your site is already patched but you didn’t do it, that can be a symptom that the site was compromised – some attacks have applied the patch as a way to guarantee they are the only attacker in control of the site.
What a unholy mess.
Of course, determining if there is a backdoor on your website isn’t going to be easy.
Clearly it’s important that vulnerable sites update to the latest version of Drupal as soon as possible, but as that’s no guarantee that they might not still have been compromised in the last couple of weeks more drastic action is required.
Drupal’s advice will be a hard pill for some sites to follow, but should mean that you can feel confident that your site is not affected.
In a nutshell, if your site wasn’t protected within a few hours of Drupal’s announcement on October 15th, you need to restore it from an old backup or rebuild it from the ground up.
The Drupal security team recommends that you consult with your hosting provider. If they did not patch Drupal for you or otherwise block the SQL injection attacks within hours of the announcement of Oct 15th, 4pm UTC, restore your website to a backup from before 15 October 2014:
- Take the website offline by replacing it with a static HTML page
- Notify the server’s administrator emphasizing that other sites or applications hosted on the same server might have been compromised via a backdoor installed by the initial attack
- Consider obtaining a new server, or otherwise remove all the website’s files and database from the server. (Keep a copy safe for later analysis.)
- Restore the website (Drupal files, uploaded files and database) from backups from before 15 October 2014 Update or patch the restored Drupal core code
- Put the restored and patched/updated website back online
- Manually redo any desired changes made to the website since the date of the restored backup
- Audit anything merged from the compromised website, such as custom code, configuration, files or other artifacts, to confirm they are correct and have not been tampered with.
While recovery without restoring from backup may be possible, this is not advised because backdoors can be extremely difficult to find. The recommendation is to restore from backup or rebuild from scratch.
Drupal is, of course, used by hundreds of thousands of websites across the internet – including some very popular ones.
Ilia Kolochenko, CEO of High-Tech Bridge, offered these thoughts about why hackers are so keen to exploit website vulnerabilities such as this one which has impacted Drupal users:
“As soon as a vulnerability in popular CMS platforms like Drupal is discovered, millions of crawlers operated by hackers (similar to Google bots) start searching for vulnerable websites. Once a victim is identified, their website gets hacked, patched (to prevent ‘competition’ to overtake the same site) and backdoored,” explained Kolochenko.
“Within several days, access to the compromised website will be sold on the darknet market, more than likely to several different customers at the same time who each may well resell it several more times,” Kolochenko continued. “Like this, your personal blog may be easily involved in a dozen different criminal offences such as hosting illicit content, sending spam and infecting visitors, to name just a few.”
Don’t dilly-dally. If your website uses Drupal, and you are responsible for maintaining its security, check that you are running the latest version and determine if you may have patched too late to prevent possible compromise.
You may also want to check with your boss whether you’ll get paid for working overtime. After all, they wouldn’t want you to down tools at 6pm, right?
Further reading:
- Drupal Core – Highly Critical – Public Service announcement – PSA-2014-003.
- Your Drupal website got hacked, now what.
I suggest patching then running a basic command in htdocs:
find ./ -mtime -30 -print
this will give you a list of all modified files in the last 30 days, narrowing files that have to be audited.
Next check your users table for modifications, or elevated permissions.
This alleviated most issues of file based backdoors (often used) and administrative login, unfortunately it doesn't touch the issues of modified DB based HTML content leaving XSS vulnerabilities in place. A simple DB restore from before the 15th would be the easiest option if possible. Else get auditing, unless anyone knows how to query mysql for modified content markers without relying on fields updated by Drupal.
(Yes, this will be a long post. If nothing else, check the example I give below on just how useful modification time is not… indeed you can change modification time forward, backward (days, weeks, ..) and that is without any modification. Consider what it can conceal: modified and then changed to be the modification time of other files.)
I would advise rather serious caution on your invocation of find as a way to determine what was tampered with. Mind, don't take this as anything but what I offer which is that modification time is, ironically enough, not indicative of any modification at all.
– Patching versus update entirely. Do they patch? (Don't know but it is relevant) If they do there is even worse of an issue (only part of file modified? You never know). Also is the reverse: is the file clean now but maybe wasn't?
– You can update files modification time fairly easily – you can even create (or modify files) to be in the past or the future. Combine that with "modification" time doesn't mean it was or wasn't modified and you essentially have the truth that modification time is not safe at all to see what was or wasn't tampered with. I'll give an example below on my build (programming) computer showing how easy it is. Even clock drifting can make some utilities (that make – pardon a hidden pun right there – use of timestamps) think something was updated when it actually wasn't touched at all. Yes, I've had exactly that problem: clock drift causing me having to rebuild (read: compile) some source code that already had the proper objects built.
Relying on modification time as a starting point of tamper evidence, is risky business. Better would be to have file verification (of course, after the fact does little). And consider that if they compromise your website you should (like any other service) assume they also compromised something else (or intend to… including indeed what I just suggested!)[1]. And while the administrators might hate it, the general rule is that the only way you clean up a compromised system is starting all over[2] and doing things right (in this case having sql filtering as default and not rely on, e.g., drupal – to do it for you, as one additional thing).
In summary, the points given by Drupal:
[1,2]"While recovery without restoring from backup may be possible, this is not advised because backdoors can be extremely difficult to find. The recommendation is to restore from backup or rebuild from scratch. "
[1,2]"Consider obtaining a new server, or otherwise remove all the website’s files and database from the server. (Keep a copy safe for later analysis.)"
[1,2]"Notify the server’s administrator emphasizing that other sites or applications hosted on the same server might have been compromised via a backdoor installed by the initial attack."
and as Graham offers:
"In a nutshell, if your site wasn’t protected within a few hours of Drupal’s announcement on October 15th, you need to restore it from an old backup or rebuild it from the ground up."
Example of modification time fun (and keep in mind you can do this to files created, you can also modify access time and… you can stop it from creating files, etc.):
$ touch -m -d "+10 years" mtime
$ ls -l mtime
-rw-rw-r– 1 coyote coyote 0 Oct 31 2024 mtime
$ touch -m -d "-10 years" mtime
$ ls -l mtime
-rw-rw-r– 1 coyote coyote 0 Oct 31 2004 mtime
But look it:
$ date
Fri Oct 31 16:30:24 PDT 2014
(As an aside: there is also status change time on files but I would strongly recommend against relying on that, too, even if only because it doesn't show WHAT changed other than it changed and it might or might not be harmless).
In the end, some will go for the easier route but I feel it is at least something I should point out: the easiest route is often the wrong route (there are so many turns, twists, mountains, lakes, … in computing)… and wrong can be any number of things (or show itself in numerous ways).
Stuart,
While this is a good trick to catch sloppy breakins it's easy to reset the created time on a file so that mtime doesn't work.
touch -d "2 weeks ago" myfile
True. That's what my post was about. But even modification time is not enough nor is change time. Even if the file is changed, how do you know what it is without a full audit? And if it is patched how do you know (without looking at the diff output) that all parts are valid? You cannot, not if you're aware of the many – pun, I know – variables.
Shortly: it is of use for some things but not much, certainly not security. mtime, ctime, atime, they're all irrelevant. Indeed, you can change the time on the files (access, change and modified all at once) simply by opening a file (e.g., in vi) and then saving it, as if you just edited it (even if you changed nothing). Similarly is that removing a space by itself will be picked up by diff and you can indeed patch the file – but it isn't much value, other than to sync up to the repository (say).
"Drupal says that some attackers may have actually applied the patch for website admins to keep out rival hackers, and potentially throw IT teams off the scent that their site has been compromised."
I'm reminded of some worms doing this… Maybe in the CodeRed (or close-by) era? Do I have the name right? It seems right but at the same time it seems somehow wrong… (brain-dead I fear) Either way, one worm worming their way in to a system to fix another worm (or some other malware). The difference here, of course, is that there is twist that is somewhat insidious: instead of removing it outright (although I seem to remember the worm in question caused harm indirectly if not otherwise) it only keeps others out and appears fixed (so essentially as you put it, throwing the IT team off). Or that's how I interpret the above. In which case, nasty yet not at all surprising, is it? I daresay this isn't all that ingenious and not even close to it. It is just misdirection which is much older than all of us…
I also seem to remember that there was a Unix worm that did similar only in the case I'm thinking of, it fixed a lot of bad configurations and at the same time made it very obvious there was actually things that were changed (including fixed)… because while fixing it it would also deface the websites on the server. Ramen worm maybe? Maybe it was two different worms… either way, it is always interesting when this type of thing happens. Some of it is a pretty clever in design but in the end, those who are compromised are generally not going to be impressed.