Cleanup Major Research Site Hack

Reported issues:

The site owners shared that in attempting to load either website:

» Initially, strange displays of garbled code symbols and broken layouts (reported in screenshots) on the front-end site pages
» Google warnings of probable hacks and dangers with visit days later
» Emails not delivering and bouncing back
» Hosting by GoDaddy Level 2 Plan;
» Numerous emails about exceeding resources and Terms of Use.
» GoDaddy throttled but did not block.

Our involvement begins:

We agreed that an Initial Corrective Audit was appropriate. Given near certainty based on the communications received as above that they were hacked. We shared their options for a hack cleanup with a dedicated service such as Sucuri or to proceed with Creative by Clark. They promised to enter into an annual Maintenance Care Plan if satisfied with our services.

• Log in credentials were received for their GoDaddy hosting account.
• We requested Google credentials for their Analytics and Search Console (formerly known as Webmaster Tools) account.
• No separate credentials were required for their domain/DNS management as on the same GoDaddy account
• We secure a list of authorized WordPress Usernames, especially those who have Administrator roles, and learn that we should not have more than two users who are administrators.

Log in credentials were received for two separate WordPress installs:

• The primary domain using www.abcdef.com in the English language
• Second, using a subdomain esp.abcdef.com in the Spanish language.

Technical Process:

To alleviate the risk to our computer system, we avoided loading the website to include logging in to the WordPress admin until fully assessed.

• We ran an initial Securi Site Check scan of the two websites and both were identified as infected.
• MX Toolbox reported the domain as blacklisted, indicating the reason for mail failures.
• Google search results on the domains indicated the website hacked pronouncement and sites/domains were blacklisted and considered dangerous. Sites were removed from the Google index and likely the same with other search engines.

Locate the infection:

The Sucuri Security Scan returned some information, including some of the infection payloads and locations

We began by accessing the hosting account and cPanel

• SFTP:  We created our own FTP credentials to use via SFTP with the two directories
• We log in with our new credentials via SFTP and create a backup of the two sites.
• We verified the SSL Certificates installation
• Examine server error logs
• GoDaddy had not blocked access to the File Manager – so with FTP in our hip pocket, we started here for our initial file inspection.
• We immediately noticed folders that do not belong in the root directory (and esp folder).
• We found 21 folders (19 in esp) containing hundreds of malicious HTML files and images directed to porn websites.  In addition to malicious code buried elsewhere in the sites, it is not uncommon for these files to include malware, including infected images.
• These folders were permanently deleted from the sites.

While many people might know to look at physical files that don’t belong or might include malicious code, this is not the end.

As the site is already essentially offline, we decided to remove all the WordPress core files.  It’s not going to impact anything at this point.

• We renamed /wp-content to /wp-content-salvage and zip compressed
• We downloaded wp-config.php, .htaccess, robots.txt, and the various google verification files after inspecting each for malicious code.
• We make a recorded note of the following information for each website in wp-config.php:  Database name, Database user, Database password, and the Table Prefix.
• We deleted all root WordPress PHP files and the wp-admin and wp-includes folder.
• Replace WordPress core by downloading a brand-new original copy from wordpress.org and upload…
• We add new files in wp-content (themes) to the wp-content-salvage folder
• We will need to examine plugins, themes, and other folders, but that is on hold for now.  We’ll make note of recently modified files.
• Delete the wp-content folder that came with the new WordPress and rename wp-content-salvage back to wp-content.  For now, keep the zip archive as a backup in case we accidentally delete something necessary in our further efforts.

Dealing with the Database!

As WordPress is dynamic, we must clean the database of any bad URLs or code.  Many only looking for traditional infected files miss the database connection and have a false sense of security.

• We return to cPanel to access phpMyAdmin
• As we know the precise databases in use, we can locate the database and correct table prefix actually in use on each site.
• We export all tables to a single SQL file and repeat to export to CSV
• We installed a piece of software called PHPMinAdmin (now it’s called Adminer), and this will help us look through the many tables for bad URLs and specific snippets of code that are generally known to be malicious.

Once the database is “cleaned, ” we are ready to look at the Users.  Likely there will be at least one unauthorized administrator user added to the site — whether added by a human or a bot.  While we can delete them directly from database tables, after we verify the usernames will need to be removed, we will wait until we access WordPress admin for the deletion.

First Test of Functionality on the English version using the primary domain www.abcdefg.com

(We repeat all processes documented below two times to handle the second WordPress site in the esp.abdefg.com for the Spanish language. As the contents of wp-content will be the same, we’ll discuss a shortcut below.)

After all these changes, we will check to see if we can access the WordPress admin.

Yes… Successful.

We’ll install BackupBuddy and run a set of backups (database, plugins, themes, media, & complete). We’ll delete these backups and run clean sets once we’ve got everything remediated.

Deal with the User Situation

• We create a new administrator username for ourselves.  We log out and log in with our credentials.
• Delete the bad users and all their content.  Hopefully, there won’t be too much more if we manage to get a good clean on the database.  When prompted by WordPress to assign that content to another user, we just have it deleted.
• We force change passwords of the remaining users to strong passwords.  Later we’ll send them reset links.

We install two more plugins:

• We install the File Manager plugin, which will provide us similar powers as working with the cPanel version of File Manager.
• We install WordFence Security — we won’t activate it quite yet.

Plugins & Themes:

• Examine Dashboard | Updates for the list of Plugins and Themes with new versions.  Don’t take action yet — this is just an overview.
• We prefer going to Plugins and displaying the list of all plugins – active and inactive and printing this entire list as it displays installed version numbers. We can see those plugins that have an update.
• We then use the plugin list as a check against published lists of plugins that have current security vulnerabilities in the installed versions and if the issue is resolved in a patched update.
• We focus initially on outdated plugins.
• The process is slow and tedious, but these actions will ensure we remove anything potentially infected.
• We subscribe to developer resources that keep us informed about plugins and themes with security threats. We search those databases for plugins by name.  Most plugins installed are free and can be further researched at the WordPress Plugin Repository.
• For safety, we download clean versions of all free plugins after assessing that these are stable even at the latest version.
• We found only one premium plugin installed:  Gravity Forms, and voila, we found that what was installed was quite old and unlicensed.  And the version installed had a known XSS vulnerability.  Could this be the pathway used for the hack?  We own a developer’s elite license of Gravity Forms. We download a copy of the latest version to have on hand.

What is Cross-Site Scripting (XSS)

WordPress provides a range of functions that help plugin developers ensure that user-supplied data is safe. Trusted users, that is Administrators on a single WordPress installation, site administrators in WordPress Multisite, and users with the permission to edit existing content, can publish unfiltered HTML or JavaScript as they need to, such as inside a page, post, or in Gravity forms: form confirmation or field label. Untrusted users and user-submitted content is filtered by default to remove dangerous entities by escaping output and stripping tags as appropriate depending on the context. When the plugin is not updated to enable protections after an XSS vulnerability is known in a version, that can become the conduit for a hack.

We cannot be 100% certain that the very outdated and known-to-be XSS vulnerable version of Gravity Forms, but strongly suspect this was indeed the vehicle for this massive exploit.

Our approach is conservative.  We’re not taking chances with anything old remaining that might have infected code.

Let the deletion and replacement begin.

We delete the physical plugin folders one at a time using the File Manager plugin.  After we delete, we do an Add New to upload the latest version.  Data from the earlier configuration will be in the database using this method rather than risking any table deletions upon deleting a plugin from the plugin page.

We triage any of the plugins that have outdated versioning but only after we verify that the version from the WordPress Repository is well-supported and show no issues that might indicate a security risk.  We replaced a couple of items with other plugins that we felt were not being updated with any frequency.  For those plugins that we deleted permanently, we went into the database tables via phpMyAdmin to delete any tables.  This avoids bloat and ensures that nothing malicious is embedded in those items.

Finally, we followed this process to remove an outdated standard Gravity Forms license.  We strongly suspect that the previously vulnerable scripts were a likely entry point for malware insertion.  We connected our premium elite license, and this plugin will update easily.

On themes, we deleted extraneous themes keeping only the active Premium Parent & Child Theme installed.  We obtained login credentials to the purchase source, replaced the parent theme with the most current version, selectively installed updates on necessary bundled items, and removed others.  On default WordPress themes, it is our practice to keep only one default theme installed, and we choose the most recent theme as it is designed for maximum compatibility with the latest WordPress core version.  We permanently deleted all others; in this case, there were five old default themes and one other free theme installed.

Other Housekeeping:

Via the File Manager – either the plugin or cPanel version – delete any Cache files – these are usually located in the /wp-content/uploads folder.

Activate WordFence:

• Set up the Firewall
• Configure to enable notifications to come to our email,
• Block the usernames that the hacker used
• Block all common usernames used by bots and hackers to crack into sites to include admin, administrator, test, the domain name as a username with and without www and admin_domain domain_admin admin-domain domain-admin, etc.
• Whitelist our IP address and that of the client’s physical location

We also installed another security plugin to harden further — one that would add more security headers and prevent information leaks on the actual WordPress configuration.  We will have three tools to scan the website for changed files and malware.

Finally, from the host cPanel, we created a user database user and password with a strong password.  We then attached the new user to the database and updated the wp-config.php settings.  While editing that critical file, we changed the security salts.  Changing security sales forces logouts by modifying the session cookies.  We then log back into the WordPress dashboard to verify all is still working well.

We verify that the site is loading properly on the front end.  We’ll put this site on hold while we repeat most of the above on the Spanish language version.

Shortcut for the Spanish Language Version:

As the Spanish language version of this client’s website is identical save for the actual written content (and that is in the database), we can copy the entire wp-content folder to the esp directory.  This saves a lot of redundant processes.

• When logged in to this WordPress dashboard, plugins will be current but will need to be activated.
• Gravity Forms must be activated and configured with the premium license key.
• WordFence will have to be configured for this site separately.