Googlebot Cannot Access CSS and JS Files
Beginning on 7/27/15 Google started sending email notifications to site managers via the Google Search Console (Webmaster Tools). These emails stated that:
Fetch as Google/Fetch and Render
In Google Search Console under Crawl → Fetch as Google are tools allowing you to see what code Googlebot fetches for a page, how it renders for the bot and finally if any resources were missing or unable to be retrieved.
In the case of one site that received the email notification, Google listed only two images from a WordPress theme as ‘Temporarily Unreachable’.
In a second case, four external scripts were listed as blocked. One from createsend.com (Campaign Monitor), two from doubleclick.net and one from google.com/uds.
A third case showed two CSS files in the WordPress theme folder and a similar .less file were ‘Temporarily Unreachable’.
A fourth case showed one .less file in WordPress theme folder and a dozen images (2x Retina images) as ‘Temporarily Unreachable’.
You can see the diversity of apparent issues. Many others have reported similarly widespread files being returned for their own homepages when using this tool.
The Google Search Console also offers information on any blocked resources that “can impair the indexing of your webpages” under Google Index → Blocked Resources.
Of the four sites mentioned above only one had any indication of a blocked resource in this section (the external Campaign Monitor file). That file has little bearing on the rendering of the site either for desktop or smartphones.
The Robot.txt File
Many sites contain a robots.txt file in its web root, for example:
This file can contain various recommendations for search engines regarding which files to index and which to disallow. Note: these are generally considered as recommendations by the search engines, not absolute directives. There are many legitimate uses for the robots.txt files.
In the case of our example sites above, the robots.txt file for each read:
This is very common configuration for a WordPress site, requesting search engines not crawl or index the WordPress admin directory (/wp-admin/). It is not particularly restrictive.
The tool will highlight any errors or warnings found in your robots.txt file.
In our case none of the four sites above showed any errors or warnings using this tool. This is not surprising given the paucity of the robots.txt file.
So What Do I Do?
Option 1: Trim Down Your Robots.txt File
If you have a more complex robots.txt file, for example something like:
…you should consider trimming it down to the essentials if possible. This of course requires some knowledge of which assets truly need to be blocked and which can slip by (at least for now). Generally speaking, if it’s a publicly accessible directory or file, or if the directory is listed in one of the alerts given by the Fetch as Google or Blocked Resources tools in Google Search Console, try removing it from the robots.txt file and retesting using Fetch and Render.
NOTE: If Fetch and Render shows a block to an external resource (one not on your site), you can safely ignore it.
Option 2: Allowing Additional Access via Robots.txt
a) You could remove the robot.txt file completely. Perhaps not the best solution unless you have no other option.
b) Use a “blank” robot.txt file:
The above allows all access.
Please do not use:
…which will block all access to the site for bots following the rule.
c) The robots.txt file can be used to allow additional access, in this case to .js, .css and various images file types:
The “Allow:” instruction tells a bot that it is okay to index a file in a directory that has been “Disallowed” by other instructions. In this case .js and four other file types would be allowed even within /wp-admin/.
The use of wildcards * in the Allow: directives matches any filename as well as any version numbers appended to end of the filename.
This seems like a pretty rough hack, but again if nothing else works, you might consider it.
You can always do nothing for a spell and see how it all settles out in the coming weeks. If your only issues reported by Fetch as Google for both desktop and smartphone are marked ‘Temporarily Unreachable’, the error may resolve itself. You can check whether those files are available in your browser, or in a browser emulating a mobile device (Chrome allows for this in the Developer Tools console).
Otherwise, check back here and we will update this post as options reveal themselves.
Have a question? Leave it for us in the comments below.