Should I disallow crawling of all of my site’s JavaScript files?

Should I disallow crawling of all of my site’s JavaScript files?


MATT CUTTS: Today’s question
comes from Zurich, in Switzerland. John Mueller wants to know,
“Googlebot keeps crawling my JavaScript and thinking
the text in my scripts refers to URLs. Can I just disallow crawling of
my JavaScript files to fix that?” Well, you can disallow
all JavaScript. But I really, really, really
would not recommend that. If there’s perhaps one
individual JavaScript file that’s the source of
the problem, you could disallow that. But in general, it can be really
helpful for Google to be able to fetch, and process,
and execute that JavaScript to learn what other links
are on the page. So in general, I would not block
JavaScript, and I would not block CSS, those
sorts of things. It turns out, as we’re executing
JavaScript, we do look at the attributes. So you can actually use
JavaScript and put, like, a no follow attribute on
individual URLs. And so it is the sort of thing
where you can do you link-level granularity there. And you can block, for example,
an individual JavaScript file. But in general, I would not
recommend blocking all of your JavaScript files. Those can be really helpful for
Google to understand your site and be able to find more
pages deeper within your site. So I wouldn’t recommend
blocking them.

10 comments

  • Clear answer, thx!

  • thanks Sir

  • No…for the first time, Matt replied to a question with an answer that even the great unwashed could interpret as an answer. He's answered the vast majority of the other questions…it's just that most people are too stupid and/or selfish to read between the lines properly.

  • Matt – sent you this underground blog network on Twitter but maybe you missed it. Don't let these guys keep manipulating your search results.

  • I'd agree, were it not for umpteen cases where nosy Googlebot picks up nonsense pseudo urls from js code that's on-page.

  • You'd improve your ease of maintenance of the site and that's about all.
    You're better off to just ensure your css is clean and devoid of redundancy and not overloaded with umpteen versions of styles that aren't being used.

  • Hello, what happened to Google's related search option? I can't see it under search tools now. Or is it my browser playing up?

  • Please UNN needs more explanation on this
    Loveth Onuigbo

  • Great to know – thanks!

  • Disallowing Google from crawling the entire site's Javascript files can be very harmful to your site. Instead of doing that, going the extra effort of making sure that you CSS has no redundancy is a better game plan.

Leave a Reply

Your email address will not be published. Required fields are marked *