Slow Website? Consider Robots As Being The Culprit

This is something I’ve already posted about on my forum and on the Xenforo board forums, but I wanted to share it here as well. So, as many of you are aware, I’ve been busy working on building up our new forum, Office Outlaw.

As such, I’ve run into some very unique issues as I’ve gotten more involved with running a forum and being so active with maintaining a website like that. More specifically, maintaining a website that has many active members and lurkers alike.

One of these issues actually involves site crawlers. Now, many of you are familiar with Googlebot, Bingbot, and other search engine crawlers. But, did you know that there are a lot of crawlers out there that might be slowing down your website whilst offering no real benefit to you? This was happening to my forum, but upon further investigation, I realised that this is also happening to my regular websites as well.

Bots You Should Consider Blocking & Why

The bots below are ones that were specifically plaguing my forum. Before blocking them, I researched each one to make sure I wasn’t blocking one that was going to benefit my site, like a search engine crawler.

Note that every site is different, so if you actually benefit from, let’s say Amazon Alexa users, then you don’t want to block the Amazonbot from crawling your website. Likewise, as your site grows, it’s worth revisiting which spiders you’ve blocked to make sure it’s still worth blocking them.

  • Bytespider – I kid you not, this one alone was pinging my forum up to a hundred times a minute. In fact, I noticed that other Xenforo forum owner found this one particularly annoying as they reported that it was ignoring their robots.txt files. If you don’t know what it is, it’s Bytedance AKA TikTok’s bot that trains their language learning models. In other words, there’s absolutely no benefit to you for allowing this one to crawl your website.
  • Amazonbot – This is another big offender. Like I mentioned before, you can consider allowing this one if you’ve found people use Alexa with your site and you want to keep that feature working.
  • Semrush – This one is useful if you’re using tool or if you plan on selling your website in the future as many in the industry feel they’re fairly accurate with collecting data. Since I don’t use the tool and I don’t plan on selling my forum, I’ve blocked it.
  • Ahrefs – This is the situation as SemRush: unless you’re using the tool, it’s best to block it.
  • Mozlinkexplorer – This one is a lot more reasonable compared to the other bots, but again, it’s only useful if you plan on using the tool.

I’ve shared this information on the Xenforo forums as I’m an active participant there these days, and I plan on sharing it on my forum as well.

Alistair Kavalt
Latest posts by Alistair Kavalt (see all)

Related Posts: