This has long been a puzzle for me. We can’t tell from the requests if it’s someone on the server. I can understaand sites, hosted at servers, visiting and poking around, looking to spam, advertise, or just do something mean. But, a legit web hosting company? That doesn’t figure
I’ve sent emails like this…
“I thought I’d ask why this “18.104.22.168 Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/) /robots.txt 403″ keeps showing up in my logs. I’ve wondered about web host visits for some time and just recently began contacting people to ask what it’s all about.”
I sent that message to WholeSale Internet, Inc., here are some other outfits that constantly show up in logs:
Amazon Technologies Inc., comes through using “22.214.171.124 BUbiNG ( http://law.di.unimi.it/BUbiNG.html) /robots.txt 403”. So it’s really a visit from “law.di.unimi.it”?
Hetzner Online GmbH – Contact Role, using “126.96.36.199 Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/) /robots.txt 403”. An open source crawler, crawling websites for…? Puzzle.
GoDaddy.com, LLC, comes round with “188.8.131.52 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36 /wp-login.php 403” and wants to log in
MAIL.RU NOC, using “184.108.40.206 Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; http://go.mail.ru/help/robots) / 403”
So you see that these visits all got denied and met the 403 page, which records to visit. The question people have might be “Is Go Daddy really sending a log in bot?”. The answer would be that the IP logged is an IP assigned dynamically to any number of sites on one of GoDaddys servers.
So if you block 220.127.116.11 you’d likely block one site sending log in bots and 78 other sites not sending log in bots. This is why, after a while, people with websites stop blocking IP’s and turn to other methods. Request blocking is my choice. I will block legitimate bots that I choose not to participate with. Like, ahrefs and MJ12bot.
On wordpress sites I install the BBQ (Jeff Starr plugin-planet.com) and Rename wp-login.php (iseulde.com) plugins. WP-SpamShield (redsandmarketing.com) takes care of the comment critters and some harvesters