223 Spiders online.
That's a lot!!,of trackers any chance that we can get the mods to tell the admin about this. This website only needs one spider and that's Google.
Lets get real, NO ONE should be using AOL and all the other 1990s and 2000s websites
For those new to the robots.txt file, it is merely a text file implementing what is known as the Standard for Robot Exclusion. The file is placed in the main directory of a website and advises spiders and other robots which directories or files they should not access. The file is purely advisory — not all spiders bother to read it let alone heed it. However, most, if not all, the spiders sent by the major search engines to index your site will read it and obey the rules contained within the file (provided those rules make sense).
Whose online - Forum Stats
- Thread starter 285south
- Start date
- Status
Not really sure why 223 spiders is a problem... why do you want this fixed? And not everyone uses google, why would you chose to lock out other search engines?
What is a spider? Not all of us know the jargon you use.
If you mean the bot web crawler, I fail to see what the big deal is if the site still works OK.
If you are not affiliated with the owners of this forum, why are you telling them how to run it? Is that not presumptuous?
Too many Web crawlers can crash or slow a website. I'm guessing that's part of therreason we've had something many server upgrades. 223 is a high number and probably slowing the website.
Also Google does have the best search features for finding Web pages on the forum.
If worry more about the North Korean's than a few search engine web bot spiders! If they got ahold of all this oil technology we'd be shafted! Besides, never let a monopoly run the world!
- Status