TulipTools Internet Business Owners and Online Sellers Community

Quote: Spiders active in past 15 minutes:
Googlebot (2), WISENutbot, Mediapartners-Google

et tu?

WISENutbot  hasn't left my sites all week. 

WISENutBot is LookSmart

There was a thread about the little guy running rampant at Webmaster's world:
http://www.webmasterworld.com/forum16/1607.htm

Oh, by the way . . . it visited one of my sites, stayed for days, ignored robots.txt - basically was a pain. 

I wasn't selling anything on that sight, so I banned the little monster's IP blocks . . . you can find them at http://www.banbots.com/looksmart.htm

I don't know how up to date that is now, but it worked for me in the past.

I hate banning something that could help in a search engine, but it was a nuisance - and I wasn't selling anything on that site.

OH - never mind.  I just saw your posts on "attack of the spiders" and it looks like you could probably teach me a thing or two about this stuff.  I thought you were just asking what in the heck the thing was.    :-[

Yes, WiseNutJob has taken a liking to some of our sites this week and yes it is ignoring the robots.txt file on one of our sites which is more than a pain because the file it is indexing is the cache file of the latest search terms which people have used on our metasearch engine.

Our metasearch engine simultaneously searches 8 search engines every time someone performs a search.  Earlier tonight wisenutbot had 275 separate searches going (which is actually 275x8 or 2200 searches that then get written to the database)...a few to many connections at once for MYSQL to handle...well actually it does handle them by rejecting additional connections.    .  All of our searches also return a list of related search terms, and so unWISENUTJOB is also performing searches using those terms. 

We could always ban the spider but I don't want to because we have a couple of sites in the LookSmart directory.