Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[bargainbloodhound]

Does anyone else have a problem with search engine spiders completely slowing their servers to a crawl when they're indexing your sites?

The TulipTools forums for now are sharing space on one of our dedicated servers with one of our directory sites which is why you might notice a slowdown at times when spiders are attacking (indexing) the directory site.

 The directory site has huge databases (over 2 GB for the directory and over 1 GB for the metasearch) and when multiple spiders hit that site doing deep crawls it puts a huge load on the server. The normal sever load on the server ranges from 0.3-0.8, but when multiple spiders hit it jumps way way up: today MSN and LookSmart were indexing at the same time and the load average jumped to 18  , yesterday afternoon the load average was running 35 to 40  when MSN/Yahoo/Ask Jeeves all arrived near the same time and spent about 45 minutes indexing the site.

The spider related spikes in server load only occur  for short periods of time, but depending on the length of the crawl can last 45 minutes to an hour on rare occasions.

...soo, 1. is it time to stick the databases on a separate dedicated server and just keep the actual site and its programs/pages/etc on this dedicated server?  2. would converting the large databases from MySQL to PostGres maybe help? huh, huh, would it? 

[Tags:]

Tags:

[Jen]

Never had that problem. . . but I'm small fries in comparison to you guys.

Just curious, is one of those spiders named "ginger"? 

[ginger]

 

[bargainbloodhound]

Just curious, is one of those spiders named "ginger"? 

No spiders named Ginger on the main site on this server. 

...on this site though, I found this in our stats:

Hits                   Files                    KBytes                 Visits                 Hostname
22875    7.45%    16403    19.34%    33640    6.79%    96    5.75%     lilgingeebot
2677    0.87%       2669    3.15%      25351   5.12%      28  1.68%   crawl-66-249-65-180.googlebot.com


 

[scampy7]

Why does everyone pick on Ginger?

[amy]

lilgingeebot

SHE gets her own spider AND smilie?!?!?! 






 

[jezebel]

...soo, 1. is it time to stick the databases on a separate dedicated server and just keep the actual site and its programs/pages/etc on this dedicated server?

One.
Two.  wars have been started over the MySQL vs. PostGres question 

[jezebel]

Did you move the databases?   

[bargainbloodhound]

Did you move the databases?   

Not yet.    but we have an excuse.    We just finished changing the backend of the metasearch engine (the spider) so its now a command line C program instead of being PHP based.  The part the user sees is still written in PHP.  We just put it into service early Saturday and now we're playing with the timing of the searches so it could be a while before we move the databases.  So far the move to C seems to be helping cut down on the server load...we're still getting hit hard when certain spiders visit (and decide they'd like to use the related search terms that are returned to index the whole friggin'  Internet    ) but not as hard as before.

[ginger]

SHE gets her own spider AND smilie?!?!?! 

 

[jezebel]

Did you move the databases?   

Not yet.    but we have an excuse.    We just finished changing the backend of the metasearch engine (the spider) so its now a command line C program instead of being PHP based.  The part the user sees is still written in PHP.  We just put it into service early Saturday and now we're playing with the timing of the searches so it could be a while before we move the databases.  So far the move to C seems to be helping cut down on the server load...we're still getting hit hard when certain spiders visit (and decide they'd like to use the related search terms that are returned to index the whole friggin'  Internet    ) but not as hard as before.

Seems to be working.  No more gazillion minute waits for this board to load.