08-11-2005, 06:44 PM
Does anyone else have a problem with search engine spiders completely slowing their servers to a crawl when they're indexing your sites?
The TulipTools forums for now are sharing space on one of our dedicated servers with one of our directory sites which is why you might notice a slowdown at times when spiders are attacking (indexing) the directory site.
The directory site has huge databases (over 2 GB for the directory and over 1 GB for the metasearch) and when multiple spiders hit that site doing deep crawls it puts a huge load on the server. The normal sever load on the server ranges from 0.3-0.8, but when multiple spiders hit it jumps way way up: today MSN and LookSmart were indexing at the same time and the load average jumped to 18 :blinkie:, yesterday afternoon the load average was running 35 to 40 :blinkie: when MSN/Yahoo/Ask Jeeves all arrived near the same time and spent about 45 minutes indexing the site.
The spider related spikes in server load only occur for short periods of time, but depending on the length of the crawl can last 45 minutes to an hour on rare occasions.
...soo, 1. is it time to stick the databases on a separate dedicated server and just keep the actual site and its programs/pages/etc on this dedicated server? 2. would converting the large databases from MySQL to PostGres maybe help? huh, huh, would it?
The TulipTools forums for now are sharing space on one of our dedicated servers with one of our directory sites which is why you might notice a slowdown at times when spiders are attacking (indexing) the directory site.
The directory site has huge databases (over 2 GB for the directory and over 1 GB for the metasearch) and when multiple spiders hit that site doing deep crawls it puts a huge load on the server. The normal sever load on the server ranges from 0.3-0.8, but when multiple spiders hit it jumps way way up: today MSN and LookSmart were indexing at the same time and the load average jumped to 18 :blinkie:, yesterday afternoon the load average was running 35 to 40 :blinkie: when MSN/Yahoo/Ask Jeeves all arrived near the same time and spent about 45 minutes indexing the site.
The spider related spikes in server load only occur for short periods of time, but depending on the length of the crawl can last 45 minutes to an hour on rare occasions.
...soo, 1. is it time to stick the databases on a separate dedicated server and just keep the actual site and its programs/pages/etc on this dedicated server? 2. would converting the large databases from MySQL to PostGres maybe help? huh, huh, would it?
