TulipTools Internet Business Owners and Online Sellers Community
›
Webmaster Issues
›
Webmaster Issues
›
General WebMaster Issues Discussion
Attack of the Spiders
Attack of the Spiders
|
Attack of the Spiders
|
|
08-11-2005, 06:44 PM,
 Post: #1 |
|||
|
|||
|
Attack of the Spiders
Does anyone else have a problem with search engine spiders completely slowing their servers to a crawl when they're indexing your sites?
The TulipTools forums for now are sharing space on one of our dedicated servers with one of our directory sites which is why you might notice a slowdown at times when spiders are attacking (indexing) the directory site. The directory site has huge databases (over 2 GB for the directory and over 1 GB for the metasearch) and when multiple spiders hit that site doing deep crawls it puts a huge load on the server. The normal sever load on the server ranges from 0.3-0.8, but when multiple spiders hit it jumps way way up: today MSN and LookSmart were indexing at the same time and the load average jumped to 18 :blinkie:, yesterday afternoon the load average was running 35 to 40 :blinkie: when MSN/Yahoo/Ask Jeeves all arrived near the same time and spent about 45 minutes indexing the site. The spider related spikes in server load only occur for short periods of time, but depending on the length of the crawl can last 45 minutes to an hour on rare occasions. ...soo, 1. is it time to stick the databases on a separate dedicated server and just keep the actual site and its programs/pages/etc on this dedicated server? 2. would converting the large databases from MySQL to PostGres maybe help? huh, huh, would it? 
"Well, Jay was so giddy that someone named Jay was involved with this site we posted our first non-eBay listing in 3 years here at Lunarbid (we tried two items at Yahoo once upon a time, they bombed)" -Marie posting in a LunarBid thread at OTWA in 2005 wins the award for 'most moronic reason ever given for choosing a venue"
"thanks twat u must have nothing better 2 do. do u talk to all your members like that. will not be recomending your site. best way to put it is TULIPTOOLS.COM IS REALLY SHIT. DONT JOIN." -pubescent owner of rinky dink off2auction.com in 2011 |
|||
![[+]](https://community.tuliptools.com/images/1/collapse_collapsed.gif)
|
08-11-2005, 07:07 PM,
Post: #2 |
|||
|
|||
|
Re: Attack of the Spiders
Never had that problem. . . but I'm small fries in comparison to you guys.
Just curious, is one of those spiders named "ginger"? 
|
|||
|
08-11-2005, 09:47 PM,
Post: #3 |
|||
|
|||
Re: Attack of the Spiders
|
|||
|
08-12-2005, 06:05 PM,
Post: #4 |
|||
|
|||
|
Re: Attack of the Spiders
[quote author=Jen link=topic=162.msg593#msg593 date=1123787262]
Just curious, is one of those spiders named "ginger"? [/quote] No spiders named Ginger on the main site on this server. :  ...on this site though, I found this in our stats: Hits Files KBytes Visits Hostname 22875 7.45% 16403 19.34% 33640 6.79% 96 5.75% lilgingeebot 2677 0.87% 2669 3.15% 25351 5.12% 28 1.68% crawl-66-249-65-180.googlebot.com :blinkie: :blinkie: :blinkie: :blinkie: :blinkie:
"Well, Jay was so giddy that someone named Jay was involved with this site we posted our first non-eBay listing in 3 years here at Lunarbid (we tried two items at Yahoo once upon a time, they bombed)" -Marie posting in a LunarBid thread at OTWA in 2005 wins the award for 'most moronic reason ever given for choosing a venue"
"thanks twat u must have nothing better 2 do. do u talk to all your members like that. will not be recomending your site. best way to put it is TULIPTOOLS.COM IS REALLY SHIT. DONT JOIN." -pubescent owner of rinky dink off2auction.com in 2011 |
|||
|
08-12-2005, 09:33 PM,
Post: #5 |
|||
|
|||
|
Re: Attack of the Spiders
Why does everyone pick on Ginger?
|
|||
|
08-13-2005, 12:10 AM,
Post: #6 |
|||
|
|||
Re: Attack of the Spiders
Quote: lilgingeebot SHE gets her own spider AND smilie?!?!?! :blinkie:  
|
|||
|
08-13-2005, 01:13 AM,
Post: #7 |
|||
|
|||
Re: Attack of the Spiders
Quote:...soo, 1. is it time to stick the databases on a separate dedicated server and just keep the actual site and its programs/pages/etc on this dedicated server? One.  Two. wars have been started over the MySQL vs. PostGres question 
|
|||
|
08-21-2005, 11:13 PM,
Post: #8 |
|||
|
|||
|
Re: Attack of the Spiders
Did you move the databases?
|
|||
|
08-22-2005, 11:14 AM,
Post: #9 |
|||
|
|||
|
Re: Attack of the Spiders
[quote author=jezebel link=topic=162.msg884#msg884 date=1124665984]
Did you move the databases? [/quote] Not yet. :-[ but we have an excuse. We just finished changing the backend of the metasearch engine (the spider) so its now a command line C program instead of being PHP based. The part the user sees is still written in PHP. We just put it into service early Saturday and now we're playing with the timing of the searches so it could be a while before we move the databases. So far the move to C seems to be helping cut down on the server load...we're still getting hit hard when certain spiders visit (and decide they'd like to use the related search terms that are returned to index the whole friggin' Internet : ) but not as hard as before.
"Well, Jay was so giddy that someone named Jay was involved with this site we posted our first non-eBay listing in 3 years here at Lunarbid (we tried two items at Yahoo once upon a time, they bombed)" -Marie posting in a LunarBid thread at OTWA in 2005 wins the award for 'most moronic reason ever given for choosing a venue"
"thanks twat u must have nothing better 2 do. do u talk to all your members like that. will not be recomending your site. best way to put it is TULIPTOOLS.COM IS REALLY SHIT. DONT JOIN." -pubescent owner of rinky dink off2auction.com in 2011 |
|||
|
08-22-2005, 11:07 PM,
Post: #10 |
|||
|
|||
Re: Attack of the Spiders
Quote:SHE gets her own spider AND smilie?!?!?! 8) |
|||
|
« Next Oldest | Next Newest »
|
Users browsing this thread: 2 Guest(s)
