ShopWiki and the new breed of crawler based shopping comparison engines

[mandy]

ShopWiki is a shopping search engine that indexes retail products from thousands of online stores. ShopWiki is not a store and does not sell anything. The purpose of the site is to help consumers find the products they are looking for online with ease. ShopWiki offers:

More Stores

    * ShopWiki crawls more than 125,000 stores to ensure you find the best product and price available.

Complete Searches

    * The search function on ShopWiki offers a variety of options from a simple product search to one with more details. Our system allows you to narrow down your search by numerous variables, including price, special features and brand. You can even find a particular product from a specific store. See our Sample Searches page for some examples.

Wiki Buying Guides

    * The wiki premise is simple: Every user has the ability to change, edit and update content on the site. It also means that every user that contributes to the site becomes a part of our wiki community.
    * This site features numerous buying and gift guides to help narrow down your search options. Written by users like yourself, ShopWiki buying guides define some of the complex terms and give the insight you desire to find the best product for your needs.
    * Want to contribute? Check out our writing wiki content page for some helpful tips.

http://www.shopwiki.com/wiki/Help:Why+Shopwiki

ShopWiki is almost ready to launch. The site claims to crawl “more than 120,000 stores” which, if true, would make it one of the most comprehensive comparison shopping engines.

http://www.comparisonengines.com/2006/03...-horowitz/

Which brings me back to Dulance and similar shopping search engines which don’t depend on merchant feeds but rather crawl the web for merchants and associated products. These shopping engines don’t impose price floors and get paid through affiliate programs or reasonable pay per click fees. Which means that these services can move past 5,000 merchants, past 50,000 merchants, and maybe past 100,000 merchants…hitting the long tail and possibly becoming an order of magnitude better than the current field of shopping comparison engines. Yes, there are still many issues to deal with around building an incredible user experience, but hopefully you’re getting the picture.

full article: http://www.comparisonengines.com/2006/03...-knew-you/

ShopWiki web site: http://www.shopwiki.com/

[thentavius]

Interesting! Thanks for sharing.

[regic]

Shopwiki's search crawler is a menace to webservers. We had to add a delay to our robots.txt file on one site to slow down their spider.

User-agent: ShopWiki
Crawl-Delay: 5

[bargainbloodhound]

We have 9 ShopWiki spiders racing around on one of our directories as I type (they're frequent visitors) and it isn't even a shopping related directory (although the web site is in a shopping related category in the Yahoo directory due to some Yahoo Directory category changes several years ago)...since their crawler is crawling nonshopping related directories like ours I reallllllly have to question how relevant ShopWiki's search results will be for the average shopper.

[eliot]

I am an employee at ShopWiki and would like to respond to the above post.

ShopWiki actively crawls more than 120,000 stores to ensure consumers find the best product and price available. However, our crawlers periodically scan the Internet, searching more than 300,000 Web sites, to find more stores and to ensure we only show results from true retail outlets.

Because we are looking to gain access to every retail outlet on the Web, we are sensitive to the needs of every site we crawl. In an effort to make our crawlers as site-friendly as possible, we regularly check our crawling speeds adjust the settings as needed.

[regic]

Hi Eliot,

The crawl delay worked and there's no problem with your spiders now.  Your crawler is much more obedient to robots.txt rules than a few others like Looksmart's WiseNut 

Can shopping sites submit their sites directly to ShopWiki to be crawled or does your crawler automatically find them by following links on other sites?

[eliot]

Glad to hear that the crawler is now behaving nicely.

Anyone can add sites for the crawler to look at on this page:
http://www.shopwiki.com/addSite.jsp

-Eliot

[mandy]

A review of ShopWiki:

ECommerce Guide tested ShopWiki and found some of its features to be quite innovative. The one tool that stood out is the sliding price range finder, located at the top of every search. Rather than repeatedly typing in additional price criteria for an item you are searching for, all you have to do is move either tab (see image above) to change the price range you are interested in. For example, if you happen to be looking for USB flash drives, the default search shows you all options from $1 to $101. If $101 is too high for you, simply slide the higher tab down until the maximum price target is $75. The page instantly refreshes showing you all available drives up to $75...

full article: http://www.ecommerce-guide.com/news/news...hp/3600166

According to the article, ShopWiki plans to introduce a toolbar later this year and expand its service to 25 countries by 2007.

[mandy]

Yet another ShoppingWiki review:

“We asked users what their fantasy online shopping engine looked like, and they told us they wanted the whole universe of products on there. That’s the point of shopping on the Internet; otherwise, people would just go to the store down the block.” At the same time, testers told Ryan and Merriman that they were tired of seeing screen after screen of merchant affiliate listings of the same product. So ShopWiki weeds out affiliates and lists only the parent merchants, showing the handful (ten or so) of the most clicked-on product listings first.

ShopWiki may still be in beta, but Ryan says it has already crawled the sites and extracted product data from some 120,000 Web merchants. Asked if any merchants have objected to having their sites scraped or their affiliates dissed, Ryan says not—and that since clicking on a ShopWiki product listing brings the user to the merchant Web site, most have been appreciative of the traffic boost. (A little snail icon next to some listings denotes a merchant page that’s particularly slow to load...

full article: http://multichannelmerchant.com/webchann...-ShopWiki/

[amy]

http://www.shopwiki.com/wiki/mothers+day+gifts 

they were tired of seeing screen after screen of merchant affiliate listings of the same product. So ShopWiki weeds out affiliates and lists only the parent merchants

I wish everyone removed affiliate listings from their search results.