Adult SEO Blog

1/16/2005

• Adult SEM Map
  • Adult SEM Forum
• Recommended Blogs
  • Baddog's Bytes
  • Business Voyeur
  • Free Adult Website Traffic
• Recommended Links
  • AVN Online
  • Herbal Affiliate Program
  • Multiple Class C Hosting
• Categories:
  • general
  • adult seo tactics
  • affiliate marketing
  • link strategies
  • adult seo tutorials
  • adult seo training
  • adult content optimization
  • adult seo articles
  • search engine news
• Search:
  
  
• Archives:
  • September 2005
  • August 2005
  • July 2005
  • May 2005
  • April 2005
  • March 2005
  • February 2005
  • January 2005
• Adult Search Engine Newsletter
  
  name:
  email:
  Yes! I want to recieve Adult SEM Newsletter. I agree with AdultSEM.com's Privacy Statement and Terms. I also understand that I will recieve the newsletter by email.
  
• 

• January 2005

  S	M	T	W	T	F	S
  				Feb »
  						1
  2	3	4	5	6	7	8
  9	10	11	12	13	14	15
  16	17	18	19	20	21	22
  23	24	25	26	27	28	29
  30	31

• Other:
  • Login
  • Register
• Meta:
  • 
  • 
  • 
  • 
  • Comments RSS 2.0
  • Valid XHTML
  • WP

  • Yahoo! Search Support for X-Robots-Tag Directive to Simplify Webmaster
    

    Today we're announcing support for tags that give webmasters even more flexibility over which pages and documents are crawled and indexed by Yahoo! Search. Specifically, we're extending our support of page level exclusion tags -- NOINDEX, NOARCHIVE, NOSNIPPET, NOFOLLOW -- to provide additional control for archiving and summarization of ANY file type. Previously, these page level tags could only be expressed within html pages through the META directive (for e.g. ), but based on feedback from our webmasters, Yahoo! now enables these tags to be expressed through X-Robots-Tag directive in the http header, giving webmasters the flexibility to achieve exclusions on PDF, Word documents, PowerPoint, video, and other file types, including html files, and increasing their coverage through a simplified process. Additionally, webmasters no longer need access to html templates in order to express exclusions for html files. To take advantage of this feature, simply add the following page level tags to the X-Robots-Tag directive in the HTTP Header. Here are a few examples:

    • X-Robots-Tag: NOINDEX -- If you don't want to show the URL in the Yahoo! Search results.

      Note: We'll still need to crawl the page to see and apply the tag, so if you don't wish to have the page crawled, use robots disallow on robots.txt.

    • X-Robots-Tag: NOARCHIVE -- If you don't want to display cache link in the search results page.
    • X-Robots-Tag: NOSNIPPET -- If you don't want to display summary in the search results page.
    • X-Robots-Tag: NOFOLLOW -- If you don't want Yahoo! to crawl links in the page.

    Along with this change, we'll be rolling out additional changes to our crawling, indexing and ranking algorithms over the next few days. We expect the update will be completed early next week, but you may see some changes in ranking as well as some shuffling of the pages in the index during this process.

    We're at SES in Chicago and WebmasterWorld's PubCon in Las Vegas, participating in a few different panels this week. Please find us if you have any questions or suggestions or drop us your feedback here.

    
    Sharad Verma
    Yahoo! Search

    Yahoo! Search Crawler (Yahoo! Slurp) - Supporting wildcards in robots.
    I was going through my notes from Danny Sullivan's Open Feedback sessions that occur during the ‘Meet the Crawlers’ panel at Search Engine Strategies. One of the items on my list was a request for enhanced syntax in robots.txt to...

    Congratulations Hastings, Nebraska -- Winner of our "Be a Better Plane
    

    What do the birthplace of Kool-Aid and green living have in common? More than you think.

    Back in April, we announced Yahoo!'s "Be a Better ____" challenge to encourage people to use Yahoo! tools to become better fanatics, advertisers, entrepreneurs, best friends, gardeners... however they want to become "better." Part of this program included Yahoo!'s "Be a Better Planet" Greenest City in America Challenge to see how communities across the country can take action against climate change and use Yahoo! tools to become better planetarians.

    Today, the winner of the "Be a Better Planet" Greenest City in America Challenge was revealed and we want to give a shout out to the nation's green leader. The competition started with more than 350 cities across the country, and the winner is the home of Kool-Aid itself -- Hastings, Nebraska! In addition to their grand prize of $250,000 for city greening projects, Hastings and the top five finalist cities will receive a slew of other green prizes.

    Hastings secured the top spot by rallying together a good portion of its residents to earn participation credits on Yahoo! Green through pledging to reduce carbon diets, Yahoo! Answers through answering green questions and Yahoo! oneSearch through conducting eco-friendly mobile searches. And now the city's mayor is tapping into the Answers community to solicit thoughts and advice for how to invest the $250,000 prize. If you have some great green ideas, don't hesitate to share.

    Here's a bit more information on Hastings' contribution to the challenge and what they did to stand out. Congrats, Hastings.

    
    Yahoo! Search Blog team

    That Cunt Pamela Paul is at it Again
    

    Start Monkeying Around
    

    In February, we began talking about our plans to open up Yahoo! Search to Web site owners and all third party developers. This new developer platform, which we're calling SearchMonkey, uses data web standards and structured data to enhance the functionality, appearance and usefulness of search results. Specifically, with SearchMonkey:

    

    
    

  • Site owners can build enhanced search results that will provide searchers with a more useful experience by including links, images and name-value pairs in the search results for their pages (likely resulting in an increase in traffic quantity and quality)
  • Developers can build SearchMonkey apps that enhance search results, access Yahoo! Search's user base and help shape the next generation of search
  • Users can customize their search experience with apps built by or for their favorite sites
  
  

  
  As Ari Balogh, Yahoo!'s new CTO, will discuss in his Web 2.0 Expo keynote this morning, we're rolling out a limited preview of the SearchMonkey Developer Tool starting today. With this online tool, developers can build data services that can be used to present richer, more useful search results. These data services can be constructed using structured data either from the Yahoo! Search index or from publicly available sources (such as APIs).

  In addition to signing up for the preview, head over to the Yahoo! Developer Network booth at Web 2.0 Expo (#901 in the Expo Hall) to check out a demo.

  And what's a new product without a party to kick it off? We're celebrating this component of our open platform with a Developer Event at our Sunnyvale Campus on Thursday, May 15. If you're interested in joining us, here's more information. Hope to see you there.

  
  Amit Kumar
  Chief SearchMonkey

  Giving Birth - 1st Trimester
  

  2-fer Tech Tuesday #37: Printable, laminatable cheat sheets: JavaScri
  

  One Standard Fits All: Robots Exclusion Protocol for Yahoo!, Google an
  

  Over the last couple of years, we've been collaborating with Google and Microsoft to make webmasters' efforts more effective across the major search engines. By bringing standards such as Sitemaps and improvements like auto-discovery and cross-host submission, webmasters can simplify their account management across the different search engines.

  The Robots Exclusion Protocol (REP) lets content publishers specify which parts of their site they want public and which parts they want to keep private from robots, whether it's controlling the visibility of their content across their site (via robots.txt) or at the level of individual pages (via META tags). REP was introduced in the early 1990's and is the de facto standard. Its strengths lie in its flexibility to evolve in parallel with the web, its universal implementation across major search engines and all major robots and in the way it works for any publisher, no matter how large or small. We've heard that there is some confusion around the specific implementation of REP supported by each engine. Since we've never detailed the specifics of implementing the protocol, today we're releasing detailed documentation on how REP directives will be handled by the three major search providers.

  Common REP Directives
  The following are all the major REP features currently implemented by Google, Microsoft and Yahoo!. Each of these directives can be specified to be applicable for all crawlers or for specific crawlers by targeting them to specific user-agents, which is how any crawler identifies itself. Each of us also supports Reverse DNS based authentication of our crawler,and you can use this validate the identity of any crawlers claiming a particular user-agent.

  1. Robot.txt Directives

  DIRECTIVE	IMPACT	USE CASE(s)
  Disallow	Tells a crawler not to crawl your site or parts of your site -- your site's robots.txt still needs to be crawled to find this directive, but the disallowed pages will not be crawled.	'No crawl' pages from a site. This directive in the default syntax prevents specific path(s) of a site from crawling.
  Allow	Tells a crawler the specific pages on your site you want indexed so you can use this in combination with Disallow. If both Disallow and Allow clauses apply to a URL, the most specific rule - the longest rule - applies.	This is useful in particular in conjunction with Disallow clauses, where a large section of a site is disallowed, except a small section within it.
  $ Wildcard Support	Tells a crawler to match everything from the end of a URL -- large number of directories without specifying specific pages.	'No Crawl' files with specific patterns, for eg., files with certain filetypes that always have a certain extension, say pdf; etc.
  Sitemap Location	Tells a crawler where it can find your sitemaps.	Point to other locations where feeds exist to point the crawlers to the site's content.

  2. HTML META Directives

  These directives can either be placed in the HTML of a page or in the HTTP header for non-HTML content like PDF, video, etc. using an X-Robots-Tag. The X-Robots-Tag mechanism allows these directives to be available for all types of documents -- HTML or otherwise. If both forms of the tag, HTML META and X-Robots-Tag in the header are present, the most restrictive one applies.

  DIRECTIVE	IMPACT	USE CASE(s)
  NOINDEX META Tag	Tells a crawler not to index a given page.	Don't index the page. This allows pages that are crawled to be kept out of the index.
  NOFOLLOW META Tag	Tells a crawler not to follow a link to other content on a given page.	Prevent publicly writeable areas from being abused by spammers looking for link credit. By NOFOLLOW, you let the robot know that you are discounting all outgoing links from this page.
  NOSNIPPET META Tag	Tells a crawler not to display snippets in the search results for a given page.	Present no abstract for the page on search results.
  NOARCHIVE META Tag	Tells a search engine not to show a "cached" link for a given page.	Do not make a copy of the page available to users from the search engine cache.
  NOODP META Tag	Tells a crawler not to use a title and snippet from the Open Directory Project for a given page.	Do not use the ODP (Open Directory Project) title and abstract for this page in Search.

  Other REP Directives
  Yahoo!-specific REP directives that are not supported by Microsoft and Google include:

  • Crawl-Delay: Allows a site to delay the frequency with which a crawler checks for new content
  • NOYDIR META Tag: This is similar to the NOODP META Tag above but applies to the Yahoo! Directory, instead of the Open Directory Project
  • Robots-nocontent Tag: Allows you to identify the main content of your page so that our crawler targets the right pages on your site for specific search queries by marking out non content parts of your page. We won't use the sections tagged as such for indexing the page or for the abstract in the search results.

  Apart from these tools in the REP, Yahoo! Site Explorer also provides further ways to tell Yahoo! to Delete URLs, or Rewrite Dynamic URLs to remove spurious parameters. You can learn more about our crawler at the Slurp Help page.

  We plan to continue coordinating with the leading search engines to ensure simplicity for webmasters, so stay tuned for more developments in the future. You can also find information on the Official Google Webmaster Central Blog or the Microsoft Live Search Webmaster Center Blog. And, feel free to share your thoughts with the webmaster community on the Site Explorer Suggestion Board.

  
  Priyank Garg
  Director, Product Management
  Yahoo! Search

  The Best Laid Plans Of Mice And Men
  

  Use Yahoo! Shopping Data on Your Site
  We're excited to announce the Yahoo! Shopping API, a new product that enables Web site owners and software developers to use Yahoo! comparison shopping data in their sites and applications. Using the Shopping API, developers can create applications that search...