If you don’t want your content to appear in the search engines’ indexes, you can them from crawling your content by using the robots.txt protocol. That will give the robots instructions on which pages they are allowed to crawl and/or index.
However, even if the search engines don’t crawl or index the content of pages blocked by robots.txt, they might index the URLs by discovering references to the excluded URLs in other sources on the Web.
The most effective way to remove your site URLs from search engines’ indexes is to use the following tools:
Google URL removal tool
Yahoo! Site Explorer
Bing Content removal
Tags: robots.txt