Gün: 30 Ocak 2009

Dear Nurettin, Twiceler is the crawler for our new search engine. It is important to us that it obey robots.txt, and that it not crawl sites that do not wish to be crawled. Recently we have seen a number of crawlers masquerading as Twiceler, so please check that the IP address of the crawler in…