Useragent YandexBot allow a command thanks to which the Yandex search bot can index all pages of the resource. Crawldelay the definition of the time period that the robot must withstand in the process of loading pages. It is neee in order to eliminate excessive load on the server. However the nee for this command […]