How to create Robots.txt File for SEO and WordPress and website|Ramdas khillare|SEO
1. What is Robots.txt file
These are the very First file which is inspect and crawled by Google or other search engine bots
these are a simple text file with the show the whole idea of our website with including accessing permission and overall sitemap
Web site owners use the /robots.txt file to give instructions about their site to web robots this we called The Robots Exclusion Protocol.
The robots.txt file is part of the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users. The REP also includes directives like meta robots, as well as page-, subdirectory or site-wide instructions
To which pages we have to be a private from the google searches
Basic Format is Like
2. Importance of Robots.txt file
It is very important to Create the Robots.txt file Because it sets the permission to the Google Robots to what is to access and which part of the website is to index in website
2.1 which pages we want disallow for google robots it includes
1.WP-admin area
2.Comment area
which is on your preference But
Admin and the Root Folder you must Want to Daniel the access Using
Disallow: [URL string not to be crawled] keyword
2.2 provide website sitemap and overall architecture of your website
3. How to see a robots.txt file of any website
It is the very simple method we can see the robots.txt file of any website its same like checking sitemap
4. How robots.text file look like
4. where to add sitemap into the robots.txt file
After describing the different google robots agent with allowing and disallow page
we have to add the whole sitemap of the website in the bottom of your robots.txt file
It includes
4.1 sitemap.xml
It simply shows the hierarchy pages which included in your site it also contains all internal of website pages
4.2 sitemap-image.xml
it includes the sitemap of image file because if you want to rank your page with the specific image it also added importance to rank in google as specific content we must include all captioned and ALT tag images
4.3 sitemap-video.xml
it same as the image it shows all architecture of included video which is in your website
5. where to add robots.txt
this file you have to add in your root folder of Website or blog
.Public/index.html
etc
So if you have a Website/blog must create Robots.txt fie and add to it
0 Comments