Search engine spider crawl your
periodically. How to limit the access of the spider? So the search
engine only crawled the page pages you want to show and leave others
ignored. What we need is a file called “robots.txt”. The robot text file
is used to disallow the search engine spider’s access to folders or
pages that you don’t want to be indexed.
Why you need “robots.txt”?
For example, you
may create some personal pages or you may have some company web pages,
which is for internal use only. You may also do not want to expose the
image folder.
How to use “robots.txt”?
You just need to upload “robots.txt” to the root of your web site.
How to create “robots.txt”?
Use a text editor (e.g. NotePad) to create a new file named “robots.txt”. The syntax is
User-agent: spiders_name_here
Disallow: filename_here
User-agent
You can use * acts as a wildcard and disallows all spiders.
Visit http://www.robotstxt.org/wc/active/html/ to find the name of user
agent.
Disallow
To disallow an entire directory use
Disallow: /directory_name/
To disallow an individual file use
Disallow: /filename
For multiple files, you need to write in separate lines.
Disallow:/file1.htm
Disallow:/file2.htm
Yes, it’s simple.
All you need is the above, and you can restrict the access of the spider.
We had been attempting to start your website with the latest google android mobile and for some reason the menu would start blinking but I wasn’t able to choose any of the options. Do you have any kind of idea what could be the issue?