Mercurial > hg > ywww

annotate robots.txt @ 45:8bc395c87c6f

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
less built-in use of full site address
author Henry S. Thompson <ht@inf.ed.ac.uk>
date Sun, 06 Jan 2019 08:09:47 -0500
parents 970a4834afff
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
1 User-agent: *
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
2 Disallow: /images/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
3 Disallow: /facebook/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
4 Disallow: /php/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
5 Disallow: /tt/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
6 Disallow: /twitter/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
7 Disallow: /user/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
8 Disallow: /xml/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
9 Disallow: /books_test/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
10 Disallow: /books_24th/
12
970a4834afff Testing hg - add folder to robots.txt
Rob Boland <robert@metail.co.uk>
parents: 6
diff changeset
11 Disallow: /goodreads/