Mercurial > hg > ywww

annotate robots.txt @ 50:99a730ffeaf6

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
more parameterisation
author Charlie Root
date Sun, 27 Jan 2019 14:24:02 -0500
parents 970a4834afff
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
1 User-agent: *
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
2 Disallow: /images/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
3 Disallow: /facebook/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
4 Disallow: /php/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
5 Disallow: /tt/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
6 Disallow: /twitter/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
7 Disallow: /user/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
8 Disallow: /xml/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
9 Disallow: /books_test/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
10 Disallow: /books_24th/
12
970a4834afff Testing hg - add folder to robots.txt
Rob Boland <robert@metail.co.uk>
parents: 6
diff changeset
11 Disallow: /goodreads/