Mercurial > hg > ywww

annotate robots.txt @ 35:86f79bc1d142

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
refactor to split out shareable fn to actually do an Amazon API request
author Charlie Root
date Fri, 04 Jan 2019 12:54:12 -0500
parents 970a4834afff
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
6
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
1 User-agent: *
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
2 Disallow: /images/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
3 Disallow: /facebook/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
4 Disallow: /php/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
5 Disallow: /tt/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
6 Disallow: /twitter/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
7 Disallow: /user/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
8 Disallow: /xml/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
9 Disallow: /books_test/
077b0a0a3e6d remaining originals according to dependency walk
Robert Boland <robert@markup.co.uk>
parents:
diff changeset
10 Disallow: /books_24th/
12
970a4834afff Testing hg - add folder to robots.txt
Rob Boland <robert@metail.co.uk>
parents: 6
diff changeset
11 Disallow: /goodreads/