: Robots.txt Disallow command How do I use robots.txt to disallow folders which are being crawled due to a bad URL structure? They're currently causing a duplicate page error. The URL which been

Posted in: #DuplicateContent #RobotsTxt #SearchEngines

How do I use robots.txt to disallow folders which are being crawled due to a bad URL structure? They're currently causing a duplicate page error.

The URL which been crawled incorrectly is:
abc.com/forum/index.php?option=com_forum

However, The correct page is:
abc.com/index.php?option=com_forum

Is robots.txt a correct way of excluding this? I'm thinking about using:

Disallow: /forum/

Will that not block legitimate content in the /forum/ folder of my site?

10.01% popularity Vote Up Vote Down

: Connecting to my local WampServer from mobile? I want to make an app for file transfer between mobile and a web server. I don't have web space yet and want to simulate that. The mobile part

@Kaufman445

Posted in: #Android #Php #Wamp

1 Comments

: What are the SEO Benefits of adding a Tumblr feed to a website? A client of ours has a CMS driven Blog in his hotel site - he would like to use the blog to add depth top his site and

@Kaufman445

Posted in: #Blog #Feeds #Seo #Tumblr

2 Comments

: How to make an apDiv show up in same spot on all browsers I'm pretty new to the whole webmaster stuff and am just trying to format a slideshow on my homepage to be in a certain area with

@Kaufman445

Posted in: #Flash #Html #Php

1 Comments

: Breadcrumbs RDFA Have implemented Breadcrumb RDFA http://www.mycarhelpline.com/index.php?option=com_forms&view=pages&layout=sellcar&Itemid=4 While checking the page , the RDFA Data shows property:

@Kaufman445

Posted in: #Breadcrumbs #Rdfa #RichSnippets

1 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Dunderdale272

If abc.com/forum/index.php?option=com_forum is the only link that's been mis-read then you can just add Disallow: /forum/index.php to your robots.txt file. Adding Disallow: /forum/ will tell robots to ignore everything in that directory, which doesn't sound like what you want.

You could also add a 301 redirect from abc.com/forum/index.php?option=com_forum to abc.com/index.php?option=com_forum to tell robots what the URL should be. This will also help any users who accidentally land on the wrong URL.

10% popularity Vote Up Vote Down

Feed

: Robots.txt Disallow command How do I use robots.txt to disallow folders which are being crawled due to a bad URL structure? They're currently causing a duplicate page error. The URL which been

More posts by @Kaufman445

: Connecting to my local WampServer from mobile? I want to make an app for file transfer between mobile and a web server. I don't have web space yet and want to simulate that. The mobile part

: What are the SEO Benefits of adding a Tumblr feed to a website? A client of ours has a CMS driven Blog in his hotel site - he would like to use the blog to add depth top his site and

: How to make an apDiv show up in same spot on all browsers I'm pretty new to the whole webmaster stuff and am just trying to format a slideshow on my homepage to be in a certain area with

: Breadcrumbs RDFA Have implemented Breadcrumb RDFA http://www.mycarhelpline.com/index.php?option=com_forms&view=pages&layout=sellcar&Itemid=4 While checking the page , the RDFA Data shows property:

Login to post a comment!

1 Comments

Back to top | Use Dark Theme