: Robots.txt for a CodeIgniter based website? I am working on a php site that uses Code Igniter framework and has the following folder structure on the server. public_html application

Posted in: #Codeigniter #Googlebot #RobotsTxt

I am working on a php site that uses Code Igniter framework and has the following folder structure on the server.

public_html

application
public
sql
style
system
design
images
js
plugin

.htaccess

robots.txt

sitemap.xml

index.php

The bulleted list of items are subfolders under public_html.

The application folder contains the code consisting of views and controllers.

My question is,

should googlebot be able to crawl folders other than application?

OR

should I Disallow all folders other than application so that no search engine bot gets to crawl other folders?

10.02% popularity Vote Up Vote Down

: Website Name shown at the end of HTML Titles A google search for site:cnn.com cnn returns results like Kenyans one dance away from semifinals - Sports - CNN.com Health News

@Bryan171

Posted in: #GoogleSearch #SearchEngines

3 Comments

: Want to use the HTTP-REFERER of in an HTML page through Javascript I Want to use the HTTP-REFERER of in an HTML page. Is it possible to do it with a JavaScript included in the page itself?

@Bryan171

Posted in: #HttpHeaders #Javascript

1 Comments

: You're on shared hosting, there are other sites on the same server, and the server serves different websites based on the domain name the browser requests. No, it won't affect your SEO.

@Bryan171

0 Comments

: Might it be the fact that you have a last rewrite flag on the first rule? I would remove the L from [NC, QSA, L]. RewriteRule ^gallery(/((([a-zA-Z0-9-]+)(/(d+))?)/?)?)?$ index.php?page=77&groupId=&showpage=

@Bryan171

0 Comments

Login to post a comment!

2 Comments

Sorted by latest first Latest Oldest Best

@Yeniel560

No search engine spider would even know about the application folder and thus not be able to crawl it anyway. They would only see what URLs you provide from the home page.

Actually IIRC the .htaccess file CI supplies should only allow access to specific folders like images so the application folder should never be accessible. So you don't really need to block anything in robots.txt.

10% popularity Vote Up Vote Down

@Bryan171

Your images folder might be useful if you want images to appear in google image search.

Everything else should probably be disallowed.

10% popularity Vote Up Vote Down

Feed

: Robots.txt for a CodeIgniter based website? I am working on a php site that uses Code Igniter framework and has the following folder structure on the server. public_html application

More posts by @Bryan171

: Website Name shown at the end of HTML Titles A google search for site:cnn.com cnn returns results like Kenyans one dance away from semifinals - Sports - CNN.com Health News

: Want to use the HTTP-REFERER of in an HTML page through Javascript I Want to use the HTTP-REFERER of in an HTML page. Is it possible to do it with a JavaScript included in the page itself?

: You're on shared hosting, there are other sites on the same server, and the server serves different websites based on the domain name the browser requests. No, it won't affect your SEO.

: Might it be the fact that you have a last rewrite flag on the first rule? I would remove the L from [NC, QSA, L]. RewriteRule ^gallery(/((([a-zA-Z0-9-]+)(/(d+))?)/?)?)?$ index.php?page=77&groupId=&showpage=

Login to post a comment!

2 Comments

Back to top | Use Dark Theme