https://wordpress.org/support/topic/no-robotstxt-file/page/2
you have a self-hosted blog, here are some of the things that can ease up your blogging experience-
create a new blank file on your desktop named "robots.txt" in there put this text (taken from mine, works fine):
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads
User-agent: Adsbot-Google
Allow: /
User-agent: Googlebot-Mobile
Allow: /
User-agent: ia_archiver-web.archive.org
Disallow: /
# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*
# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*
# digg mirror
User-agent: duggmirror
Disallow: /
Sitemap: http://yoursitename/sitemap.xml**********
Note 1.
change the last line to your sitemap.xml or sitemap.xml.gz file location, which is usually in the root of your wordpress folder, along with wp-admin folder and wp-content.
if you don't know it, keep reading 'till the end, I am explaining it again. if you use the "XML Sitemap Generator for WordPress" plugin (I do) then un-Tick the 'V' from the line "Add sitemap URL to the virtual robots.txt file." in your plugin configuration page.
*********
Note 2.
WordPress uses a "virtual" robots.txt file as long you don't have a real one in your wordpress blog-root. so once you upload it to your wordpress blog-root from your desktop all your problems should be gone.
*********
Note 3.
make sure you give a read permissions to every one (754 permissions is good) you can change it using ftp-client or you host's-cPanel.
O.K. lets continue...
once you've done all of the steps above, make sure you've properly registered in those two sites:
https://www.google.com/webmasters/tools
https://www.google.com/analyticsfollow the steps to add your website in the google-search index,
you'll need to verified that you've got access the the root by uploading a simple html file with a name like google0b11136ba11c6118.html once finished up you'll need to make your blog a more SEO-friendly, so make sure you've got the following plugins to run things smooth (no code insertion/ editing is needed):
-"All in One SEO Pack"
-"Google Analytics for WordPress"
-"Google XML Sitemaps"
-"Shockingly Simple Favicon" (recommended so you'll have an icon in the google-search page)
-"WordPress Mobile Pack" (so you can get into the google mobile results)you should re-visit the google links above in a few days to refine some of the settings, also copy-paste the google-id string from Google Analytics to "Google Analytics for WordPress" so it will track the visits in your blog.
after you've got a few posts in your blog, take a look at the "Google XML Sitemaps" settings page, rebuild manually, make sure the 'V' of the virtual robots.txt is UNTICKED and finds your location to the sitemap.xml and gz files, created by this plugin, usually its in your blog-root, using a ftp-client such as FileZilla browse your blog's folders find the robots.txt and make sure the line at the end having the right location to your sitemap.xml.gz (gz if a compressed file, of the sitemap.xml saving you extra web-traffic).
list your blog-website using some free-services you can find in google, that auto-submit your websites to many search engines around the world,
another way to make your website more search-engine friendly is to make sure it has Permalink (under settings in the admin page), selecting the buttom "Custom Structure" and putting "/%postname%/" and applying (without the semicolons, just /%postname%/) will make your posts look prettier, and listed better in the search engine's results. and when you are posting or editing a blog make sure to fill up the extra information that "All in One SEO Pack" plugin has put in for you to fill.
Google Webmaster Tool
https://www.google.com/webmasters/tools/home?hl=nl
Google Site verification
https://www.google.com/webmasters/verification/home?hl=nl
WebSite aanmelden
http://www.bing.com/toolbox/submit-site-url
https://www.google.com/webmasters/tools/submit-url
http://www.wpbeginner.com/beginners-guide/how-to-add-your-wordpress-site-to-google-webmaster-tools/
How to Add your WordPress Site to Google Webmaster Tools
Go to Google Webmaster tools website and sign in with a Google account. After signing in, add your website’s URL and hit the add new site button.
On the next screen, you will be asked to verify the ownership of the website you are adding. You can either upload an HTML file to your website’s root directory using FTP or click on Alternate method to obtain a meta tag which you can then add to your WordPress site.
To easily add the meta tag to your website you can install and activate our Insert Headers and Footers plugin. Copy the meta tag line provided by webmaster tools which will look something like this:
<meta name="google-site-verification" content="VerificationKeyCode" />
Now go to Settings » Insert Headers and Footers and paste the meta tag line into the header field. Save your changes and go back to Google Webmaster tools website and click on Verify site button.
Alternately if you are using WordPress SEO plugin, then you can copy the verification key in content part of the meta tag and paste it in WordPress SEO plugin. Click on SEO in your WordPress admin sidebar and then scroll down to webmaster tools section to paste the unique key in the Google Webmaster Tools field. Save your changes and go back to Google Webmaster tools website to verify your website.
Once you have verified your website, the next thing you would want to do is to add your XML sitemap. Don’t have a sitemap or not sure what is a sitemap? Check out our guide on how to create XML sitemaps in WordPress. After you have finished adding your sitemap, it may take a while for Google to display your site data.
That’s all you need to do to add your WordPress site to Google webmaster tools. Once your data starts coming in, you can use it to your advantage.
We hope that you found this article helpful. For more WordPress and SEO tips, you can tweet to us @WPBeginner or join WPBeginner on Facebook. For questions and feedback about this article please leave a comment below.
No Comments