How do I prevent specific pages from getting indexed by search engines?
To prevent specific pages from being indexed by search engines, follow these steps:
Click on the gear icon in the top menu to access the "Page Settings".
Within the page settings, you will find an option to enable or disable "Show page in search engines" for the selected page. Disable this option to prevent the page from being indexed.
After saving the changes and republishing your site, this action will add (or remove) a meta tag in the page HTML.
Why doesn't my robots.txt file change to "Disallow" when I disable indexing for a page?
The short answer is that excluding individual pages from indexing is managed through noindex meta robots tags, as recommended by Google. You can read more about this in Google’s documentation: Introduction to Robots.txt.
Using a robots.txt file to hide your web pages from Google search results is not recommended because if other pages link to your page with descriptive text, Google could still index the URL without visiting the page. To effectively block a page from search results, use the noindex meta tag or other methods such as password protection. More information can be found here: Block Search Indexing.
Please note that for the noindex directive to be effective, the page or resource must be accessible to the crawler and not blocked by a robots.txt file. If the page is blocked by a robots.txt file or the crawler cannot access it, the noindex directive will not be seen. Consequently, the page can still appear in search results if other pages link to it.
By following these steps, you can control which pages are indexed by search engines and manage your site's visibility effectively.