Using Robots.Txt to Hide Azure Blob Storage Data from Search Engines

Hello,

What if you want to make an Azure Blob storage container public so that you can quickly share data with someone – but don’t really want the rest of the world to discover that it’s in place? A simple approach would be to drop a robots.txt file in the directory with the following contents inside:

User-agent: *
Disallow: /

This will ensure that the directory isn’t crawled by search engines – which will reduce the risk of exposing the data to unwanted parties.

Azure blob storage robots.txt search engine

Azure blob storage robots.txt search engine

Ira Bell

COO & Co-Founder at Nimbo
Ira is a technology leader, with over 15 years in enterprise advisory space. He has deep technical knowledge of cloud computing, mergers and acquisitions, and integration. Ira co-founded Nimbo with the goal of leveraging cloud computing, business processes and information systems as a mechanism to help organizations become more competitive and successful in their field.
Posted in:
About the Author

Ira Bell

Ira is a technology leader, with over 15 years in enterprise advisory space. He has deep technical knowledge of cloud computing, mergers and acquisitions, and integration. Ira co-founded Nimbo with the goal of leveraging cloud computing, business processes and information systems as a mechanism to help organizations become more competitive and successful in their field.