User-agent: * Disallow: /common/ Disallow: /core/ Disallow: /docs/ Disallow: /domains/ Disallow: /pages/