User-agent: * Disallow: /clients/ Disallow: /public/ Disallow: /click-tracking* # Certain social media sites are whitelisted to allow crawlers to access page markup when links to subscription forms are shared. User-agent: Twitterbot Allow: /clients/ User-agent: facebookexternalhit Allow: /clients/ User-agent: LinkedInBot Allow: /clients/