The following example shows you how to allow blank referers for social network crawlers when HTTP Referer protection is turned on. Social networks use different user agents to access content, so we need to allow blank referer for these user agents only. Any other user agents will receive a 403 Forbidden status code, as well as requests with non-listed referers.
EdgeRules Configuration
NGINX Configuration
location / {
valid_referers *.example.com;
set $flag “1”;
if ($http_user_agent ~* (Twitterbot|Pinterest|facebookexternalhit|LinkedInBot|Google) {
set $flag “0”;
}
if ($http_referer ~* ^$) {
set $invalid_referer $flag;
}
if ($invalid_referer = 1) {
return 403;
}
cURL Examples
In the examples below, the desired behavior can be verified via the cURL command, passing the appropriate request parameters to mimic different use cases.
No referrer and no user agent should return 403:
$ curl -I http://cdn.example.com/MaxCDN.png HTTP/1.1 403 Forbidden Date: Thu, 10 Mar 2016 19:10:23 GMT Content-Type: text/html Content-Length: 169 Connection: keep-alive Server: NetDNA-cache/2.2
user agent “Google” should be allowed:
$ curl -I http://cdn.example.com/MaxCDN.png -A "Google" HTTP/1.1 200 OK Date: Thu, 10 Mar 2016 19:10:47 GMT Content-Type: image/png Content-Length: 46538 Connection: keep-alive Last-Modified: Thu, 10 Mar 2016 18:24:10 GMT ETag: "56e1bbca-b5ca" Server: NetDNA-cache/2.2 X-Cache: HIT Accept-Ranges: bytes
user agent “Twitterbot” should be allowed:
$ curl -I http://cdn.example.com/MaxCDN.png -A "Twitterbot" HTTP/1.1 200 OK Date: Thu, 10 Mar 2016 19:10:53 GMT Content-Type: image/png Content-Length: 46538 Connection: keep-alive Last-Modified: Thu, 10 Mar 2016 18:24:10 GMT ETag: "56e1bbca-b5ca" Server: NetDNA-cache/2.2 X-Cache: HIT Accept-Ranges: bytes
user agent “Pinterest” should be allowed:
$ curl -I http://cdn.example.com/MaxCDN.png -A "Pinterest" HTTP/1.1 200 OK Date: Thu, 10 Mar 2016 19:10:58 GMT Content-Type: image/png Content-Length: 46538 Connection: keep-alive Last-Modified: Thu, 10 Mar 2016 18:24:10 GMT ETag: "56e1bbca-b5ca" Server: NetDNA-cache/2.2 X-Cache: HIT Accept-Ranges: bytes
user agent “facebookexternalhit” should be allowed:
$ curl -I http://cdn.example.com/MaxCDN.png -A "facebookexternalhit" HTTP/1.1 200 OK Date: Thu, 10 Mar 2016 19:11:05 GMT Content-Type: image/png Content-Length: 46538 Connection: keep-alive Last-Modified: Thu, 10 Mar 2016 18:24:10 GMT ETag: "56e1bbca-b5ca" Server: NetDNA-cache/2.2 X-Cache: HIT Accept-Ranges: bytes
user agent “LinkedInBot” should be allowed:
$ curl -I http://cdn.example.com/MaxCDN.png -A "LinkedInBot" HTTP/1.1 200 OK Date: Thu, 10 Mar 2016 20:09:31 GMT Content-Type: image/png Content-Length: 46538 Connection: keep-alive Last-Modified: Thu, 10 Mar 2016 18:24:10 GMT ETag: "56e1bbca-b5ca" Server: NetDNA-cache/2.2 X-Cache: HIT Accept-Ranges: bytes
As always, If you have any questions or concerns about any of the topics mentioned in this article, please feel free to reach out to support. Live chat and ticket support are available 24/7.