Jon McPartland

Web developer

Implementing SSL with Varnish and Apache

In my server setup post, I mentioned that I sat Apache behind Varnish, so that the site would be nice and quick. In this post, I’ll detail how I set up the server for HTTPS-only connectivity—I’m a firm believer in HTTPS everywhere—which is normally straightforward, until Varnish is added into the mix. Caveat: I’m a big fan of Varnish, and I completely understand why the creator didn’t want to include support for HTTPS connections, so please don’t think that I’m criticising them; you can read their justification here.

So, because Varnish does not support connections over HTTPS, I needed to layer another package in front of it—a reverse proxy—which will handle the secure client-server communication, and then speak in plain HTTP to Varnish. This doesn’t cause any security issues, though, as the HTTP communication will be through the loopback interface directly on the server, so there’s no worry about leaking unencrypted data.

Flow overview

The reverse proxy I’m currently using is NGINX, which is an amazing piece of kit, though the same result could be achieved with HAProxy or Pound (I like Pound, but it isn’t as easy to configure SSL correctly as with NGINX, sadly). Once I’ve set up NGINX, the flow will be as follows:

  • Client-Server HTTPS communication handled by NGINX
  • NGINX decrypts and passes requests off to Varnish
  • If Varnish hasn’t cached the requested page, Varnish sends a request off to Apache
  • Apache will, with the help of Google’s Pagespeed module, serve an optimised page back to Varnish
  • Varnish will cache the page, and serve it to NGINX
  • NGINX serves the page to the Client

Installing NGINX

NGINX does have a Debian package already listed in the Ubuntu repositories, but it's an older version. I wanted to install a version greater than 1.9.5, as that is the earliest version of NGINX which includes HTTP/2 out of the box (HTTP/2 is awesome, but that's a story for another post). In order to install NGINX with the minimum version requirements, I needed to add the official, external repository into Aptitude.

  • I grabbed NGINX's signing key: curl http://nginx.org/keys/nginx_signing.key | sudo apt-key add nginx_signing.key
  • I added the source: sudo add-apt-repository "deb http://nginx.org/packages/ubuntu $(lsb_release -s -c) nginx"
  • I installed NGINX: sudo apt-get update && sudo apt-get install nginx

In the configuration file (located at /etc/nginx/nginx.conf), I needed to add SSL parameters, and proxy all connections to Varnish. So, before I finalised the NGINX config, I had to obtain an SSL certificate. For that, I used Let's Encrypt.

Obtaining an SSL Certificate from Let's Encrypt

The Let's Encrypt (henceforth LE) initiative is fantastic, and has great flexibility in the way that you can obtain a certificate through them. At the time of writing, they support auto-install for Apache, but there is no official install target for NGINX. However, since my configuration for NGINX isn't as a standard virtual host, I would want to configure it myself, anyway. To that end, I merely wanted certificate generation. My process was as follows.

  • Grab the repository from Github: cd ~ && git clone https://github.com/letsencrypt/letsencrypt.git && cd letsencrypt
  • Disable Varnish (LE binds to port 80 for verification)
  • Generate the certificate: ./letsencrypt-auto certonly --agree-tos -d mcpart.land -d www.mcpart.land -d jon.mcpart.land --rsa-key-size 4096
  • Re-enable Varnish (for now)

And that's it. LE doesn't just provide free SSL Certificates, it makes the process of obtaining one painless. I didn't even need to generate a Certificate Signing Request (CSR) - LE did it for me. If you use a different SSL Certificate provider, you'd generate a CSR with openssl req -nodes -newkey rsa:4096 -sha256 -keyout server.key -out server.csr

Configuring NGINX with SSL termination in reverse proxy mode

First, I backed up the default configuration file: cd /etc/nginx && mv nginx.conf nginx.conf.old

Next, I generated a set of Diffie-Hellman parameters, so that I could support Perfect Forward Secrecy in my SSL configuration. I chose 4096 bit, as that's the key size of my SSL certificate. Important: make sure you run this with sudo (if not running as root), as it took around 30 mins to generate on my box, and if the process can't write to the destination file due to permissions, you'll have to run it again!sudo openssl dhparam -out /etc/nginx/dhparam.pem 4096

  • In /etc/nginx/nginx.conf:
user www-data www-data;
pid  /var/run/nginx.pid;

worker_processes 4;

error_log  /var/log/nginx/error.log;
access_log /var/log/nginx/access.log combined;

events {
    worker_connections 512;
}

http {
    gzip          on;
    sendfile      on;
    server_tokens off;
    tcp_nodelay   on;
    tcp_nopush    on;
    keepalive_timeout   70;
    types_hash_max_size 2048;

    include /etc/nginx/mime.types;

    # redirect all (sub-)domain HTTP requests to target sub-domain as HTTPS
    server {
        server_name ~^((?!jon).)*\.?mcpart.land$;

        return 301 https://jon.mcpart.land$request_uri;
    }

    # redirect all (sub-)domain HTTPS requests to target sub-domain
    server {
        listen      443 ssl http2;
        listen      [::]:443 ssl http2;
        server_name ~^((?!jon).)*\.?mcpart.land$;
        include     /etc/nginx/conf.d/ssl.conf;

        return 301 https://jon.mcpart.land$request_uri;
    }

    # target sub-domain over HTTPS
    server {
        listen      443 ssl http2 default_server;
        listen      [::]:443 ssl http2 default_server;
        server_name jon.mcpart.land;
        include     /etc/nginx/conf.d/ssl.conf;
 
        # proxy to Varnish
        location / {
            proxy_set_header   Host $host;
            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header   X-Forwarded-Proto $scheme;
            proxy_set_header   X-Real-IP $remote_addr;
            proxy_pass         http://127.0.0.1:8080;
            proxy_redirect     http://127.0.0.1:8080 https://jon.mcpart.land;
            proxy_read_timeout 70;
        }
    }
}
  • In /etc/nginx/conf.d/ssl.conf:
ssl_certificate           /etc/letsencrypt/live/mcpart.land/fullchain.pem;
ssl_certificate_key       /etc/letsencrypt/live/mcpart.land/privkey.pem;
ssl_protocols             TLSv1.2;
ssl_ciphers               "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_prefer_server_ciphers on;
ssl_dhparam               /etc/nginx/dhparam.pem;
ssl_session_cache         shared:SSL:10m;
ssl_session_timeout       10m;
ssl_session_tickets       off;
ssl_stapling              on;
ssl_stapling_verify       on;

add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;

Note that this configuration only supports modern browsers; to include support for old Android, Java, IE, etc, you'll need to modify the ssl_protocols and ssl_ciphers parameters.

Note the headers I've added; the Strict-Transport-Security one is important if you want to declare your domain on the HSTS Preload list. In simple terms, this list (supported by all modern browsers) is a list stored within the browser; if someone visits http://yoursite.com—and yoursite.com is in the HSTS Preload list—the browser will perform a 307 redirect to https://yoursite.com before even sending a request to the server. This is especially useful if you're moving a long-standing, existing site to HTTPS-only, as it will bypass the need for the server to redirect HTTP connections to HTTPS, since it is done at the browser level.

Varnish config updates

Next, I had to modify my Varnish config to listen for connections on port 8080, as specified in the above NGINX config, rather than on port 80. I edited /etc/default/varnish, and replaced the line DAEMON_OPTS="-a :80 \ with DAEMON_OPTS="-a :8080 \

And finally

I had to modify my ufw rules to allow connections over port 443.

  • sudo ufw allow 443/tcp
  • sudo ufw disable
  • sudo ufw enable

I restarted the appropriate services (sudo service varnish restart and sudo service nginx restart), and the new configuration was up and running.

Let's Encrypt SSL Certificates have a lifetime of only 90 days so, to ensure that my certificate never expired, I set up a cron job to auto-renew my certificate within that period. Due to LE requiring a bind to port 80 whilst it's running, I had to create a shell script to stop NGINX, run LE, and restart NGINX. In my home folder, I created the script, and called it renewSSLCert.sh:

#!/bin/sh

service nginx stop
/home/jon/letsencrypt/letsencrypt-auto certonly -n --renew-by-default --rsa-key-size 4096
service nginx start

Note that some of these commands require sudo but, rather than add those to the list of commands my user can run with sudo without requiring a password, I decided to add the script to the root user's crontab. Using sudo crontab -e to add the cron job, I added the below line (I have added extra spacing here for reference only). This will run the renewal script at midnight every 2.8 months.

# m h dom  mon   dow command
  0 0 *    */2.8 *   /home/jon/renewSSLCert.sh

if you have any questions or feedback, please leave them in the comments below.