At Shopblocks, each customer receives several temporary subdomains so that they can access their website, admin system, and static assets. Part of the challenge of building Shopblocks was in providing all customers, by default, an SSL certificate.
During our prelaunch stage, our system was based heavily on Apache Virtual Hosts, with each customer getting their own Virtual Host file. This was necessary because of the SSL certificate configuration being required on a per-customer domain basis.
Addressing Limitations When Scaling
However, a problem with using Apache Virtual Hosts started to crop up when we grew our customer base and started getting into the thousands and tens of thousands of virtual host files.
We were finding the memory usage (without any request throughput) was increasing with the number of Virtual Host files loaded; graceful reloads were taking longer to perform. Each time a customer registered, a process was set off to create their config files and then to reload Apache.
Further issues appeared when we started building out our infrastructure for a hosted application, such as reloading multiple servers, ensuring that all servers are capable of serving all sites, handling a server that's gone offline, and ensuring server configurations were synchronized when they came back online.
Shifting SSL Termination Responsibility to HAProxy
Many of these issues were solved by removing the requirement of a Virtual Host per customer, but this left open the issue of SSL termination. We solved this by moving the responsibility for terminating SSL further up the chain to the HAProxy load balancer and away from Apache itself.
In HAProxy, we use map files and header rewrites to handle all domains without a change to any Apache configuration for new customers.
When you sign up to Shopblocks, you actually receive three subdomains. For example, if you registered with the name bowersbros, you will receive bowersbros.myshopblocks.com, bowersbros-admin.myshopblocks.com, and bowersbros-static.myshopblocks.com.
This will create a few entries in the map file for our HAProxy configuration. These lines look like this:
... bowersbros.myshopblocks.com <id> bowersbros-admin.myshopblocks.com <id> bowersbros-static.myshopblocks.com <id> ...
Replacing <id> with a specific identifier for your customer, you can reference the correct config/database as necessary. This will be passed through to your application as a HTTP header X-Customer-ID.
When registering, you cannot register with a dash in your name, so we can safely assume that any hyphens with -admin or -static are intended to hit those specific routes in the application (and no hyphen being the customers public website).
Our HAProxy configuration now looks something like this.
frontend http
bind :80
option forwardfor
# Redirect all http requests to be https with a 301 redirect
redirect scheme https code 301
frontend https
# Check for your PEM certificates
bind :443 ssl crt /path/to/certificates
# Check if the hostname is recognised in the map file
acl is_customer hdr(host),lower,map_str(/path/to/map/file.map) -m found
# If the domain is not recognised, then silently drop the conneciton
# You have the option of deny if you want to immediately reject this connection
http-request silent-drop unless is_customer
# Delete the X-Customer-ID header incase a header is sent through unexpectedly
http-request del-header X-Customer-ID
# Add a X-Forwarded-Host header with the original hostname
http-request add-header X-Forwarded-Host %[req.hdr(Host)]
http-request add-header X-Customer-ID %[hdr(host),lower,map_str(/path/to/map/file.map)]
default_backend core
backend core
balance roundrobin
http-request set-header Host app.<your domain>.com</your>This is a simplified version of the configuration we use, of course, and will need to be modified for your use case.
All requests that hit the HTTPS frontend get their SSL certificate checked to decrypt the request, then we check their domain against the map file. If we do not find them, they are dropped silently from the request.
We then add their ID to the request under X-Customer-ID, which will be sent with the request as it continues to your web server. Inside of your web server, you can then use the correct config and database connections based on the value of that header.
Conclusion
The Host header is also overwritten so that the correct Apache Virtual Host can be used, with a generic server name in Apache. This now allows your Apache Virtual Host configuration to be generic with one per application, rather than one per server. The only requirement when adding a new customer, domain, or SSL certificate is to modify the file.map and perform a reload in HAProxy, which is a zero-downtime action.
And there we have it, an overview of using HAProxy to setup a SaaS app simply.