Octeth v6.0.7 Now Available! See What's New

Stage 3 of 12

Server Infrastructure for an Email Service Provider

An ESP on Octeth runs on four servers: the app server, the MTA server, a link tracking edge and a monitoring server. This guide covers what each one does, the sizes in our runbooks, the operating system, SSH hardening, the open ports per server and how to put Cloudflare in front of the app without breaking mail.

By Cem Hurturk, founder of Octeth Updated 10 min read

Skills
  • Linux administration
  • SSH and firewall hardening
  • Docker
  • Cloudflare
On this page
  1. The four server roles
  2. Server sizes
  3. Operating system
  4. Harden every server
  5. Firewall per server
  6. Cloudflare in front of the app

Run the ESP on separate servers with separate jobs. The app server stays closed behind Cloudflare. The MTA server talks to mailbox providers. A small tracking edge takes the public traffic from tracked links. A monitoring server watches all three. Each server opens only the ports its job needs.

If you send for one business only, start with the VPS guide for email marketing. This guide covers the multi-tenant case, where many customers send through your platform.

The four server roles

ServerWhat it runsWho connects to it
App serverOcteth in Docker: the customer app, staff area, API, database and queuesCustomers and staff through Cloudflare, the tracking edge, your admin IPs
MTA serverThe mail transfer agent, such as PowerMTA, and the sending IPsThe app server, to hand over mail. The MTA connects out to mailbox providers.
Tracking edgeCaddy with on-demand TLS, proxying to the app serverRecipients clicking tracked and unsubscribe links
Monitoring serverPrometheus, Grafana, AlertmanagerYour team, through Cloudflare. It scrapes the other servers.
The four server roles of an ESPCustomers and staff reach the app server through Cloudflare, which proxies the app, staff area and API in Full (strict) mode on ports 80 and 443. Recipients who click tracked and unsubscribe links reach the tracking edge, Caddy with on-demand TLS, on ports 80 and 443, and the edge proxies those requests to the app server over HTTPS on port 443. The app server runs Octeth in Docker: the customer app, staff area, API, database and queues. It hands mail to the MTA server over SMTP on port 587. The MTA server holds the mail transfer agent and the sending IPs, connects out to mailbox providers on port 25, and receives bounce and feedback loop mail on port 25. The monitoring server runs Prometheus, Grafana and Alertmanager. It scrapes the app server and the tracking edge, and scrapes the MTA server or receives the metrics the MTA pushes.Customers, staffApp, staff area, APIRecipientsTracked links80, 443CloudflareProxied, Full (strict)Tracking edgeCaddy, on-demand TLS80, 443443App serverOcteth in Docker: customer app,staff area, API, database, queuesSMTP, port 587MTA serverMail transfer agent, sending IPsPort 25 outBounce and feedbackloop mail, port 25Mailbox providersMonitoring serverPrometheus, Grafana, AlertmanagerScrapes or receives metrics
The four server roles. Only the app server holds Octeth, only the MTA server holds the sending IPs, and the tracking edge takes the public link traffic so the app server stays behind Cloudflare.

Why the MTA gets its own server

The MTA holds your sending IPs, their reputation and the MTA license. Keep it apart from the app, so you can work on one without touching the other.

There is a second reason if you want help running the app. Octeth Care maintains the Octeth server and requires a server used only for Octeth. The MTA therefore runs on its own server on every path where we maintain the app.

Why tracking runs on its own edge

With a verified sender domain, Octeth builds campaign links on a host under the customer’s domain. That host is a CNAME to a shared host you own, which is DNS only. Recipients connect straight to whatever that host points at, and a certificate must be issued for each customer host on first use, over port 80.

The app server accepts web traffic only from Cloudflare and your admin IPs, so it cannot serve those links. A tracking edge with Caddy on-demand TLS serves them and proxies the requests to the app over HTTPS. Caddy asks Octeth before it issues each certificate, so it only gets certificates for domains Octeth knows. The edge keeps the app closed and takes abuse traffic on a server you can replace.

The other option is to open 80 and 443 on the app server to everyone and keep the admin host Cloudflare-only with a local HAProxy rule. Upgrades overwrite that rule, so we use the edge.

Why monitoring runs apart

Monitoring on the app server goes down with the app server. Put it on its own server. If that server shares hardware with what it watches, add an outside check that alerts when monitoring itself stops. Stage 12 covers the monitoring setup.

A separate domain for sending infrastructure

Run the app on one domain, and put the sender, tracking and SMTP hosts on a second domain used only for sending infrastructure. A blocklisting of the sending domain then cannot reach the domain your customers log in to.

Server sizes

Size the app server from Octeth’s server requirements page, which lists sizes by subscriber count for each version. Our runbooks give the monitoring server 2 vCPU, 4 GB RAM and 60 GB of disk or more, and the tracking edge 1 vCPU and 1 to 2 GB RAM.

What we learned about sizing:

  • CPU. Octeth’s docs call 2 vCPUs the minimum, for testing, and 4 the recommended size. Octeth sets CPU limits for the send engine and link proxy. On a server with fewer than 4 cores, check those limits against the core count before and after the install. A limit above the core count stops the container from starting. The installation docs explain the keys.
  • Memory. MySQL is the heaviest container. Leave memory for the rest of the stack and add swap.
  • Disk on the monitoring server. The metrics database shares the root disk. Cap its retention by time and by size, for example 90 days and 60 GB.

Operating system

Use Ubuntu 24.04 LTS on every server. Octeth supports Ubuntu 24.04 LTS on x86_64 and tests each release on it, and Octeth Care requires it. Start from a fresh install with no control panel, web server, database or mail server on it. Octeth installs its own services in Docker.

Install Docker from Docker’s own apt repository, not Ubuntu’s docker.io package. Octeth’s server setup page has the commands for your version.

Slow provider DNS resolvers break calls the server makes to its own app domain and to other hosts. Pin public resolvers if you see resolver timeouts.

Harden every server

Apply the same baseline to the app, MTA, tracking edge and monitoring server. Keep a second SSH session open while you change SSH, so a mistake does not lock you out.

  1. Make SSH key-only. Put the settings in a drop-in file:

    # /etc/ssh/sshd_config.d/99-hardening.conf
    PermitRootLogin prohibit-password
    PasswordAuthentication no
    KbdInteractiveAuthentication no
    X11Forwarding no
    MaxAuthTries 3
    LoginGraceTime 30
    ClientAliveInterval 300
    ClientAliveCountMax 2

    sshd uses the first value it reads, and drop-in files load in name order. Set PasswordAuthentication no in any cloud-init file that enables it. Check the result with sshd -T, then try a password login from a new connection. It must be refused.

  2. Run fail2ban on sshd:

    # /etc/fail2ban/jail.local
    [DEFAULT]
    bantime = 1h
    findtime = 10m
    maxretry = 5
    backend = systemd
    
    [sshd]
    enabled = true
  3. Allow SSH only from your admin IPs, in the firewall script below.

  4. Put /etc in git and commit after each change. Exclude the SSH host private keys: they have no file extension, so a *.key rule misses them. If the first commit includes them, rebuild the repository without them.

  5. Install updates before you install anything else.

Firewall per server

How we run the firewall

  • Deny by default. Deny incoming and routed traffic, allow outgoing.
  • One script per server. Every rule lives in a script that resets ufw and applies all rules. A rule added by hand disappears on the next run. Run the scripts from cron, monthly for example, and fetch Cloudflare’s current IP ranges each time. The script checks the downloaded list before it resets anything.
  • Firewall before Octeth. Docker publishes container ports past ufw’s normal rules. Install the ufw-docker helper and run the firewall script before you install Octeth. Rules for container ports are route rules, and they match the container’s port, not the host port it is mapped to.
  • Test from outside. From an IP that is not on your admin list, every closed port must time out. Use the open port checker or a multi-location TCP check.

Open ports by server

Each table lists every inbound port. Everything else stays closed.

App server

PortOpen toPurpose
22Admin IPsSSH
80, 443Cloudflare’s IP ranges, admin IPsApp, staff area and API through Cloudflare
443Tracking edgeTracked links proxied from the edge
9100, 9104, 9121, 9280, 15692Monitoring serverHost, MySQL, Redis, container and RabbitMQ metrics
587Customers, once you offer SMTP relayCustomer SMTP submission
25NobodyOcteth publishes it. Keep it blocked.

MTA server

The MTA gets the same baseline: deny by default, and SSH only from your admin IPs, as in the hardening section. Beyond that it must be reachable for two jobs:

  • Mail from the app. Octeth hands mail to the MTA over SMTP on port 587. Open 587 on the MTA to the app server.
  • Mail for the MTA’s own domains. Bounces and feedback loop mail arrive on port 25. Limit what the MTA accepts in its relay configuration: accept inbound mail only for its own relay domains and refuse relaying for anything else.

Tracking edge

PortOpen toPurpose
22Admin IPsSSH
80EveryoneHTTPS redirect and certificate checks
443EveryoneTracked and unsubscribe links
9100, 9114Monitoring serverHost and Caddy metrics

Monitoring server

PortOpen toPurpose
22Admin IPsSSH
80, 443Cloudflare’s IP ranges, admin IPsGrafana through Cloudflare

Notes on the MTA:

  • Outbound port 25. The MTA must reach mailbox providers on port 25. Some providers block it on new servers. Vultr blocks outbound port 25 on new instances by default, which is fine for a tracking edge and wrong for an MTA. Check your provider’s policy before you order the MTA server.
  • The MTA’s web console. Keep PowerMTA’s web console on localhost and reach it through an SSH tunnel.
  • Metrics. When the monitoring server cannot scrape an MTA, have the MTA push its metrics instead, and open the receiving port on the monitoring server to the MTA only.

Before you open 587 on the app server to customers, load a TLS certificate into Octeth’s SMTP relay and check its per-IP connection limit (Stage 5, sending infrastructure and IP warm-up). Without a certificate, the relay offers no STARTTLS and accepts passwords in plain text.

Cloudflare in front of the app

Settings for the app domain

  • Proxied, Full (strict). Proxy the app domain through Cloudflare and use Full (strict) mode with a Cloudflare Origin CA certificate on the server. Never use Flexible for the app: Flexible sends traffic from Cloudflare to the server unencrypted, API keys included.

  • Origin closed to everyone else. The firewall above admits only Cloudflare’s ranges, your admin IPs and the tracking edge on 80 and 443.

  • Always Use HTTPS on, minimum TLS 1.2.

  • A redirect from / and /login to /user/login, the customer login.

  • A WAF rule on the admin paths. Octeth’s classic admin answers on /admin/ and /app/admin/, and the staff area is under /user/staff. Block them for everyone but your admin IPs. On 6.0.0, also allow the server’s own IP, because that version’s health check loads the admin login page through the public URL:

    (lower(http.request.uri) contains "/admin"
      or starts_with(http.request.uri.path, "/user/staff"))
    and not ip.src in {192.0.2.10 198.51.100.20 203.0.113.5}

    Replace the addresses with your admin IPs, plus the app server’s IP on 6.0.0. If the zone has other admin-only hosts, such as Grafana on reports.example.com, add or http.host eq "reports.example.com" to the same rule.

  • The visitor’s real IP. Behind Cloudflare, Octeth must know which proxies to trust, or it records a Cloudflare edge IP for every login and click. Which setting applies depends on your Octeth version and on how TLS ends at the server. See Octeth’s configuration docs. Include the tracking edge in the trusted list, then check that your own login shows your real IP. Cloudflare’s ranges change rarely, but when they do, update the list.

  • Optional: Authenticated Origin Pulls. The firewall admits any request from Cloudflare’s network. Zone-level Authenticated Origin Pulls, with a certificate you upload, makes the server accept only requests that came through your zone. On Octeth this is a local HAProxy change, so re-apply it after each upgrade.

What must not be proxied

Cloudflare’s proxy carries web traffic. Mail and the hosts recipients reach directly stay DNS only.

RecordProxy statusWhy
App domain, such as app.example.comProxiedWeb traffic
Shared sender host, such as mailer.example.netDNS onlyPoints at the tracking edge
SMTP host for customer relay, such as smtp.example.netDNS onlySMTP is not web traffic
MTA hostnames that match your sending IPs’ reverse DNSDNS onlyMust resolve to the real IPs
MX, SPF, DKIM and DMARC recordsDNS onlyOnly A, AAAA and CNAME can be proxied

Check the sender host and your MX records with the MX inspector after each DNS change. Stage 5 covers the MTA’s reverse DNS and Stage 6 the sender domain records.

Before you move on

  • You have four servers, or a plan for them: app, MTA, tracking edge and monitoring.
  • The MTA runs on its own server, not on the app server.
  • Every server runs Ubuntu 24.04 LTS, installed fresh, with no control panel, web server or mail server on it.
  • SSH is key-only, password login is refused when you test it, and fail2ban watches sshd.
  • SSH is open only to your admin IPs on every server.
  • /etc is in git on every server, without the SSH host private keys.
  • One firewall script per server holds every rule, and it ran before Octeth was installed.
  • From outside, every port you did not open times out.
  • The app domain is proxied by Cloudflare in Full (strict) mode, and the origin accepts web traffic only from Cloudflare, your admin IPs and the tracking edge.
  • A Cloudflare WAF rule blocks the admin paths for everyone but your admin IPs, and the server itself on 6.0.0.
  • Your own login shows your real IP in Octeth, not a Cloudflare address.
  • The tracking host, the SMTP host and the MTA hostnames are DNS only, and the tracking host points at the tracking edge.

What we've seen go wrong

Mail ports open before the firewall

On one install, Octeth went in before the firewall. Docker publishes ports 25, 80, 443 and 587 on all addresses and bypasses plain ufw, so those ports stayed open to the internet until the firewall step. Our runbook now applies the firewall right after Docker and before the Octeth install.

Password login that stayed on

After SSH hardening on one server, sshd -T still showed password login on. The cloud image had a drop-in file, 50-cloud-init.conf, that enabled it, and sshd uses the first value it reads, so it won over our 99-hardening.conf. Check every file in sshd_config.d, then try a password login from a new connection and confirm it is refused.

A hostname that broke the health check

When the server's hostname is the app domain, cloud-init maps that domain to 127.0.1.1 in /etc/hosts, and calls from the server to its own app, such as Octeth's health check, fail. Mapping the domain to the public IP does not work either, because the Cloudflare Origin CA certificate is trusted only by Cloudflare. Comment out the line in the cloud-init template and in /etc/hosts, so the domain resolves through Cloudflare.

Tracking links that pointed at the MTA

On one platform, the shared sender host that customer domains point to had its A record on the MTA's IP. The MTA serves no web traffic, so every tracked link and every unsubscribe link failed with a TLS error. That host must point at the tracking edge. Click a real tracked link and the unsubscribe link from a sent email before you call the setup done.

A database that took the server's memory

On an install we support, MySQL's buffer pool was set close to the server's total RAM, with no swap. The kernel killed MySQL again and again, and after one kill a campaign went out about three times. Leave memory for the other containers and add swap.

Provider DNS that timed out

On one app server, on a version before 6.0.1, the hosting provider's DNS resolvers timed out for hours. That version's health check resolved the app URL with a 3-second limit, so it returned 503 sixteen times in an hour and a half. From 6.0.1 the health endpoint probes the login pages inside the container. The `health:check` command still calls the app URL. We pinned public resolvers in the network configuration and turned off the resolvers handed out by DHCP.

Cloudflare changes that did nothing

On one domain, rule changes made through the Cloudflare API succeeded and read back correctly, but live traffic did not change. The zone was still pending in that account, and the domain's nameservers belonged to a different Cloudflare account. Check that the zone is active and that the live nameservers match it, then test every change with a live request.

We build ESPs

Rather have us build this?

We set up, harden, maintain and monitor the app server and the MTA server for you.