Stage 3 of 12
Server Infrastructure for an Email Service Provider
An ESP on Octeth runs on four servers: the app server, the MTA server, a link tracking edge and a monitoring server. This guide covers what each one does, the sizes in our runbooks, the operating system, SSH hardening, the open ports per server and how to put Cloudflare in front of the app without breaking mail.
By Cem Hurturk, founder of Octeth Updated 10 min read
- Linux administration
- SSH and firewall hardening
- Docker
- Cloudflare
On this page
Run the ESP on separate servers with separate jobs. The app server stays closed behind Cloudflare. The MTA server talks to mailbox providers. A small tracking edge takes the public traffic from tracked links. A monitoring server watches all three. Each server opens only the ports its job needs.
If you send for one business only, start with the VPS guide for email marketing. This guide covers the multi-tenant case, where many customers send through your platform.
The four server roles
| Server | What it runs | Who connects to it |
|---|---|---|
| App server | Octeth in Docker: the customer app, staff area, API, database and queues | Customers and staff through Cloudflare, the tracking edge, your admin IPs |
| MTA server | The mail transfer agent, such as PowerMTA, and the sending IPs | The app server, to hand over mail. The MTA connects out to mailbox providers. |
| Tracking edge | Caddy with on-demand TLS, proxying to the app server | Recipients clicking tracked and unsubscribe links |
| Monitoring server | Prometheus, Grafana, Alertmanager | Your team, through Cloudflare. It scrapes the other servers. |
Why the MTA gets its own server
The MTA holds your sending IPs, their reputation and the MTA license. Keep it apart from the app, so you can work on one without touching the other.
There is a second reason if you want help running the app. Octeth Care maintains the Octeth server and requires a server used only for Octeth. The MTA therefore runs on its own server on every path where we maintain the app.
Why tracking runs on its own edge
With a verified sender domain, Octeth builds campaign links on a host under the customer’s domain. That host is a CNAME to a shared host you own, which is DNS only. Recipients connect straight to whatever that host points at, and a certificate must be issued for each customer host on first use, over port 80.
The app server accepts web traffic only from Cloudflare and your admin IPs, so it cannot serve those links. A tracking edge with Caddy on-demand TLS serves them and proxies the requests to the app over HTTPS. Caddy asks Octeth before it issues each certificate, so it only gets certificates for domains Octeth knows. The edge keeps the app closed and takes abuse traffic on a server you can replace.
The other option is to open 80 and 443 on the app server to everyone and keep the admin host Cloudflare-only with a local HAProxy rule. Upgrades overwrite that rule, so we use the edge.
Why monitoring runs apart
Monitoring on the app server goes down with the app server. Put it on its own server. If that server shares hardware with what it watches, add an outside check that alerts when monitoring itself stops. Stage 12 covers the monitoring setup.
A separate domain for sending infrastructure
Run the app on one domain, and put the sender, tracking and SMTP hosts on a second domain used only for sending infrastructure. A blocklisting of the sending domain then cannot reach the domain your customers log in to.
Server sizes
Size the app server from Octeth’s server requirements page, which lists sizes by subscriber count for each version. Our runbooks give the monitoring server 2 vCPU, 4 GB RAM and 60 GB of disk or more, and the tracking edge 1 vCPU and 1 to 2 GB RAM.
What we learned about sizing:
- CPU. Octeth’s docs call 2 vCPUs the minimum, for testing, and 4 the recommended size. Octeth sets CPU limits for the send engine and link proxy. On a server with fewer than 4 cores, check those limits against the core count before and after the install. A limit above the core count stops the container from starting. The installation docs explain the keys.
- Memory. MySQL is the heaviest container. Leave memory for the rest of the stack and add swap.
- Disk on the monitoring server. The metrics database shares the root disk. Cap its retention by time and by size, for example 90 days and 60 GB.
Operating system
Use Ubuntu 24.04 LTS on every server. Octeth supports Ubuntu 24.04 LTS on x86_64 and tests each release on it, and Octeth Care requires it. Start from a fresh install with no control panel, web server, database or mail server on it. Octeth installs its own services in Docker.
Install Docker from Docker’s own apt repository, not Ubuntu’s docker.io package. Octeth’s server setup page has the commands for your version.
Slow provider DNS resolvers break calls the server makes to its own app domain and to other hosts. Pin public resolvers if you see resolver timeouts.
Harden every server
Apply the same baseline to the app, MTA, tracking edge and monitoring server. Keep a second SSH session open while you change SSH, so a mistake does not lock you out.
-
Make SSH key-only. Put the settings in a drop-in file:
# /etc/ssh/sshd_config.d/99-hardening.conf PermitRootLogin prohibit-password PasswordAuthentication no KbdInteractiveAuthentication no X11Forwarding no MaxAuthTries 3 LoginGraceTime 30 ClientAliveInterval 300 ClientAliveCountMax 2sshd uses the first value it reads, and drop-in files load in name order. Set
PasswordAuthentication noin any cloud-init file that enables it. Check the result withsshd -T, then try a password login from a new connection. It must be refused. -
Run fail2ban on sshd:
# /etc/fail2ban/jail.local [DEFAULT] bantime = 1h findtime = 10m maxretry = 5 backend = systemd [sshd] enabled = true -
Allow SSH only from your admin IPs, in the firewall script below.
-
Put
/etcin git and commit after each change. Exclude the SSH host private keys: they have no file extension, so a*.keyrule misses them. If the first commit includes them, rebuild the repository without them. -
Install updates before you install anything else.
Firewall per server
How we run the firewall
- Deny by default. Deny incoming and routed traffic, allow outgoing.
- One script per server. Every rule lives in a script that resets ufw and applies all rules. A rule added by hand disappears on the next run. Run the scripts from cron, monthly for example, and fetch Cloudflare’s current IP ranges each time. The script checks the downloaded list before it resets anything.
- Firewall before Octeth. Docker publishes container ports past ufw’s normal rules. Install the ufw-docker helper and run the firewall script before you install Octeth. Rules for container ports are route rules, and they match the container’s port, not the host port it is mapped to.
- Test from outside. From an IP that is not on your admin list, every closed port must time out. Use the open port checker or a multi-location TCP check.
Open ports by server
Each table lists every inbound port. Everything else stays closed.
App server
| Port | Open to | Purpose |
|---|---|---|
| 22 | Admin IPs | SSH |
| 80, 443 | Cloudflare’s IP ranges, admin IPs | App, staff area and API through Cloudflare |
| 443 | Tracking edge | Tracked links proxied from the edge |
| 9100, 9104, 9121, 9280, 15692 | Monitoring server | Host, MySQL, Redis, container and RabbitMQ metrics |
| 587 | Customers, once you offer SMTP relay | Customer SMTP submission |
| 25 | Nobody | Octeth publishes it. Keep it blocked. |
MTA server
The MTA gets the same baseline: deny by default, and SSH only from your admin IPs, as in the hardening section. Beyond that it must be reachable for two jobs:
- Mail from the app. Octeth hands mail to the MTA over SMTP on port 587. Open 587 on the MTA to the app server.
- Mail for the MTA’s own domains. Bounces and feedback loop mail arrive on port 25. Limit what the MTA accepts in its relay configuration: accept inbound mail only for its own relay domains and refuse relaying for anything else.
Tracking edge
| Port | Open to | Purpose |
|---|---|---|
| 22 | Admin IPs | SSH |
| 80 | Everyone | HTTPS redirect and certificate checks |
| 443 | Everyone | Tracked and unsubscribe links |
| 9100, 9114 | Monitoring server | Host and Caddy metrics |
Monitoring server
| Port | Open to | Purpose |
|---|---|---|
| 22 | Admin IPs | SSH |
| 80, 443 | Cloudflare’s IP ranges, admin IPs | Grafana through Cloudflare |
Notes on the MTA:
- Outbound port 25. The MTA must reach mailbox providers on port 25. Some providers block it on new servers. Vultr blocks outbound port 25 on new instances by default, which is fine for a tracking edge and wrong for an MTA. Check your provider’s policy before you order the MTA server.
- The MTA’s web console. Keep PowerMTA’s web console on localhost and reach it through an SSH tunnel.
- Metrics. When the monitoring server cannot scrape an MTA, have the MTA push its metrics instead, and open the receiving port on the monitoring server to the MTA only.
Before you open 587 on the app server to customers, load a TLS certificate into Octeth’s SMTP relay and check its per-IP connection limit (Stage 5, sending infrastructure and IP warm-up). Without a certificate, the relay offers no STARTTLS and accepts passwords in plain text.
Cloudflare in front of the app
Settings for the app domain
-
Proxied, Full (strict). Proxy the app domain through Cloudflare and use Full (strict) mode with a Cloudflare Origin CA certificate on the server. Never use Flexible for the app: Flexible sends traffic from Cloudflare to the server unencrypted, API keys included.
-
Origin closed to everyone else. The firewall above admits only Cloudflare’s ranges, your admin IPs and the tracking edge on 80 and 443.
-
Always Use HTTPS on, minimum TLS 1.2.
-
A redirect from
/and/loginto/user/login, the customer login. -
A WAF rule on the admin paths. Octeth’s classic admin answers on
/admin/and/app/admin/, and the staff area is under/user/staff. Block them for everyone but your admin IPs. On 6.0.0, also allow the server’s own IP, because that version’s health check loads the admin login page through the public URL:(lower(http.request.uri) contains "/admin" or starts_with(http.request.uri.path, "/user/staff")) and not ip.src in {192.0.2.10 198.51.100.20 203.0.113.5}Replace the addresses with your admin IPs, plus the app server’s IP on 6.0.0. If the zone has other admin-only hosts, such as Grafana on
reports.example.com, addor http.host eq "reports.example.com"to the same rule. -
The visitor’s real IP. Behind Cloudflare, Octeth must know which proxies to trust, or it records a Cloudflare edge IP for every login and click. Which setting applies depends on your Octeth version and on how TLS ends at the server. See Octeth’s configuration docs. Include the tracking edge in the trusted list, then check that your own login shows your real IP. Cloudflare’s ranges change rarely, but when they do, update the list.
-
Optional: Authenticated Origin Pulls. The firewall admits any request from Cloudflare’s network. Zone-level Authenticated Origin Pulls, with a certificate you upload, makes the server accept only requests that came through your zone. On Octeth this is a local HAProxy change, so re-apply it after each upgrade.
What must not be proxied
Cloudflare’s proxy carries web traffic. Mail and the hosts recipients reach directly stay DNS only.
| Record | Proxy status | Why |
|---|---|---|
App domain, such as app.example.com | Proxied | Web traffic |
Shared sender host, such as mailer.example.net | DNS only | Points at the tracking edge |
SMTP host for customer relay, such as smtp.example.net | DNS only | SMTP is not web traffic |
| MTA hostnames that match your sending IPs’ reverse DNS | DNS only | Must resolve to the real IPs |
| MX, SPF, DKIM and DMARC records | DNS only | Only A, AAAA and CNAME can be proxied |
Check the sender host and your MX records with the MX inspector after each DNS change. Stage 5 covers the MTA’s reverse DNS and Stage 6 the sender domain records.
Before you move on
- You have four servers, or a plan for them: app, MTA, tracking edge and monitoring.
- The MTA runs on its own server, not on the app server.
- Every server runs Ubuntu 24.04 LTS, installed fresh, with no control panel, web server or mail server on it.
- SSH is key-only, password login is refused when you test it, and fail2ban watches sshd.
- SSH is open only to your admin IPs on every server.
- /etc is in git on every server, without the SSH host private keys.
- One firewall script per server holds every rule, and it ran before Octeth was installed.
- From outside, every port you did not open times out.
- The app domain is proxied by Cloudflare in Full (strict) mode, and the origin accepts web traffic only from Cloudflare, your admin IPs and the tracking edge.
- A Cloudflare WAF rule blocks the admin paths for everyone but your admin IPs, and the server itself on 6.0.0.
- Your own login shows your real IP in Octeth, not a Cloudflare address.
- The tracking host, the SMTP host and the MTA hostnames are DNS only, and the tracking host points at the tracking edge.
What we've seen go wrong
Mail ports open before the firewall
On one install, Octeth went in before the firewall. Docker publishes ports 25, 80, 443 and 587 on all addresses and bypasses plain ufw, so those ports stayed open to the internet until the firewall step. Our runbook now applies the firewall right after Docker and before the Octeth install.
Password login that stayed on
After SSH hardening on one server, sshd -T still showed password login on. The cloud image had a drop-in file, 50-cloud-init.conf, that enabled it, and sshd uses the first value it reads, so it won over our 99-hardening.conf. Check every file in sshd_config.d, then try a password login from a new connection and confirm it is refused.
A hostname that broke the health check
When the server's hostname is the app domain, cloud-init maps that domain to 127.0.1.1 in /etc/hosts, and calls from the server to its own app, such as Octeth's health check, fail. Mapping the domain to the public IP does not work either, because the Cloudflare Origin CA certificate is trusted only by Cloudflare. Comment out the line in the cloud-init template and in /etc/hosts, so the domain resolves through Cloudflare.
Tracking links that pointed at the MTA
On one platform, the shared sender host that customer domains point to had its A record on the MTA's IP. The MTA serves no web traffic, so every tracked link and every unsubscribe link failed with a TLS error. That host must point at the tracking edge. Click a real tracked link and the unsubscribe link from a sent email before you call the setup done.
A database that took the server's memory
On an install we support, MySQL's buffer pool was set close to the server's total RAM, with no swap. The kernel killed MySQL again and again, and after one kill a campaign went out about three times. Leave memory for the other containers and add swap.
Provider DNS that timed out
On one app server, on a version before 6.0.1, the hosting provider's DNS resolvers timed out for hours. That version's health check resolved the app URL with a 3-second limit, so it returned 503 sixteen times in an hour and a half. From 6.0.1 the health endpoint probes the login pages inside the container. The `health:check` command still calls the app URL. We pinned public resolvers in the network configuration and turned off the resolvers handed out by DHCP.
Cloudflare changes that did nothing
On one domain, rule changes made through the Cloudflare API succeeded and read back correctly, but live traffic did not change. The zone was still pending in that account, and the domain's nameservers belonged to a different Cloudflare account. Check that the zone is active and that the live nameservers match it, then test every change with a live request.
Related tools
Related glossary terms
We build ESPs
Rather have us build this?
We set up, harden, maintain and monitor the app server and the MTA server for you.