Stage 12 of 12
Operations and Launch for an Email Service Provider
Back up the database and files, test restores every week, plan disaster recovery, monitor the servers, upgrade Octeth and apply security patches. Then work through the pre-launch checklist that ties every stage of the build together.
By Cem Hurturk, founder of Octeth Updated 14 min read
- Prometheus
- Grafana
- Alertmanager
- XtraBackup
- Octeth upgrades
On this page
The build ends when the first customer sends. The work does not. Backups, restore tests, monitoring, upgrades and patches keep the platform up for every customer on it, and a lost database is lost for all of them at once. Set these up before the first signup, then run the pre-launch checklist at the end of this page.
If you would rather not run this part yourself, Octeth Care maintains the Octeth server: backups, restore tests, patches, upgrades and, on Plus, monitoring. It does not cover the MTA, deliverability, DNS or sender domains.
Back up the database and the files
Back up two things on separate schedules: the database, which changes every second, and the files that turn a fresh install into yours. We recommend this schedule:
| Kind | What it holds | How often |
|---|---|---|
| Database | A hot full backup of MySQL with Percona XtraBackup | Every 6 hours |
| Files | Octeth’s env files, project config, compose files, certificates, MySQL config, inbound SMTP config, the data folder, UI storage, brand assets, /etc, the monitoring exporters, the firewall script, and ClickHouse metadata and a snapshot | Daily |
| Restore test | The newest database and files backup, restored from offsite storage | Weekly |
XtraBackup compresses and encrypts the database backup itself with AES-256. The files backup is a tar archive, compressed with zstd and encrypted with AES-256. XtraBackup copies InnoDB tables without blocking writes (Percona). On a fresh v6 install every table is InnoDB, so the backup runs while customers send. Check an upgraded install.
Octeth’s own backup add-on sets up XtraBackup with daily backups and optional upload to S3. Use it for the commands on your version.
Decide how much you can lose
With MySQL binary logging off, there is no point-in-time recovery. A restore loses up to six hours of data, the gap between two database backups. If that is too much, turn binary logging on and change the backup settings in the same step (see the pitfalls).
The schedule also depends on database size. A large database may need a daily database backup instead of one every six hours.
Encrypt before upload, and keep the key elsewhere
Encrypt each backup on the server before it leaves. Keep the key in a password manager as well as on the server. Without it, every backup is unreadable, and the server that holds the only copy may be the server you lost.
Store copies at another provider, and lock them
Keep three layers:
- Local copies for 2 days, inside the hosting provider’s daily VM snapshot.
- Offsite copies at a different provider from the hosting, such as Cloudflare R2. A lifecycle rule deletes them after 35 days (R2 object lifecycles).
- A bucket lock that keeps every object for 30 days. A delete inside that window fails (R2 bucket locks).
Give the server a token that reads and writes objects but cannot read or change the bucket rules. A compromised server therefore cannot shorten retention or delete a backup younger than 30 days. It can still read the backups, because it holds the key. Set the lifecycle and lock rules in the dashboard, then test a delete and confirm it fails.
Alert on every backup
Each run writes its result to the server’s metrics, and the monitoring server alerts on them. Each run also pushes up or down to an outside uptime monitor, so a failed backup reaches you even when monitoring is down.
| Alert | Fires when | Severity |
|---|---|---|
| Database backup stale | No good database backup for 8 hours | Critical |
| Files backup stale | No good files backup for 26 hours | Warning |
| Backup run failed | Any run fails | Warning |
| Restore test failed | The weekly test fails | Critical |
| Restore test stale | No restore test for 8 days | Warning |
Test restores every week
A backup you never restored is a guess. Run a restore test every week without anyone:
- Download the newest database and files backups from offsite storage.
- Decrypt the database backup and prepare it with XtraBackup.
- Start it in a throwaway MySQL container of the same version as production.
- Compare its table count with the live database.
- List the files archive to prove it decrypts and unpacks.
Time the restore on a small database and again as the database grows, because that time is how long a real restore keeps customers waiting.
One rule decides the order of a real restore: Octeth’s env file holds the salts the database was written with, and they must match. If the env file is gone too, restore it from the files archive before the database.
Plan for disaster recovery
Write down what you do in each case before you need it:
| What failed | What you do |
|---|---|
| The database is lost or damaged | Stop the whole Octeth stack so nothing writes. Move the old data folder aside, put the restored one in its place with the MySQL container’s owner, start the stack and run Octeth’s health check. |
| Files or settings are lost | Copy back only what you need from the files archive. ClickHouse tables come back from the snapshot in it. |
| An upgrade broke the install | Roll back to the snapshot or backups taken in the upgrade window (see Upgrades below). |
| The server is gone | Build a new server (Stage 3, server infrastructure), install the same Octeth version (Stage 4, installing Octeth), recreate the backup key and offsite credentials from your password manager, restore the files archive, then the database. Copy each enabled plugin back, because the release package ships none. |
Keep the backup scripts, their settings and the alert rules in a git repository away from the servers. Leave the backup secrets folder out of the files archive on purpose. The key and the offsite credentials then come from your password manager.
The MTA holds little state of its own. Its configuration, and the DKIM private keys if you keep them under /etc/pmta as in Stage 5, sending infrastructure and IP warm-up, live in /etc, which Stage 3, server infrastructure, puts in git on every server. Treat that repository as a secret. Mail in the queue and the accounting log are state too. The backups above do not hold the accounting log, so if you need it beyond the MTA’s own retention for reports or disputes, ship it elsewhere.
Monitor the servers and alert on failures
This section covers system health: servers, containers and the app. Reputation, complaints and blocklists belong to Stage 10, deliverability operations.
What to watch
Run monitoring on its own server, as Stage 3, server infrastructure, explains. Run Prometheus, Grafana, Alertmanager and a blackbox exporter on it, and scrape:
- the host metrics of every server
- MySQL, Redis and RabbitMQ on the app server
- every container, through cAdvisor
- the tracking edge’s host and web server metrics
Run the app server’s exporters as a separate compose project, so Octeth restarts and upgrades leave them alone. Open their ports to the monitoring server only. Cap Prometheus retention by time and size, because its database shares the root disk (Prometheus storage), for example 90 days and 60 GB.
The blackbox exporter probes the login page and the app’s health URL for uptime and certificate expiry.
Alert rules
Alert on:
- a target that stopped answering, and an endpoint that is down
- a certificate close to expiry
- disk and memory
- MySQL or Redis down
- a container in a restart loop
- a RabbitMQ backlog
- the backup rules above
- the bounce canary from Stage 5, sending infrastructure and IP warm-up, which alerts when bounces stop reaching Octeth
Octeth checks its own services too. Its health check covers MySQL, Redis, RabbitMQ, ClickHouse, the send engine, cron, supervisor, file permissions and more, from the CLI or an API endpoint (monitoring docs). That endpoint needs a credential: the admin API key or a dedicated health check token (SYSTEM_HEALTH_CHECK_TOKEN). Give the monitor the token, so rotating the admin key does not break it. The token works when SYSTEM_HEALTH_CHECK_AUTH_REQUIRED is true, the default on a fresh install.
Route alerts by severity
- Critical alerts go to a chat on the team’s phones and to the team’s Slack channel.
- Everything else goes to Slack only.
Use a dedicated bot for alerts, not one shared with other systems.
Watch the watcher
Monitoring that stops sends no alert. Give Alertmanager a Watchdog alert that is always firing. Have a timer on the monitoring server send a heartbeat to an uptime monitor outside the build every minute, but only while the Watchdog is active. When the heartbeat stops, the outside monitor alerts.
The same outside monitor checks the parts customers touch:
- the app’s login page and its certificate
- Octeth’s health endpoint, with its credential
- the tracking edge on port 443, and one real tracked link through it
- the SMTP ports of the delivery and system mail relays
- DNS resolution of the host that customer sender domains point to
Prove the alerts arrive
Before you rely on monitoring:
- Check the Prometheus and Alertmanager configs with their check tools before each reload.
- Confirm every target is up and every rule is loaded.
- Send one test critical alert and confirm it reached each channel, then resolve it.
- Confirm the outside heartbeat monitor shows up.
Upgrade Octeth
Octeth ships often, so plan for an upgrade every few weeks. The upgrade runs we timed took two to three minutes. The preparation before it takes longer, and it is where the risk is.
Read the upgrade notes as an ESP
An upgrade changes behaviour for every customer on your platform at once. The upgrader adds each new setting your env file lacks, with the new version’s default, and many default to on. Examples from 2026:
- 6.0.4: API requests that send an array for a single-value parameter are refused with HTTP 400. Customer integrations that did this break. Until 6.0.7 it also refused the PowerMTA accounting batches sent to Octeth’s
process_pmta_log_fileAPI command, so delivered, bounce and complaint events stopped arriving. On 6.0.4 to 6.0.6, setAPI_SCALAR_PARAMETER_CHECK=logif your MTA feeds that command. - 6.0.5: The SMTP relay requires STARTTLS before sign-in on ports 587 and 2525. Before that upgrade we listed the relay users who signed in without TLS and warned them.
- 6.0.7: RSS in emails became a user group permission, granted to every existing group on upgrade. Remove it from any plan that should not have it.
Read the upgrade notes in the changelog for every version you cross. Decide the value of each new setting that changes behaviour, and add it to the env file before the upgrade, because the upgrader never overwrites a setting that is already there. Tell affected customers before the window.
The runbook
The commands change between versions. The upgrade docs hold them. Our runbook has six steps:
- Plan. Read the release notes and the target’s upgrade script. Note local patches and plugins, because the upgrade overwrites patches and never updates plugins. Pick a window from the transactional queue by hour and the active campaigns.
- Prepare. Copy the release package to the server and check its checksum. Save a baseline: version, running containers, open ports, plugins, and checksums of the certificate and compose files. Pause the outside monitors and silence the alerts for the window.
- Back up. Take a tar of the install, a dump of both databases, a copy of the env file and a VM snapshot. Run the scheduled database backup too, so offsite storage holds a fresh copy.
- Upgrade. Run a dry run first. It shows the installed version’s logic, not the target’s, so use it only for the version path and the list of new settings. Then run the upgrade detached from your session. From 6.0.4, the upgrader offers to pause sending campaigns and resumes them when it finishes.
- Verify. Run the health check, request the system container’s ping endpoint and the UI health URL, fix permissions, check every plugin, and confirm no env value changed that you did not change. Check that the tracking edge and the bounce webhook answer and that monitoring shows nothing down. Then do a real sign-in and a real send, and confirm the login recorded the visitor’s real IP.
- Clean up. Resume the monitors, expire the silences, and delete the rollback set after about a week.
On a large database, take your own backup in step 3 and skip the upgrader’s backup step. The upgrader dumps the whole database while sending is stopped.
Apply security patches
Security fixes reach an Octeth server three ways: a new release, a hotfix bundle for one install, or an interim fix while a release is on its way.
In October 2026 two PHP vulnerabilities affected the PHP builds inside Octeth’s containers before the fixed release shipped. We checked the PHP versions on every server we run in one pass, upgraded PHP inside the containers that night, and upgraded to a release with the fix four days later. Two lessons came out of it:
- An interim fix inside a container is lost when the container is recreated, and an upgrade recreates them. Track it as an open item until a release replaces it.
- Check the running process, not the package version. A reload can leave the old library loaded (see the pitfalls).
For a hotfix bundle, stop if any preflight check fails, and do not adapt the bundle on the server. Record each applied hotfix with its commits, so the next upgrade can check that the release includes them.
Kernel and system library updates take effect only after a reboot. Track a pending reboot as an open item and give it a window, as you do for upgrades. Server hardening itself is in Stage 3, server infrastructure.
Set up a status page
Octeth’s signed-out pages have a Status link in the footer, set by the status page field in the branding settings (UI_BRAND_STATUS_URL). Left empty, the link does not show. You can launch without a status page and leave it empty.
If you publish one, host it away from your own servers, so it still loads when they are down. Then link it from the branding settings and from your help docs (Stage 11, customer support).
Plan the first customers
Run a readiness review a few days before the first customer: health check, disk, monitoring targets, current backups and the last restore test. Then plan how the first accounts go live:
- Review each signup and mark it Trusted only after review. While an account is Untrusted, its first campaign must show as waiting for approval. Check that once (Stage 8, abuse prevention).
- Start new accounts low. New customers go on warm IPs with low limits that rise with clean sends (Stage 5, sending infrastructure and IP warm-up).
- Watch every early send. Read bounces, deferrals and complaints per account after each campaign of the first customers (Stage 10, deliverability operations).
- Keep the response targets you published for their questions (Stage 11, customer support).
Pre-launch checklist
Copy this list into your tracker. Each line links to the stage that covers it.
Business and policies
- Plans, prices, the overage rule and the license tier are decided. (Stage 1, business planning)
- Terms, acceptable use policy, privacy policy, DPA and sub-processor list are published and linked from signup. (Stage 2, legal and policies)
- abuse@ and the other role mailboxes reach a person, and the report-abuse page is live. (Stage 2, legal and policies)
Servers and the app
- App, MTA, tracking edge and monitoring servers are hardened, and every port you did not open times out from outside. (Stage 3, server infrastructure)
- Octeth is installed, branded, sends its own mail, and records real visitor IPs behind Cloudflare. (Stage 4, installing Octeth)
Sending
- The MTA has its pools, reverse DNS, and a bounce path into Octeth with a daily canary. (Stage 5, sending infrastructure and IP warm-up)
- The warm-up schedule is written, and new accounts start on warm IPs. (Stage 5, sending infrastructure and IP warm-up)
- A test sender domain is verified, and a real campaign passes SPF, DKIM and DMARC with one-click unsubscribe. (Stage 6, sender domain authentication)
Plans, abuse and billing
- Each plan is a user group with its limits, permissions and delivery server. (Stage 7, plans and sending limits)
- Signups start Untrusted, the review questions are on, and staff know how to approve and suspend. (Stage 8, abuse prevention)
- Stripe runs in live mode, its webhook is verified, and a real card payment moved an account to its plan. (Stage 9, Stripe billing)
Deliverability and support
- Postmaster Tools, SNDS, JMRP and the Yahoo feedback loop are set up for your domains and IPs. (Stage 10, deliverability operations)
- A seed test of a real campaign showed nothing broken, and seed traffic is filtered out of reports. (Stage 10, deliverability operations)
- The support desk, onboarding emails, help docs and response targets are live. (Stage 11, customer support)
Operations and launch day
- Backups run, a restore test passed, and their alerts reach you. (this stage)
- Monitoring alerts reached every channel, and the outside heartbeat is up. (this stage)
- A test customer signed up, was reviewed, paid, sent a campaign, clicked a tracked link, unsubscribed, and saw a hard bounce reach the suppression list. (Stage 8, abuse prevention, Stage 9, Stripe billing, Stage 6, sender domain authentication)
- The abuse desk is staffed: someone reads abuse@ and the signup review queue every work day. (Stage 2, legal and policies, Stage 8, abuse prevention)
- The status page is live and linked, or the Status link is empty on purpose. (this stage)
- The first customers have a plan: review, low starting limits and a daily look at their sends. (this stage)
Before you move on
- The database is backed up every few hours and the files daily, encrypted on the server, and stored at a provider other than your host.
- The backup encryption key is in a password manager, not only on the server.
- The offsite bucket deletes old backups on a schedule and locks recent ones against deletion.
- A weekly restore test runs on its own and has passed at least once.
- Each backup kind and the restore test alert you when they fail or go stale.
- Monitoring scrapes every server, critical alerts reach a phone, and a test alert arrived on every channel.
- A check outside your servers alerts you when monitoring itself stops.
- You have an upgrade runbook and have read the upgrade notes of the next Octeth release.
- Every line of the pre-launch checklist is done.
What we've seen go wrong
A files backup that failed at random
On one install, the daily files backup failed with only an exit code in the journal. GNU tar exits 1 when a file changes while it is read, and Octeth's logs change every few seconds. Hiding the warning does not change the exit code. Our script now accepts tar exit 1 and fails on exit 2 or on any compression or encryption error.
A backup that broke when binary logging went on
On one server, binary logging was turned on for replication. MySQL runs in Docker and reports its binary log path inside the container. A backup run on the host streamed for several minutes, then failed because that path does not exist on the host. The scheduled backup would have failed the same way. When you turn on binary logging, change the backup settings in the same step.
Offsite uploads that logged an error every time
With backups uploaded to Cloudflare R2, rclone 1.60 sends a request after the upload to check the object version, and R2 answers it with 501. Setting no_head = true in the R2 remote fixes it.
A heartbeat that reported success and recorded nothing
Some uptime monitors accept heartbeats only by GET. A webhook that posts gets 404 and raises a false outage. A wrong push path can return the web app's page with HTTP 200, so the sender sees success while nothing is recorded. Send heartbeats by GET and count only an explicit ok reply as success.
A security fix that was installed but not running
After a PHP package upgrade inside Octeth's containers, a graceful web server reload kept the old library loaded in one of the two web containers, on every server we patched. The package version looked fixed while the old code still ran. We restarted that container and checked the running process for the deleted library.
We build ESPs
Rather have us build this?
We run the backups, restore tests, monitoring and server maintenance after launch.