Octeth v6.0.7 Now Available! See What's New

Stage 12 of 12

Operations and Launch for an Email Service Provider

Back up the database and files, test restores every week, plan disaster recovery, monitor the servers, upgrade Octeth and apply security patches. Then work through the pre-launch checklist that ties every stage of the build together.

By Cem Hurturk, founder of Octeth Updated 14 min read

Skills
  • Prometheus
  • Grafana
  • Alertmanager
  • XtraBackup
  • Octeth upgrades
On this page
  1. Back up the database and the files
  2. Test restores every week
  3. Plan for disaster recovery
  4. Monitor the servers and alert on failures
  5. Upgrade Octeth
  6. Apply security patches
  7. Set up a status page
  8. Plan the first customers
  9. Pre-launch checklist

The build ends when the first customer sends. The work does not. Backups, restore tests, monitoring, upgrades and patches keep the platform up for every customer on it, and a lost database is lost for all of them at once. Set these up before the first signup, then run the pre-launch checklist at the end of this page.

If you would rather not run this part yourself, Octeth Care maintains the Octeth server: backups, restore tests, patches, upgrades and, on Plus, monitoring. It does not cover the MTA, deliverability, DNS or sender domains.

Back up the database and the files

Back up two things on separate schedules: the database, which changes every second, and the files that turn a fresh install into yours. We recommend this schedule:

KindWhat it holdsHow often
DatabaseA hot full backup of MySQL with Percona XtraBackupEvery 6 hours
FilesOcteth’s env files, project config, compose files, certificates, MySQL config, inbound SMTP config, the data folder, UI storage, brand assets, /etc, the monitoring exporters, the firewall script, and ClickHouse metadata and a snapshotDaily
Restore testThe newest database and files backup, restored from offsite storageWeekly

XtraBackup compresses and encrypts the database backup itself with AES-256. The files backup is a tar archive, compressed with zstd and encrypted with AES-256. XtraBackup copies InnoDB tables without blocking writes (Percona). On a fresh v6 install every table is InnoDB, so the backup runs while customers send. Check an upgraded install.

Octeth’s own backup add-on sets up XtraBackup with daily backups and optional upload to S3. Use it for the commands on your version.

Decide how much you can lose

With MySQL binary logging off, there is no point-in-time recovery. A restore loses up to six hours of data, the gap between two database backups. If that is too much, turn binary logging on and change the backup settings in the same step (see the pitfalls).

The schedule also depends on database size. A large database may need a daily database backup instead of one every six hours.

Encrypt before upload, and keep the key elsewhere

Encrypt each backup on the server before it leaves. Keep the key in a password manager as well as on the server. Without it, every backup is unreadable, and the server that holds the only copy may be the server you lost.

Store copies at another provider, and lock them

Keep three layers:

  1. Local copies for 2 days, inside the hosting provider’s daily VM snapshot.
  2. Offsite copies at a different provider from the hosting, such as Cloudflare R2. A lifecycle rule deletes them after 35 days (R2 object lifecycles).
  3. A bucket lock that keeps every object for 30 days. A delete inside that window fails (R2 bucket locks).

Give the server a token that reads and writes objects but cannot read or change the bucket rules. A compromised server therefore cannot shorten retention or delete a backup younger than 30 days. It can still read the backups, because it holds the key. Set the lifecycle and lock rules in the dashboard, then test a delete and confirm it fails.

Alert on every backup

Each run writes its result to the server’s metrics, and the monitoring server alerts on them. Each run also pushes up or down to an outside uptime monitor, so a failed backup reaches you even when monitoring is down.

AlertFires whenSeverity
Database backup staleNo good database backup for 8 hoursCritical
Files backup staleNo good files backup for 26 hoursWarning
Backup run failedAny run failsWarning
Restore test failedThe weekly test failsCritical
Restore test staleNo restore test for 8 daysWarning

Test restores every week

A backup you never restored is a guess. Run a restore test every week without anyone:

  1. Download the newest database and files backups from offsite storage.
  2. Decrypt the database backup and prepare it with XtraBackup.
  3. Start it in a throwaway MySQL container of the same version as production.
  4. Compare its table count with the live database.
  5. List the files archive to prove it decrypts and unpacks.
Backups, the weekly restore test and the restore order On the app server, a hot database backup with Percona XtraBackup runs every 6 hours and a files backup runs daily. Both are encrypted with AES-256 on the server before upload. Copies go offsite to storage at a different provider from the hosting, such as Cloudflare R2, where a lifecycle rule deletes them after 35 days and a bucket lock keeps every object for 30 days. Every week a restore test downloads the newest backups, restores the database in a throwaway MySQL container, compares its table count with the live database and lists the files archive. Every backup run and every restore test reports its result to the monitoring server and to an outside uptime monitor, which alert when a run fails or a backup goes stale. In a real restore after losing the server, recreate the backup key and offsite credentials from your password manager, restore the files archive, which holds the env file with the salts, and then the database. App server Database Every 6 hours Files Daily Both encrypted with AES-256 before upload Offsite storage At another provider. Deleted after 35 days, locked against deletes for 30 days Restore test, every week Newest backups into a throwaway MySQL. Compare the table count with live. List the files archive. Every run reports up or down Monitoring and an outside monitor Alert when any run fails or a backup or restore test goes stale A real restore after losing the server 1. Keys from your password manager The backup key and the offsite credentials 2. The files archive It holds the env file with the salts 3. The database Written with those salts
The backup loop. A backup you never restored is a guess, so the test runs on its own every week, and every run alerts when it fails. The env file comes back before the database because it holds the salts the database was written with.

Time the restore on a small database and again as the database grows, because that time is how long a real restore keeps customers waiting.

One rule decides the order of a real restore: Octeth’s env file holds the salts the database was written with, and they must match. If the env file is gone too, restore it from the files archive before the database.

Plan for disaster recovery

Write down what you do in each case before you need it:

What failedWhat you do
The database is lost or damagedStop the whole Octeth stack so nothing writes. Move the old data folder aside, put the restored one in its place with the MySQL container’s owner, start the stack and run Octeth’s health check.
Files or settings are lostCopy back only what you need from the files archive. ClickHouse tables come back from the snapshot in it.
An upgrade broke the installRoll back to the snapshot or backups taken in the upgrade window (see Upgrades below).
The server is goneBuild a new server (Stage 3, server infrastructure), install the same Octeth version (Stage 4, installing Octeth), recreate the backup key and offsite credentials from your password manager, restore the files archive, then the database. Copy each enabled plugin back, because the release package ships none.

Keep the backup scripts, their settings and the alert rules in a git repository away from the servers. Leave the backup secrets folder out of the files archive on purpose. The key and the offsite credentials then come from your password manager.

The MTA holds little state of its own. Its configuration, and the DKIM private keys if you keep them under /etc/pmta as in Stage 5, sending infrastructure and IP warm-up, live in /etc, which Stage 3, server infrastructure, puts in git on every server. Treat that repository as a secret. Mail in the queue and the accounting log are state too. The backups above do not hold the accounting log, so if you need it beyond the MTA’s own retention for reports or disputes, ship it elsewhere.

Monitor the servers and alert on failures

This section covers system health: servers, containers and the app. Reputation, complaints and blocklists belong to Stage 10, deliverability operations.

What to watch

Run monitoring on its own server, as Stage 3, server infrastructure, explains. Run Prometheus, Grafana, Alertmanager and a blackbox exporter on it, and scrape:

  • the host metrics of every server
  • MySQL, Redis and RabbitMQ on the app server
  • every container, through cAdvisor
  • the tracking edge’s host and web server metrics

Run the app server’s exporters as a separate compose project, so Octeth restarts and upgrades leave them alone. Open their ports to the monitoring server only. Cap Prometheus retention by time and size, because its database shares the root disk (Prometheus storage), for example 90 days and 60 GB.

The blackbox exporter probes the login page and the app’s health URL for uptime and certificate expiry.

Alert rules

Alert on:

  • a target that stopped answering, and an endpoint that is down
  • a certificate close to expiry
  • disk and memory
  • MySQL or Redis down
  • a container in a restart loop
  • a RabbitMQ backlog
  • the backup rules above
  • the bounce canary from Stage 5, sending infrastructure and IP warm-up, which alerts when bounces stop reaching Octeth

Octeth checks its own services too. Its health check covers MySQL, Redis, RabbitMQ, ClickHouse, the send engine, cron, supervisor, file permissions and more, from the CLI or an API endpoint (monitoring docs). That endpoint needs a credential: the admin API key or a dedicated health check token (SYSTEM_HEALTH_CHECK_TOKEN). Give the monitor the token, so rotating the admin key does not break it. The token works when SYSTEM_HEALTH_CHECK_AUTH_REQUIRED is true, the default on a fresh install.

Route alerts by severity

  • Critical alerts go to a chat on the team’s phones and to the team’s Slack channel.
  • Everything else goes to Slack only.

Use a dedicated bot for alerts, not one shared with other systems.

Watch the watcher

Monitoring that stops sends no alert. Give Alertmanager a Watchdog alert that is always firing. Have a timer on the monitoring server send a heartbeat to an uptime monitor outside the build every minute, but only while the Watchdog is active. When the heartbeat stops, the outside monitor alerts.

The same outside monitor checks the parts customers touch:

  • the app’s login page and its certificate
  • Octeth’s health endpoint, with its credential
  • the tracking edge on port 443, and one real tracked link through it
  • the SMTP ports of the delivery and system mail relays
  • DNS resolution of the host that customer sender domains point to

Prove the alerts arrive

Before you rely on monitoring:

  1. Check the Prometheus and Alertmanager configs with their check tools before each reload.
  2. Confirm every target is up and every rule is loaded.
  3. Send one test critical alert and confirm it reached each channel, then resolve it.
  4. Confirm the outside heartbeat monitor shows up.

Upgrade Octeth

Octeth ships often, so plan for an upgrade every few weeks. The upgrade runs we timed took two to three minutes. The preparation before it takes longer, and it is where the risk is.

Read the upgrade notes as an ESP

An upgrade changes behaviour for every customer on your platform at once. The upgrader adds each new setting your env file lacks, with the new version’s default, and many default to on. Examples from 2026:

  • 6.0.4: API requests that send an array for a single-value parameter are refused with HTTP 400. Customer integrations that did this break. Until 6.0.7 it also refused the PowerMTA accounting batches sent to Octeth’s process_pmta_log_file API command, so delivered, bounce and complaint events stopped arriving. On 6.0.4 to 6.0.6, set API_SCALAR_PARAMETER_CHECK=log if your MTA feeds that command.
  • 6.0.5: The SMTP relay requires STARTTLS before sign-in on ports 587 and 2525. Before that upgrade we listed the relay users who signed in without TLS and warned them.
  • 6.0.7: RSS in emails became a user group permission, granted to every existing group on upgrade. Remove it from any plan that should not have it.

Read the upgrade notes in the changelog for every version you cross. Decide the value of each new setting that changes behaviour, and add it to the env file before the upgrade, because the upgrader never overwrites a setting that is already there. Tell affected customers before the window.

The runbook

The commands change between versions. The upgrade docs hold them. Our runbook has six steps:

  1. Plan. Read the release notes and the target’s upgrade script. Note local patches and plugins, because the upgrade overwrites patches and never updates plugins. Pick a window from the transactional queue by hour and the active campaigns.
  2. Prepare. Copy the release package to the server and check its checksum. Save a baseline: version, running containers, open ports, plugins, and checksums of the certificate and compose files. Pause the outside monitors and silence the alerts for the window.
  3. Back up. Take a tar of the install, a dump of both databases, a copy of the env file and a VM snapshot. Run the scheduled database backup too, so offsite storage holds a fresh copy.
  4. Upgrade. Run a dry run first. It shows the installed version’s logic, not the target’s, so use it only for the version path and the list of new settings. Then run the upgrade detached from your session. From 6.0.4, the upgrader offers to pause sending campaigns and resumes them when it finishes.
  5. Verify. Run the health check, request the system container’s ping endpoint and the UI health URL, fix permissions, check every plugin, and confirm no env value changed that you did not change. Check that the tracking edge and the bounce webhook answer and that monitoring shows nothing down. Then do a real sign-in and a real send, and confirm the login recorded the visitor’s real IP.
  6. Clean up. Resume the monitors, expire the silences, and delete the rollback set after about a week.

On a large database, take your own backup in step 3 and skip the upgrader’s backup step. The upgrader dumps the whole database while sending is stopped.

Apply security patches

Security fixes reach an Octeth server three ways: a new release, a hotfix bundle for one install, or an interim fix while a release is on its way.

In October 2026 two PHP vulnerabilities affected the PHP builds inside Octeth’s containers before the fixed release shipped. We checked the PHP versions on every server we run in one pass, upgraded PHP inside the containers that night, and upgraded to a release with the fix four days later. Two lessons came out of it:

  • An interim fix inside a container is lost when the container is recreated, and an upgrade recreates them. Track it as an open item until a release replaces it.
  • Check the running process, not the package version. A reload can leave the old library loaded (see the pitfalls).

For a hotfix bundle, stop if any preflight check fails, and do not adapt the bundle on the server. Record each applied hotfix with its commits, so the next upgrade can check that the release includes them.

Kernel and system library updates take effect only after a reboot. Track a pending reboot as an open item and give it a window, as you do for upgrades. Server hardening itself is in Stage 3, server infrastructure.

Set up a status page

Octeth’s signed-out pages have a Status link in the footer, set by the status page field in the branding settings (UI_BRAND_STATUS_URL). Left empty, the link does not show. You can launch without a status page and leave it empty.

If you publish one, host it away from your own servers, so it still loads when they are down. Then link it from the branding settings and from your help docs (Stage 11, customer support).

Plan the first customers

Run a readiness review a few days before the first customer: health check, disk, monitoring targets, current backups and the last restore test. Then plan how the first accounts go live:

  1. Review each signup and mark it Trusted only after review. While an account is Untrusted, its first campaign must show as waiting for approval. Check that once (Stage 8, abuse prevention).
  2. Start new accounts low. New customers go on warm IPs with low limits that rise with clean sends (Stage 5, sending infrastructure and IP warm-up).
  3. Watch every early send. Read bounces, deferrals and complaints per account after each campaign of the first customers (Stage 10, deliverability operations).
  4. Keep the response targets you published for their questions (Stage 11, customer support).

Pre-launch checklist

Copy this list into your tracker. Each line links to the stage that covers it.

Business and policies

Servers and the app

Sending

Plans, abuse and billing

Deliverability and support

Operations and launch day

  • Backups run, a restore test passed, and their alerts reach you. (this stage)
  • Monitoring alerts reached every channel, and the outside heartbeat is up. (this stage)
  • A test customer signed up, was reviewed, paid, sent a campaign, clicked a tracked link, unsubscribed, and saw a hard bounce reach the suppression list. (Stage 8, abuse prevention, Stage 9, Stripe billing, Stage 6, sender domain authentication)
  • The abuse desk is staffed: someone reads abuse@ and the signup review queue every work day. (Stage 2, legal and policies, Stage 8, abuse prevention)
  • The status page is live and linked, or the Status link is empty on purpose. (this stage)
  • The first customers have a plan: review, low starting limits and a daily look at their sends. (this stage)

Before you move on

  • The database is backed up every few hours and the files daily, encrypted on the server, and stored at a provider other than your host.
  • The backup encryption key is in a password manager, not only on the server.
  • The offsite bucket deletes old backups on a schedule and locks recent ones against deletion.
  • A weekly restore test runs on its own and has passed at least once.
  • Each backup kind and the restore test alert you when they fail or go stale.
  • Monitoring scrapes every server, critical alerts reach a phone, and a test alert arrived on every channel.
  • A check outside your servers alerts you when monitoring itself stops.
  • You have an upgrade runbook and have read the upgrade notes of the next Octeth release.
  • Every line of the pre-launch checklist is done.

What we've seen go wrong

A files backup that failed at random

On one install, the daily files backup failed with only an exit code in the journal. GNU tar exits 1 when a file changes while it is read, and Octeth's logs change every few seconds. Hiding the warning does not change the exit code. Our script now accepts tar exit 1 and fails on exit 2 or on any compression or encryption error.

A backup that broke when binary logging went on

On one server, binary logging was turned on for replication. MySQL runs in Docker and reports its binary log path inside the container. A backup run on the host streamed for several minutes, then failed because that path does not exist on the host. The scheduled backup would have failed the same way. When you turn on binary logging, change the backup settings in the same step.

Offsite uploads that logged an error every time

With backups uploaded to Cloudflare R2, rclone 1.60 sends a request after the upload to check the object version, and R2 answers it with 501. Setting no_head = true in the R2 remote fixes it.

A heartbeat that reported success and recorded nothing

Some uptime monitors accept heartbeats only by GET. A webhook that posts gets 404 and raises a false outage. A wrong push path can return the web app's page with HTTP 200, so the sender sees success while nothing is recorded. Send heartbeats by GET and count only an explicit ok reply as success.

A security fix that was installed but not running

After a PHP package upgrade inside Octeth's containers, a graceful web server reload kept the old library loaded in one of the two web containers, on every server we patched. The package version looked fixed while the old code still ran. We restarted that container and checked the running process for the deleted library.

We build ESPs

Rather have us build this?

We run the backups, restore tests, monitoring and server maintenance after launch.