Octeth v5.9.5 is out today. Like the release before it, it contains no new features.
Twenty-one commits, six fixes, two database migrations. That is the second small release in a row, which is what a two-week cadence is supposed to produce, and I wrote about why I have made my peace with that when v5.9.4 shipped a fortnight ago.
What I want to write about this time is one specific bug, because it taught me something about how we find problems and how long we can go without finding them.
A Report That Returned Nothing, To Everybody, For Its Entire Life
Octeth has a report that breaks your Email Gateway sending down by recipient domain. How much went to Gmail, how much to Outlook, how each of them performed. It is exactly the sort of thing you look at when deliverability feels off and you are trying to work out whether the problem is you or one particular mailbox provider.
For every customer sending through the Email Gateway API, that report returned an empty result. Not a low number. Nothing at all. Since the day it shipped.
The mechanism, once we looked, was almost boring. The report was reading from our event stream and grouping on a recipient-domain field. That field gets populated by our inbound SMTP relay. If you send through the API instead, nothing ever writes it, so there was nothing to group and the report correctly returned an empty list. Every piece of that chain was working exactly as written. The answer was just wrong.
Here is the part I keep thinking about: nobody told us.
I do not think that is because nobody looked. I think it is because an empty report does not look like a bug. It looks like an answer. You open a breakdown, you see nothing there, and your brain supplies a reason before you have consciously formed the question. Maybe the data has not caught up. Maybe this only populates after a certain volume. Maybe I need to configure something. Maybe we are just not sending enough for it to be interesting.
A crash gets reported in ten minutes. A wrong number gets reported when somebody cross-checks it. Emptiness gets rationalised, and then it gets ignored, and then it stops being a thing anyone expects to work.
It now reads from the send queue instead, which records the recipient domain on every message regardless of how it was sent. That table has no retention window, so the report is populated the moment you upgrade and it covers your existing send history. There is nothing to backfill and nothing to wait for. If you have been quietly assuming that report was not for you, open it after you upgrade.
One Number Is Going To Go Down
While we were in there, we found the sibling of that bug, and this one changes a number you may already be looking at.
There is a companion endpoint that gives you statistics for one recipient domain. It takes a sender domain as a parameter. It was using that parameter to check you owned the sender domain, and then reporting across every sender domain on your account anyway.
So if you send from one domain, nothing changes for you. If you send from several, the numbers you have been reading were the combined total, and after this upgrade they will be lower, because they are finally answering the question you actually asked.
I want to be direct about that rather than describe it as improved accuracy and let you discover it. A number that drops after an upgrade looks like something broke. Nothing broke. Your sending has not changed. The report was including traffic you did not ask about, and now it is not. The two reports also agree with each other for the first time, which they never did before, because they were measuring different things while appearing to measure the same one.
We Finished The Audit
The security work in this release closes the last finding from the internal audit that produced the v5.9.4 fixes.
Segments in Octeth are built from rules, and a rule names a field. If you build segments through the interface, that field always comes from a fixed list and nothing was ever wrong. But a rule submitted directly through the API could name a field that reached the query layer without being properly escaped, which could let an authenticated account on an installation read data belonging to other accounts on the same installation.
Same shape as the two we closed last time, same audit, same account-isolation boundary. We reproduced it under controlled conditions, fixed it at the point where rules are saved and again at the point where they are used, and verified the fix by reverting the patch and watching the test go red. There is no indication it was ever exploited.
When we shipped v5.9.4 we knew this one was still open. We said so internally, tracked it, and it went to the top of this sprint. I mention that because "we found three and fixed two" is a more accurate description of two weeks ago than "we found and fixed two," and the difference matters if you are trying to decide how much to trust what we tell you.
The Fix That Cost Us Twenty-Seven Hours
There is one more fix worth describing, because it is the kind of bug that makes every other bug harder.
When a database query failed inside part of our backend, the code took the error code off the exception and used it as the HTTP status code for the response. That is a reasonable thing to do, right up until the exception is a database exception, because those carry a SQLSTATE code that is a string, not a number. The string reached a function that required an integer and threw a different error entirely.
The result was that a query failure produced a generic server error page instead of the JSON error it meant to send, and the original database error was wiped out of the log and replaced by the secondary one. The thing that told you what actually went wrong was destroyed by the code trying to report it.
We know precisely what that cost, because it happened to us. A recent issue took twenty-seven hours to diagnose, and essentially all of that time was spent looking at the wrong error. Failures now report properly, and the original exception is written to the log before anything else gets a chance to go wrong.
There is a general lesson in there that I have relearned more than once. Your error handling is code too, and it runs on your worst day. It deserves at least as much care as the code it is reporting on, and probably more, because when it fails it takes your ability to see with it.
The Rest, Briefly
Upgrades are more reliable. A check we run during database upgrades could mistake a missing table for an existing one, which could stop an upgrade partway through. It is now exact, and an upgrade that cannot verify the state of your database stops cleanly and can be safely re-run rather than recording itself as finished.
Your logs are quieter. Scheduled tasks belonging to optional add-ons were running on every installation whether or not the add-on was present, writing roughly 1,750 useless error lines a day into the log an administrator checks first when something is genuinely wrong. Signal buried in noise is not signal.
Export files are cleaned up properly. Completed exports missing a finish timestamp were being skipped by the retention cleanup and kept indefinitely. Those records are repaired during the upgrade.
One Note On Planning This Upgrade
Unlike v5.9.4, this release ships database migrations, and one of them builds an index on the Email Gateway queue. On an installation with a large send history that is the only step with meaningful cost. It is not a long or risky operation, but it is worth putting inside a change window rather than running it in the middle of a send. The changelog has the detail.
That is the honest summary of two weeks: one report that was lying by omission, one that was answering a broader question than it was asked, the last of a security audit, and an error handler that ate its own evidence. Nothing here will make a demo more impressive. All of it makes the platform more truthful, which over a long enough period is the thing I would rather be selling.
Next release is September 11th.
