After running 2 split tests, I have verified that the subscriber selection process is not randomized. The first test email is sent to the first X subscriber, the next test email is sent to the next X subscribers in the list, and so on.
This completely invalidates any test I want to do, as my most responsive subscribers are in the first group.
To deal with the issue, I now must run tests to a much larger percentage of my list, virtually eliminating any gain I may realize from the test itself.
True split tests select subscribers from the whole list, randomized so that each test version gets a virtually identical cross-section of the list, without bias towards long-term or short-term subscribers, and without bias towards any particular ISP or subscriber type.
Please, please, please........ make subscriber selection random for split tests.



Reply With Quote