Contents
|
Overview
Email is Complex
The process of sending and receiving email is more complex than most people realize. Consider all the steps involved in a simple email scenario: Alice@alpha.com wants to send a message to Bob@beta.com.
Most desktop mail programs -- Outlook, Mail, Netscape, Eudora, Lotus Notes, etc. -- contain both a Mail Submission Agent (MSA) for composing and sending mail and a Mail User Agent (MUA) for downloading and displaying it. Alice composes her message and sends it using her MSA.
Alice's MSA hands the message over to the Mail Transfer Agent (MTA) at her ISP. The MTA at alpha.com then tries to pass the message to the MTA at beta.com. The beta.com MTA runs the message through a spam filter and if it passes, drops it into Bob's mailbox.
The Mail User Agent on Bob's computer logs into beta.com and retrieves the messages from Bob's mailbox. Many modern MUAs have built in spam filters, so the message may get tested again before Bob actually gets to see it.
Notice that the information has so far been moving in one direction only. Alice's MSA knows nothing about what happened to the message after the alpha.com MTA accepted it. The alpha.com MTA doesn't know what happened to the message after the beta.com MTA accepted it. The beta.com MTA doesn't know what became of the message after it was deposited in Bob's mailbox. The POP/IMAP server at beta.com that pulls the message out of the mailbox when Bob's MUA asks for new mail doesn't communicate with the MTA at all.
This example assumes that everything works perfectly, but what happens when things go wrong?
Every time the message moves from one machine to another or even between programs on the same machine, there is a chance that it will get mangled, lost, or rejected. In most cases rejection will result in a "bounce" message being sent back to Alice, which explains why the delivery is being cancelled.
Since spammers tend to use fake return addresses, most spam filters do not send bounce notices when they reject or quarantine messages.
To further complicate matters, the bounce messages are themselves email and thus subject to all of the same perils as the original message. (The only real difference in how bounce messages are handled is that no bounce message is created when a bounce message fails -- for obvious reasons).
If Alice's message is delivered safely, she will never hear anything back. If the message fails, she might find out about it. If Alice were to use ePriority she could verify her message's delivery with much greater confidence.
How We've Made It Simple for You
ePriority's Mail Submission Agent (MSA) and tracking features capture information on your message's progress and present it in easy to read reports. The tracking features can continue to gather data even after a message has been delivered. If a message cannot be delivered on the first attempt, the MSA will set it aside and try it again later.
Our MSA and tracking programs record significant events in the lifetime of every message in logs. Each log record identifies an event by its numeric code and the message it happened to by both message ID and destination address. You can download these raw logs to your own system or view the data in reports displayed on our Web site.
When a message is accepted for final delivery by a remote Mail Transfer Agent (MTA), our MSA will record an "unconfirmed delivery" event for the message.
A bounce message can turn an unconfirmed delivery into a confirmed failure. ePriority tracking features can be enabled to confirm deliveries. Tracking information embedded in the messages can be relayed back to ePriority when the recipients receive them, read them, or interact with them in other ways.
Delivery Options
ePriority provides complete control of the email process, including the delivery schedule and what tracking to be performed during and after delivery.
Delivery Schedule
The email delivery period begins on the start date and ends on the delivery stop date. If the delivery period ends for a message before it runs out of allowed delivery attempts, it will be cancelled with a 5405 event code.
The tracking stop date indicates the end of tracking event data collection for an email or batch.
Delivery blackouts can also affect a message's chances for successful delivery. A blackout, when applied to a message, prevents it from being delivered during a given time period.
Retry
Most messages are delivered on the first attempt. Those that are not successfully delivered are set aside and tried again later.
You can control the retry schedule by setting a maximum number of attempts for messages during the delivery period. The time for the next attempt is determined by the number of allowed attempts remaining and the amount of time remaining until the message has to be cancelled. Retries will be made less frequently as time goes on.
Most service interruptions on the Internet are brief, and this dynamic delivery schedule decreases the probability that a brief outage will result in a long delay.
If the number of allowed attempts runs out before the message is accepted, it will be cancelled with a 5403 event code.
Tracking and Confirmation Options
Most of ePriority's tracking features can be used to verify that a message has arrived at its destination. Here is a quick rundown of the tracking features and the information they provide.
- Bounce Tracking
Collects and tabulates bounce messages. This tracking feature produces the list of known failures.
- Receipt Tracking
Collects and tabulates Message Disposition Notification receipts (also known as read receipts or return receipts). Receipts are only returned by Mail User Agents (readers), so we know that the message the receipt was attached to reached its destination safely.
- View Tracking
The view tracking agent records an event every time the message is viewed in an MUA.
- Reply Tracking
Collects and tabulates replies to the original message. Since a reply is typically generated by an end-user's action, a reply can be be considered proof of successful delivery.
- Opt-out Tracking
Collects and tabulates opt-out requests. Like replies and receipts, opt-out requests are practically guaranteed to be caused by human action -- thus proving that your message has been received and read by a human being.
- Link Tracking
The link tracking feature records visits to Web pages that are made through links in a message. Once again, these events are human actions in response to your message that prove that it was received and read.
Recommended Settings
- Maximum Delivery Attempts
At least four, but not more than an average of one every six hours. With a 48 hour delivery period, for example, you will want at least four but not more than eight attempts.
- Delivery Period
Allow at least 24 hours from the start of delivery to the stop-delivery date.
- Tracking Period
Allow at least 48 hours for tracking after delivery is stopped. A week is optimal. If you are using tracking features other than Bounce and Receipt, you will probably want even more time.
The amount of time that you want to pass between these the stop-delivery and stop-tracking dates should be a balance between timeliness and accuracy. The longer this tracking period lasts, the more likely you are to catch responses from your slowest-to-act recipients. Some MTAs will take several days to bounce a message.
If you need to shorten the time between the start of delivery and the end of tracking, it is usually better to reduce the delivery time instead of the post-delivery tracking and reporting period.
- Bounce Tracking
Activate and either store the messages in the ePriority mailbox or forward them to another repository where you can get at them.
- Receipt Tracking
Activate, but do not save or forward the receipts. They rarely contain any useful data.
- View Tracking
No longer recommended. The communication technology that drives this feature has been abused by spammers to validate addresses in their mailing lists. As a result many mail clients now block this communication channel; some will flag as spam any message containing the code that initiates this form of communication. This tracking feature is no longer reliable and may actually hinder delivery of your messages.
We recommend using Reply Tracking or Receipt Tracking instead.
How to Make Decisions Regarding Outcomes
Confirmation events conclusively report the ultimate fate of a message. We pointed out earlier that when a message is accepted by an MTA we assume that it has been or will be delivered successfully. A returned receipt or view tracking event confirms that it was delivered successfully; a bounce message confirms that it was not.
Delivery events are recorded in the TRAC logs (trac*.txt.zip)
in the batch archive directory. Additionally, failed messages are listed
in failed.csv in the same directory.
Final Outcomes
There are three possible delivery outcomes for each message: Success, Failure, and Unconfirmed. This last state is not final in the sense that nothing more could happen; it is final in the sense that you have decided not to wait any longer for confirmation events that are not likely to happen.
The state of a message can be determined by examining the most recent event recorded for it in the TRAC log.
A confirmed success (6xxx,9xxx) is a message that we can logically prove has reached its intended recipient. Tracking events detected by the tracking features are unlikely to be caused by anything but human interaction with the message.
A confirmed failure (54xx) is a message that we know did not reach
its intended destination. The message may have been cancelled in-system
by ePriority or bounced by the recipient's systems.
Failed messages are listed in failed.csv in the batch archive
directory.
An unconfirmed success (5301) is a message that we can be reasonably certain reached its destination, but we can't prove that it did. Every message accepted by a recipient MTA is initially marked as an unconfirmed success. If the MTA decides that it cannot deliver the message, the message status will change to confirmed failure. If tracking events from the message are detected, the message status will change to confirmed success.
When to Make Decisions
Eventually a decision must be made about the unconfirmed messages: should you consider them delivered or not? When should you make that decision?
When the stop-delivery date has been reached for a batch, any messages that remain undelivered will be cancelled. After this, only confirmation events will occur for the messages.
When the stop-tracking date has been reached for the batch, the data collection programs associated with the tracking features will stop actively looking for tracking events to record.
In most cases it is safe to assume that the unconfirmed messages were indeed received. None of the tracking features are infallible and almost all of them require action on the part of the recipient. The message may simply be sitting unread and unopened in his or her inbox. The recipient's MUA may block or simply not send our tracking event notices. The recipient may be reading his or her mail offline.
Auditing Delivery Failures
A certain percentage of messages are always going to fail. People change their email addresses or let their mailboxes accumulate junk. ISPs have service outages (both planned and otherwise).
ePriority provides a variety of ways to get mailing results, some automated and some manual. Automated methods include FTP downloads and HTTP custom reports. In most cases, these will be your main source of information. Manual investigation should be reserved for those rare occassions when something truly unusual happens.
Most delivery problems are either transient (they go away quickly on their own) or have obvious causes ("destination.con" should have been "destination.com"). Some situations in which an audit might be advisable are listed below.
- Overall failure rate is unusually high.
- Is something wrong with your messages?
- Are your messages triggering spam filters?
- The failure rate for a particular domain is unusually high.
- Are your messages triggering a spam filter?
- Can you tailor your messages to increase their acceptance at that domain?
- Is the domain's MTA unreliable?
- A recipient insists on receiving documents at an unreliable address.
- Are your messages triggering a spam filter?
- Can you tailor your messages to increase their acceptance at that domain?
Audit Resources
Most of the data concerning a message and its batch is available through a Web report. The Web reports are viewable on demand and automatically updated several times an hour. Important Web reports include:
- Batch Summary
- A detailed report on a particular batch giving all available statistics.
- Batch Timeline
- A history of the delivery events by hour for a particular batch.
- Message Archive: View Message
- A display of the SMTP source code for a particular message.
- Message Archive: Package Tracking
- The history of a given message as a list of events and the times at which they happened.
In cases where delivery failures result in bounce messages, those messages can be preserved in a mailbox for you to download or forwarded to another mailbox.
The raw text files from which the Web reports are compiled are also available for download. These files all reside in the batch archive directory and may be picked up through either FTP or HTTP:
failed.csv- A list of messages that are known to have failed.
map.txt- An index of messages giving the ePriority assigned ID, the client assigned ID and destination address for each.
trac*.txt.zip- Listings of message events.
Audit Procedures
The exact procedure that you follow in auditing failed messages will vary according to the information you have and the information you need.
The first step is determining the general cause of failure by looking at the highest event code logged:
- 5404
The message was cancelled by ePriority due to structural defects. Look for 52xx events for this message in the TRAC logs. One of these events will be issued every time a fatal defect is encountered during message construction. Those event codes should be specific enough for you to identify and correct the problem.
- 5406
The message was aborted because the batch was aborted. Someone with administrative access to your ePriority account did it.
- 5403,5405
The message was cancelled by ePriority because the recipient's MTA was unable or unwilling to accept the message. These are the hardest to audit, as MTA programs aren't usually very talkative when they're refusing mail.
- Make sure that the profile you used gave the message a fair shot at being delivered. If the entire delivery period for the message was in a blackout, for example, it would expire before a first attempt could be made.
- Were any 52xx events logged for the failed delivery attempts? They might provide a clue if not an explanation.
- Was anything else delivered successfully to that domain in that time period? If so, that would make a recipient MTA outage an unlikely cause.
- 5402
The message was accepted and then bounced by the recipient's MTA. This event code means that a bounce message was sent to ePriority. If you had the bounce message stored in a mailbox or forwarded on to your organization, you can look at it and see what it says. Bounce messages are usually reports that include a human-readable explanation of why a message was refused.
- 5301
The recipient's MTA accepted the message for delivery. If a bounce message was sent we never received it. Since MTAs usually send bounce messages when they reject mail, the more likely suspect would be a spam filter connected to either the MTA or the recipient's MUA.
The content of your message is probably triggering a spam filter. See «SpamAssassin: Tests Performed» for a list of phrases and patterns to avoid.
Requesting an Audit from ePriority
If you cannot reach a satisfactory conclusion using the data available, call ePriority support. We may be able to provide more information.