Technic Blog

Technik BlogDomain Message Authentication

March 30, 2018 Maximilian Fruth

Protect your email domain and your company's reputation

Spoofing sender addresses in e-mails exploits the unawareness of the online user. A typical example is email phishing. Phishing attempts to obtain personal data such as the authentication data of an online user by sending real looking emails. Part of the camouflage is usually also the forgery of the email sender address.

Email address spoofing harms both parties. On the one hand, the online user is affected, who is threatened with financial or non-material damage as a result of identity theft. On the other hand, the person or company whose sender address has been misused. Every case of abuse will damage a company's reputation, especially if an organization's email domain has been used for criminal purposes.

Email phishing was also one of the main reasons why a group of leading organizations developed the Domain-based Message Authentication Reporting and Conformance (DMARC) process. DMARC integrates the previously developed methods SPF (Sender Policy Framework) and DKIM (Domain Keys Identified Mail) and adds a policy on how the receiver should handle unauthenticated emails. In addition, DMARC offers reporting functions that support the analysis of implementation problems and misuse attempts.

Especially for companies with an large and complex email infrastructure the implementation of a DMARC policy is not without pitfalls and should be planned exactly. Communication channels such as newsletter mailing, web servers, email gateways and mail servers must initially be captured together with organizational domains and subdomains. Then an analysis can be performed to determine the implications of a SPF and DKIM roll-out. Special attention must be paid to special cases that cannot always be solved easily.

Prerequisite for a successful DMARC introduction is detailed knowledge of email transmission. The different methods of email validation by mail servers are important, in particular SPF and DKIM, which are discussed in more detail below.

SPF (Sender Policy Framework)

The Sender Policy Framework (SPF) is one of the most important methods for authenticating e-mail messages during transmission on the Internet. The importance of SPF has grown since the introduction of DMARC, as it can solve problems such as email forwarding.

However, creating and managing DNS SPF records is not as trivial as it seems. A survey of 1 million of the TOP domains revealed that approximately one in six domains with the SPF configuration violated a definition or restriction defined in RFC7208. We would like to show you a few pitfalls.

Simply put, SPF is a method that allows the owner of a domain to specify which mail servers are authorized to send email from that domain. The definitions required for this are stored in the DNS.

During an SPF check, email providers verify the SPF entry of the domain determined from the "Envelope From" address, also known as the return path. If the IP address sending an email on behalf of the "Envelope From" domain is not listed in the SPF record, SPF authentication fails.

SPF Flow Chart

Validation of subdomain addresses

The flowchart shows the validation of the sender IP using an SPF TXT record in the DNS zone of example.com. This would prevent forged e-mails from being sent by unauthorized servers. But this does not provide complete protection. This is because the SPF configuration of example.com is not inherited (rolled up) for subdomains (foo.example.com). In other words, the SPF of an organizational domain will not be considered as a fallback during validation if a specific record is missing in the subdomain. However, an SPF None is to be treated as an SPF Fail in a DMARC policy. The reason for this is that often a subdomain address in the SMTP MAIL FROM (Return-Path) is used and the FROM header line is forged when spam or phishing emails are sent. And this must be prevented. However, the objective should be that the name of an organization cannot be interpreted as a supposed sender and that the authenticity of an email can be clearly determined.

A strict DMARC policy requires that all subdomains of an organization that are used in the return path when sending mail must also be protected with SPF. This is also an effective protection against backscatter attacks, i.e. the blocking of a company's mail server with undeliverable messages (unsolicited error messages).

Using includes in the SPF configuration in DNS

Especially with large email infrastructures and especially when 3rd Party Providers are involved in sending emails, changes can soon lead to chaos in the SPF configuration, with fatal consequences. For this reason, various additional features have been implemented to define SPF Records. For example, definitions of the organizational domain can be included in the subdomain or definitions of other domains, such as those of newsletter providers. But be careful when using includes.

According to RFC7208, SPF configurations must limit the number of mechanisms that perform DNS lookups to a maximum of 10 per SPF check, including all lookups caused by using the "include" mechanism or the "redirect" modifier. The mechanisms of: "include", "mx", "a", "ptr" and "exists" count against the limit of 10 lookups. The mechanisms "all, "ip4" and "ip6" do not count against the limit of 10, since they do not require a DNS lookup.

If the maximum number of lookups is exceeded, the SPF check returns a Fail in the result and the delivery of the email is at least at risk. The problem here is often that the limit is quickly reached, especially with nested includes. Changes in the SPF configuration of 3rd Party Providers also pose a risk, especially if these are not communicated or not checked.

Problems with Email Forwarding

If a strict SPF FAIL policy is configured in a domain, messages are rejected that are forwarded by one or more mail servers to the actual receiving address. Problems occur under the following conditions:

  • The forwarding mail server uses the original return path. This is a often configured setting for email forwarding.
  • The next mail server in the forwarding chain checks SPF and does not have the forwarder's IP in the whitelist.

Since the sender does not usually know the complex paths to the receiver, an email authentication procedure designed only for SPF is incomplete. This gap was closed with DKIM, a procedure to digitally sign the email, and DMARC, a sender policy in the DNS for the receiver.

Checking the HELO(EHLO) identity

Error notifications and other automatic email messages usually have a blank string in the return path. To verify the authenticity of these messages, the HELO identity is verified in an SPF check. For this reason, when configuring mail servers, make sure that an SPF record is configured for the host name used in the HELO.

DKIM (Domainkeys Identified Mail)

Like SPF, DKIM is a method to prevent the reception of emails with a spoofed identity. DKIM is not an alternative, but a supplement to SPF, since the specific limitations of the individual methods can be corrected by the combination.

DKIM is based on the public-key cryptography technique. An email is tagged with a digital signature that can be verified by the receiver with the public key in the DNS. In this way it can be determined whether the e-mail was forged during transport or whether the sender address was manipulated.

DKIM Flow Chart

Usually the signing of the emails is done by the sending email server. The server calculates a hash value for each mail it sends and appends this hash to each mail in the e-mail header. The receiving server can use the signature and also calculate the hash. If the hash specified in the mail header matches the calculated hash, it is ensured that the mail originates from the sender's e-mail server and has not been changed.

To prevent the hash from being changed during the transmission of the mail, the hash is encrypted by the sending mail server and decrypted by the receiving mail server. The required keys can be created by the company itself and do not have to be certified.

Canonicalization / Normalization

Since the hash for header and body calculated by the receiving mail server must exactly match the value stored by the sender in the email, no changes may occur in the email during the entire transmission path due to reformatting. However, this is not always the case, as mail gateways with antispam filters often make minimal changes to emails to simplify the check. For example, unnecessary spaces and blank lines are removed, which already leads to an invalid DKIM signature.

DKIM offers options that parameterize the algorithms for generating and checking the hash values. With the simple canonicalization method, both spaces and case-sensitivity case are handled in the header signature. This is the option with the highest level of security, but also the most sensitive method for changes by mail gateways. This increases the risk that a DKIM validation by the receiving mail server will fail. The relaxed algorithm is more flexible, so that signature verification remains successful for minor changes. In this case, normalization is executed before the hash value is calculated:

  • All header field names (for example, Content-Type) are converted to lowercase (content-type).
  • All header values separated by a carriage return line feed (CRLF) are merged and all spaces at the end of the line are removed.
  • Consecutive whitespaces are compressed to a whitespace.
  • Spaces before or after the colons of the header fields are removed.

The relaxed algorithm normalizes the email body only marginally:

  • Consecutive whitespaces are compressed to a whitespace.
  • A CRLF is added at the end of the message.

Content-Transfer-Encoding

The specification of DKIM in RFC4871 contains an important note that is often not addressed in many DKIM technical documentation:

Some messages, particularly those using 8-bit characters, are subject to modification during transit, notably conversion to 7-bit form. Such conversions will break DKIM signatures. In order to minimize the chances of such breakage, signers SHOULD convert the message to a suitable MIME content transfer encoding such as quoted-printable or base64 as described in MIME Part One before signing. Such conversion is outside the scope of DKIM; the actual message SHOULD be converted to 7-bit MIME by an MUA or MSA prior to presentation to the DKIM algorithm.

Unlike canonicalization, content transfer encoding cannot be configured in the hash algorithm. This means that the validation of signed, 8-bit encoded emails may fail in certain cases. It is therefore necessary that the e-mail client (or the signing mail server) performs a 7-bit conversion. This is especially important with program libraries for sending e-mails (example PHPMailer), since this software often uses 8-bit encoding as default setting.

DMARC (Domain-based Message Authentication)

A DMARC policy defines for a sender domain that e-mails are to be protected by SPF and/or DKIM. In addition, the policy gives the receiving mail server instructions on how to perform the identity check and how to proceed if the check is not successful.

DKIM Flow Chart

Alignment

DMARC requires not only that a message passes the DKIM or SPF validation, but also that its identity domain is aligned. For SPF, the domain in the return path (Mail From) must match the domain in the mail header (From:). Depending on the configuration, this match must be exact (strict) or relaxed (subdomains are also possible).

For DKIM, the d=-domain of the valid signature must match the domain in the mail header (From:). The strict or relaxed mode for handling subdomains must be configured in the DKIM software.

The flowchart shows that at least one of the two checking procedures must be successful for an email to be delivered. This logic makes DMARC so valuable, because due to the limitations of SPF and DKIM mentioned above, a reliable validation during an individual test is not always possible.

Subdomains

DMARC also defines validation policies as TXT records in the DNS. But how does DMARC handles subdomains if they are involved in mail delivery? Does a separate policy have to be stored in the DNS for each subdomain? Or in other words, which DNS TXT records are checked by the receiver?

The following rules have been defined so that the receiving server does not have to perform an enormous number of DNS queries:

  • The receiver first determines the DNS record of the domain in the RFC5322 From address of the email header.
  • The receiver performs a maximum of two DNS queries to find a DMARC record for an email.
  • If the sender address contains a subdomain and no explicit DMARC policy is defined for the subdomain, the DMARC policy of the organizational domain is queried. DMARC records, for all other subdomains in the DNS tree are ignored.

The DMARC processor needs a so-called TLD-suffix list to determine the organizational domain. For some DMARC implementations such as OpenDMARC, this list must be provided explicitly and defined in the configuration. If a suitable configuration is missing, a DMARC validation for subdomains is only performed if a DNS record has been defined in the sudomain. The policy in the organizational domain cannot be found if the TLD list is missing or incomplete.

During DMARC validation the following DNS lookups are performed:

From Address

sender@example.com
sender@abc.example.com
sender@xyz.abc.example.com

First Domain Lookup

example.com
abc.example.com
xyz.abc.example.com

Second Domain Lookup

N/A
example.com
example.com

DMARC Reports

Reporting is a very helpful function of DMARC, especially in the rollout phase. The reports are automatically generated by the receiving mail servers and provide the administrators of a DMARC domain with useful information about the validation results of emails sent on the name of the domain and received from the server. The reports contain information about faulty configurations or misuse of the email domain.

In a DMARC policy, reports on the receipt of emails can be requested. In practice, however, it is up to the operator of the receiving server whether this option is executed. Large email providers usually generate anonymous and aggregated reports on a daily basis, which are sent to a specified receiving address according to the rua tag.

However, detailed error reports (ruf Tag) are are usually not generated. Although detailed reports would be very useful for the forensic to analyze errors in your email configuration or to be informed about phishing using the company domain. However, the load due to the generation of these reports is considerable, since the reports are already sent after the validation has been completed. Errors are reported for each individual check, although the DMARC overall check was successful. Most critical, however, is that the forensic report can be a complete copy of the rejected email in the so-called Abuse Reporting Format (ARF). Especially in the case of phising, this option could therefore trigger a flood of extensive report e-mails. For this reason, very few email providers support the generation of these detailed error reports.

DMARC Reports vs. Privacy

An interesting question arises as to whether DMARC Reports are in compliance with the applicable data privacy laws. An expert report commissioned by the Association of the German Internet Economy (eco) came to the conclusion that the implementation of DMARC is compatible with German law, with considerable restrictions in reporting. Here is an excerpt:

  • The transmission of aggregated reports is problematic from the aspect of data privacy: The sending IPs included in the reports are to be classified legally as personal data and are therefore subject to the regulations of the Federal Data Privacy Act.
  • RFC7489 chapter 7.1 requires an authentication and verification system to ensure that the specific report recipient is actually authorized and willing to receive the data in order to prevent misuse with regard to report reception. For external report addresses, we recommend that you have the reports sent to the DMARC policy domain and then forward them to the external report address.
Detailed reports are judged very critically in the expert opinion and placed with strong restrictions, so that a forensic analysis of the reports is hardly possible. This is certainly one reason why the generation of ruf-reports should be declined.

DMARC Rechtsgutachten des eco Verbandes

A DMARC policy can initially be operated in monitor mode (p=none). This allows an report analysis without time pressure. If problems were fixed and configuration changes were successful, the policy can be strengthened (p=reject).

Summary and sample Implementation

A complete list of options and value definitions for SPF, DKIM and DMARC was not included in thos blog. Numerous documents on this topic can be found in the Internet. The aim of this blog was to point out issues with the introduction of DNS-based email authentication methods.

Implementation of domain-based authentication

The following sample configuration is our recommendation for the implementation of DMARC in your company. However, implementation should only take place once all email communication channels in the company are known and the various problems have been analysed. Then nothing should prevent a successful DMARC introduction.

The SPF DNS Record

example.com. 3600 IN TXT "v=spf1 mx a:mta.example.com/24 include:_spf.newsletter-provider.com ~all"

  • Currently, DMARC validation is mainly used by large email providers. The older SPF method, on the other hand, is used on many mail servers. To mitigate the forwarding problem here, we recommend the definition of a SoftFail Policy.
  • External 3rd Party Providers (e.g. newsletter providers) should use definitions without further DNS lookups in their SPF records if possible. Pay attention to the limit of a maximum of 10 DNS lookups!
  • If subdomains are used in the return-path, a separate SPF record is required. In contrast to DMARC, no further lookup (organizational domain) is performed at SPF.
  • Make sure that an SPF record is defined for the HELO domain name of your mail server.

The DKIM signature

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=example.com; s=mail; t=1522326427; bh=ZK8Y9bOdiS+Vz/KGba1lfZFf/soe3k5iV4BHokN+vQA=; h=Subject:Date:From:To:From; b=jTWD48qsJ3JkTJtjrfGA8DKS72i+WZHGM9bURFcexIVcBkCTZ1dS0ig08Zgav/4WE PBqBjA5/7AQOvjSxKHghVCE35p92CvuyfENTH/j6WyCwZWuekJP4gbnPNiojXDRnKC VPI5QHnuJl+XqZ2ecQO9S1DZrc6vO3SwXhjqn4jrsgY3ipkVhZKIf+lyANjPQj7VBG H3d80r9ZMRECKvDEc6JmU06bG6D7rbNXSpeKhgTkFvQzQHjNdUrvIYktCNIdnlM8WK tEAw8eApZBEvEWNTV4N1ml/V0qNX/PglDPj++KXo+IL9aVHbP+S0aZRo8ePaCIdx10 Y9pcnN7lsZbEQ==

  • To avoid the problems described under Canonicalization / Normailization when validating the DKIM hash values, the signing sender and the receiving server should normalize the header and body of the email (c=relaxed/relaxed).
  • Make sure that the mail client or alternatively the signing mail server performs a 7-bit conversion (content transfer encoding: quoted-printable ).
  • When configuring the DNS record for DKIM, please note that TXT records with more than 255 characters must be split. The resulting parts of the TXT record must be enclosed in quotation marks and separated by a space in a line.

The DMARC DNS Record

_dmarc.example.com. 3600 IN TXT "v=DMARC1\; p=reject\; rua=mailto:reports@example.com"

  • DMARC requires a match in alignment. How exactly this must be done for the SPF and the DKIM check can be defined in the policy. In the example, the options aspf and adkim are not specified. Thus the defaults (relaxed) are used. Relaxed Alignment already accepts a partial match between SPF and/or DKIM record and RFC5322 From Domain. Thus, sub-domains can also be considered as matching.
  • If emails are sent under a subdomain, DMARC performs a maximum of two DNS lookups to find the DMARC policy in the DNS. First, a query is made under the subdomain. If no DMARC record exists, it will be searched under the organizational domain.
  • Especially in the implementation phase, the received reports for DMARC validation should be checked on a regular basis. The rua option specifies the email address for receiving anonymized and aggregated status reports. If a problem analysis with these reports is not possible, detailed reports can be requested with the option ruf. However, many email providers such as google do not support ruf-tags.
  • A recommended strategy for implementing DMARC is to first publish a simple DMARC record in monitor mode (i.e. p=none).