Tag Archives: email

A fascination with special characters

In the computer world, special characters can have a certain usefulness or can be a hindrance. The very first special character I learnt 25 years ago, was the colon. It was ( still is ) used as the delimiter to change drive letters in DOS.

eg. if you were in drive C and wanted to go to drive D, you would type:

d:

Simple? Yes

But not always so. Because of the special intent of certain characters, using them in a non-content fashion can be problematic ( sometimes catastrophic ).

What do I mean by non-content? Well consider a document – the text in the document is the content and the title of the document is ( part of ) the metadata or non-content.

System administrators or programmers will know that #! has special meaning. If you want to use it in a non-content fashion, you have to take extra steps. If you don’t, you may not get the result you expect.

This brings to mind a recent story from a colleague. A client had gone to a web development firm to ask about pricing for a web site. Not having done anything similar before, they got quite a shock when the quote was presented.

The client decided to look for other avenues and ended up doing the site themselves using an online templating service. It was a reasonable site but had many links and pages, and was a bit complex. Nevertheless, the client then approached an SEO firm ( search engine optimisation ) to make sure that the site was successful from a search and marketing point of view.

But the site never appeared in search results at all. The client was quite upset with the SEO firm but after much troubleshooting the SEO firm determined the issue was not with them. So the client asked my colleague to take a look and he found the issue: the client had prefixed all page titles on the site with #! – those in the know  understand that this means “Do not index this site under any circumstances!” And so the search engines blissfully ignored the site.

And so the client had to go back and re-title around 300 web pages – quite a bit of not-so-enjoyable work.

It often occurs that someone is not happy with the pricing of a service but there might be a very good reason for that pricing – the service provider has experience and understands the ins and outs of their industry. That experience costs money and is of value. Doing things yourself can sometimes end up costing you more in the end.

Not to digress, special characters should be avoided at all cost. Do NOT use them!!!

My philosophy is to only use lower case and no special characters at all, especially when naming things. Keep it simple.

Email delivery ( and some other stuff about email )

Email is still the single most used communications tool on the internet and will probably remain so for some time to come. We send in the region of 300 Billion emails per day and for the most part, everything just works. On the face of it, email is simple: compose, address and send. But behind the scenes, the infrastructure that makes email work is not so simple ( or reliable ) as we’d like to think. Here follows a quick ( and hopefully reasonably simple ) overview of how email gets from a sender to a recipient.

The quick overview

sender -> sender MTA -> recipient MTA -> recipient

MTA = mail transfer agent

MUA = email client ( like Outlook, Thunderbird,  etc. )

In very simple terms, one composes an email with an email client ( MUA ), addresses it to one or more recipients and then sends it through the local email server ( MTA ). The sender’s local email server connects to the recipient’s email server ( MTA ), and transfers the email to the recipient’s mail store. The recipient’s email client ( MUA ) will then poll the mail store on a regular basis and retrieve any emails waiting there.

But there is actually a whole lot more that happens in between the sender and recipient.

The in-depth story – sender side

sender -> sender mta [ av check, content checks,  relay check ] -> relay

There’s quite a few steps that occur before an email even leaves your email server. Here follows some of them:

  • av check: your email will normally be anti-virus checked before leaving the network to make sure that viruses are not propogated
  • content check: the email content ( including body and subject line ) will be checked against a static list for content that should be blocked
  • relay check: your network location is checked to see if it is allowed to use the specified email server to send email; we don’t want just anyone using our email server to send email otherwise they could use it to send spam ( also called an open relay )
  • relay: once an email is ready to be sent, it may be routed through one or more relays or intermediate email servers

Relays / Smarthosts

A relay is simply an intermediate email server whose sole job is to route your email to its destination. Most ISP’s will make a relay available to their clients to use as a sending email server ( also called smtp or outgoing server ). More specifically, users who are on 3G, dialup or dynamic IP addresses will need to use a relay. For users who have their own in-house email server, this will typically be configured on the email server itself so nothing further needs to be done by individual users.

However, even if you have your own in-house email server, when traveling, you could have issues sending email for a number of reasons:

  • the ISP you are currently connected to may not allow you to use email servers other than their own
  • your in-house email server may not allow you to connect from external networks or may require you to authenticate with a username and password before sending email

You can speak to your email administrator for more information on configuring your email client when out on the road.

 The in-depth story – recipient side

relay -> recipient mta [ RBL checks, greylisting, address checks, anti-spam checks, av check, content checks ] -> recipient

  • RBL checks: real-time blackhole lists are lists of spamming servers that are maintained on the internet by a number of commercial and non-commercial companies; when an email is received, the sending server’s IP address is checked against these lists – if it matches then the sending server is considered a spammer and the email is dropped or bounced
  • greylisting: when the recipient server receives an email from the combination of a sender and a sending server that it has never seen before, it temporarily rejects the email; the sending server will then resend the email a short whole later; the reason this process stops spam is because spammers are typically not setup to receive the temporary reject notice and thus will never resend the original email
  • address checks: the recipient server will check that the sending server has a valid DNS hostname ( eg. mail.server.com )  and also that the hostname maps back to the IP address that is presented by the sending server ( reverse DNS check )
  • anti-spam checks: these include intelligent ( heuristic ) checking of email and scoring of a variety of parameters of the email; if the total score exceeds a preset threshold value, then the email is considered to be spam
  • anti-virus check: self explanatory?
  • content checks: the subject line and body of the email are checked against a preset list of content ( eg. words or phrases ) – if matched, then the email is rejected or dropped

All of the above checks are there to reduce the amount of spam ( non-solicited ) emails that are received. Considering the amount of spam that is generated and sent each day ( ~ 90% of all email ), any opportunity to stop spam email is a good thing. Unfortunately, sometimes even valid email can be blocked by recipient servers.

When an email is blocked, an error message is typically sent back to the sender so that they ( or their email administrator ) can take some action that may result in future emails to the recipient being successfully accepted.

An example of some block messages

You’ll notice that when you receive an error message for an email that was not successfully delivered, there is normally a code associated with that error message like 550 or 554. These are SMTP error codes and provide a standard way of understanding email routing and other errors.

 

Mailbox is full

Well  this one is self-explanatory – the mail store of the user you sent email to is full, they need to delete some email before they can receive more

 

Message exceeds size limit

The email you sent is larger than the recipient server is willing to accept; the size that can be received varies from server to server but you should try to keep your email size below 7MB

 

User unknown/invalid recipient/mailbox unavailable

The email address is wrong or does not exist

 

Please turn on SMTP Authentication in your mail client, or login to the IMAP/POP3 server before sending your message.  (mail.server.com) is not permitted to relay through this server without authentication

In this case, the relay or outbound server  requires your username and login to send email

 

blocked using zen.spamhaus.org; http://www.spamhaus.org/query/bl?ip=71.230.109.142

A server trying to deliver an email to us was regarded as a spam server by an RBL and blocked; in this case, the recipient server administrator needs to request the de-listing of their domain/address from the specific RBL list

 

Your access to this mail system has been rejected due to the sending MTA’s poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means.

Similar to the example above, our email to someone else was blocked because the recipient server regarded us as being a spammer

 

Recipient address rejected: Policy Rejection- Please try later

An end user will probably not see this message directly but it’s an example of greylisting

 

Could not send your message. Error 421

This one is very cryptic because it only gives an error code but no real explanation of what went wrong. In this case, error 421 means the service was not available; more specifically it could mean that the recipient server had an issue and could not accept the email.

 

 Sender address rejected: relay access denied

Quite a common one, this means that either you need to authenticate or you are not allowed to use this relay at all

 

Reading an error message carefully ( ignoring any complex addressing or other information ) can give you a clue as to why your email was rejected/bounced. Make the necessary changes ( or ask your email administrator ) and resend your email.

 

RBLs ( realtime blackhole lists )

The address of incoming emails is checked against RBLs to see if they are listed as spammers. RBLs, along with greylisting, is the biggest defense we have against spam email. However, when they fail to work correctly they can cause many issues. An RBL went off-line recently and as a result, any email server checking against this list, would block all incoming email.

Conclusion

While many assume that email is a guaranteed service, this is not the case; email can be blocked or dropped for any number of reasons including network errors, email server misconfiguration and anti-spam reasons. Some anti-spam measures can have a negative impact on the receipt of emails even when these are expected to be delivered.

Email can be made to work reasonably efficient and reliably but it requires correct configuration of both sides of the link, and anything in between, to function well. And that requires system administrators who understand the intricacies of email, something that is in short supply …

For more information on using email efficiently, please see my article Email woes and etiquette

New security issue: typo-squatting

Malware, phishing, pharming, typo-squatting, etc. There’s a long list of security issues we have to deal with every day. Keeping track of these and responding correctly in each case is a veritable minefield. That’s after our newly updated anti-virus app has completely missed the threat.

Typo-squatting is the well-known practice of serving up scams or threats via sites that have been misspelled by the user. ie. user enters www.drell.com instead of www.dell.com and gets sent to an infected web site. This has now been a move to translate this into the realm of email. It consists of using so-called “doppelganger domains” and mail servers for intercepting emails sent by mistake to them.

So you send an email to user@zadomain.com when you meant to send it to user@za.domain.com. There is a registered domain for zadomain.com with an email server waiting for your email. All email can now be harvested by the ‘fake’ server and used for nefarious purposes.

Further on from this scenario, attackers can now also execute Man-In-The-Middle ( or should I say Man-In-The-Mailbox ) attacks by sending the received email on to the correct address. The recipient replies ( without checking the reply-to-address ) and the attacker now receives the response.

Some researchers from Godai Group have also pointed out that these kind of attacks have probably been underway for a while. A number of doppelganger domains have already been set up by individuals that – judging by the domain registrant email information – are mostly (if not all) based in China.

But the worst part of it is that even though companies are aware that typosquatting can present a big problem, only one of the 30 companies for which the researchers have set up doppelganger domains have noticed them doing it and reacted appropriately. Approximately 151 out of the TOP 500 are susceptible to these attacks.

Time to be even more vigilant …

Email woes and etiquette

Based on issues that a client of mine has had in recent times with email, I decided to resurrect and rehash my XStore IT Bulletin no. 1 from July 2006.

One of the biggest issues with email is the lack of understanding of how it works. In addition, the requirement to implement anti-spam solutions falls foul of the lack of etiquette and/or knowledge of most email senders. Most email users don’t understand that their misuse ( due to ignorance or lack of knowledge ) of email solutions directly impacts the smooth operation of email systems. The more misuse, the more issues there are.

And so it’s in everyone’s interest to learn how to use email properly and effectively. What are the main issues?

The 1st cardinal sin is the forwarding of emails without merit. Chain emails are often carriers of virus and malware, and as such, the forwarding of these emails, directly impacts the security of your recipients.

The 2nd major issue is formatting of emails. This includes content, subject lines and addressing. Anti-spam solutions look at all these areas  to make a decision as to whether they think an email is spam or not.

3rd is the idea ( and expectation ) that email systems offer guaranteed delivery. SMTP ( the protocol used to send email ) was never designed with guaranteed delivery in mind. Although mechanisms were built-in to work around Internet and system failures, delivery of emails is a haphazard affair. The fact that it works well most of the time is a testament to the original specification’s quality, and additions that have been made since then. However, don’t assume that if you send an email, that it will received. And don’t count on it for critical business or life-threatening situations.

How does email work?

When you send email from your email application, it will travel as follows:

  • from your desktop to your sending/outgoing server ( you may have one at your business or use your ISP’s )
  • from your business email server or ISP to the recipient’s email server
  • the recipient then reads the email off their server

Along the way, there are a number of checks done, both by your sending server and by the receiving server. Your server would for example, make sure that the recipient domain ( @domain.co.za ) exists. The recipient server would perform a number of anti-spam checks.

Spam is an unfortunate reality of life when it comes to email usage. There are a number of ways we can reduce the spam volume received everyday using our email servers without negatively impacting our users.

a. Anti spam engine

Anti spam engines such as SpamAssassin and PureMessage work using a multitude of modes and tests including header and text analysis, Bayesian filtering, DNS blocklists, and collaborative filtering databases. A score is calculated based on these tests and if that score exceeds a preset value, the email is tagged as spam.

b. DNS blocklists ( RBL / DNSBL )

When an email is received, the email server checks the name of the sending server. If that name exists on a preset database ( there are many of these on the Internet ), the email is blocked or rejected.

c. Content filtering

Checks can be done on the header and body contents of emails to find specific instances of words, characters or phrases. If found, the email is blocked or rejected

d. Grey listing

When an email is received with a first time combination of sender, recipient and sending server, the email is temporarily rejected. On retry the email will be accepted. The basis for this mechanism is the fact the very few spam servers would resend an email.

Email servers can deal quite effectively with spam but one needs to make use of all the available options to combat this problem. The flip side is that we need to take care when sending emails, as this will assist the server actions that are taken.

Here are some basic ideas that one should use when sending email:

  • do not forward chain letters or suspicious emails
  • do not cc recipients in a general email, use bcc instead – most recipients do not want their email addresses being seen by others and in addition this is a security hazard ( most email clients automatically add cc’d address in received emails to their address books; a subsequent virus infection on one those recipients would then be sent to everyone in their address book )
  • only use cc where a recipient is directly required to respond to your email
  • do not respond to unsubscribe requests in spam
  • never reply to spam
  • use text email where possible instead of rich text email with pictures, etc.
  • clearly think about the contents of email with respect to spam filters and the like
  • check the size of an email before sending it – many companies do not have much Internet bandwidth and these emails
  • might be blocked or filtered as a result
  • check with your IT support if you’re unsure whether an email address is valid or not
  • make use of filters in your email client to filter out spam
  • learn about the built-in anti-spam features in your email client
  • make use of good email etiquette – you are representing your company
  • do not use your business email address to subscribe to websites on-line unless they are business-related

There is much more to email etiquette and use; if you’d like to know more, please check the following links:

http://www.emailreplies.com/

http://iwillfollow.com/email.htm

The misuse and incorrect use of email systems can directly cause one to receive more spam and security-related emails, so it’s in your interest to learn how to use email more effectively and safely.

New email service coming from XStore and SilcomIT

ISPs have recently been getting very strict in terms of email usage on their ADSL networks and have implemented quota management for outbound email when using their smtp relays. As a result, a client could:

  • have email rejected once a certain threshold is reached ( eg. 30 emails per sender per hour )
  • be disconnected from the internet once a 2nd higher threshold is reached

This obviously causes serious issues for those who deal with marketing emails, emailing lists and generally a higher usage than what the ISPs allow. I’ve already seen 2 clients this week be disconnected from the  internet due to UBE ( unsolicited bulk email ) notices.

As a result, XStore and SilcomIT are combining under the name of EmailStor to offer a relay service for all clients who need a customised service for sending email. The service will be available from this week coming and will include a myriad of additional email features like:

  • selectable rate and storage quotas per domain and/or per user
  • mailing lists
  • forwarding/bounces ( eg. single to multiple – simple mailing list )
  • aliases

The sequence is currently:

user -> company email server -> ISP relay -> recipient server

this will change to:

user -> company email server -> EmailStor relay -> recipient server

The system is still being built as we speak but will have initial relay services available from next week. Additional services and features will come on-line in the next few weeks.

In addition, we will be offering other internet services:

  • email hosting with POP3 and IMAP support
  • backup email service
  • web hosting
  • ftp/data storage
  • dns hosting

We’re busy putting the website, pricelists and business documentation together – this will be available from tomorrow and we’ll send out an email explaining the service.

Note, to limit the possibility of UBE being sent from your network ( eg. from a virus-infected PC ), please consult XStore to make sure your firewall allows ( and has enabled ) the ability to block outbound smtp traffic from all internal computers except your mail server.