Back when Hotmail was the biggest thing in email, WebCom deployed a secret weapon that turned the tide in the email wars. WebCom SMTP (WSMTP) was a multi-phase project to create an entirely new email server from the ground up, something that could handle thousands of emails per second and HUGE attachments. But most importantly, it was designed to allow sysadmins to sleep at night!
WebCom started out humbly using off the shelf tools for the time, Sendmail for email, NCSA httpd for web serving, PERL for our web control panel, and Sybase for our customer database. NCSA httpd was the first component that needed upgrading, it was replaced with Netscape Enterprise Server.
Netscape Enterprise Server was developed by the team that originally wrote NCSA httpd, but Netscape would be better known for their contribution known simply as Netscape. Netscape Enterprise Server was a high performance commercial product that was modular, with shared object libraries (plugins) performing many of the individual tasks, this allowed the customer to modify the behavior of the server by writing their own plugins.
On the heels of the success with their HTTP server, Netscape bought a mail server product from Innosoft, called PMDF. This was renamed to Netscape Messaging (Mail) Server and simply known as NMS. Today this product is owned by Oracle and is referred to as Oracle Communications Messaging Server.
Netscape Mail Server
In 1996 we were feeling the pain of running Sendmail in a hosting context. Sendmail wasn’t well designed for handling queues of thousands of messages and stored everything in a single directory. Queue lengths grew long and exacerbated the situation, leading to unhappy customers and frazzled sysadmins. One thing was clear: Sendmail had to go!
Given the success of NES, we decided to purchase NMS and integrate it into our systems. NMS had a very different user account model than Sendmail. In Sendmail you have aliases, .forward files, and Unix mailboxes. Mail comes in for one or more email addresses associated with a Unix mailbox and it is either delivered to the mailbox or forwarded. NMS had a much more enlightened view where you had an account that could receive email for multiple addresses AND forward to multiple addresses AND deliver to a mailbox AND do auto-replies AND support POP3. The design disconnected the Unix user from the email account and permitted a single account to have many delivery methods at once.
It took some time to integrate NMS into our services, we had to rethink how email accounts were managed, create the dashboards to manage accounts, build the database schema to track them, do resource tracking and billing, and ultimately we had to develop a program that could fetch data from the database and push it to the NMS admin server. NMS was not designed with any sort of API, you were expected to directly manage users via their web interface!
The program we wrote for moving data from our database to NMS was actually used as prior art in a litigation between Web.com and Godaddy in 2008. I find this rather fitting given that Web.com tried previously to cancel the Webcom trademark. When NTT spun off the bad part of their hosting business, Web.com snatched up the webcom.com domain name. That domain name has since been collateralized to various lenders.
NMS was not the solution we were looking for, after all of the engineering investment, hardware costs, and product costs, we found it was incapable of handling the workload we purchased it for. We were running it on a first generation Ultra 1 workstation computer, this was much faster than the previous machine we ran Sendmail on, however it could not keep up with our demand.
Our interim workaround was to configure NMS to use a mail relay for outgoing email, for this purpose we purchased an x86 tower chassis server running Solaris x86 to run Sendmail. This alleviated some of the stress, ironically by using Sendmail as a dumb queue. This actually proved to work, but we still had the problem of customer outgoing mail passing through NMS unnecessarily. All of the troubles we had with NMS got us thinking about a longer term solution.
WSMTP
The WSMTP project started out with two fronts: an interim workaround that would buy us time with the NMS, and a longer term design that would be our ultimate vision of a working email server. We undertook 2 tasks in parallel: building an SMTP proxy and designing a schema that would allow us to store email messages in a Sybase database.
The SMTP listener was fairly straightforward, it would fork multiple processes and listen for email, then it would open a connection to one of 2 email servers to send that email message to: NMS or Sendmail on the Solaris x86 box. This proxy would tap into the existing NMS account schema in our provisioning database and determine if the combination of email sender and recipient could bypass NMS and go straight to Sendmail. Since our customers used us as their outgoing mail relay, a fair amount of traffic was destined for Sendmail. This was before SMTP authentication was available, so we had rulesets to prevent spammers from using us as an open relay.
Once the initial SMTP proxy work was done and we upgraded the Solaris x86 box to Linux (FASTER!), it bought us a good while to work on the development of WSMTP proper.
Overview
WSMTP was unique in that we stored email messages in a database. We didn’t use the database for long term storage, this was strictly transient storage, like the Sendmail mqueue directory. We had experience with the Sybase database server and wanted to leverage the work we did to interface C programs with Sybase and build something that was truly capable.
The first thing you learn in hosting is that 1 server is not enough, today that’s a no brainer, but back then you had a choice of desktop computer hardware, somewhat expensive Unix machines (SGI Irix), and really expensive machines (Sun Enterprise). Well, as things went, we kept purchasing bigger and faster Sun machines, adding CPU cards, etc. A single machine that costs as much as a house is not easily replaceable or duplicatable. Rather than trying to scale a monolithic architecture, we decided on a horizontally scalable design that used commodity hardware where possible. This meant a server that cost $2000 to build instead of $5,000 or $10,0000.
By having redundant arrays of inexpensive servers we could sleep at night and not worry about failures. This approach manifested as having 4 Sun Ultra 2 servers to run the daemons, and Linux PCs running Sendmail, Qpop, and Sybase.
WSMTP was broken up into 6 different scalable tiers so that we could add machines to whatever tier needed them. Here is a breakdown of the teirs:
Listener
The listener tier started out as the SMTP proxy we used with NMS, the listener implemented the traditional SMTP server semantics, receiving messages, validating recipients, and indicating status to the sender. The listener was implemented as a pre-forked daemon with (nominally) 125 connections per incoming server. It had rudimentary blacklist and filtering capabilities to cut down on SPAM. We deployed this on 2 Ultra 2 servers for a total of 250 simultaneous SMTP connections. The speed with which the system operated did not dictate more computing power or connections than this.
The listener would parse the incoming email message, creating a list of envelope recipients, sender, message headers, and body. The body was stored in 2KB chunks as BLOB types in the database. The server would handle whatever was thrown at it, a common test was sending many emails with 100MB attachments. I recall even sending an email with a 700MB attachment, and it worked!
Because the listener was disconnected from most of the rest of the system, its only dependency was having at least 1 SQL database it could queue messages to. This meant that every other tier (except SQL) could be broken, offline, or otherwise unavailable but the end users would not see anything amiss.
Router
The routing layer was the first processing step that messages took, this was implemented as 20 processes each, running on a pair of Ultra 2 boxes. The router would pull an email message, lookup the email account it was associated with, then perform processing based on the forwarding, POP, distribution list, auto-reply, and mbox attributes of the account. It would recursively process recipient lists, so if one email goes to a distribution list, it would read and pull all those addresses in, then continue to perform recursive resolution of addresses until none were left to process. There were 3 types of auto-replies: echo, vacation, and every time, the router would generate those ancillary messages while processing the main message.
The main takeaway from router is that you had exactly 1 email message in the queue, no matter how many recipients were on that message. Contrast that to how Sendmail handled messages, it amplified the number of messages based on how many recipients there were.
The last task that router performed was to create delivery batches. We originally envisioned using WSMTP to queue messages by MX record — consider how many domains Gmail handles and just the handful of MX servers they have. We planned to create 1 delivery batch per MX record, so all of the AOL users would get their email with just 1 connection and 1 message delivered to AOL.
How do you eat an Elephant? One bite at a time. How do you build a mail server with 1 fulltime engineer, 1 part-time DBA, and 1 part-time engineer? You do it in pieces. The MX batching was put aside to get a working system in place, so we simply decided to create batches of 20 addresses per message, sorted by domain.
Deliver
The next tier in WSMTP was the delivery agent, or wdeliver as it was called. This daemon would look for messages that were marked as deliverable and it would do one or more of the following:
- Send the message batch(es) to the Sendmail relay tier
- Send a message batch to the QPOP tier
- Deliver the message to an mbox
- Handle any MAILER-DAEMON messages that were generated during processing
The delivery agent was pretty straightforward, it simply had to talk SMTP to either the Sendmail relay tier, the QPOP tier, or invoke procmail to deliver email to one of our mbox destinations. This tier also ran 20 processes per server on 2 servers, it could handle all the traffic we had with just 40 processes.
Sybase
We used Sybase ASE as our transient mail storage layer. This tier was typically composed of 4 Linux boxes that ran ASE. We started out using ASE on Windows NT, but then I attended the Linuxworld Expo in 1999 and Sybase was handing out Sybase ASE 11.9.2 CDs for Linux. I’m sure they intended for these to be demo discs, given the 1024 file descriptor limit, but there was no license or prohibition on the use of the software! I March of the following year I requisitioned some hardware from the vendor next door and I built Redeemer, not because it was our first Linux ASE server, but because I was really into Unreal Tournament at the time!
The version of Sybase ASE I got from Linuxworld Expo could handle up to 412 simultaneous connections, so that is the number we used. Our design used many long lived connections instead of the more common approach today where you open a connection, do something, then close the connection. We never had any trouble with long lived connections, in fact Sybase worked better that way.
The Sybase servers generally had about 9GB of storage, spread over several 2GB partitions. They were typically dual CPU Pentium II machines, later being upgraded to much faster Pentium III Xeon servers once our parent company started deploying Sybase.
Sendmail Relay
Remember how we used Sendmail as a relay for NMS? Well, as the development timeline wore on, NMS was at its limit and the workaround SMTP proxy just wasn’t cutting it anymore. To expedite the development process we cut out MX batching and delivery queuing. To get around this, we simply deployed a group of 5 Sendmail boxes that we would use as a relay group. We did our best to make Sendmail’s job easy by batching messages alphabetically by domain, so for the vast majority of email it could be one and done.
The Sendmail relays occasionally needed maintenance, like emptying some queues or dealing with MXs that accidentally put us on their blacklist. As I recall, my coworker actually spent a fair amount of time dealing with AOL and other major providers to ensure we were on their whitelists, sometimes one of our relays would fall off and cause backups.
QPOP
The way we handled POP3 email was to try and simplify the process as much as possible. We had a modified version of QPOP that could authenticate against a DBM file to allow customers to access their email. The DBM files were used in many places to replicate the essential data that was in the Sybase provisioning DB, thus reducing the load on that DB. QPOP would handle the POP3 duties, while Sendmail would receive email from wdeliver and deliver it to customer POP boxes. We could have just as easily eliminated the Sendmail component and used the mbox delivery mechanism of wdeliver to send mail to NFS mounted customer mail spools.
Longevity
WSMTP was brought online about a year after Skynet 😉. There were some teething issues for the first couple months, but it stabilized and worked much faster than NMS. I can’t tell you how fast it was because we never encountered a situation where it was bogged per-se. There was a bug that existed for almost 10 years, it would mark messages as ready for processing even if there was an error during the DATA phase of the SMTP dialog. This meant many thousands of bodiless messages made it into the system and could not be processed. When we were under heavy SPAM attacks, guess what phase of the SMTP dialog the sender was rejected at? Yep, the DATA phase.
WebCom ceased to be WebCom in May of 2008, in fact my co-workers and I were laid off May 30th, 2008. I wouldn’t classify it as going down with the ship, more like last one out the door. WSMTP operated continuously from 1998 to 2008, it was one heck of a piece of work!