Mail on TurboVision

From Mailboxes to Everything Internet, Part 2: Mail Migration Under Real Traffic

Tue, 27 Feb 2007 00:00:00 +0000

If Part 1 was about building a bridge, Part 2 is about learning to drive trucks across it in bad weather.

Once mail leaves “small local utility” territory and becomes a central service, the conversation changes. You stop asking “can it send and receive?” and start asking:

can it survive hostile traffic?
can it be operated by more than one person?
can policy changes be rolled out without accidental outages?
can users trust it on weekdays when everyone is overloaded?

In our case, that transition happened between 2001 and 2007. By then, Linux mail infrastructure was no longer experimental in geek circles. It was production, with all the consequences.

Why we moved away from “wizard-level config only”

Many older setups depended on one person who understood every macro, alias map, and legacy hack in a mail config. That worked until that person got sick, changed jobs, or simply slept through a pager alert.

Our first explicit migration goal in this phase was organizational, not technical:

A competent operator should be able to reason about mail behavior from plain files and runbooks.

That goal pushed us toward simpler policy expression and clearer service boundaries. Whether your final stack was sendmail, postfix, qmail, or exim mattered less than whether your team could operate it calmly.

The stack boundary model that reduced incidents

We separated the pipeline into explicit layers:

SMTP ingress/egress policy
queue and routing
content filtering (spam/virus)
mailbox delivery and retrieval (POP/IMAP)
user/admin observability

The key idea: one layer should fail in ways visible to the next, not silently mutate behavior.

When all logic is crammed into one giant config, failure states become ambiguous. Ambiguity is expensive in incidents.

Real-world migration pattern: parallel path, then cutover

Our cutovers got safer once we standardized this pattern:

deploy new MTA host in parallel
mirror relevant policy maps and aliases
run shadow traffic tests (submission + delivery + bounce paths)
cut one low-risk domain first
watch queue/error behavior for a week
migrate high-volume domains next

This sounds slow. It is fast compared to cleaning up one bad all-at-once switch.

The anti-spam era changed architecture

By 2005-2007, spam pressure made “mail server” and “mail security” inseparable. A useful configuration had to combine:

connection-level checks (HELO sanity, rate controls)
policy checks (relay restrictions, recipient validation)
reputation checks (RBLs)
content scoring (SpamAssassin-like layer)
malware scanning

A typical policy layout in that era looked conceptually like:

ingress:
  reject_non_fqdn_sender
  reject_non_fqdn_recipient
  reject_unknown_sender_domain
  reject_unauth_destination
  check_rbl zen.example-rbl.net
  pass_to_content_filter

content_filter:
  spam_score_threshold = 6.0
  quarantine_threshold = 12.0
  antivirus = enabled

The exact knobs differed by implementation. The architecture of staged decision points did not.

False positives: the quiet business outage

Most teams fear spam floods. We learned to fear false positives just as much. Aggressive filtering can silently break legitimate workflows, especially for smaller orgs where one supplier’s odd mail setup is still mission-critical.

We moved to a tiered posture:

reject only on high-confidence transport policy violations
tag/quarantine for uncertain content cases
teach users to report false positives with full headers

This reduced support friction and preserved trust.

A service users trust imperfectly is a service they route around with private inboxes, and then governance fails quietly.

Queue operations: numbers that actually mattered

People love total queue size graphs. Useful, but incomplete. We tracked a more operational set:

queue age percentile (P50/P95)
deferred reasons by top code/domain
bounce class distribution
local disk growth vs queue growth
retry success after first deferral

Why queue age percentile? Because a small queue with very old entries is often more dangerous than a large queue of fresh retries.

Submission and auth became first-class

As users moved from fixed office networks to mixed environments, authenticated submission stopped being optional. We separated trusted relay from authenticated submission explicitly and documented it in end-user instructions.

A minimal policy split looked like:

relay without auth only from managed LAN ranges
require auth for all remote submission
enforce TLS where practical
disable legacy insecure paths gradually with communication windows

People remember technical changes. They forget user communication. In migrations, communication is part of uptime.

Logging: from forensic artifact to daily dashboard

Early on, logs were mostly used after incidents. By mid-migration, we treated them as daily control instruments. We built tiny scripts that summarized:

top rejected senders
top deferred recipient domains
top local auth failures
per-hour inbound/outbound volume

Even crude summaries built operator intuition fast. If Tuesday looks unlike every previous Tuesday, investigate before users notice.

DNS and reputation maintenance discipline

Mail reliability in 2007 is tightly coupled to DNS hygiene and sending reputation. We added recurring checks for:

forward/reverse consistency
MX consistency after planned changes
SPF correctness
stale secondary records

A single stale record can cause “works for most people” failures that consume days.

Incident story: the day policy order bit us

One outage class recurred until we fixed our process: policy ordering mistakes.

A config reload with one rule moved above another can flip behavior from permissive to catastrophic. We had one deploy where recipient validation executed before a required local map was loaded in a new process context. External effect: temporary 5xx rejects for valid local recipients.

The post-incident fix was procedural:

stage config in syntax check mode
run policy simulation against known-good/known-bad test cases
reload in maintenance window
verify with live probes
keep rollback snippet ready

The technical fix was small. The process fix prevented repeats.

The human layer: runbooks and ownership

Mail operations improved when we wrote short, explicit runbooks and attached clear ownership:

“high queue depth but low queue age”
“low queue depth but high queue age”
“sudden outbound spike”
“auth failure burst”
“upstream DNS inconsistency”

Each runbook had:

first checks
known bad patterns
escalation condition
rollback or containment action

The format matters less than consistency. Under stress, consistency wins.

Migration economics: why smaller steps are cheaper

A common argument was “let’s wait and migrate everything when we also redo identity and web hosting.” We tried that once and regretted it. Bundling too many moving parts creates coupled risk and unclear root causes.

Mail migration became tractable when we treated it as its own program with clear acceptance gates:

transport reliability
policy correctness
abuse resilience
operator clarity
user communication quality

Only after those stabilized did we stack adjacent migrations.

What changes in 2007 operations

Compared with 2001, a 2007 Linux mail setup in our environment looked less romantic and much more professional:

explicit relay boundaries
documented policy layers
operational dashboards from logs
recurring DNS/reputation checks
reproducible deployment and rollback
practical abuse handling without user-hostile defaults

We did not eliminate incidents. We made incidents legible.

That is the difference between hobby administration and service operations.

Practical checklist: if you are migrating this year

If you are planning a migration this year, this is the condensed list I would tape above the rack:

define policy boundaries before touching software packages
build and test in parallel, then cut over domain-by-domain
implement anti-spam as layered decisions, not one giant hammer
measure queue age, not just queue size
separate LAN relay from authenticated submission
automate log summaries your operators will actually read
simulate policy before reload
treat user comms as part of the rollout, not afterthought

If you do only four of these, do 1, 3, 4, and 7.

Weekly review ritual that kept us honest

One habit improved this migration more than any single package choice: a short weekly mail operations review with evidence, not opinions.

The agenda stayed fixed:

queue age trend over last seven days
top five defer reasons and whether each is improving
false-positive reports with root-cause category
auth failure clusters by source network
one policy/rule cleanup item

We kept the meeting to thirty minutes and required one concrete action at the end. If there was no action, we were probably admiring graphs instead of improving service.

This ritual sounds simple because it is simple. The impact came from repetition. It turned scattered incidents into a feedback loop and gradually removed “mystery behavior” from the system.

From Mailboxes to Everything Internet, Part 1: The Gateway Years

Tue, 14 Mar 2006 00:00:00 +0000

By the time people started saying “everything is online now,” many of us had already lived through two different worlds that barely spoke the same language.

The first world was mailbox culture: dial-up nodes, message bases, Crosspoint setups, nightly rituals, packet exchanges, and local sysops who could fix a broken feed with a modem command and a pot of coffee. The second world was internet service culture: DNS, MX records, SMTP relays, POP boxes, always-on links, and users asking why the web was “slow today” as if bandwidth was weather.

This series is about that crossing.

Part 1 is the beginning of the crossing: the gateway years, when we still had one foot in mailbox software and one foot in Linux services, and we built bridges because nothing else existed yet.

The room where migration began

Our first Linux gateway did not arrive as strategy. It arrived as a beige box rescued from an office upgrade pile, with a noisy fan and a disk that sounded like it was counting down to failure. We installed a small distribution, gave it a static IP, and told ourselves this was “temporary.” It stayed in production for three years.

The old world was stable in the way old systems become stable: every sharp edge had already cut someone, so everyone knew where not to touch. Crosspoint was doing its job. Message exchange windows were predictable. Users knew when lines were busy and when downloads would be faster. Nothing was modern, but everything had shape.

The new world was not stable. It was fast and constantly changing, but not stable. Protocol expectations moved. User behavior moved. Threat models moved. Providers moved. The migration problem was not “install Linux and done.” The migration problem was preserving trust while replacing almost every layer under that trust.

That is why gateways mattered. They let us migrate behavior first and infrastructure second.

Why gateways beat big-bang migrations

The smartest decision is refusing the heroic rewrite mindset. We do not announce one switch date and burn the old stack. We insert a Linux gateway between known systems and unknown systems, then move one concern at a time:

forwarding paths
addressing and aliases
queue behavior
retries and failure visibility
user-facing tooling

That ordering was not glamorous, but it protected operations.

Big-bang migrations look fast on whiteboards and expensive in real life. Gateways look slow on whiteboards and fast in incident response.

The first practical bridge: message transport

The earliest bridge usually looked like this:

mailbox network traffic continues as before
internet-bound traffic exits through Linux SMTP path
incoming internet mail lands on Linux first
local translation/forwarding rules feed legacy mailboxes where needed

This gave us one powerful property: we could debug internet path issues without disrupting internal mailbox flows that users depended on daily.

A minimal relay policy draft from that era often looked like:

# conceptual policy, not distro-specific syntax
allow_relay_from = 127.0.0.1, 192.168.0.0/24
default_action   = reject
local_domains    = example.net, bbs.example.net
smart_host       = isp-relay.example.net
queue_retry      = 15m
max_queue_age    = 3d

You can replace every keyword above with your preferred MTA syntax. The architectural point is invariant: explicit relay boundaries, explicit domains, explicit queue policy.

Addressing drift: the hidden migration tax

The first operational pain was not modem scripts or DNS records. It was naming drift.

Mailbox-era naming conventions and internet-era address conventions were often related but not identical. We had aliases in user muscle memory that did not map cleanly to internet address rules. People had decades of habit in some cases:

old handles
area-specific routing assumptions
implicit local-domain shortcuts

The migration trick was to preserve familiar entry points while moving canonical identity to internet-safe forms.

We ended up with translation tables that looked boring and saved us hundreds of support mails:

old_alias      -> canonical_mailbox
sysop          -> admin@example.net
support-local  -> helpdesk@example.net
john.d         -> john.doe@example.net

Most migration failures are identity failures dressed as transport failures.

DNS is where we stopped improvising

In mailbox culture, many routing assumptions lived in operator knowledge. In internet culture, that same routing intent must be represented in DNS records that other systems can query and trust.

The day we moved MX handling from ad-hoc provider defaults to explicit records was the day incident triage got easier.

A tiny zone fragment captured more operational truth than many meetings:

@      IN  MX 10 mail1.example.net.
@      IN  MX 20 mail2.example.net.
mail1  IN  A  203.0.113.15
mail2  IN  A  203.0.113.16

The key is not syntax. The key is declaring fallback behavior intentionally. If primary host is down, we already know what should happen next.

Queue literacy as survival skill

Every sysadmin migrating to internet mail learns this eventually: queue behavior is where confidence is either built or destroyed.

Users do not care that a remote host gave a transient 4xx. They care whether their message disappeared.

So we trained ourselves and junior operators to answer three questions fast:

Is the message queued?
Why is it queued?
When is next retry?

Those three answers turn panic into process.

During the gateway years, we posted a laminated “mail panic checklist” near the rack:

check queue depth
sample queue reasons
verify DNS and upstream reachability
confirm local disk not full
verify daemon alive and accepting local submission

It looked primitive. It prevented chaos.

Mailbox systems had abuse, but internet-facing SMTP changed abuse economics overnight. Open relay misconfiguration could turn your server into a spam cannon before breakfast.

Our first open relay incident lasted forty minutes and felt like forty days.

We fixed it by moving from permissive defaults to deny-by-default relay policy and by testing from outside networks before every major config change. We also added tiny audit scripts that checked banner, open ports, and policy behavior from a second host. Nothing fancy. Just enough automation to avoid repeating avoidable mistakes.

The cultural shift was bigger than the technical shift: “it works” was no longer sufficient. “It works safely under hostile traffic” became baseline.

Going online changed support load

A mailbox user asking for help usually came with local context: software version, dialing behavior, known node, known timing window.

An internet user asking for help often came with “mail is broken” and no context.

So we created what we now call structured support intake, long before that phrase became common:

sender address
recipient address
timestamp and timezone
exact error text
one reproduction attempt with command output

This cut mean-time-to-triage massively.

In other words, migration forced us to formalize operations.

The tooling stack we trusted by 2001

By the end of the earliest gateway phase, a reliable small-site stack often included:

Linux host with disciplined package baseline
DNS under our control
SMTP relay with strict policy
basic POP/IMAP service for user retrieval
log rotation and disk-space monitoring
scripted daily backup of configs and queue metadata

We did not call this “platform engineering.” It was just survival with documentation.

Why these gateway lessons matter in 2006 operations

In 2006 operations, the web moves fast. Broadband is common in many places. Users assume immediacy. People discuss hosted services seriously. Yet the gateway lessons still hold:

preserve behavior during infrastructure changes
migrate one boundary at a time
make routing intent explicit
treat queues as first-class observability
never ship mail infrastructure without hostile-traffic assumptions

These are not legacy lessons. They are durable operations lessons.

Field note: the migration metric that mattered most

We tried to track many metrics during those years: queue depth, retries, bounce rates, uptime percentages. Useful, all of them. But the metric that predicted success best was simpler:

How many issues can a tired operator diagnose correctly in ten minutes at 02:00?

If your architecture makes that easy, your migration is healthy. If your architecture requires one heroic expert, your migration is brittle.

Gateways made 02:00 diagnosis easier. That is why they were the right choice.

Current migration focus areas

The same gateway discipline applies immediately to the next pressure zones:

mail stack policy and anti-spam layering without open-relay mistakes
file/print and identity migration in mixed Windows-Linux environments
perimeter/proxy/monitoring runbooks that keep incident handling predictable

Appendix: the one-page gateway notebook

One practical artifact from these years deserves to be copied directly: a one-page gateway notebook entry that every on-call operator could read in under two minutes.

Ours looked like this:

Gateway host: gw1
Critical services: smtp, dns-cache, queue-runner
Known upstreams: isp-relay-a, isp-relay-b

If mail delayed:
  1) check queue depth + oldest queued age
  2) check DNS resolution for target domains
  3) check upstream reachability and local disk free
  4) sample 5 queued messages for common reason
  5) decide: wait/retry, reroute, or escalate

Escalate immediately if:
  - queue age > 2h for priority domains
  - repeated local write errors
  - resolver timeout > threshold for 15m

That page did not make us smarter. It made us consistent. In migration work, consistency under pressure is often the difference between a bad hour and a bad weekend.