Mustang Wang

Your Server Has Been Trying to Tell You Something

The problem nobody warns you about

A compute server is happy to tell you things. A disk is failing. A cron job died at 3am. The nightly job finished, or it didn’t. The kernel logged something it really wants a human to read. Linux has been able to send this mail since before most of us were writing code.

The catch is that “send mail” and “reach an inbox” are two very different things. A headless box will cheerfully hand a message to its local mail spool and call it done. That message goes nowhere. And the day you finally wire it up to send for real, a fresh problem shows up: mail coming straight from some random server IP, with a From address like root@your-box.localdomain, gets dropped on the floor by every serious mail provider. No bounce, no warning. It just evaporates.

I wanted a small compute server to email a whole team when something needed attention. Not a full mail server, nothing with an open port 25. Just outbound notifications that actually arrive. Here’s how that works, and the two places it quietly goes wrong that the standard tutorials never mention.

The shape of the fix

The trick is to not send directly at all. Instead you configure the local mail transfer agent (Postfix, on most boxes) as a relay that authenticates to a mailbox you already own and hands every outgoing message to it. Your mail provider does the hard part: it already has a trusted IP, and its domain already publishes the SPF, DKIM, and DMARC records that make mail land in inboxes instead of spam folders. Your server just needs to log in and pass the message along, exactly like your laptop’s mail client does.

Every send-only tutorial online covers this part, and they all point you at SendGrid or Mailgun. That’s fine if you want to sign up for another SaaS. But if you already run a mailbox, or your team does, you can relay through that instead and add zero new dependencies. Almost nobody writes that up, and it’s the setup most of us are actually in.

The core of /etc/postfix/main.cf:

# hand everything to a mailbox we already own
relayhost = [mail.example.net]:587

# authenticate to it, over TLS, like any mail client would
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_security_level = encrypt
Bash

The credentials live in their own file, /etc/postfix/sasl_passwd:

[mail.example.net]:587    alerts@example.com:your-app-specific-password
Bash

Use an app-specific password if the provider offers one, never your account login. Then compile it to a hashed map and lock down the permissions, because that file is a plaintext password until you do:

sudo postmap /etc/postfix/sasl_passwd
sudo chmod 600 /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db
sudo systemctl reload postfix
Bash

One thing to settle while you’re in here, since it’s the only way this setup can hurt anyone but you: keep the box send-only. Postfix should listen on loopback and nothing else, inet_interfaces = loopback-only in main.cf. You want a client that dials out, not a server the internet can dial into. A relay that accepts mail from the outside world is how you wake up as somebody’s spam cannon.

The From address is a trap

Here’s where the SaaS tutorials quietly cheat. When you relay through SendGrid, it rewrites your sender for you. When you relay through a mailbox you own, it won’t. Your provider will only let you send as an address you’re actually allowed to use, and your server’s default sender, root@your-box.localdomain, is not that address. The relay rejects it, and you spend twenty minutes reading 550 errors in the log wondering what you broke.

The fix is to rewrite outbound addresses on their way to the relay. Create /etc/postfix/generic:

root@your-box.localdomain      alerts@example.com
@your-box.localdomain          alerts@example.com
Bash

Point Postfix at it and compile:

# in main.cf
smtp_generic_maps = hash:/etc/postfix/generic
Bash
sudo postmap /etc/postfix/generic && sudo systemctl reload postfix
Bash

Now anything the box sends leaves as alerts@example.com, an address the relay recognizes and is willing to sign. This one line is the difference between “it works on my LAN” and “it works.”

root turns out to be everyone

The point of all this is that a human reads the alert. On a real team that’s several humans, and you’d rather not hard-code a list of addresses into every script and monitor on the box. You also don’t want each recipient to see everyone else’s address every time a disk gets warm.

Linux already has the answer, and it’s older than most of the tools it’s warning you about: /etc/aliases. Everything on a Unix box that has something to say sends it to root. So point root at your people:

# /etc/aliases
root:   you@example.com, teammate@example.com, oncall@example.com
Bash
sudo newaliases
Bash

The nice property here, and the reason I reach for this instead of a BCC rule, is what the recipients see. Each person on that list gets the message addressed to root@your-box. They don’t see each other. You get team-wide delivery without leaking a contact list into every notification, and without fighting Postfix’s BCC handling, which pointedly skips automatic BCC after an alias expands (so the “obvious” always_bcc approach doesn’t even fire here). Add or remove a teammate by editing one line and running newaliases. Every script on the box picks up the change for free, because none of them ever knew the addresses in the first place.

Testing, and the moment it looks broken

Send yourself something:

echo "If you can read this, the relay works." | mail -s "test from $(hostname)" you@example.com
Bash

Then watch the log:

sudo tail -f /var/log/maillog   # or /var/log/mail.log on Debian/Ubuntu
Bash

You’re looking for one word: status=sent. If you see it and the mail arrives, you’re done.

Now the part that fooled me for a minute. I went to check the local mail spool to confirm delivery, and /var/mail was empty. Every mailbox, zero bytes. That looks exactly like failure. It isn’t. It’s the whole point. Because the box relays everything out to a real provider, nothing is ever stored locally. An empty /var/mail on a send-only relay means the mail left the building, which is precisely what you asked it to do. If you want a local paper trail, log a copy from your own scripts. Don’t expect the spool to hold one.

The moment it genuinely worked was seeing a message land in my inbox with To: root@…. Nothing on the box had to know my address, or anyone’s. It just told root, the way Unix has told root for forty years, and root turned out to be all of us.

When the mailbox is down

Providers have bad mornings, and the relay will be unreachable at exactly the wrong moment eventually. Postfix takes this in stride better than you’d expect. A message it can’t hand off doesn’t disappear, it drops into the queue and gets retried on a backoff, so your 3am disk warning arrives at 3:14 instead of never. See what’s waiting with mailq, and if you’ve just fixed the relay and don’t want to sit through the next retry cycle, sudo postqueue -f flushes everything now. The failure mode is late, not lost, which for a notification channel is the right way to fail.

Hooking up the actual senders

With the pipe working, everything that already speaks mail on a Linux box just starts reaching you:

  • Cron honors a MAILTO= line at the top of a crontab, and mails you any job that prints to stderr.
  • Package auto-updates (dnf-automatic, or unattended-upgrades on Debian) will email a report when they patch or when a patch needs a reboot. Point their config at root and you inherit the alias.
  • Disk and RAID health (smartd, mdadm --monitor) send exactly the alert you want to never ignore, straight to the same list.
  • Your own scripts can call mail in three lines and get the same reach, no library, no API key.

None of these need to know a single email address. They send to root; the alias fans it out; the relay makes it deliverable. That’s the entire architecture, and it’s mostly configuration files that predate the cloud.

When it works too well

There’s a second failure waiting on the far side of success, and it’s the one that actually got me. A healthy channel is quiet. A failing disk is not. smartd re-checks on a timer, a crash-looping job restarts every few seconds, and every cycle is glad to send one more identical email. By lunchtime you’ve got four hundred messages that all say the same thing, an inbox rule that quietly files them out of sight, and a channel you no longer trust. The alert worked perfectly. That’s how it stopped working.

The fix is small and worth building once: don’t let the same alert fire more than once per cooldown window. A lockfile and a timestamp are enough.

#!/usr/bin/env bash
# notify <key> <subject> — at most one mail per key per hour, body on stdin
key=$1; subject=$2; cooldown=3600
stamp="/tmp/notify-$key.last"
now=$(date +%s)
last=$(cat "$stamp" 2>/dev/null || echo 0)
if 2; then
  mail -s "$subject" root
  echo "$now" > "$stamp"
fi
Bash

Now a flapping sensor can shout all it likes and you hear it once an hour, until you fix the disk it’s shouting about. The goal was never to send mail. It was to stay worth reading.

Why bother, when a webhook exists

You could POST to a chat webhook instead, and sometimes that’s the right call. But mail has a property that’s easy to undervalue: everything on a Linux server already knows how to send it. The kernel, cron, the package manager, the RAID driver, a shell one-liner. You don’t instrument anything. You wire up one relay, forward root, and a machine that had no voice starts telling you what it needs, through infrastructure you already run and already trust.

Placeholder domains and addresses throughout (example.com, mail.example.net). Swap in your own mailbox, your own relay, your own people.

Print 🖨 eBook 📱
  1. now – last >= cooldown [↩]
  2. now – last >= cooldown [↩]

Posted

in

by

Xiaomeng Wang

Tags:

, , , ,

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

Table of Contents

← 🧭