Building a 3-Server Mail Platform with Rspamd, Proxmox Mail Gateway, and MailLab (Roundcube, DKIM, Fail2Ban)

Linux

Tested on: Ubuntu 22.04/24.04, Proxmox Mail Gateway 8.2

What You Will Build

  • Inbound flow: Gmail (or the Internet) → Rspamd → Proxmox Mail Gateway → MailLab → Maildir
  • Outbound flow: Roundcube → MailLab → Proxmox (port 26) → Gmail/Internet
  • Webmail: Roundcube (IMAP 993 + SMTP 587 TLS)
  • Security: DKIM, Fail2Ban, TLS
  • Scalability: Easy to add users (per-user Maildir)

Server Inventory

Server IP Hostname Role
Rspamd 91.98.95.46 rspamd.mailinglab.online MX + inbound filter (Postfix + Rspamd)
Proxmox 37.27.249.93 mailscanner.mailinglab.online Filter + relay (listens on port 25/26)
MailLab 23.88.42.117 mail.mailinglab.online Delivery (Postfix + Dovecot) + Roundcube

Prerequisites

  • Three clean Linux VMs with the IPs/hostnames above.
  • DNS control in Namecheap (or similar).
  • Firewall open for: 25/TCP (SMTP), 587/TCP (Submission), 993/TCP (IMAPS), and web ports for Roundcube/PMG UI.

Step 1 — DNS (Namecheap)

Create these records:

A mailinglab.online. 91.98.95.46 
A mail.mailinglab.online. 23.88.42.117 
A rspamd.mailinglab.online. 91.98.95.46 
MX @ 10 rspamd.mailinglab.online.

Why: Mail for @mailinglab.online first lands on Rspamd (MX). Webmail and IMAP/SMTP submission will go to mail.mailinglab.online.

Step 2 — Rspamd Server (91.98.95.46)

Install base packages

sudo apt update sudo apt install postfix rspamd clamav clamav-daemon spamassassin -y

Why: Postfix receives mail on port 25 (MX). Rspamd tags/spam-scores, ClamAV scans attachments, SpamAssassin is optional/complimentary.

Postfix: relay inbound mail to Proxmox

Edit /etc/postfix/main.cf:

myhostname = rspamd.mailinglab.online 
mydomain = mailinglab.online
myorigin = $mydomain 
inet_interfaces = 
all mynetworks = 127.0.0.0/8 91.98.95.46/32 37.27.249.93/32 
relay_domains = mailinglab.online 
transport_maps = hash:/etc/postfix/transport
  • myhostname: must match the server’s FQDN (affects EHLO banner).
  • mydomain/myorigin: base domain; used by Postfix for internal addressing.
  • inet_interfaces: listen on all interfaces.
  • mynetworks: trusted networks (localhost + your Rspamd + Proxmox IPs). Keep this tight.
  • relay_domains: this Postfix will accept mail only for these domains.
  • transport_maps: tells Postfix where to send mail for listed domains (our Proxmox).

Create /etc/postfix/transport:

mailinglab.online smtp:[37.27.249.93]:25
  • Square brackets prevent MX lookup; Postfix will connect directly to that host:port.

Build the map and restart:

sudo postmap /etc/postfix/transport sudo systemctl restart postfix

Rspamd: add helpful headers

Edit /etc/rspamd/local.d/milter_headers.conf:

use = ["x-spam", "authentication-results"];
  • x-spam: injects spam score headers (useful for downstream decisions).
  • authentication-results: writes SPF/DKIM/DMARC verdicts when available.
sudo systemctl restart rspamd

Step 3 — Proxmox Mail Gateway (PMG) (37.27.249.93)

Install PMG 8.2+

Download ISO from Proxmox, install with ≥4 GB RAM. Access the UI at https://37.27.249.93:8006.

System → DNS

  • Primary DNS: 8.8.8.8
  • Secondary DNS: 8.8.4.4
  • Search Domain: mailinglab.online

Mail Proxy → Networks

Network Allow as sender Allow as relay
91.98.95.46/32 Yes Yes
23.88.42.117/32 Yes Yes

Why: Rspamd (inbound) and MailLab (outbound via port 26) are trusted peers.

Mail Proxy → Relay Domains

  • Domain: mailinglab.online
  • Transport: mail.mailinglab.online (or 23.88.42.117)
  • Port: 25
  • Use MX: No (go directly to MailLab)

Mail Proxy → Transports

  • Domain: mailinglab.online
  • Host: 23.88.42.117
  • Port: 25
  • Protocol: SMTP, Use MX: No

Mail Proxy → DKIM Keys

  1. Generate DKIM key for mailinglab.online.
  2. Copy the suggested TXT record to Namecheap (see Step 7).

Apply Configuration

Optional: force IPv4 in Postfix inside PMG if you have IPv6 quirks.

# /etc/postfix/main.cf inet_protocols = ipv4 
sudo systemctl restart postfix pmg-smtp-filter

Step 4 — MailLab (23.88.42.117)

Install core services

sudo apt update sudo apt install postfix dovecot-imapd dovecot-pop3d roundcube apache2 openssl fail2ban -y

Postfix: Submission (587) + Relay to PMG (port 26)

/etc/postfix/main.cf

myhostname = mail.mailinglab.online 
mydomain = mailinglab.online 
myorigin = $mydomain 
inet_interfaces = all 
mydestination = $myhostname, localhost.$mydomain, localhost 
relayhost = [37.27.249.93]:26 
home_mailbox = Maildir/
  • relayhost: all outbound mail goes to PMG on port 26 (so PMG can filter/queue).
  • home_mailbox: deliver locally per user to ~/Maildir/ (IMAP-friendly).

/etc/postfix/master.cf — enable submission (587) with TLS + SASL via Dovecot:

submission inet n - y - - smtpd 
-o syslog_name=postfix/submission 
-o smtpd_tls_security_level=encrypt 
-o smtpd_sasl_auth_enable=yes 
-o smtpd_sasl_type=dovecot 
-o smtpd_sasl_path=private/auth 
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
  • smtpd_tls_security_level=encrypt: forces STARTTLS on 587.
  • smtpd_sasl_*: hands off authentication to Dovecot’s UNIX socket.
  • permit_sasl_authenticated,reject: only authenticated users can send on 587.
sudo systemctl restart postfix

Dovecot: IMAP (993) + SASL for Postfix

/etc/dovecot/conf.d/10-auth.conf

disable_plaintext_auth = no 
auth_mechanisms = plain login 
auth_username_format = %Ln !include auth-system.conf.ext
  • disable_plaintext_auth = no: allow PLAIN/LOGIN because we will require TLS elsewhere.
  • %Ln: usernames like alice (lowercased local part).
  • auth-system.conf.ext: use system users (/etc/passwd) for simplicity.

/etc/dovecot/conf.d/10-master.conf — expose auth socket for Postfix:

service auth { unix_listener /var/spool/postfix/private/auth { mode = 0660 user = postfix group = postfix } }

/etc/dovecot/conf.d/10-mail.conf

mail_location = maildir:~/Maildir

/etc/dovecot/conf.d/10-ssl.conf

ssl = required 
ssl_cert = </etc/ssl/certs/mail.mailinglab.online.crt 
ssl_key = </etc/ssl/private/mail.mailinglab.online.key 
sudo systemctl restart dovecot

Let’s Encrypt TLS for Apache (Roundcube)

sudo apt install certbot python3-certbot-apache -y sudo certbot --apache -d mail.mailinglab.online

Roundcube: connect via IMAPS 993 + SMTP 587

/etc/roundcube/config.inc.php

$config['db_dsnw'] = 'mysql://roundcube:heslo@localhost/roundcube'; 
$config['default_host'] = 'tls://mail.mailinglab.online'; 
$config['default_port'] = 993; 
$config['smtp_server'] = 'tls://mail.mailinglab.online'; 
$config['smtp_port'] = 587; 
$config['smtp_user'] = '%u'; 
$config['smtp_pass'] = '%p'; 
$config['des_key'] = 'random24charstringhere123456';
  • default_host/port: IMAPS endpoint for inbox.
  • smtp_server/port: Submission endpoint for sending (587, STARTTLS).
  • %u/%p: pass through the user’s login and password.
  • des_key: set a long random string; used to encrypt sensitive prefs.

Create Roundcube DB and user:

CREATE DATABASE roundcube; 
CREATE USER 'roundcube'@'localhost' IDENTIFIED BY 'heslo'; 
GRANT ALL ON roundcube.* TO 'roundcube'@'localhost'; 
FLUSH PRIVILEGES;

Load Roundcube schema and restart Apache:

sudo mysql -u roundcube -p roundcube < /usr/share/roundcube/SQL/mysql.initial.sql 
sudo systemctl restart apache2

Fail2Ban: basic mail jails (all three servers)

sudo apt install fail2ban -y sudo tee /etc/fail2ban/jail.local >/dev/null <<'EOF' [postfix] enabled = true [postfix-sasl] enabled = true [dovecot] enabled = true EOF 
sudo systemctl restart fail2ban

Why: blocks brute-force attempts against SMTP auth and IMAP logins.

Step 5 — Add a New User (MailLab)

sudo adduser --home /home/alice --shell /bin/bash alice 
sudo mkdir -p /home/alice/Maildir/{cur,new,tmp} 
sudo chown -R alice:alice /home/alice/Maildir 
sudo chmod -R 700 /home/alice/Maildir

Tip: Dovecot/Postfix will also auto-create Maildir after first delivery, but pre-creating helps with early testing.

Step 6 — Testing

Inbound (deliver to Maildir)

echo "Test inbound" | swaks --to alice@mailinglab.online --from test@gmail.com --server localhost 
# Then on MailLab: 
ls /home/alice/Maildir/new/

Outbound (Roundcube)

Log in to Roundcube at http://23.88.42.117/roundcube as alice@mailinglab.online and send to an external mailbox (e.g., Gmail). Verify delivery headers show PMG in the path.

Step 7 — DKIM DNS (Namecheap)

From PMG’s DKIM key screen, publish the TXT record. It usually looks like:

default._domainkey.mailinglab.online. TXT "v=DKIM1; k=rsa; p=MIIBIjANBgk..."

After publishing, check with an external DKIM tester or by sending to Gmail and inspecting the Authentication-Results header.

Step 8 — Backups (Optional)

On MailLab, back up all user Maildir trees (adjust host/path/keys to your environment):

sudo rsync -avz /home/*/Maildir/ backup@backup:/backup/mail/

Operational Notes & Hardening

  • TLS everywhere: Ensure valid certs for your domain. Enforce TLS on submission (587) and IMAPS (993).
  • SPF/DMARC: Add an SPF TXT for your domain and consider a DMARC policy once DKIM/SPF pass consistently.
  • Firewall: Restrict 25/TCP on Rspamd/PMG to the Internet as needed; 26/TCP only between MailLab ↔ PMG.
  • Logs to watch:
    • Postfix: /var/log/mail.log (Ubuntu) or journalctl -u postfix
    • Dovecot: /var/log/dovecot.log (or journal)
    • Rspamd: /var/log/rspamd/rspamd.log
    • PMG: UI → Tracking Center and /var/log/ services
    • Fail2Ban: /var/log/fail2ban.log
  • User provisioning: For many users, consider virtual mailboxes (Dovecot SQL) instead of system users.
  • ClamAV tuning: Enable on PMG and/or Rspamd host; consider on-access scanning for uploads.

Quick Port Reference

  • 25/TCP — SMTP inbound (Rspamd, PMG, MailLab internal)
  • 26/TCP — Custom relay from MailLab → PMG (outbound)
  • 587/TCP — SMTP Submission (STARTTLS required)
  • 993/TCP — IMAPS
  • 80/443 — Roundcube (Apache), PMG UI (8006/TCP for PMG admin)

Troubleshooting Checklist

  • MX correctness: dig mx mailinglab.online should point to rspamd.mailinglab.online.
  • Postfix relays: Rspamd → PMG via transport_maps; PMG → MailLab via Relay Domains/Transports.
  • Auth failures: Check Dovecot SASL (private/auth socket permissions) and that submission is enabled.
  • DKIM fails: Confirm TXT selector/name, remove quotes wrapping the entire record incorrectly, wait for DNS TTL.
  • Fail2Ban locks you out: Temporarily stop the service: sudo systemctl stop fail2ban, adjust jail regexes/ignoreip.

Conclusion

You now have a production-style mail stack with clear separation of duties: Rspamd for initial MX + headers, Proxmox for policy/AV/antispam and DKIM, and MailLab for mailbox storage + client access via Roundcube/IMAP/Submission. Extend with SPF/DMARC, better auth policies, and virtual mailbox backends as your user count grows.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.