The Hail Mary Cloud was a widely distributed, low intensity password guessing botnet that targeted Secure Shell (ssh) servers on the public Internet.
The first activity may have been as early as 2007, but our first recorded data start in late 2008. Links to full data and extracts are included in this article.
We present the basic behavior and algorithms, and point to possible policies for staying safe(r) from similar present or future attacks.
But first, a few words about the devil we knew before the incidents that form the core of the narrative.
The Traditional SSH Bruteforce Attack
If you run an Internet-facing SSH service, you have seen something like this in your logs:
Sep 26 03:12:34 skapet sshd: Failed password for root from 22.214.171.124 port 40992 ssh2 Sep 26 03:12:34 skapet sshd: Failed password for root from 126.96.36.199 port 40992 ssh2 Sep 26 03:12:35 skapet sshd: Received disconnect from 188.8.131.52: 11: Bye Bye Sep 26 03:12:44 skapet sshd: Invalid user admin from 184.108.40.206 Sep 26 03:12:44 skapet sshd: input_userauth_request: invalid user admin Sep 26 03:12:44 skapet sshd: Failed password for invalid user admin from 220.127.116.11 port 41484 ssh2 Sep 26 03:12:44 skapet sshd: Failed password for invalid user admin from 18.104.22.168 port 41484 ssh2 Sep 26 03:12:45 skapet sshd: Connection closed by 22.214.171.124 Sep 26 03:13:10 skapet sshd: Failed password for root from 126.96.36.199 port 43344 ssh2
This is the classic, rapid-fire type of bruteforce attack, with rapid-fire login attempts from one source. (And yes, skapet is the Internet-facing host on my home network.)
The Likely Business Plan
These attempts are often preceded by a port scan, but in other cases it appears that the miscreants are just blasting away at random. In my experience, with the gateway usually at the lowest-numbered address, the activity usually turns up first there, before trying higher-numbered hosts. I'm not really in a mind to offer help or advice to the people running those scripts, but it might be possible to scan the internet from 255.255.255.255 downwards next time. Anyway, looking at the log excerpts, miscreants' likely plan is
- Try for likely user names, hope for guessable password, keep guessing until successful.
Traditional Anti-Bruteforce Rules
Rapid-fire bruteforce attacks are easy to head off. I tend to use OpenBSD on internet facing hosts, so first we present the technique as it has been available in OpenBSD since version 3.5 (released in 2005), where state tracking options are used to set limits we later act on:
In your /etc/pf.conf, you add a table to store addresses, block access for all traffic coming from members of that table, and finally amend your typical pass rule with some state tracking options. The result looks something like this:
table <bruteforce> persist block quick from <bruteforce> pass inet proto tcp to $int_if:network port $tcp_services \ keep state (max-src-conn 100, max-src-conn-rate 15/5, \ overload <bruteforce> flush global)
Here, max-src-conn is the maximum number of concurrent connections allowed from one host
max-src-conn-rate is the maximum allowed rate of new connections, here 15 connections per 5 seconds.
overload <bruteforce> means that any hosts that exceed either of these limits are have their adress added to this table
and, just for good measure, flush global means that for host that is added to our overload table, we kill all existing connections too.
Traditional Anti-Bruteforce Rules, Linux Style
For those not yet converted to the fine OpenBSD toolset (available in FreeBSD and other BSDs too, with only minor if any variations in details for this particular context), the Linux equivalent would be something like
sudo iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH sudo iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 5 \ --hitcount 15 --rttl --name SSH -j DROP
But be warned: this will still be minus the maximum number of connections limit, plus the usual iptables warts. And you'd need a separate set of commands for ip6tables.
It's likely something similar is doable with other tools and products too, including possibly some proprietary ones. I've made something of an effort to limit my exposure to the non-free tools, so I can't offer you any more detail. To find out what your present product can do, please dive into the documentation for whichever product you are using. Or come back for some further OpenBSD goodness.
But as you can see, for all practical purposes the rapid-fire bruteforce or floods problem has been solved with trivial configuration tweaks.
But then something happened.
What's That? Something New!
On November 19th, 2008 (or shortly thereafter), I noticed this in my authentication logs:
Nov 19 15:04:22 rosalita sshd: error: PAM: authentication error for illegal user alias from s514.nxs.nl Nov 19 15:07:32 rosalita sshd: error: PAM: authentication error for illegal user alias from c90678d3.static.spo.virtua.com.br Nov 19 15:10:20 rosalita sshd: error: PAM: authentication error for illegal user alias from 207-47-162-126.prna.static.sasknet.sk.ca Nov 19 15:13:46 rosalita sshd: error: PAM: authentication error for illegal user alias from 125-236-218-109.adsl.xtra.co.nz Nov 19 15:16:29 rosalita sshd: error: PAM: authentication error for illegal user alias from 188.8.131.52 Nov 19 15:19:12 rosalita sshd: error: PAM: authentication error for illegal user alias from 184.108.40.206 Nov 19 15:22:29 rosalita sshd: error: PAM: authentication error for illegal user alias from 220.127.116.11 Nov 19 15:25:14 rosalita sshd: error: PAM: authentication error for illegal user alias from 130.red-80-37-213.staticip.rima-tde.net Nov 19 15:28:23 rosalita sshd: error: PAM: authentication error for illegal user alias from 70-46-140-187.orl.fdn.com Nov 19 15:31:17 rosalita sshd: error: PAM: authentication error for illegal user alias from gate-dialog-simet.jgora.dialog.net.pl Nov 19 15:34:18 rosalita sshd: error: PAM: authentication error for illegal user alias from 18.104.22.168 Nov 19 15:37:23 rosalita sshd: error: PAM: authentication error for illegal user alias from 22.214.171.124 Nov 19 15:40:20 rosalita sshd: error: PAM: authentication error for illegal user alias from 70-46-140-187.orl.fdn.com Nov 19 15:43:39 rosalita sshd: error: PAM: authentication error for illegal user alias from 126.96.36.199 Nov 19 15:46:41 rosalita sshd: error: PAM: authentication error for illegal user amanda from 188.8.131.52 Nov 19 15:49:31 rosalita sshd: error: PAM: authentication error for illegal user amanda from host116-164.dissent.birch.net Nov 19 15:55:47 rosalita sshd: error: PAM: authentication error for illegal user amanda from robert71.lnk.telstra.net Nov 19 15:59:08 rosalita sshd: error: PAM: authentication error for illegal user amanda from static-71-166-159-177.washdc.east.verizon.net
... and so on. The alphabetic progression of user names went on and on.
The pattern seemed to be that several hosts, in widely different networks, try to access our system as the same user, up to minutes apart. When any one host comes back it's more likely than not several user names later. The full sequence (it stopped December 30th), is available here.
Take a few minutes to browse the log data if you like. It's worth noting that rosalita was a server that had a limited set of functions for a limited set of users, and basically no other users than myself ever logged in there via SSH, even if they for various reason had the option open to them. So in contrast to busier sites where sequences like this might have drowned in the noise, here it really stood out. And I suppose after looking at the data, you can understand my initial reaction.
The Initial Reaction
My initial reaction was pure disbelief.
For the first few days I tried tweaking PF rules, playing with the attempts/second values and scratching my head, going, "How do I make this match?"
I spent way too much time on that, and the short version of the answer to that question is, you can't. With the simple and in fact quite elegant state tracking options, you will soon hit limits (especially time limits) that interfere with normal use, and you end up blocking legitimate traffic.
So I gave up on prevention (which really only would have rid me of a bit of noise in my authentication logs), and I started analyzing the data instead, trying to eyeball patterns that would explain what I was seeing. After a while it dawned on me that this could very well be a coordinated effort, using a widely distributed set of compromised hosts.
So there was a bit of reason in there after all. Maybe even a business plan or model. Next, I started analyzing my data, and came up with -
Bruteforcer Business Plan, Distributed Version
The Executive Summary would run something like this: Have more hosts take turns, round robin-ish, at long enough intervals to stay under the radar, guessing for weak passwords.
The plan is much like before, but now we have more host on the attacking side, so
- Pick a host from our pool, assign a user name and password (picked from a list, dictionary or pool)
- For each host,
- Try logging in to the chosen target with the assigned user name and password
- If successful, report back to base (we theorize); else wait for instructions (again we speculate)
- Go to 1).
- For each success at 2.2), PROFIT!
You're The Target
Let's recap, and take step back. What have we learned?
To my mind at least, it all boils down to the basics
- Your Unix computer (Linux, OpenBSD, FreeBSD or other) is a desirable, powerful thing.
- If your password is weak, you will be 0WN3D, sooner rather than later.
- There's a whole fleet out there, and they're coordinated.
Initial Public Reaction
On December 2, 2008, I published the first blog post in what would become a longish sequence, A low intensity, distributed bruteforce attempt, where I summarized my findings. It's slightly more wordy than this piece, but if I've piqued your interest so far, please go ahead and read. And as to a little public attention, I got my wish. The post ended up
- more disbelief, (see slashdot and other comments) but also
- confirmation, via comments and email, that others were seeing the same thing, and that the first occurrences may have been seen up to a year earlier (November-ish 2007).
The slow bruteforcers were not getting in, so I just went on collecting data. I estimated they'd be going on well past new year's if they were going to reach the end of the alphabet.
On December 30th, 2008, The Attempts Stopped
The attempts came to an end, conveniently while I was away on vacation. The last entries were:
Dec 30 11:03:08 rosalita sshd: error: PAM: authentication error for illegal user sophia from 184.108.40.206 Dec 30 11:05:08 filehut sshd: error: PAM: authentication error for illegal user sophia from 220.127.116.11 Dec 30 11:06:35 rosalita sshd: error: PAM: authentication error for illegal user sophia from static-98-119-110-139.lsanca.dsl-w.verizon.net Dec 30 11:09:03 filehut sshd: error: PAM: authentication error for illegal user sophia from static-98-119-110-139.lsanca.dsl-w.verizon.net
That is, not even completing a full alphabetic cycle.
By then they had made 29916 attempts, all failed. You can find the full listing at here).
Trying 6100 user IDs (list by frequency here). More than likely you can guess the top one without even looking.
From a total of 1193 different hosts (list by frequency here).
As I said earlier, there were no successful penetrations. Zero.
The slashdot story brought comments and feedback, with some observations from other sites. Not a lot of data, but enough that the patterns we had observed were confirmed. The attempts were all password authentication attempts, no other authentication methods attempted.
For the most part the extended incident consisted of attempts on an alphabetic sequence of 'likely' user names, but all sites also saw at least one long run of root only attempts. This pattern was to repeat itself, and also show up in data from other sources.
There would be anything from seconds to minutes between attempts, but attempts from any single host would come at much longer intervals.
First Round Observations, Early Conclusions
Summing up what we had so far, here are a few observations and attempts at early conclusions.
At the site where I had registered the distributed attempts, the Internet-reachable machines all ran either OpenBSD or FreeBSD. Only two FreeBSD boxes were contacted.
The attackers were hungry for root, so having PermitRootLogin no in our
We hadn't forced our users to keys only, but a bit of luck and John the Ripper (/usr/ports/security/john) saved our behinds.
The number of attempts per user name had decreased over time (as illustrated by this graph), so we speculated in the second article Into a new year, slowly pounding the gates (on slashdot as The Slow Bruteforce Botnet(s) May Be Learning) that success or not was measured at a command and control site, with resources allocated accordingly.
With the sequence not completed, we thought they'd given up. After all, the odds against succeeding seemed monumental.
After all, a couple of slashdotted blog posts couldn't have hurt, could they?
But Of Course They Came Back
As luck would have it, whoever was out there had not totally admitted defeat just yet. In the early hours CET, April 7th, 2009, the slow brutes showed up again:
Apr 7 05:02:07 rosalita sshd: error: PAM: authentication error for root from ruth.globalcon.net Apr 7 05:02:15 rosalita sshd: error: PAM: authentication error for root from ip-206-83-192-201.sterlingnetwork.net Apr 7 05:02:54 rosalita sshd: error: PAM: authentication error for root from cyscorpions.com Apr 7 05:02:59 rosalita sshd: error: PAM: authentication error for root from smtp.bancomorada.com.br Apr 7 05:03:10 rosalita sshd: error: PAM: authentication error for root from 18.104.22.168 Apr 7 05:03:25 rosalita sshd: error: PAM: authentication error for root from 22.214.171.124 Apr 7 05:03:52 rosalita sshd: error: PAM: authentication error for root from rainha.florianonet.com.br Apr 7 05:04:00 rosalita sshd: error: PAM: authentication error for root from 126.96.36.199 Apr 7 05:04:34 rosalita sshd: error: PAM: authentication error for root from s1.serverhex.com Apr 7 05:04:38 rosalita sshd: error: PAM: authentication error for root from mail.pitnet.com.br
Starting with 2318 attempts at root before moving on to admin and proceeding with the alphabetic sequence. The incident played out pretty much like the previous one, only this time I was sure I had managed to capture all relevant data before my logs were rotated out of existence.
The data is available in the following forms: Full log here, one line per attempt here, users by frequency here, hosts by frequency here.
I couldn't resist kicking up some more publicity, and indeed we got another slashdot storm out of the article The slow brute zombies are back, on slashdot as The Low Intensity Brute-Force Zombies Are Back.
And shortly afterwards, we learned something new -
Introducing dt_ssh5, Linux /tmp Resident
Of course there was a piece of malware involved.
A Linux binary called dt_ssh5 did the grunt work.
The dt_ssh5 file was found installed in /tmp on affected systems. The reason our perpetrators chose to target that directory is likely because the /tmp directory tends to be world readable and world writeable.
Again, this points us to the three basic lessons:
- Stay away from guessable passwords
Watch for weird files (stuff you didn't put there yourself) anywhere in your file system, even in
- Internalize the fact that PermitRootLogin yes is a bad idea.
dt_ssh5: Basic Algorithm
The discovery of dt_ssh5 made for a more complete picture. A rough algorithm suggested itself:
- Pick a new host from our pool, assign a user name and password
For each host,
- Try user name and password
- drop the dt_ssh5 binary in /tmp; start it
- report back to base
Go to 1.
- For each success at 2.2, PROFIT!
I never got myself a copy, so the actual mechanism for communicating back to base remains unclear.
The Waves We Saw, 2008 - 2012
We saw eight sequences (complete list of articles in the References section at the end),
|From - To||Attempts||User IDs||Hosts||Successful Logins|
|2008-11-19 15:04:22 - 2008-12-30 11:09:03||29916||6100||1193||0|
|2009-04-07 03:56:25 - 2009-04-12 21:01:37||12641||249||1104||0|
|2009-09-30 21:15:36 - 2009-10-15 13:42:07||9998||1||1071||0|
|2009-10-28 23:58:35 - 2010-01-22 09:56:24||44513||8110||4158||0|
|2010-06-17 01:55:34 - 2010-08-11 13:23:01||23014||3887||5568||0|
|2011-10-23 04:13:00 - 2011-10-29 05:40:07||4773||944||338||0|
|2011-11-03 20:56:18 - 2011-11-26 17:42:19||4907||2474||252||0|
|2012-04-01 12:33:04 - 2012-04-06 14:52:11||4757||1081||23||0|
The 2009-09-30 sequence was notable for trying only root, the 2012-04-01 sequence for being the first to attempt access to OpenBSD hosts.
We may have missed earlier sequences, early reports place the first similar attempts as far back as 2007.
For A While, The Botnet Grew
From our point of view, the swarm stayed away for a while and came back stronger, for a couple of iterations, possibly after tweaking their code in the meantime. Or rather, the gaps in our data represent times when it focused elsewhere.
Clearly, not everybody was listening to online rants about guessable passwords.
For a while, the distributed approach appeared to be working.
It was (of course) during a growth period I coined the phrase "The Hail Mary Cloud".
Instantly, a myriad of "Hail Mary" experts joined the insta-punditry on slashdot and elsewhere.
It Went Away Or Dwindled
Between August 2010 and October 2010, things either started going badly for The Hail Mary Cloud, or possibly they focused elsewhere.
I went on collecting data.
There wasn't much to write about, except possibly that the botnet's command and control was redistributing effort based on past success. Aiming at crackable hosts elsewhere.
And Resurfaced In China?
Our last sighting so far was in April 2012. The data is preserved here.
This was the first time we saw Hail Mary Cloud style attempts at accessing OpenBSD systems.
The majority of attempts were spaced at at least 10 seconds apart, and until I revisited the data recently, I thought only two hosts in China were involved.
In fact, 23 hosts made a total of 4757 attempts at 1081 user IDs, netting 0 successful logins.
The new frequency data I thought interesting enough to write about, so I wrote up If We Go One Attempt Every Ten Seconds, We're Under The Radar, and netted another slashdotting. I took another look at the data later and slightly amended the conclusions, the article has been corrected with proper data extracted.
Then What To Do?
The question anybody reading this far will be asking is, what should we do in order to avoid compromise by the password guessing swarms? To my mind, it all boils down to common sense systems administration:
Mind your logs. You can read them yourself, or train a robot to. I use
Keep your system up to date. If not OpenBSD, check openssh.org for the latest version, check what your system has and badger the maintainer if it's outdated.
And of course, configure your applications such as
sshd_config: 'PermitRootLogin no' and a few other items
These two settings in your
PermitRootLogin no PasswordAuthentication no
Make your users generate keys, add the *.pub to their ~/.ssh/authorized_keys files.
For a bit of background, Michael W. Lucas: SSH Mastery (Tilted Windmill Press 2013) is a recent and very readable guide to configuring your SSH (server and clients) sensibly. It's compact and affordable, too, and you can even buy it via the OpenBSD site.
Keep Them Out, Keep Them Guessing
At this point, most geeks would wax lyrical about the relative strengths of different encryption schemes and algorithms.
Being a simpler mind, I prefer a different metric for how good your scheme is, or effectivness of obfuscation (also see entropy):
How many bytes does a would-be intruder have to get exactly right?I've summed up the answer to that question in this table:
|Authentication method||Number of bytes|
|Password||Password length (varies, how long is yours?)|
|Alternate Port||Port number (2 bytes, it's a 16 bit value, remember)|
|Port Knocking||Number of ports in sequence * 2 (still a 16 bit value)|
|Single Packet Authentication||2 bytes (the port) plus Max 1440 (IPv4/Ethernet) or 1220 (IPv6/Ethernet)|
|Key Only||Number of bytes in key (depending on key strength, up to several kB)|
You can of course combine several methods (with endless potential for annoying your users), or use two factor authentication (OpenSSH supports several schemes).
Keys. You've Got To have Keys!
By far the most effective measure is to go keys only for your ssh logins. In your sshd_config, add or uncomment
Restart you sshd, and have all users generate keys, like this:
$ssh-keygen -C "email@example.com"
There are other options to play with, see ssh-keygen(1) for inspiration.
Then add the *.pub to their ~/.ssh/authorized_keys files.
And I'll let you in on a dirty little secret: you can even match on interface in your sshd config for things like these
Why Not Use Port Knocking?
Whenever I mention the Hail Mary Cloud online, two suggestions always turn up: The iptables example I mentioned earlier (or link to the relevant slide), and "Why not use port knocking?". Well, consider this:
Port knocking usually means having all ports closed, but with a daemon reading your firewalls logs for a predetermined sequence of ports. Knock on the correct ports in sequence, your're in.
Another dirty little secret: It's possible to implement port knocking with only the tools in an OpenBSD base system. No, I won't tell you how.
Executive Summary: Don't let this keep you from keeping your system up to date.
To my mind port knocking gives you:
- Added complexity or, one more thing that will go wrong. If the daemon dies, you've bricked your system.
- An additional password that's hard to change. There's nothing magical about TCP/UDP ports. It's a 16 bit number, and in our context, it's just another alphabet. The swarm will keep guessing. And it's likely the knock sequence (aka password) is the same for all users.
- You won't recognize an attack until it succeeds, if even then. Guessing attempts will be indistinguishable from random noise (try a raw tcpdump of any internet-facing interface to see the white noise you mostly block drop anyway), so you will have no early warning.
If you want a longer form or those arguments, my November 4, 2012 rant Why Not Use Port Knocking? was my take (with some inaccuracies, but you'll live).
There's No Safety In High Ports Anymore
Another favorite suggestion is to set your sshd to listen on some alternate port instead of the default port 22/TCP.
People who did so have had a few years of quiet logs, but recent reports show that whoever is out there have the resources to scan alternate ports too.
Once again, don't let running your
Of course I've ranted about this too, in February 2013, There's No Protection In High Ports Anymore. If Indeed There Ever Was. (which earned me another slashdotting).
Reports with logs trickle in from time to time of such activity at alternate ports, but of course on any site with a default deny packet filtering policy will not see any traces of such scans unless you go looking specifically at the mass of traffic that gets dropped at the perimeter.
Final thoughts, for now
Microsoftish instapundits were quick to assert that
They're wrong. OpenSSH (which is what essentially everyone uses) is maintained as an integral part of the OpenBSD project, and as such is a very thoroughly audited mass of code. And it keeps improving with every release.
I consider the Hail Mary Cloud an example of distributed, parallel problem solving, conceptually much like SETI@Home but with different logic and of course a more sinister intent.
Computing power is cheap now, getting cheaper, and even more so when you can leverage other people's spare cycles.
The huge swarm of attackers concept is as I understand it being re-used in the recent WordPress attacks. We should be prepared for swarm attacks on other applications as soon as they reach a critical mass of users.
There may not be a bullseye on your back yet (have you looked lately?), but you are an attractive target.
Fortunately, sane system administration practices will go a long way towards thwarting intrusion attempts, as in
- keep your system up to date,
- allow only what's necessary for the task at hand and
- keep watching your logs for weirdness.
UPDATE 2013-11-21: A recent ACM Conference on Computer and Communication Security paper, "Detecting stealthy, distributed SSH brute-forcing," penned by Mobin Javed and Vern Paxson, references a large subset of the data and offers some real analysis, including correlation with data from other sites (Spoiler alert: in some waves, almost total overlap of participating machines). One interesting point from the paper is that apparently attack matching our profile were seen at the Lawrence Berkeley National Laboratory as early as 2005.
And in other news, it appears that GitHub has been subject to an attack that matches the characteristics we have described. A number of accounts with weak passwords were cracked. Investigations appears to be still ongoing. Fortunately, GitHub appear to have started offering other authentication methods.
UPDATE 2014-09-28: Since early July 2014, we have been seeing similar activity aimed at our POP3 service, with usernames taken almost exclusively from our spamtrap list. The article Password Gropers Take the Spamtrap Bait has all the details and log data as well as references to the spamtrap list.
UPDATE 2014-12-10: My Passwords14 presentation, Distributed, Stealthy Brute Force Password Guessing Attempts - Slicing and Dicing Data from Recent Incidents has some further data as well as further slicing and dicing of the earlier data (with slightly different results.
The slides for the talk this article is based on live at http://home.nuug.no/~peter/hailmary2013/, with a zipped version including all data at http://home.nuug.no/~peter/hailmary2013.zip (approx. 26MB) for your convenience.
Mobin Javed and Vern Paxson, "Detecting stealthy, distributed SSH brute-forcing," ACM International Conference on Computer and Communication Security (CCS), November 2013.
The blog posts (field notes) of the various incidents, data links within:
Peter N. M. Hansteen, (2008-12-02) A low intensity, distributed bruteforce attempt (slashdotted)
Peter N. M. Hansteen, (2008-12-06) A Small Update About The Slow Brutes
Peter N. M. Hansteen, (2008-12-21) Into a new year, slowly pounding the gates (slashdotted)
Peter N. M. Hansteen, (2009-01-22) The slow brutes, a final roundup
Peter N. M. Hansteen, (2009-04-12) The slow brute zombies are back (slashdotted)
Peter N. M. Hansteen, (2009-10-04) A Third Time, Uncharmed (slashdotted)
Peter N. M. Hansteen, (2009-11-15) Rickrolled? Get Ready for the Hail Mary Cloud! (slashdotted)
Peter N. M. Hansteen, (2011-10-23) You're Doing It Wrong, Or, The Return Of The Son Of The Hail Mary Cloud
Peter N. M. Hansteen, (2011-10-29) You're Doing It Wrong, Returning Scoundrels
Peter N. M. Hansteen, (2012-04-06) If We Go One Attempt Every Ten Seconds, We're Under The Radar (slashdotted)
Peter N. M. Hansteen, (2012-04-11) Why Not Use Port Knocking?
Peter N. M. Hansteen, (2013-02-16) There's No Protection In High Ports Anymore. If Indeed There Ever Was. (slashdotted)
Other Useful Texts
Marcus Ranum: The Six Dumbest Ideas in Computer Security, September 1, 2005
Michael W. Lucas: SSH Mastery, Tilted Windmill Press 2013 (order direct from the OpenBSD bookstore here)
Michael W. Lucas: Absolute OpenBSD, 2nd edition No Starch Press 2013 (order direct from the OpenBSD bookstore here)
Peter N. M. Hansteen, The Book of PF, 2nd edition, No Starch Press 2010 (order direct from the OpenBSD bookstore here), also the online PF tutorial it grew out of, several formats http://home.nuug.no/~peter/pf/, more extensive slides at http://home.nuug.no/~peter/pf/newest/
OpenBSDs web http://www.openbsd.org/ -- lots of useful information.
If you enjoyed this: Support OpenBSD!
If you have enjoyed reading this, please buy OpenBSD CDs and other items, and/or donate!
Useful links for this are:
OpenBSD.org Orders Page: http://www.openbsd.org/orders.html
OpenBSD Donations Page: http://www.openbsd.org/donations.html.
OpenBSD Hardware Wanted Page: http://www.openbsd.org/want.html.
Remember: Free software takes real work and real money to develop and maintain.
If you want to support me, buy the book! (if you want to give the OpenBSD project a cut of that, this is the link you want).