Saturday, April 23, 2016

Does Your Email Provider Know What A "Joejob" Is?

Anecdotal evidence seems to indicate that Google and possibly other mail service providers are either quite ignorant of history when it comes to email and spam, or are applying unsavoury tactics to capture market dominance.

The first inklings that Google had reservations about delivering mail coming from my bsdly.net domain came earlier this year, when I was contacted by friends who have left their email service in the hands of Google, and it turned out that my replies to their messages did not reach their recipients, even when my logs showed that the Google mail servers had accepted the messages for delivery.

Contacting Google about matters like these means you first need to navigate some web forums. In this particular case (I won't give a direct reference, but a search on the likely keywords will likely turn up the relevant exchange), the denizens of that web forum appeared to be more interested in demonstrating their BOFHishness than actually providing input on debugging and resolving an apparent misconfiguration that was making valid mail disappear without a trace after it had entered Google's systems.

The forum is primarily intended as a support channel for people who host their mail at Google (this becomes very clear when you try out some of the web accessible tools to check domains not hosted by Google), so the only practial result was that I finally set up DKIM signing for outgoing mail from the domain, in addition to the SPF records that were already in place. I'm in fact less than fond of either of these SMTP addons, but there were anyway other channels for contact with my friends, and I let the matter rest there for a while.

If you've read earlier instalments in this column, you will know that I've operated bsdly.net with an email service since 2004 and a handful of other domains from some years before the bsdly.net domain was set up, sharing to varying extents the same infrastructure. One feature of the bsdly.net and associated domains setup is that in 2006, we started publishing a list of known bad addresses in our domains, that we used as spamtrap addresses as well as publising the blacklist that the greytrapping generates.

Over the years the list of spamtrap addresses -- harvested almost exclusively from records in our logs and greylists of apparent bounces of messages sent with forged From: addresses in our domains - has grown to a total of 29757 spamtraps, a full 7387 in the bsdly.net domain alone. At the time I'm writing this 31162 hosts have attempted to deliver mail to one of those spamtrap addresses. The exact numbers will likely change by the time you read this -- blacklisted addresses expire 24 hours after last contact, and new spamtrap addresses generally turn up a few more each week. With some simple scriptery, we pick them out of logs and greylists as they appear, and sometimes entire days pass without new candidates appearing. For a more general overview of how I run the blacklist, see this post from 2013.

In addition to the spamtrap addresses, the bsdly.net domain has some valid addresses including my own, and I've set up a few addresses for specific purposes (actually aliases), mainly set up so I can filter them into relevant mailboxes at the receiving end. Despite all our efforts to stop spam, occasionally spam is delivered to those aliases too (see eg the ethics of running the traplist page for some amusing examples).

Then this morning a piece of possibly well intended but actually quite clearly unwanted commercial email turned up, addressed to one of those aliases. For no actually good reason, I decided to send an answer to the message, telling them that whoever sold them the address list they were using were ripping them off.

That message bounced, and it turns out that the domain was hosted at Google.

Reading that bounce message is quite interesting, because if you read the page they link to, it looks very much like whoever runs Google Mail doesn't know what a joejob is.

The page, which again is intended mainly for Google's own customers, specifies that you should set up SPF and DKIM for domains. But looking at the headers, the message they reject passes both those criteria:

Received-SPF: pass (google.com: domain of peter@bsdly.net designates 2001:16d8:ff00:1a9::2 as permitted sender) client-ip=2001:16d8:ff00:1a9::2;
Authentication-Results: mx.google.com;
       dkim=pass (test mode) header.i=@bsdly.net;
       spf=pass (google.com: domain of peter@bsdly.net designates 2001:16d8:ff00:1a9::2 as permitted sender) smtp.mailfrom=peter@bsdly.net
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bsdly.net; s=x;
 h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date:Message-ID:From:References:To:Subject; bh=OonsF8beQz17wcKmu+EJl34N5bW6uUouWw4JVE5FJV8=;
 b=hGgolFeqxOOD/UdGXbsrbwf8WuMoe1vCnYJSTo5M9W2k2yy7wtpkMZOmwkEqZR0XQyj6qoCSriC6Hjh0WxWuMWv5BDZPkOEE3Wuag9+KuNGd7RL51BFcltcfyepBVLxY8aeJrjRXLjXS11TIyWenpMbtAf1yiNPKT1weIX3IYSw=;

Then for reasons known only to themselves, or most likely due to the weight they assign to some unknown data source, they reject the message anyway.

We do not know what that data source is. But with more than seven thousand bogus addresses that have generated bounces we've registered it's likely that the number of invalid bsdly.net From: addresses Google's systems has seen is far larger than the number of valid ones. The actual number of bogus addresses is likely higher, though: in the early days the collection process had enough manual steps that we're bound to have missed some. Valid bsdly.net addresses that do not eventually resolve to a mailbox I read are rare if not entirely non-existent. But the 'bulk mail' classification is bizarre if you even consider checking Received: headers.

The reason Google's systems most likely has seen more bogus bsdly.net From: addresses than valid ones is that by historical accident faking sender email addresses in SMTP dialogues is trivial.

Anecdotal evidence indicates that if a domain exists it will sooner or later be used in the from: field of some spam campaign where the messages originate somewhere else completely, and for that very reason the SPF and DKIM mechanisms were specified. I find both mechanisms slightly painful and inelegant, but used in their proper context, they do have their uses.

For the domains I've administered, we started seeing log entries, and in the cases where the addresses were actually deliverable, actual bounce messages for messages that definitely did not originate at our site and never went through our infrastructure a long time before bsdly.net was created. We didn't even think about recording those addresses until a practical use for them suddenly appeared with the greytrapping feature in OpenBSD 3.3 in 2003.

A little while after upgrading the relevant systems to OpenBSD 3.3, we had a functional greytrapping system going, at some point before the 2007 blog post I started publishing the generated blacklist. The rest is, well, what got us to where we are today.

From the data we see here, mail sent with faked sender addresses happens continuously and most likely to all domains, sooner or later. Joejobs that actually hit deliverable addresses happen too. Raw data from a campaign in late 2014 that used my main address as the purported sender is preserved here, collected with a mind to writing an article about the incident and comparing to a similar batch from 2008. That article could still be written at some point, and in the meantime the messages and specifically their headers are worth looking into if you're a bit like me. (That is, if you get some enjoyment out of such things as discovering the mindbogglingly bizarre and plain wrong mail configurations some people have apparently chosen to live with. And unless your base64 sightreading skills are far better than mine, some of the messages may require a bit of massaging with MIME tools to extract text you can actually read.)

Anyone who runs a mail service and bothers even occasionally to read mail server logs will know that joejobs and spam campaigns with fake and undeliverable return addresses happen all the time. If Google's mail admins are not aware of that fact, well, I'll stop right there and refuse to believe that they can be that incompentent.

The question then becomes, why are they doing this? Are they giving other independent operators the same treatment? If this is part of some kind of intimidation campaign (think "sign up for our service and we'll get your mail delivered, but if you don't, delivering to domains that we host becomes your problem"). I would think a campaign of intimidation would be a less than useful strategy when there are alread antitrust probes underway, these things can change direction as discoveries dictate.

Normally I would put oddities like the ones I saw in this case down to a silly misconfiguration, some combination of incompetence and arrogance and, quite possibly, some out of control automation thrown in. But here we are seeing clearly wrong behavior from a company that prides itself in hiring only the smartest people they can find. That doesn't totally rule out incompetence or plain bad luck, but it makes for a very strange episode. (And lest we forget, here is some data on a previous episode involving a large US corporation, spam and general silliness. (Now with the external link fixed via the wayback machine.))

One other interesting question is whether other operators, big and small behave in any similar ways. If you have seen phenomena like this involving Google or other operators, I would like to hear from you by email (easily found in this article) or in comments.

Update 2016-05-03: With Google silent on all aspects of this story, it is not possible pinpoint whether the public whining or something else that made a difference, but today a message aimed at one of those troublesome domains made it through to its intended recipient.

Much like the mysteriously disappeared messages, my logs show a "Message accepted for delivery" response from the Google servers, but unlike the earlier messages, this actually appeared in the intended inbox. In the meantime one of one of the useful bits of feedback I got on the article was that my IPv6 reverse DNS was in fact lacking. Today I fixed that, or at least made a plausible attempt to (you're very welcome to check with tools available to you). And for possibly unrelated reasons, my test message made it through all the way to the intended recipient's mailbox.

[Opinions offered here are my own and may or may not reflect the views of my employer.]


I will be giving a PF tutorial at BSDCan 2016, and I welcome your questions now that I'm revising the material for that session. See this blog post for some ideas (note that I'm only giving the PF tutorial this time around).

Wednesday, March 9, 2016

Domain Name Scams Are Alive And Well, Thank You

Is somebody actually trying to register your company name as a .cn or .asia domain? Not likely. And don't pay them.

It has been a while since anybody tried to talk me into registering a domain name I wasn't sure I wanted in the first place, but it has happened before. Scams more or less like the Swedish one are as common as they are transparent, but apparently enough people take the bait that the scammers keep trying.

After a few quiet years in my backwater of the Internet, in March of 2016, we saw a new sales push that came from China. The initial contact on March 4th, from somebody calling himself Jim Bing read (preserved here with headers for reference, you may need MIME tools to actually extract text due to character set handling),

Subject: Notice for "bsdly"

Dear CEO,

(If you are not the person who is in charge of this, please forward this to your CEO, because this is urgent, Thanks)

We are a Network Service Company which is the domain name registration center in China.
We received an application from Huabao Ltd on March 2, 2016. They want to register " bsdly " as their Internet Keyword and " bsdly.cn "、" bsdly.com.cn " 、" bsdly.net.cn "、" bsdly.org.cn " 、" bsdly.asia " domain names, they are in China and Asia domain names. But after checking it, we find " bsdly " conflicts with your company. In order to deal with this matter better, so we send you email and confirm whether this company is your distributor or business partner in China or not?

Best Regards,

Jim
General Manager
Shanghai Office (Head Office)
8006, Xinlong Building, No. 415 WuBao Road,
Shanghai 201105, China
Tel: +86 216191 8696
Mobile: +86 1870199 4951
Fax: +86 216191 8697
Web: www.cnweb-registry.com


The message was phrased a bit oddly in parts (as in, why would anybody register an"internet keyword"?), but not entirely unintelligible as English-language messages from Asians sometimes are.

I had a slight feeling of deja vu -- I remembered a very similar message turning up in 2008 while we were in the process of selling the company we'd started a number of years earlier. In the spirit of due diligence (after asking the buyer) we replied then that the company did not have any plans for expanding into China, and if my colleagues ever heard back, it likely happened after I'd left the company.

This time around I was only taking a break between several semi-urgent tasks, so I quickly wrote a reply, phrased in a way that I thought would likely make them just go away (also preserved here):

Subject: Re: Notice for "bsdly"
 
Dear Jim Bing,

We do not have any Chinese partners at this time, and we are not
currently working to establish a presence in Chinese territory. As to
Huabao Ltd's intentions for registering those domains, I have no idea
why they should want to.

Even if we do not currently plan to operate in China and see no need
to register those domains ourselves at this time, there is a risk of
some (possibly minor) confusion if those names are to be registered
and maintained by a third party. If you have the legal and practical
authority to deny these registrations that would be my preference.

Yours,
Peter N. M. Hansteen


Then on March 7th, a message from "Jiang zhihai" turned up (preserved here, again note the character set issues):

Subject: " bsdly "
Dear Sirs,

Our company based in chinese office, our company has submitted the " bsdly " as CN/ASIA(.asia/.cn/.com.cn/.net.cn/.org.cn) domain name and Internet Keyword, we are waiting for Mr. Jim's approval. We think these names are very important for our business in Chinese and Asia market. Even though Mr. Jim advises us to change another name, we will persist in this name.

Best regards

Jiang zhihai

Now, if they're in a formal process of getting approval for a that domain name, why would they want to screw things up by contacting me directly? I was beginning to smell rat, but I sent them an answer anyway (preserved here):

Subject: Re: " bsdly "

Dear Jiang zhihai,

You've managed to make me a tad curious as to why the "bsdly" name
would be important in these markets.

While there is a very specific reason why I chose that name for my
domains back in 2004, I don't see any reason why you wouldn't be
perfectly well served by picking some other random sequence of characters.

So out of pure curiosity, care to explain why you're doing this?

Sincerely,
Peter N. M. Hansteen

Yes, that domain name has been around for a while. I didn't immediately remember exactly when I'd registered the domain, but a quick look at the whois info (preserved here) confirmed what I thought. I've had it since 2004.

Anyone who is vaguely familiar with the stuff I write about will have sufficient wits about them to recognize the weak pun the domain name is. If "bsdly" has any other significance whatsoever in other languages including the several Chinese ones, I'd genuinely like to know.

But by now I was pretty sure this was a scam. Registrars may or may not do trademark searches before registering domains, but in most cases the registrar would not care either way. Domain registration is for the most part a purely technical service that extends to making sure whether any requested domains are in fact available, while any legal disputes such as trademark issues could very easily be sent off to the courts for the end users at both ends to resolve. The supposed Chinese customer contacting me directly just does not make sense.

Then of course a few hours after I'd sent that reply, our man Jim fired off a new message (preserved here, MIME and all):

Subject: CN/ASIA domain names & Internet Keyword

Dear Peter N. M. Hansteen,

Based on your company having no relationship with them, we have suggested they should choose another name to avoid this conflict but they insist on this name as CN/ASIA domain names (asia/ cn/ com.cn/ net.cn/ org.cn) and internet keyword on the internet. In our opinion, maybe they do the similar business as your company and register it to promote his company.
According to the domain name registration principle: The domain names and internet keyword which applied based on the international principle are opened to companies as well as individuals. Any companies or individuals have rights to register any domain name and internet keyword which are unregistered. Because your company haven't registered this name as CN/ASIA domains and internet keyword on the internet, anyone can obtain them by registration. However, in order to avoid this conflict, the trademark or original name owner has priority to make this registration in our audit period. If your company is the original owner of this name and want to register these CN/ASIA domain names (asia/ cn/ com.cn/ net.cn/ org.cn) and internet keyword to prevent anybody from using them, please inform us. We can send an application form and the price list to you and help you register these within dispute period.

Kind regards

Jim
General Manager
Shanghai Office (Head Office)
8006, Xinlong Building, No. 415 WuBao Road,
Shanghai 201105, China
Tel: +86 216191 8696
Mobile: +86 1870199 4951
Fax: +86 216191 8697
Web: www.cnwebregistry.com

So basically he's fishing for me to pony up some cash and register those domains myself through their outfit. Quelle surprise.

I'd already checked whether my regular registrar offers .cn registrations (they don't), and checking for what looked like legitimate .cn domain registrars turned up that registering a .cn domain would likely cost to the tune of USD 35. Not a lot of money, but more than I care to spend (and keep spending on a regular basis) on something I emphatically do not need.

So I decided to do my homework. It turns out that this is a scam that's been going on for years. A search on the names of persons and companies turned up Matt Lowe's 2012 blog post Chinese Domain Name Registration Scams with a narrative identical to my experience, with only minor variations in names and addresses.

Checking whois while writing this it turns out that apparently bsdly.cn has been registered:

[Wed Mar 09 20:34:34] peter@skapet:~$ whois bsdly.cn
Domain Name: bsdly.cn
ROID: 20160229s10001s82486914-cn
Domain Status: ok
Registrant ID: 22cn120821rm22yr
Registrant: 徐新荣
Registrant Contact Email: 1725093@qq.com
Sponsoring Registrar: 浙江贰贰网络有限公司
Name Server: ns1.22.cn
Name Server: ns2.22.cn
Registration Time: 2016-02-29 20:55:09
Expiration Time: 2017-02-28 20:55:09
DNSSEC: unsigned

But it doesn't resolve more than a week after registration:

[Wed Mar 09 20:34:47] peter@skapet:~$ host bsdly.cn
Host bsdly.cn not found: 2(SERVFAIL)


That likely means they thought me a prospect and registered with an intent to sell, and they've already spent some amount of cash they're not getting back from me. I think we can consider them LARTed, however on a very small scale.

What's more, none of the name servers specified in the whois info seem to answer DNS queries:

[Wed Mar 09 20:35:36] peter@skapet:~$ dig @ns1.22.cn bsdly.cn any

; <<>> DiG 9.4.2-P2 <<>> @ns1.22.cn bsdly.cn any
; (2 servers found)
;; global options:  printcmd
;; connection timed out; no servers could be reached
[Wed Mar 09 20:36:14] peter@skapet:~$ dig @ns2.22.cn bsdly.cn any

; <<>> DiG 9.4.2-P2 <<>> @ns2.22.cn bsdly.cn any
; (2 servers found)
;; global options:  printcmd
;; connection timed out; no servers could be reached



So summing up,
  • This is a scam that appears to have been running for years.
  • If something similar to those messages start turning up in your inbox, the one thing you do not want to do is to actually pay for the domains they're offering.

    Most likely you do not need those domains, and it's easy to check how far along they are in the registration process. If you have other contacts that will cheaply and easily let you register those domains yourself, there's an element of entertainment to consider. But keep in mind that automatic renewals for domains you don't actually need can turn irritating once you've had a few laughs over the LARTing.
  • If you are actually considering setting up shop in the markets they're offering domains for and you receive those messages before you've come around to registering domains matching your trademarks, you are the one who's screwed up.
If this makes you worried about Asian cyber-criminals or the Cyber Command of the People's Liberation Army out to get your cyber-whatever, please calm down.

Sending near-identical email messages to people listed in various domains' whois info does not require a lot of resources, and as Matt says in his article, there are indications that this could very well be the work (for some values of) of a single individual. As cybercrime goes, this is the rough equivalent of some petty, if unpleasant, street crime.

I'm all ears for suggestions for further LARTing (at least those that do not require a lot of effort on my part), and if you've had similar experiences, I'd like to hear from you (in comments or email). Do visit Matt Lowe's site too, and add to his collection if you want to help him keep track.

And of course, if "Jim Bing" or Jiang zhihai" actually answer any of my questions, I'll let you know with an update to this article.

Update 2016-03-15: As you can imagine I've been checking whether bsdly.cn resolves and the registration status of the domain via whois at semi-random intervals of at least a few hours since I started the blog post. I was a bit surprised to find that the .cn whois server does not answer requests at the moment:

[Tue Mar 15 10:23:31] peter@portal:~$ whois bsdly.cn
whois: cn.whois-servers.net: connect: Connection timed out


It could of course be a coincidence and an unrelated technical issue. I'd appreciate independent verification. 

Friday, July 24, 2015

The OpenSSH Bug That Wasn't

Much has been written about a purported OpenSSH vulnerability. On closer inspection, the reports actually got most of their facts wrong. Read on for the full story.

It all started with a blog post dated July 16, 2015, titled OpenSSH keyboard-interactive authentication brute force vulnerability (MaxAuthTries bypass), where the TL;DR is that it's possible to get an almost infinite number of tries at authentication -- good for bruteforce password guessing, for example -- if you only tickle the OpenSSH server just so.

This sounded interesting and scary enough that I wanted to try it out myself. The blog quite helpfully supplies a one-liner that you can cut and paste to your own conmand line to check whether the systems you have within reach are indeed vulnerable.

Here's a transcript of running those tests on the machines I happened to try (Disclaimer: The recorded sessions here are from a second try, a few days after the first). First, my home gateway, running a recent OpenBSD 5.8-beta:


[Fri Jul 24 14:58:31] peter@elke:~$ ssh -lrazz -oKbdInteractiveDevices=`perl -e 'print "pam," x 10000'` skapet.bsdly.net
Host key fingerprint is SHA256:maeVFpNMibnkcwPSmjV4QBXfz5J97XLta6e2CrzsAYQ
+---[ECDSA 256]---+
|     .o=o+..     |
|      o.X+.o     |
|      EO.+* .    |
|      o.+oo+ =  .|
|        So=.o + o|
|         B   . o.|
|        . +   . +|
|         . +  .=.|
|         .+ .o+++|
+----[SHA256]-----+

razz@skapet.bsdly.net's password: 
Permission denied, please try again.
razz@skapet.bsdly.net's password: 
Permission denied, please try again.
razz@skapet.bsdly.net's password: 
Permission denied (publickey,password,keyboard-interactive).
[Fri Jul 24 16:53:06] peter@elke:~$ 

Here, razz is a non-existent user, and as we can see we get exactly three password prompts before the connection is shut down. Now we know that an essentially untweaked SSH server configuration on a recent OpenBSD does not behave as described in the article.

But what about earlier OpenBSD releases? I have one box I should have upgraded a while ago (to my enduring shame, it's still on 5.3-stable, but don't tell anyone). So here's the same thing pointed at that box:

[Fri Jul 24 16:53:06] peter@elke:~$ ssh -lrazz -oKbdInteractiveDevices=`perl -e 'print "pam," x 10000'` delilah         
Host key fingerprint is SHA256:AO8rn6Va9+b3+7gdVUxby5zWQFaLnkIA6wcEsOVHukA
+---[ECDSA 256]---+
|   Eoo.+..   .+.+|
|  . +o+ . .  .++B|
|   o oo+   . . O+|
|    ..+.. . . o .|
|     ...S. . o  .|
|       ..   .   .|
|    . o o      . |
|     + = .. .  o.|
|    ..+  oo. .=+o|
+----[SHA256]-----+

razz@delilah's password: 
Permission denied, please try again.
razz@delilah's password: 
Permission denied, please try again.
razz@delilah's password: 
Permission denied (publickey,password,keyboard-interactive).
[Fri Jul 24 16:59:37] peter@elke:~$ 

razz is not a valid user here either, and even the old OpenSSH version here shuts down the connection after three failures. I don't have any OpenBSD boxes with older versions than this anywhere I know about, and we can be reasonably confident that at least close to default configurations on OpenBSD are not vulnerable.

But several of the articles hint that OpenSSH on Linux is vulnerable. I do have a few CentOS boxes within reach. I'll repeat the test on one of these, then:


[Fri Jul 24 17:05:13] peter@elke:~$ ssh -lrazz -oKbdInteractiveDevices=`perl -e 'print "pam," x 10000'` nms
Host key fingerprint is SHA256:fdFxpvSDLq3W9D1d8U6RzuYQcd0WzAmIFfJAzcIkD8I
+---[RSA 2048]----+
|    .. o+==ooo*oB|
|     E. ++++ o+X=|
|         ....oo*.|
|         .  o.+ =|
|        S ...= ++|
|           .= =o+|
|           o . ++|
|          .     .|
|                 |
+----[SHA256]-----+

razz@nms's password: 
Permission denied, please try again.
razz@nms's password: 
Permission denied, please try again.
razz@nms's password: 
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

nms is a CentOS 6.6 box, and the result is as we can see here, pretty much identical to what we saw on the OpenBSD machines. So far we haven't seen anything like what the blogger kingcope had us to expect to see.

But looking back to the original article, he seems to have tested only on FreeBSD. I have some FreeBSD boxes within reach too, and rosalita runs a recent FreeBSD 10.1, freshly upgraded via freebsd-update. Here's what our experiment looks like pointed at rosalita:

[Fri Jul 24 17:15:03] peter@elke:~$ ssh -lrazz -oKbdInteractiveDevices=`perl -e 'print "pam," x 10000'` rosalita           
Host key fingerprint is SHA256:Ig6F8Au3f0KYNrzuc5qRrpZgY4Q/tz0bJrS0NZMxp1g
+---[ECDSA 256]---+
|.                |
| o .             |
|o + . E .        |
|.= * o *         |
|..X * B S        |
|.=o@.= +         |
|+ *oBo+          |
| =.oo=o.         |
|oo*+  .o         |
+----[SHA256]-----+

Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
^C

Bingo! We have finally seen the reported vulnerability in action on a live system. The ^C at the end is me pressing Ctrl-C to abort after ten tries, my heart wasn't quite in it for the full ten thousand. So far, it looks like this behavior is specific to FreeBSD, but of course it is conceivable that other systems ship with their sshds configured in a similar way.


After a bit of back and forth and reading articles elsehere, it seems that only OpenSSH servers that are set up to use PAM for authentication and with a very specific (non-default on OpenBSD and most other places) setup are in fact vulnerable. Even though there is a patch available which tightens up the code a bit in the PAM-specific parts, OpenBSD users don't actually need to apply it. One big reason being that OpenBSD does not use PAM for its authentication.

The question also came up in a thread on OpenBSD-misc, titled Alleged OpenSSH bug, where several OpenBSD developers commented. Do read the whole thread, but as we've already seen, it's easy to test whether your systems behave as described in the original blog post as well as this one.

And as OpenBSD developer Marc Espie says in his message,

Not surprisingly, as the patch clearly shows, the problem is right smack in the middle of USE_PAM code.

I wouldn't call that an OpenSSH bug. I would call it a systemic design flaw in PAM. As usual. LOTS of security holes in authentication systems stem from PAM. Why ? Because that stuff is over designed. Difficult to configure. Gives you MORE than you need to hang yourself several times over. It's been that way for as long as I can remember.

As they say, do read the whole thing. TL;DR this time around is: OpenBSD is not vulnerable, and on the systems that are, changing the configuration to close this particular bruteforcing opportunity is trivial. As is checking the facts before writing up a story. (And as several correspondents have reminded me already -- switching your sshd to keys only authentication will let you sleep better at night.)

Sunday, April 12, 2015

Solaris Admins: For A Glimpse Of Your Networking Future, Install OpenBSD

Yet another proprietary tech titan turns to the free OpenBSD operating system as their source of innovation in the networking and security arena.

Roughly a week ago, on April 5th, 2015, parts of Oracle's roadmap for upcoming releases of their Solaris operating system was leaked in a message to the public OpenBSD tech developer mailing list. This is notable for several reasons, one is that Solaris, then owned and developed by (the now defunct) Sun Microsystems, was the original development platform for Darren Reed's IP Filter, more commonly known as IPF, which in turn was the software PF was designed to replace.

IPF was the original firewall in OpenBSD, and had at the time also been ported to NetBSD, FreeBSD and several other systems. However, over time IPF appears to have fallen out of favor almost everywhere, and as the (perhaps not quite intended as such) announcement has it,

IPF in Solaris is on its death row.
Which we can reasonably be taken to mean that Oracle, like the OpenBSD project back in 2001 but possibly not for the same reasons, are abandoning the legacy IP Filter code base, and moving on to something newer:

PF in 11.3 release will be available as optional firewall. We hope to make PF default (and only firewall) in Solaris 12. You've made excellent job, your PF is crystal-clear design.
Perhaps due to Oracle's practice of putting beta testers under non-disclosure agreements, or possibly because essentially no tech journalists ever read OpenBSD developer-focused mailing lists, Oracle's PF plans have not generated much attention in the press.

I personally find it quite interesting that the Oracle Solaris team are apparently taking in the PF code from OpenBSD. As far as I'm aware release dates for Solaris 11.3 and 12 have not been announced yet, but looking at the release cycle churn (check back to the Wikipedia page's Version history section), it's reasonable to assume that the first Solaris release with PF should be out some time in 2015.

The OpenBSD packet filter subsystem PF is not the first example of OpenBSD-originated software ending up in other projects or even in commercial, proprietary products.

Basically every Unix out there ships some version of OpenSSH, which is developed and maintained as part of the OpenBSD project, with a -portable flavor maintained in sync for others to use (a model that has been adopted by several other OpenBSD associated projects such as the OpenBGPD routing daemon, the OpenSMTPD mail daemon, and most recently, the LibreSSL TLS library. The portable flavors have generally seen extensive use outside the OpenBSD sphere such as Linux distributions and other Unixes.

The interesting thing this time around is that Oracle are apparently now taking their PF code directly from OpenBSD, in contrast to earlier code recipients such as Blackberry (who became PF consumers via NetBSD) and Apple, whose main interface with the world of open source appears to be the FreeBSD project, except for the time when the FreeBSD project was a little too slow in updating their PF code, ported over a fresher version and added some features under their own, non-compatible license.

Going back to the possibly unintended announcement, the fact that the Oracle developers produced a patch against OpenBSD-current, which was committed only a few days later, indicates that most likely they are working with fairly recent code and are probably following OpenBSD development closely.

If Oracle, or at least the Solaris parts of their distinctly non-diminutive organization, have started waking up to the fact that OpenBSD-originated software is high quality, secure stuff, we'll all be benefiting. Many of the world's largest corporations and government agencies are heavy Solaris users, meaning that even if you're neither an OpenBSD user or a Solaris user, your kit is likely interacting intensely with both kinds, and with Solaris moving to OpenBSD's PF for their filtering needs, we will all be benefiting even more from the OpenBSD project's emphasis on correctness, quality and security in the released OpenBSD code.

If you're a Solaris admin who's wondering what this all means to you, you can do several things to prepare for the future. One is to install OpenBSD somewhere (an LDOM in a spare corner of your T-series kit or an M-series domain will do, as will most kinds of x86ish kit) - preferably, also buying a CD set.

A second possibly smart action (and I've been dying to say this for a while to Solaris folks) is to buy The Book of PF -- recently updated to cover new features such as the traffic shaping system.

And finally, if you're based in North America (or if your boss is willing to fly you to Ottawa in June anyway), there's a BSDCan tutorial session you probably want to take a closer look at, featuring yours truly. Similar sessions elsewhere may be announced later, watch the Upcoming talks section on the upper right. If you're thinking of going to Ottawa or my other sessions, you may want to take a peek at my notes on tutorials originally for two earlier BSDCan sessions.

Update 2015-04-15: Several commenters and correspondents have asked two related questions: "Will Oracle contribute code and patches back?" and "Will Oracle donate to OpenBSD?". The answer to the first question is that it looks like they've already started. Which is of course nice. Bugfixes and well implemented feature enhancements are welcome, as long as they come under an acceptable license. The answer to the second question is, we don't know yet. It probably won't hurt if the Oracle developers themselves as well as Solaris users start pointing the powers that be at Oracle in the direction of the OpenBSD project's Donations page, which outlines several useful approaches to help financing the project.

Update 2015-07-07: The first public Solaris 11.3 beta is out, and it contains a port of PF circa OpenBSD 5.5. Sasha Nedvedicky (Oracle's main PF on Solaris developer) offers some more details on his blog.


If you find this or other articles of mine useful, irritating, enlightening or otherwise and want me and the world to know about it, please use the comments feature. Response will likely be quicker there than if you send me email.

Thursday, December 11, 2014

The Password? You Changed It, Right?

Right at this moment, there's a swarm of little password guessing robots trying for your router's admin accounts. Do yourself a favor and do some logs checking right away. Also, our passwords are certainly worth a series of conferences of their own.

As my Twitter followers may be aware, I spent the first part of this week at the Passwords14 conference in Trondheim, Norway. More about that later, suffice for now to say that the conference was an excellent one, and my own refreshed Hail Mary Cloud plus more recent history talk was fairly well received.

But the world has a way of moving on even while you're not looking, and of course when I finally found a few moments to catch up on my various backlogs while waiting to board the plane for the first hop on the way back from the conference, a particular sequence stood out in the log extracts from one of the Internet-reachable machines in my care:


Dec  9 19:00:24 delilah sshd[21296]: Failed password for invalid user ftpuser from 81.169.131.221 port 37404 ssh2
Dec  9 19:00:25 delilah sshd[6320]: Failed password for invalid user admin from 81.169.131.221 port 38041 ssh2
Dec  9 19:00:26 delilah sshd[10100]: Failed password for invalid user D-Link from 81.169.131.221 port 38259 ssh2
Dec  9 19:03:53 delilah sshd[26709]: Failed password for invalid user ftpuser from 83.223.216.46 port 43261 ssh2
Dec  9 19:03:55 delilah sshd[23796]: Failed password for invalid user admin from 83.223.216.46 port 43575 ssh2
Dec  9 19:03:56 delilah sshd[12810]: Failed password for invalid user D-Link from 83.223.216.46 port 43833 ssh2
Dec  9 19:06:36 delilah sshd[14572]: Failed password for invalid user ftpuser from 87.106.66.165 port 52436 ssh2
Dec  9 19:06:37 delilah sshd[427]: Failed password for invalid user admin from 87.106.66.165 port 53127 ssh2
Dec  9 19:06:38 delilah sshd[28547]: Failed password for invalid user D-Link from 87.106.66.165 port 53393 ssh2
Dec  9 19:14:44 delilah sshd[31640]: Failed password for invalid user ftpuser from 83.223.216.46 port 35760 ssh2

Yes, you read that right. Several different hosts from widely dispersed networks, trying to guess passwords for the accounts they assume exist on your system. One of the user names is close enough to the name of a fairly well known supplier of consumer and SOHO grade network gear that it's entirely possible that it's a special account on equipment from that supplier.

Some catching up on sleep and attending to some high priority tasks later, I found that activity matching the same pattern turned up in a second system on the same network.

By this afternoon (2014-12-11), it seems that all told a little more than 700 machines have come looking for mostly what looks like various manufacturers' names and a few other usual suspects. The data can be found here, with roughly the same file names as in earlier episodes. Full list of attempts on both hosts here, with the rather tedious root only sequences removed here, hosts sorted by number of attempts here, users sorted by number of attempts here, a CSV file with hosts by number of attempts with first seen and last seen dates and times, and finally hosts by number of attempts with listing of each host's attempts. Expect updates to all of these at quasi-random intervals.

The pattern we see here is quite a bit less stealthy than the classic Hail Mary Cloud pattern. In this sequence we see most of the hosts trying all the desired user names only a few seconds apart, and of course the number of user IDs is very small compared to the earlier attempts. But there seems to be some level of coordination - the attackers move on to the next target in their list, and at least some of them come back for a second try after a while.

Taken together, it's likely that what we're seeing is an attempt to target the default settings on equipment from a few popular brands of networking equipment. It's likely that the plan is to use the captured hosts to form botnets for purposes such as DDOSing. There is at least one publicly known incident that has several important attributes in common with what we're seeing: Norwegian ISP and cable TV supplier GET found themselves forced to implement some ad hoc countermeasures recently (article in Norwegian, but you will find robots) in a timeframe that fits with the earliest attempts we've seen here. I assume similar stories will emerge over the next days or weeks, possibly with more detail that what's available in the digi.no article.

If you're seeing something similar in your network and you are in a position to share data for analysis similar to what you see in the files referenced abovee, I would like to hear from you.



A conference dedicated to passwords and their potential replacements.

Yes, such a thing exists. All aspects of passwords and their potential replacements have been the topics of a series of conferences going back to 2011. This year I finally had a chance to attend the European one, Passwords14 in Trondheim, Norway December 8-10.

The conference has concluded, but you can find the program up still here, and the video from the live stream is archived here (likely to disappear for a few days soon, only to reappear edited into more manageable chunks of sessions or individual talks). You'll find me in the material from the first day, in a slightly breathless presentation (58 slides to 30 minutes talking time), and my slides with links to data and other material are available here.

Even if you're not in a position to go to Europe, there is hope: there will be a Passwords15 conference for the Europe-challenged in Las Vegas, NV, USA some time during the summer of 2015, and the organizers are currently looking for a suitable venue and time for the 2015 European one. I would strongly recommend attending the next Passwords conference; both the formal talks and the hallway track are bound to supply enlightening insights and interesting ideas for any reasonably security oriented geek.

Now go change some passwords!

I'll be at at least some of the BSD themed conferences in 2015, and I hope to see you there.










Saturday, October 25, 2014

The Book of PF, 3rd Edition is Here, First Signed Copy Can Be Yours

Continuing the tradition started by Michael Lucas with the Absolute OpenBSD, 2nd edition auction, I will be auctioning off the first signed copy of the Book of PF, 3rd edition.

Updated - the ebay auction has concluded, final bid was US $3,050.00 - see below
 
Today I took delivery of two boxes full of my The Book of PF, 3rd edition author copies. They are likely the first to arrive in Norway as well (a few North Americans received their copies early last week), but of course this is somewhere in the range hard to impossible to verify.

Anyway, here is the long anticipated with book selfie:


(larger size available here)

The writing process and the subsequent editing and proofing steps that you, dear reader, will know to appreciate took significantly longer than I had expected, but this edition of the book has the good luck to become available just before the release of OpenBSD that it targets. My original plan was to be in sync with the OpenBSD 5.5 release, but to nobody's surprise but mine the process took longer than I had wanted it to.

As regular readers will know already, the main reason this edition exists is that from OpenBSD 5.5 on, we have a new traffic shaping system to replace the more than 15 years old experimental ALTQ code. The book is up to date with OpenBSD 5.6 (early preorderers have received their disks already, I hear) and while it gives some hints on how to migrate to the new queues and priorities system, it also notes that ALTQ is no longer part of OpenBSD as of version 5.6.

And of course there have been various improvements in OpenBSD since 2010 and version 4.8, which were the year and version referenced in the second edition. You will see updates reflecting at least some of those changes in various parts of the book.

Even if you're not on OpenBSD at all, this edition is an improvement over previous versions, we've taken some care to include information relevant to FreeBSD and NetBSD as well, and where there are significant differences between the systems, it's noted in the text and examples.

It could have been tempting to include specific references to Apple's operating system as well, but I made a decision early on to stay with the free systems. I have written something about PF and Apple, but not in the book -- see my Call for Testing article How Apple Treats The Gift Of Open Source: The OpenBSD PF Example for a few field notes.

But now for the main item. For this edition, for a limited time only, there will be a

Book of PF Auction

You have a chance to own the first author signed copy of The Book of PF, 3rd edition.

The auction is up at http://www.ebay.com/itm/The-Book-of-PF-3rd-ed-signed-by-the-author-First-Copy-signed-/321563281902? - I'll look into extending the auction period, for some odd reason the max offered was 10 days. If your bid is not the successful one, I strongly urge you to make a direct donation of the same amount to the OpenBSD Foundation instead.

I've signed the book, and will fill in the missing spaces once we have the name and amount:




UPDATE 2014-10-26 01:00 CEST: Whatever it was that stopped ebay from listing the auction was resolved. The auction is up at http://www.ebay.com/itm/The-Book-of-PF-3rd-ed-signed-by-the-author-First-Copy-signed-/321563281902? - I'll look into extending the auction period, for some odd reason the max offered was 10 days. If your bid is not the successful one, I strongly urge you to make a direct donation of the same amount.to the OpenBSD foundation instead.

The first signed copy, and incidentally also the first copy my wife picked out of the first box we opened, will come with this inscribed in my handwriting on the title page:

FOR (your name)
Winner of the 2014 Book of PF Auction
Thank you for Supporting OpenBSD with your
(CAD, USD or EUR amount) donation

Bergen, (date), (my signature)

That's just for your reference. My handwriting is not a pretty sight at the best of times, and when you, the lucky winner, receive the book it's entirely reasonable that you will not be able to dechipher the scrawls at all.

If you think your chances of actually winning are not worth considering, please head over to the OpenBSD donations or orders page and spend some of your (or your boss') hard earned cash!

My speaking schedule has not been set for the upcoming months, but there is a reasonable chance I'll attend at least a few BSD events in the near future. See you there!

UPDATE 2014-11-26: The auction concluded on November 4th, with Bill Allaire as the successful bidder. He paid via PayPal (as you almost inevitably will on an Ebay auction) immediately, and I sent the signed book to him two days later.

As the lady at the post office said, the package took about a week to turn up in Bill's mailbox. But it took 21 days before PayPal finally made the funds available to me, and after a bit of wrestling with the possibly very intuitive (to someone else) PayPal interface, I transferrred the amount to the OpenBSD Foundation today. PayPal of course racked up fees both incoming and outgoing, to a degree that I think if the fees we paid there ar considered at all competitive, whoever coined the phrase "the giant vampire squid" to describe US banks must have been trying desperately to make the traditional US banks sound all nice and cuddly.

For the unsuccessful bidders, I urge you to head over to the OpenBSD Foundation's Donations page and make a donation equal to your highest bid.

Tuesday, August 12, 2014

Password Gropers Take the Spamtrap Bait

We have just seen a new level of password gropers' imbecility. Or, Peak Stupid.

Regular readers of this column know that I pay attention to my logs, as any sysadmin worth his or her salt would. We read logs, or at least skim summaries because in between the endless sequence of messages that were successfuly delivered, users who logged in without a hitch and web pages served, from time to time unexpected things turn up. Every now and then, the unexpected log entries lead to discoveries that are useful, entertaining or even a bit bizarre.

In no particular order, my favorite log discoveries over the years have been
  • the ssh bruteforcer password guessers, a discovery that in turn lead to some smart people developing smart countermeasures that will shut each one of them down, automatically.
  • the faked sender addresses on spam, leading to bounces to non-existent users. Once again, thanks to the OpenBSD developers, I and others have been feeding those obviously fake addresses back to spamdb, slowing spammers down and generating publishable blacklists.
  • and never forgetting the relatively recent slow, distributed ssh bruteforcers, also known as "The Hail Mary Cloud", who in all likelihood had expected to avoid detection by spreading their access attempts over a long time and a large amount of hosts in coordination that was only detectable by reading the logs in detail.

After the Hail Mary Cloud cycle of attempted attacks, I thought I'd seen it all. 

But of course I was wrong.  Now we have more likely than not found evidence of another phenomenon which by perverse coincidence or fate appears to combine features of all these earlier activities.

Early in July 2014, my log summaries started turning up failed logins to the pop3 service (yes, I still run one, for the same bad reasons more than a third of my readers probably do: inertia).

At our site (which by now serves mainly as my personal lab in addition to serving a few old friends and family), pop3 activity besides successful logins by the handful of users who still use the service had been quite low for quite a while. And if we for now ignore the ssh bruteforcers who somehow never seem to tire of trying to log in as root, my log summaries had been quite boring for quite a while. But then the failed pop3 logins starting this July were for the unlikely targets admin, sales and info, user names that have never actually existed on that system, so I just kept ignoring the irregular and infrequent attempts.

My log summaries seemed to indicate that whoever is in charge of superonline.net (see the log summary) should tighten up a few things, but then it's really not my problem.

But then on July 18th, this log entry presented itself:

Jul 18 17:01:14 skapet spop3d[28606]: authentication failed: no such user: malseeinvmk - host-92-45-149-176.reverse.superonline.net (92.45.149.176)

That user name malseeinvmk is weird enough that I remembered adding it as part of one of the spamtrap addresses at one point. I had to check:

$ grep malseeinvmk sortlist
malseeinvmk@bsdly.net

which means yes, I remembered correctly. That user name is part of one of the more than twenty-eight thousand addresses on my traplist page, added at some point between 2006 and now to my spamdb database as a spamtrap, and as such known to be non-deliverable. So I started paying a bit more attention, and sure enough, over the next few days the logs (preserved here) were still turning up more entries from the traplist page. We would see local parts (usernames) only, of course, but grep would find them for us.

Now, trying to log on to a system with user names that are known not to exist sounds more than a little counterproductive at the best of times. But from my perspective, the activity was in fact quite harmless and could be potentially fun to watch, so I adjusted my log rotation configuration to preserve the relevant logs for a little longer than the default seven days.

Coming back to peek at the results every now and then, I noticed fairly soon that the attempts in almost all cases were concentrated in the first two minutes past every full hour. There are a couple of hour long periods with attempts spread more or less evenly with a few minutes in between, but the general pattern is a anything from one to maybe ten attempts in the :00:00-:02:00 interval.

The pattern was not exactly the Hail Mary Cloud one, but similar enough in that the long silent intervals could very well be an attempt at hiding in between other log noise.

But that returns us to the question, Why would anybody treat a list of known to be non-existent user names as if they actually offered a small chance of access?

They could be trying to weed out bad entries. One possible explanation would be that whoever is at the sending end here is trying to weed out the bad addresses in a long list that may or may not be the wanted quality. If an attempted login gives an indication whether the user exists or not, it might be worth trying.

They could be thinking the list is all good, and they want access. Brute force password guessing is not limited to ssh. We will explore this option further in a bit.

This could be an elaborate joke. The Hail Mary Cloud got passing mention in some mainstream press, and there are people out there who might be able to pull this off just for the hell of it.

Let's put each of those hypotheses to the test.

First, when you try to log in to the service, do you get any indication whether the user you attempt to log in as exists?

Here's what it looks like when a valid user logs in:

$ telnet skapet.bsdly.net pop3
Trying 213.187.179.198...
Connected to skapet.bsdly.net.
Escape character is '^]'.
+OK Solid POP3 server ready
USER peter
+OK username accepted
PASS n0neof.y3RB1Z
+OK authentication successful

At this point, I would have been able to list or retrieve messages or even delete them. But since my regular mail client does that better than I do by hand, I close the session instead:

quit
+OK session ended
Connection closed by foreign host.

And in case you were wondering, that string is not my current password, if it ever was.

Next, let us compare with what happens when you try logging in as a user that does not exist:

$ telnet skapet.bsdly.net pop3
Trying 213.187.179.198...
Connected to skapet.bsdly.net.
Escape character is '^]'.
+OK Solid POP3 server ready
USER jallaballa
+OK username accepted
PASS Gakkazoof
-ERR authentication failed
Connection closed by foreign host.

Here, too, the user name is tentatively accepted, but the login fails anyway without disclosing whether the password was the only thing that was invalid. If weeding out bad entries from the list of more than twenty-eight thousand was the objective, they're not getting any closer using this method.

Unless somebody actually bothered to compromise several hundred machines in order to pull off a joke that would be funny to a very limited set of people, the inescapable conclusion is that we are faced with would-be password guessers who
  • go about their business slowly and in short bursts, hoping to avoid detection
  • base their activity on a list that was put together with the explicit purpose of providing no valid information

If throwing semi-random but 'likely' user names and passwords at random IP addresses in slow motion had monumental odds against succeeding, I'm somewhat at a loss to describe the utter absurdity of this phenomenon. With trying to sneak under the radar to guess the passwords of users that have never existed, I think we're at the point where the Internet's bottom-feeding password gropers have finally hit peak stupidity.

More likely than not, this is the result of robots feeding robots with little or no human intervention, also known as excessive automation. Automation in IT is generally a good thing, but but I have a feeling somebody is about to discover the limits of this particular automation's usefulness.

I hate to break it to you, kids, but your imaginary friends, listed here, never actually existed, and they're not coming back.

And if this is actually a joke that has somebody, somewhere rolling on the floor laughing, now is a good time to 'fess up. There is still the matter of a few hundred compromised hosts to answer for, which may be a good idea to clear up as soon as possible.

As usual, I'll be tracking the activities of the miscreants and will refresh these resources at semi-random intervals as long as the activity persists:

At the rate they're going at the moment, we could be seeing them hang on for quite a while. And keep in mind that the list generally expands by a few new finds every week.

Update 2014-08-19: The attempts appear to have stopped. After some 3798 access attempts by 849 hosts trying 2093 user IDs, the last attempt so far was

Aug 17 22:40:58 skapet spop3d[20058]: authentication failed: no such user: jonatas-misbruke - host-92-45-135-208.reverse.superonline.net (92.45.135.208)

I take this as an indication that these access attempts are at least to some extent monitored, and with those numbers of attempts with a total of 0 successes, any reasonably constructed algorithm would have found reason to divert resources elsewhere. We can of course hope that some of the participating hosts have been cleaned up (although nobody actually wrote me back about doing that), and of course you can't quite rule out the possibility that whoever runs the operations reads slashdot.

But then again, the fact that the pop3 password gropers have moved on from my site should not lead you to believe that your site is no longer a target.

Update 2014-08-21: I spoke too soon about the pop3 password gropers giving up and moving on. In fact, only a few hours after the last update (in the early morning CEST hours of August 20th), two more attempts occured:

Aug 20 05:07:41 skapet spop3d[2943]: authentication failed: no such user: info - 94.102.63.160
Aug 20 05:31:58 skapet spop3d[15882]: authentication failed: no such user: info - host-92-45-150-182.reverse.superonline.net (92.45.150.182)


before another long silence set in and I decided to wait a bit before making any new announcements.

But at just after ten in the morning CEST on August 21st, the slow and distributed probing with usernames lifted from my spamtraps page resumed. At the time of this writing, the total number of attempts has reached 3822, with 856 participating hosts and attempts on 2103 distinct user names. I'll publish refreshed files at quasi-random intervals, likely once per day if not more frequently.

Update 2014-09-01: Statistics improved, estimates revised.

A couple of days ago, I tweeted:
which, based on some quick calculations at the time, seemed a reasonable number. Now we've reached the end of the first full month of this extended incident, and it's time to present some preliminary statistics, lightly polished.

For the estimated end date, which will be loosely based on the rate of attempts at new user names, the calculation is as follows: On the spamtraps page, roughly half the addresses belong to domains whose primary MX is not skapet.bsdly.net (we're secondary or further out). Stripping those addresses from our total, we're left with 14809 possible usernames. Of course, we have no idea when the list they're working from was sucked in, but this is our latest data. Next, by the time I started writing this update, a total of 2523 usernames had been attempted. We're now well into the 57th day, so dividing 2523 usernames by 57, we get a little more than 44 usernames per day on average. Dividing our average per day with total number of usernames, we get approximately 334 days to try them all, which means that the first attempt on the last username in the list is likely 277 (344 - 57) days in the future. This means that at the current rate, it will be early June 2015 before they run out of usernames. How long they stick around will likely depend on their strategy for selecting passwords to match.

If you want to see some more detail on the attackers' progress, here are some data I generated while actively procrastinating and putting off other tasks with a defined and fixed deadline:

A graph of attempts per day, based on this .csv file and massaged via LibreOffice in this spreadsheet:



(larger .png here). I also tried feeding LibreOffice this .csv file with per hour data, but getting the data graphed in any sensible manner seems to require more effort than I'm inclined to put into that particular task.

Hosts participating with first and last seen dates: Possibly more useful is this .csv file, which has participating hosts in the same order as the List of compromised hosts participating but expanded to comprise the fields Attempts,Host,User Names,First Seen,Last Seen, where the last two are dates in formats that your spreadsheet should be able to parse and use as a sort key with only minor efforts. The host with the most attempts as of the last dump was only first seen on August 29th, while several of the top 10 have been with us since some time in July. And of course, at the other end of the scale, quite a few have made only one attempt to date. I'll be updating the data at semi-random intervals, likely at least once a day. Raw data and generated files can be found here, along with scripts used and even a few temporary files. The data can be used for any purpose as long as proper attribution is included. See the Hail Mary Cloud Data page for details, and the Hail Mary Cloud overview article for background information.

Update 2014-09-04: A separate effort detected Perceptive readers will have noticed that the data now includes traces of activity from hosts in the dynamic.adsl.gvt.net.br domain starting on September 2, 2014, which deviates far enough from the general distributed pattern that it's likely it represents a separate, if not overly successful, effort.

These four (so far) hosts have tried relatively long sequences of attempts at single user names at a time, targeting what I assume are 'probable' user names for that part of the world, 22 user IDs so far.  None of those user IDs expand to valid addresses in our domains, and just because I can, I've now added these user IDs to the spamtraps page as presumed members of our nxdomain.no domain. If you feel those hosts should be removed from the data for purposes of your own analyses, it's fairly easy to strip them away. In fact, a simple

$ grep -v dynamic.adsl.gvt.net.br filename

where filename is  either the raw authentication log or the extracted files that include hostnames will remove them from sight.

Update 2014-12-10: Data from this incident as well as some others have been included in my Passwords14 presentation, PDF version available too, here.

Update 2015-08-24: Since this article was originally posted, there have been several shorter episodes of pop3 groping activity, but mostly very low volume and short lived enough to be not really worth noting. However, at the very end of July 2015, the intensity increased sligtly, and I've preserved the log entries starting July 27 in a convenient place -  raw log data, overview .csv file, user names by frequency and finally hosts by frequency.

A very preliminary analysis of the data says we have a total of 268 hosts attempting access at least once, and apparent coordination (several hosts processing the same user name for a short while before moving on to the next) in several long sequences. We also see long sequences of one single host attempting one user name or an assortment of user names, and while we have a number of 'likely' user names (including my first name) as attempted user IDs, we also see an assortment of user IDs apparently lifted from the spamtraps page.

If you're seeing something like this in your logs, I'm interested in hearing from you. And if you're responsible for one or more of the systems that appear in those logs, you have a bit of cleaning up to do.


Book of PF News: All ten chapters and the two appendixes have been through layout and proofing, front and back matter is still a work in progress, expected arrival for physical copies is still not set. Those most eager to read the thing could try early access or preorder.

It's almost certain that physical copies will not be available at EuroBSDCon in Sofia, but If you make it to one of my sessions there, a special discount code worth 40% off on both the ebook and print version will be disclosed to attendees. (And I'm told it's active already, so if you guess what it is, you can use it.)

(And of course, as this blog post shows, the book came out and was well received, at least in some quarters.)