Note: When I began writing this, I was going to go into a lot more detail, explaining ntpd (the reference implementation), OpenNTPD, the NTP Pool Project, etc. I may do that in a follow-up post but, for now, I’ll keep it short and to the point.

The NTP Pool Project has a set of “basic guidelines” whose intended audience is “anyone distributing an appliance, operating system or some other kind of software using NTP”. This almost certainly includes OpenBSD (an operating system) which ships with OpenNTPD (an NTP implementation).

The NTP Pool Project‘s “Basic guidelines” clearly state:

“Do not use the standard pool.ntp.org names as a default configuration in your system.”

A little further down, to reiterate the importance of this, they again state (emphasis in the original):

“You must absolutely not use the default pool.ntp.org zone names as the default configuration in your application or appliance.”

The next sentence informs the reader, “You can apply for a vendor zone here on the site.”

Just like the label on a hair dryer that warns one not to use it while in the shower, these guidelines exist for a reason (cf. “Flawed Routers Flood University of Wisconsin Internet Time Server“, for example).

OpenBSD, however, has chosen to flagrantly ignore them.

OpenNTPD is included as part of OpenBSD’s “base system”. Version 1.13 (dated 2015/05/18) of it’s default configuration file, /etc/ntpd.conf, includes the following:


# use a random selection of NTP Pool Time Servers
# see http://support.ntp.org/bin/view/Servers/NTPPoolServers
servers pool.ntp.org


N.B.: This isn’t a recent change, either — it’s been there since 2004.

I was going to write to one of the OpenBSD mailing lists about this, thinking that perhaps it was simply an oversight. Before I did, however, I searched the archives to see if it had been discussed before. I discovered that it had — almost five years ago.

In that thread, Theo de Raadt wrote:

We don’t intend to change anything.

Their decrees are meaningless. They don’t provide the ntp traffic,
they only provide DNS records. As written, those rules are designed
to let the people at ntp.org impose a punishing policy against those
they don’t like.

Those rules do not improve time distribution. It is just control
freak behaviour.

This blatant violation of the guidelines is made even worse by what I discovered while writing this:


$ dig +short {0,1,2,3}.openbsd.pool.ntp.org
173.44.32.10
208.75.88.4
66.228.35.252
98.191.213.12
108.61.73.243
96.44.142.5
70.35.113.43
199.7.177.206
129.6.15.28
69.50.219.51
66.175.209.17
65.182.224.39
216.229.4.66
129.250.35.251
192.155.90.13
132.163.4.101


An “openbsd” vendor zone is already available for their use. Adherence to the guidelines would require, at minimum, a one-line change to /etc/ntpd.conf:


-servers pool.ntp.org
+servers 0.openbsd.pool.ntp.org


I’m a pretty easy guy to get along with, I think. I’m fairly laid back and there’s not a whole lot that really bothers me. It takes a lot to truly upset me. While some people (especially many of the women I have dated) would, when questioned, insist that I am a “grade A” asshole, I don’t really think I am.

There are some things, though, that really just annoy the shit outta me.

This block of text, which appeared at the bottom of an e-mail from a “System Operations Specialist” at Aclara, a company that provides technologies like TWACS,  is one of ‘em:

This e-mail is intended for the addressee shown. Should you have
received it in error, please delete this message and contact the sender.
This e-mail and any attachments thereto may contain information that is
privileged, confidential or proprietary. Any review, dissemination or
use of this transmission or its contents by persons other than the
addressee or authorized employees of the intended organizations is
strictly prohibited.

You might have heard me rant about this bullshit that I’m certain some overpriced air thief of a lawyer came up with. Later, some suit at another company saw it and from there it spread faster than the Morris worm. The morons who insist that similar messages be automatically slapped on to every piece of e-mail that leaves their company servers obviously spent less time consciously thinking the repercussions through than I spent trying to find Aclara’s CEO’s e-mail address on their web site (hint: about 30 seconds).

Unfortunately, I didn’t find it. I was hoping that I would because I wanted to point out just what those idiotic “disclaimers” actually accomplish.

Let’s rewind to yesterday.

Note: I’ve changed the name of the “System Operations Specialist” who almost certainly is an innocent bystander and didn’t choose to have these unenforceable “legal disclaimers” tagged on to every message he sends out.

So yesterday afternoon, “Bob” sends an e-mail to “Sally” (also not her real name). Sally is employed by one of my customers, an electric utility who is also a customer of Aclara (to be clear, my company handles all I.T. for the utility). The e-mail contains some information about an ongoing issue that they’ve been experiencing. Sally, who is not a technical person, passes the e-mail along to my boss and, along with a question from the bossman, it ends up in my mailbox shortly afterward.

What is one to do?

Obviously, I should respond to the e-mail from my boss along with the answer to his question, right? My answer will provide insight into the issue and I’ll probably even offer up a solution since I know exactly what is causing the anomaly that is being seen. At this point, no one else does. Clearly, I should respond with this useful information.

“… delete this message … contact the sender … use of this transmission … by persons other than … authorized employees of the intended organizations is strictly prohibited.”

Yes, I should respond with this useful information. Because I am not “the addressee shown” or even an employee of the intended organization (not to mention an authorized one!), I am “strictly prohibited” from doing so.

Thanks to some asshat lawyer, I am “prohibited” from providing assistance and, quite possibly, resolving this issue.

I’m sure that you can see the major malfunction here so, please, if your company is ever considering slapping these meaningless, completely unenforceable contracts onto the bottom of every employee’s e-mail messages, please point them here so that they might understand why doing so is a bad idea.

Now, if you’ll excuse me, I must try to find contact information for Aclara’s general counsel so that I can request, in writing and notarized, an exception to this policy they have implemented.

Maybe I really am an asshole.

Now that Ryan Werber (“Scrye” on Freenode #cisco) has passed the CCIE exam and became #38168, he apparently has some extra free time on his hands.

Update: This “Star Wars Traceroute” has been propagating (see what I did there?) like wildfire in the nerd-o-sphere since I first published this post. Since then, Ryan has originated (get it!?) a Tumbler page where he explains how and why he did it and also posted the configs that he used. Contrary to the assumptions of many on Hacker News (who may be “rock star hackers” but obviously aren’t network engineers), he did not deploy a shit-ton of virtual machines to do it. Instead, he accomplished it with a pair of low-end Cisco ISR routers and a bunch of VRFs (and, of course, a DNS server). Scrye will neither confirm nor deny whether he had the assistance of the “VRF Smurf” in putting it all together.

$ traceroute -m 100 -q 1 216.81.59.173
traceroute to 216.81.59.173 (216.81.59.173), 100 hops max, 60 byte packets

...

 3  ae6-449.edge2.Chicago2.Level3.net (4.30.13.41)  10.838 ms
 4  vl-3602-ve-226.ebr2.Chicago2.Level3.net (4.69.158.54)  10.983 ms
 5  ae-1-51.edge3.Chicago3.Level3.net (4.69.138.136)  10.982 ms
 6  gblx-level3-10g.Chicago3.Level3.net (4.68.63.66)  11.197 ms
 7  HURRICANE-ELECTRIC-LLC-Ashburn.TenGigabitEthernet4-4.ar3.DCA3.gblx.net (64.214.121.170)  35.826 ms
 8  10gigabitethernet1-2.core1.atl1.he.net (184.105.213.110)  51.048 ms
 9  216.66.0.26 (216.66.0.26)  44.041 ms
10  *
11  Episode.IV (206.214.251.1)  94.905 ms
12  A.NEW.HOPE (206.214.251.6)  96.390 ms
13  It.is.a.period.of.civil.war (206.214.251.9)  97.131 ms
14  Rebel.spaceships (206.214.251.14)  93.888 ms
15  striking.from.a.hidden.base (206.214.251.17)  100.856 ms
16  have.won.their.first.victory (206.214.251.22)  99.782 ms
17  against.the.evil.Galactic.Empire (206.214.251.25)  98.152 ms
18  During.the.battle (206.214.251.30)  97.891 ms
19  Rebel.spies.managed (206.214.251.33)  101.350 ms
20  to.steal.secret.plans (206.214.251.38)  110.337 ms
21  to.the.Empires.ultimate.weapon (206.214.251.41)  109.110 ms
22  the.DEATH.STAR (206.214.251.46)  107.664 ms
23  an.armored.space.station (206.214.251.49)  110.316 ms
24  with.enough.power.to (206.214.251.54)  114.483 ms
25  destroy.an.entire.planet (206.214.251.57)  112.441 ms
26  Pursued.by.the.Empires (206.214.251.62)  113.439 ms
27  sinister.agents (206.214.251.65)  115.253 ms
28  Princess.Leia.races.home (206.214.251.70)  116.631 ms
29  aboard.her.starship (206.214.251.73)  117.439 ms
30  custodian.of.the.stolen.plans (206.214.251.78)  119.306 ms
31  that.can.save.her (206.214.251.81)  119.458 ms
32  *
33  freedom.to.the.galaxy (206.214.251.89)  120.390 ms
34  0-------------------0 (206.214.251.94)  122.645 ms
35  0------------------0 (206.214.251.97)  123.462 ms
36  0-----------------0 (206.214.251.102)  123.462 ms
37  0----------------0 (206.214.251.105)  126.516 ms
38  0---------------0 (206.214.251.110)  126.885 ms
39  0--------------0 (206.214.251.113)  133.179 ms
40  0-------------0 (206.214.251.118)  132.983 ms
41  0------------0 (206.214.251.121)  134.323 ms
42  0-----------0 (206.214.251.126)  134.317 ms
43  0----------0 (206.214.251.129)  136.663 ms
44  0---------0 (206.214.251.134)  134.644 ms
45  0--------0 (206.214.251.137)  138.128 ms
46  0-------0 (206.214.251.142)  137.741 ms
47  0------0 (206.214.251.145)  138.064 ms
48  0-----0 (206.214.251.150)  131.350 ms
49  0----0 (206.214.251.153)  111.471 ms
50  0---0 (206.214.251.158)  113.649 ms
51  0--0 (206.214.251.161)  110.592 ms
52  0-0 (206.214.251.166)  111.370 ms
53  00 (206.214.251.169)  109.261 ms
54  I (206.214.251.174)  112.135 ms
55  By.Ryan.Werber (206.214.251.177)  117.487 ms
56  When.CCIEs.Get.Bored (206.214.251.182)  111.883 ms
57  CCIE.38168 (206.214.251.185)  117.883 ms
58  FIN (206.214.251.190)  111.251 ms

Apparently /31s aren’t used on the CCIE lab exam. *grin*

Tom Hollingsworth laid down a nice little rap in the Day 1 Wrap-Up Video at Network Field Day 4, but it looks like he may have some competition — from inside Cisco, no less!

From the video description:

The group of new hires at cisco were told to make a parody music video for fun. This is what our young talent is doing and having fun at it!

In a previous life, I worked at a .edu and lived, roughly, across the street from the building that housed IT. I don’t recall what it was that caused me to go to work that Monday evening, the third day of a three-day (holiday) weekend, but as anyone who works in IT would likely agree, it’s not unusual to be working at the office or data center at weird hours.

As I unlocked the door to the server room and pushed it open, I was met by a huge blast of HOT air — not just warm; HOT.

Unbeknownst to anyone in IT or our Facilities department, the dedicated cooling system for the data center had failed. It’s been several years ago now but, if memory serves, it was wintertime and the air conditioning unit on the roof had frozen up.

Exactly when it failed wasn’t known but, if the 109 degree (Fahrenheit) temperature was any indication, it had almost certainly been much earlier in the weekend. It certainly didn’t happen ten minutes before I arrived.

A phone call and perhaps 30 minutes later, our Executive Director of Facilities walked into the server room, where I was — now shirtless and sweating — shutting down as many non-critical systems as I could.

I don’t recall many of the details (if I ever even knew them) regarding how they fixed it but that’s somewhat irrelevant to the story anyways.

Fortunately for us, no equipment suffered any damage (as far as we could tell).

As you might imagine, my request to purchase environmental monitoring equipment was quickly approved and, in short order, we had a device from APC installed and configured.

Note: I highly recommend the APC NetBotz line of products.

The APC environmental monitor supported SNMP and we used our (Nagios-based) monitoring system to query the temperature (and graph it) every five minutes. The device itself also had the ability to generate e-mail alerts when thresholds were exceeded. We configured those to be sent to mailing lists of which every employee in the IT and Facilities departments was a member. In addition, I whipped up a little script that pulled the temperature via SNMP and also generated alerts when the thresholds we defined were exceeded — you know, just in case. We certainly didn’t want to have a repeat of this incident.

History Repeats Itself

Fast forward several years to this past Tuesday night. I’m working for a different company in a different city, but the story is similar.

I’m at home and searching for my phone to send a text message to a friend. When I find it, I notice that I had received an e-mail alert from our monitoring system about an hour before. One of the two VMware ESXi servers in our data center had stopped responding to ICMP echo requests.

Note: I use the term data center loosely. The building I’m referring to in this article is far from a “real” data center — “equipment hut” would be more accurate — but it (roughly) meets Wikipedia’s definition of “data center” so I’m rolling with it.

I get dressed, find my keys, jump in my vehicle, and head to the site.

As I unlock the door and open it, I am once again greeted with a blast of hot air — the cooling system, as you might have guessed by now, wasn’t.

I’m not sure what the actual temperature was as the air conditioning system itself was off and the remote control unit has only a two-digit read-out on it. It was certainly well above the indicated 99 degrees Fahrenheit.

The cooling system was powered off.

I decided that perhaps I should turn it on.

Luckily for us, it was somewhat cold outside. I left the external door standing wide open and the combination of the cold outside air and an air conditioning system that was now running balls to the wall managed to cool it down relatively quickly.

Once I was satisfied that everything was okay, I powered the VMware ESXi server back up, waited for the “all clear” alerts to arrive, and headed home.

Arriving home, I sent out an e-mail to my team to let them know what had happened. Early the next morning, I received a reply from one of my co-workers. It read, in part:

“I’ll take the fall for this one. When we [installed some equipment] sometime last week it was like 60 degrees in there. So I shut off the AC and left the door open while we we’re doing stuff. I totally forgot to turn it back on.”

Now, I’m certainly not trying to call him out because we all make mistakes and screw up at times. I can understand how he overlooked forgetting to turn the cooling system back on before he left.

As he wrote in his e-mail, “I guess we wouldn’t be human if we didn’t make a few mistakes.”

He’s absolutely right, of course. Shit happens. I’d be willing to bet, however, that he’ll never make that same mistake again. It was a learning experience and, fortunately, nothing seemed to be damaged. No harm, no foul.

Now, if you’ll excuse me, I need to go install some environmental monitoring equipment.

Note: The events described herein took place many, many years ago (the statute of limitations has long since expired!), but the moral of the story remains the same. This topic came up on IRC a few nights ago and since a) several were interested in the story and b) I’m heading out to DerbyCon shortly, the timing seemed appropriate.

It’s 2 a.m. and I’m the stereotypical inquisitive teenager furiously typing away on the PC in the corner of my darkened bedroom.

I had connected — via dialup — to my ISP and used SSH to access one of their servers (let’s call it “alpha“), an x86 box running Red Hat Linux (RHL, not RHEL). This server was running telnet, SSH and FTP daemons in order to provide remote access to both e-mail and the files in the user’s home directory (mostly so that one could dump stuff in ~/public_html/ and make it available via HTTP). In addition, this was the only server that users could directly log in to.

Though I had logged in to alpha hundreds of times before to send and receive email, I’d never really “looked around” the system. For whatever reason, I became curious and started “exploring” in an attempt to see how my ISP had set their environment up.

One of the first things I noticed was that user home directories were mounted via NFS, which made sense since the HTTP server ran on another machine. Likewise, /var/spool/mail was mounted via an NFS share from the mail server.

In all, there were four filesystems mounted via NFS on alpha:

• /home/a/ mounted over NFS from server bravo,
• /home/b/ mounted over NFS from server charlie,
• /home/c/ mounted over NFS from server delta, and
• /var/spool/mail/ mounted over NFS from server echo.

After a bit of looking around, I discovered the meaning behind the layout of /home.

Regular customers had their home directories under /home/a/, business customers had their home directories under /home/b/, and the home directories of the administrators lived under /home/c/.

Naturally, I was drawn to /home/c/.

The home directories of the system administrators themselves were accessible by anyone, although they contained many subdirectories that weren’t accessible due to permissions. I spent a bit of time exploring these files and directories that I did have access to but, alas, there was not much of interest to be found.

But I wanted access to the rest of the files and, luckily, it wasn’t hard to get.

A local root vulnerability for older versions of the Linux kernel had been discovered a good while before. Unfortunately for me, it had long since been fixed and new versions made available. Fortunately, however, the administrators at my ISP (we’ll call them “Alice” and “Bob“) hadn’t bothered updating their servers.

A little bit later, I had tweaked the exploit code slightly in order to work on this server, compiled it on my local machine and transferred it to alpha. It only took a few seconds to execute it.

# id
uid=0(root) gid=0(root) groups=0(root)

I had gained root access on alpha, the server belonging to my ISP!

My job wasn’t done yet, however.

You see, there’s an NFS server option called root_squash. This option prevents the root user on an NFS client from actually have root privileges to the mounted filesystems. Thus, while I was root on alpha, I still couldn’t access any of those files I wanted to. Luckily, there’s an easy fix for that:

# su alice
$ id
uid=1000(alice) gid=100(users) groups=100(users)

I was now, in effect, logged in as one of the system administrators and had access to all of their files.

This, of course, gave me access to all of their files which led to an interesting discovery — they kept all of their user’s passwords in plain-text!

When I had noticed that they were using NFS to share filesystems across servers, I had naturally assumed that they were also using NIS — the two often go hand-in-hand. Instead, when a user’s password needed changed, they’d update it locally on a central server. Custom scripts on that server then took care of pushing it out to all of their other servers (including their RADIUS servers), thus keeping everyone’s passwords synchronized across their servers.

I’d found the treasure — getting plain-text versions of every user’s passwords is quite an achievement!

Looking at their script, I gained insight into how the process worked. The script would be invoked on demand whenever necessary and it would then carry out its job without any interaction required from the user running it.

But where was it getting the passwords from? MySQL.

While quite popular today, MySQL was still fairly new at the time and, I’ll confess, I didn’t know much about it. Luckily, I had a couple of PCs at home running Linux and, soon afterwards, was intimately familiar with MySQL from an administrator perspective.

Armed with the MySQL credentials from their scripts, I connected to the MySQL server and started looking around. I had already found the proverbial pot of gold, but I was about to find another pot as well.

Like pretty much every other ISP (at the time), this one had a large infrastructure running — almost exclusively — Cisco gear. And, like pretty much every other ISP, they used SNMP (mostly via homebrew Perl scripts) to manage it.

The credentials that their custom script used to access their password database in MySQL had superuser privileges on the database server. This script that only needed to read data from rows in a single table in a single database actually had full permissions to every database in existence.

This oversight granted me access to a separate database on the MySQL server — the one where they stored their per-device SNMP community strings. Amazingly, in stark contrast to violating every other best practice in existence, every Cisco device on their network used separate read-only and read-write community strings. Unfortunately for them, I was staring at all of ‘em.

I don’t recall the name now but there existed a program which, given an SNMP read-write community string, would quickly download the running configuration off of Cisco devices and save it locally (I’m sure that, today, there are probably hundreds of these). Armed with valid user credentials to every server on their network, information gathering wasn’t a problem. The zone files on their primary DNS server, for example, were well commented and described what every device was.

A short time later, I was reading through the running configurations of their core infrastructure devices and, armed with the read-write community strings, I could have made any changes I wanted to.

Summary

In a short period of time, I had escalated my privileges from those of a normal user to the superuser.

Not only had I gained root access on one of their servers, but I had also acquired the plain-text passwords of every single customer. None of us customers could SSH into any server except for alpha, but alice and bob could. Equipped with their passwords, now I could too.

That local root vulnerability I mentioned? Yep, it was present on every other Red Hat Linux server they were running too.

I had root access to their shared hosting servers, which hosted hundreds of customer websites.

I had access to the stored e-mail of every single customer — and the system administrators themselves.

Last, I had administrative access to every switch and router on their network.

Pwned? Fuck yeah.

Note: Next week I’ll post a “shit they should have done differently” article, explaining how following best security practices could have mitigated my attempts to compromise their environment.

Hot on the heels of the new Arista Networks 7150S series top-of-rack switches that offer latencies as low as 350 nanoseconds, Cisco came out with an announcement of their own, shattering the land speed record in the process.

The newest addition to the Nexus line of high-performance datacenter switches, the Cisco Nexus 3548 is a top-of-rack switch that crams 48 SFP+ ports of up to 10 GbE goodness into a 1RU form factor.

“Industry-standard latency is 380+ nanoseconds. With the 3548, we are now able to break the 200 nanosecond barrier, which is a 60% performance improvement.” –Rajan Panchanathan, director of product management

The Cisco Nexus 3548 is the first of the Nexus line of switches to incorporates Cisco’s new Algorithm Boost (“Algo Boost”) Technology in custom-built ASICs, bringing port-to-port latency down to the 190 nanosecond range, even when forwarding frames OVER NINE THOUSAAAAAAAAAAND bytes in size.

Targeted at High Frequency Trading (HFT), High Performance Computing (HPC), and “Big Data” environments, and seemingly positioned to compete directly with Arista’s 7150S series, the Cisco Nexus 3548 comes equipped with a 4,000 ACL TCAM and 18 MB packet buffers.

In addition to supporting 10 GbE L2/L3 at line-rate across all 48 ports simultaneously and integrated Network Address Translation (NAT), the 3548 also supports IEEE 1588, the Precision Time Protocol, and the ability to combine four physical 10 GbE ports into a single 40 GbE interface (this is not your typical 4x10GbE link aggregation group).

The Nexus 3548 is officially being announced at 5 a.m. PDT (noon GMT) today. Pricing information was not available from Cisco.

Full disclosure: I’ve been a guest at the corporate offices of both Arista and Cisco and covered their products previously as part of the Gestalt IT Tech Field Day events, which both companies have sponsored.

If you communicate with me via e-mail, IRC, or Twitter, you’d know that I haven’t really left. A question recently asked on the Evil Routers Facebook page, however, made me realize that I really need to start hanging out in the blogosphere again:

Indeed, the last post here on Evil Routers, “IPv6 — It’s Not An Option“, which was simply a cheap shot at some Marketing folks, was published just over three months ago. My last “real” post, explaining a Cisco ASA error message, is now nearly five months old. That is, by far, the longest hiatus in Evil Routers’ history.

The reason for this is quite simple: it’s been a beautiful (albeit abnormally hot) summer here in the midwest and, because of that, I’ve devoted much more time to the first four items in the title of this post, at the expense of the last one.

   

Seriously, though, when my options are to a) stay at home and write up some articles or b) go out riding the bike or spend the weekend on the lake, it really isn’t a tough decision to make! I hope that you’ll forgive me. =)

Anyway, when I started my current job, I was working mostly on the VAR/MSP side of the house. In the last six months or so, I’ve mostly been working on the ISP side of things. We have an existing network to maintain, of course, but my major project is, basically, designing and building out a completely new network encompassing our 50+ sites.

Our original network started out small, of course, but grew rapidly. Unfortunately, the folks who built it out didn’t have any real-world experience designing or managing networks and mostly learned as they went along. This led to several decisions that, well, let’s just say they weren’t the best decisions that have ever been made in the history of networking decisions. Whenever I started in this role, we were beginning to experience major issues because of those decisions.

The company officers — my bosses — are businessmen, of course. One has a good technical background, in general, but the other is “all business”. Fortunately, I’ve been able to explain the issues that were occurring and we all agreed that we needed to “start from scratch”, in effect, and do things right.

Luckily, right before I came on board, they had procured fiber to a new site that was going to be built — our new “datacenter”. This has allowed us to build the new network in parallel to the existing one, at a comfortable place, while making sure that the network is properly designed.

Several months ago, we installed the firewall, deployed the servers, and lit up the fiber at our new datacenter. Shortly thereafter, we brought a completely new site online and, I’m proud to say, the network has been extremely stable and much more reliable than the “old network”.

We’re making good progress on this project and slowly have been moving our POPs (and customers, of course) over from the “old network” to the “new network”. Redundancy is an integral part of the design and my bosses are starting to see the value of a properly designed network. As we expand our offerings to include things such as Voice-over-IP (VoIP), a reliable, stable network is more important than ever (not to mention that our customers will leave us if we can’t provide them with a reliable connection). Eventually, everything will be switched over to the “new network”, we’ll redo a few things on what’s left of the “old network”, and then connect the two together to provide even more redundancy.

Working on this project has, as you might guess, provided me with the inspiration for several networking-related articles. I already have some drafts written up, but they need finished and tweaked a bit before they can be published.

What you might not have guessed, however, is that I also have ideas for many articles that aren’t strictly networking focused. Many, many years ago, before Evil Routers ever existed, my blog lived at a different domain — linuxwiz.net. If you’ve met me in person, you probably already know that I’m a huge fan of Linux, having been introduced to it in 1996. If I have a server to deploy and running Linux on it is an option, I’ll choose Linux about 90% of the time (the remaining 10% is split roughly 50/50 between FreeBSD and Microsoft Windows).

I mention this because I’ve been debating whether to publish any Linux-focused articles here on Evil Routers. I thought about resurrecting the linuxwiz.net domain (which I still own) and posting them there, but I’d rather not have yet another web site to maintain. Ultimately, I decided that there’s a lot of overlap between the “network engineer” and “Linux sysadmin” realms and came to the decision that I’ll post those articles here.

To give you an idea of the types of articles that are in the pipeline, remember that I am, in effect, building a (wireless) internet service provider network from scratch and I’m also rebuilding the entire server infrastructure that goes along with it. Thus, you can expect articles on everything from BGP (transit customer, transit provider, and peering), OSPF (what we run internally), spanning tree, “non-mainstream” networking devices (i.e. Mikrotik), deploying Linux servers (CentOS and Debian, almost exclusively) via kickstart to make things easy, using puppet to manage those same servers, and installing, configuring, and maintaining various server software — Apache, BIND, FreeRADIUS, MySQL, OpenLDAP, Postfix, PowerDNS, and many more — on the Linux platform. I am a huge proponent of anonymity and privacy as well, and I’ve been lucky enough to have some major influence on our policies in that regard. You’ll hear about that too.

This article ended up more long-winded than I planned on but, for those of you who’ve been fans of the site, I wanted to provide an explanation for my absence and let you know that I’ll soon be serving up fresh, new content for your enjoyment once again.

***

On a side note, Evil Routers passed the four year mark this past weekend and I want to thank all of you for helping me make the site into what it is today. When it first started, I didn’t expect that the number of RSS subscribers would ever even come close to the three-digit mark. As of today, there are over 1,700 of you and the site has served up well over one million views! That absolutely blows my mind.

In addition, I certainly never would have expected that my writings here would result in me getting to visit the campuses of Cisco, HP, Juniper, and the many other companies that I’ve since been to. I didn’t even hold a single networking certification when Evil Routers was born (Network+ doesn’t count) — yet, since then, I’ve been fortunate enough to meet, in real life, some of the biggest names in the networking industry.

My sincere thanks goes out to each of you who I’ve interacted with in any way over the last four years. Here’s to the next four!

Those of you who know me in real life are probably quite aware of just how much of a dislike I have for sales and marketing people. That’s why whenever they make themselves look like idiots, I like to acknowledge it and recognize them for it. Besides, marketing folks tend to believe that any publicity is good publicity so I’m certain that they don’t mind me pointing out just how retarded they so often are.

Today’s winner is BT Diamond IP (the missing hyperlink is intentional), who obviously bought my e-mail address from some other assholes a while back. With the World IPv6 Launch now less than a month away, somebody in marketing at BT Diamond IP apparently decided to try to capitalize on the occasion. Thus, they’ve put together an event where they’ll graciously give us, among other things, “recommendations from our experts on how to address” “the key issues that we are all facing as we move to IPv6″ as well as explain “how BT Diamond IP can help you through your migration plan”. How nice of them.

Claire Ritchie, who probably wouldn’t know an IPv6 address from a type 5 LSA, is the “Head of Network Services Marketing” for BT Global Services and the individual responsible for the most recent spam e-mail to land in my inbox. The e-mail attempts to catch our attention with a large graphic at the top by asking “Are you ready for IPv6?” and then declaring quite loudly that “IPv6 — it’s not an option”.

Later in Claire’s e-mail, we’re reassured that BT Diamond IP “has vast experience on a global scale …”.

I was almost impressed. Even this late in the game, it’s quite uncommon for a large enterprise (such as BT) to have even started deploying IPv6. I mean, to have “vast experience on a global scale” and be in a position to offer up “experts” and IPv6 deployment advice, they must certainly be way ahead of the curve and miles ahead of their competitors, right?

Yeah, um, about that… well, let’s use the UNIX dig command to find out!

$ dig +short btdiamondip.com. AAAA
$ dig +short www.btdiamondip.com. AAAA

Hmm, that’s weird. dig doesn’t return any output. Normally it would return the associated IPv6 addresses (the “AAAA” resource records). Perhaps I’m a little rusty or maybe I’m just straight up using it wrong — let’s try the host command instead. It will either return the records we’re looking for or explicitly tell us that they simply don’t exist.

$ host -t AAAA btdiamondip.com.
btdiamondip.com has no AAAA record
$ host -t AAAA www.btdiamondip.com.
www.btdiamondip.com has no AAAA record

No AAAA resource record for their website? That’s not quite what I expected. Hmm, well, certainly their e-mail system supports IPv6! Claire’s e-mail address is claire.ritchie@bt.com so let’s check what mail servers handle e-mail for the bt.com domain.

$ host -t MX bt.com.
bt.com mail is handled by 10 smtp61.intersmtp.com.
bt.com mail is handled by 10 smtp62.intersmtp.com.
bt.com mail is handled by 10 smtp63.intersmtp.com.
bt.com mail is handled by 10 smtp64.intersmtp.com.
bt.com mail is handled by 10 smtpe1.intersmtp.com.

Wow, five MX hosts — with that many set up it’s obvious they want you to be able to e-mail them and considering the “vast experience on a global scale” that BT has, I bet they’re all running IPv6!

$ for S in `host -t MX bt.com.|awk '{ print $7 }'`
> do
> host -t AAAA $S
> done
smtp64.intersmtp.com has no AAAA record
smtpe1.intersmtp.com has no AAAA record
smtp61.intersmtp.com has no AAAA record
smtp62.intersmtp.com has no AAAA record
smtp63.intersmtp.com has no AAAA record

Interesting, I guess they were right after all. If you’re on an IPv6-only network or want to access their website or send them e-mail over IPv6, well…

Yep, BT, the company with “vast experience on a global scale”, the ones who are offering up recommendations from their “experts”, the ones who “can help you through your migration plan”, are not even running IPv6 on their public network. Since most of the real experts that I’ve talked to generally recommend beginning your IPv6 deployment with your public-facing servers, it’s probably safe to assume that they’re not running IPv6 internally either. That doesn’t make me feel very confident about their services, what about you?

And yet, at the same time, evilrouters.net has been accessible over IPv6 since last year.

For spamming me, BT, I sentence you to 40 lashes with my clue-by-four.

Now, if you’re interested in IPv6 advice from guys who really know what they’re talking about, check out Ivan Pepelnjak’s IPv6 webinars and Brandon Carroll’s IPv6 video tutorials.

Note: I absolutely hate vendors cold calls and unsolicited spam and the sales and marketing people behind them (and, yes, I absolutely do keep a “do not ever buy from” list). While you may think that I’ve written this article just to be or because I am an asshole, rest assured that it isn’t true! I used to think I was an asshole too, but Mrs. Y informed me at Net Field Day 3 “you’re not really an asshole” and, although most of my ex-girlfriends would disagree, I’m not going to argue with her. If you’re smart, you won’t either.

Recently, while upgrading a Cisco ASA that belongs to a customer and hasn’t been touched in a long time, the following took place:

ciscoasa# copy tftp flash

Address or name of remote host []? 10.10.10.42

Source filename []? asa842-k8.bin

Destination filename [asa842-k8.bin]?

Accessing tftp://10.10.10.42/asa842-k8.bin...!!!!!!!!!!!!!!!!!!!!!!!

...

Writing file disk0:/asa842-k8.bin...

%Error opening disk0:/asa842-k8.bin (Read-only file system)
ciscoasa#

Being fairly fluent in teh Lunix, I’ve seen a filesystem get mounted read-only several times — mostly when a server hasn’t been shut down correctly. But on an ASA? A read-only filesystem?

I thought to myself, “How the hell do you fix this?” and then, almost immediately, “I wonder if I can fsck the filesystem?”. And, as it turns out, I could.

fsck is short for “filesystem check” and is similar to chkdsk from the DOS/Windows world. I decided to give it a shot.

ciscoasa# fsck disk0
dosfsck 2.11, 12 Mar 2005, FAT32, LFN
Reclaimed 26 unused clusters (106496 bytes) in 3 chains.
Performing changes.
/dev/hda1: 49 files, 10065/31033 clusters

fsck of disk0: complete
ciscoasa#

Was it really that easy? There was only one way to find out.

ciscoasa# copy tftp flash

Address or name of remote host []? 10.10.10.42

Source filename []? asa842-k8.bin

Destination filename [asa842-k8.bin]?

Accessing tftp://10.10.10.42/asa842-k8.bin...!!!!!!!!!!!!!!!!!!!!!!!

...

Writing file disk0:/asa842-k8.bin...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
...
25159680 bytes copied in 27.350 secs (931840 bytes/sec)
ciscoasa#