Sat, 31 Dec 2005

Reboot after kernel panic in Linux

Sometimes it’s better to reboot after a kernel panic instead of dumping to console and waiting for the admin to go to the colocation facility and reboot the machine. The Linux sysctl kernel.panic lets you specify the number of seconds after a panic until a hard reboot. For example, to reboot 5 seconds after a panic, put this line into /etc/sysctl.conf:

kernel.panic = 5

To load your change, run sysctl -p as root.

[/config/linux-kernel] permanent link

Mon, 19 Sep 2005

MySQL InnoDB Defrag HOWTO

So you’re running MySQL and you have a lot of data in an InnoDB. Over time, your InnoDB files will fragment just like an old FAT based Windows box. Unlike that Windows box MySQL doesn’t ship with a defrag utility so you have to brute force it. The fragmentation problem gets much worse when you want to drop tables; the free space that generates is generally scattered all throughout the InnoDB which causes performance to get worse and worse as writes occur. The following procedure will demonstrate how to delete large tables and defrag your database with a minimal amount of downtime.

This HOWTO assumes that you have a master/slave replication setup; if you don’t you’ll need to incur enough downtime to dump and reload your database. This can take several hours for a 20GB database, and therefor isn’t practical for many organizations. This HOWTO also assumes that you’re using MySQL 4.0.24 or higher (it should work fine with 4.1, too).

We’re going to call your master database master, and your replicant database slave in this document.

On master, create a new database to hold your garbage tables called “trash”:

master# mysqladmin -u root create trash

Then, rename all of your tables that you would otherwise DROP into trash:

master# mysql -u root -e "RENAME TABLE mydb.mytable TO trash.mytable"

The idea here is that this will get them out of your database (so you can verify that you don’t actually need them), but doesn’t free up the InnoDB blocks, so your master DB performance won’t take a hit during the defrag process.

Now, we move over to slave, and do the rest of the work. Make sure that nothing is referencing your slave, as it will have to be offline for the duration of the dump and reload.

On your slave:

slave# mysql -u root -e "STOP SLAVE IO_THREAD"

This stops the slave from pulling binlog data from master but allows it to complete processing all of the binlogs that it has already pulled.

Now, run:

slave# mysql -u root -e "SHOW PROCESSLIST"

until the state for the SQL thread says “Has read all relay log; waiting for the I/O slave thread to update it”.

Now, run:

slave# mysql -u root -e "SHOW SLAVE STATUS\G"

and note the Master_Log_File and Read_Master_Log_Pos values; you’ll need them later.

Now shutdown mysql on slave' (/etc/init.d/mysql stop on Debian) and edit the my.cnf. Addskip-networking(to minimize the chance of someone accidentaly connecting to your DB while you're working on it) andskip-slave-start` (so it doesn’t automatically resume replication when you bring it back online).

Start mysql up again, and prepare to dump your data:

For every database except for trash, mysqldump it to somewhere safe:

slave# mysqldump -u root --opt --verbose --databases db1 db2 db3 ... > /var/tmp/mysql-dump.sql

This step will take a very long time, potentially.

Once the dump completes, shut mysql down again, and move the mysql data directory out of the way:

slave# mv /var/lib/mysql /var/lib/mysql-fragmented

Then create an empty mysql datadir, and populate the `mysql’ database (so you have users and passwords):

slave# mkdir /var/lib/mysql
slave# chown mysql:mysql /var/lib/mysql
slave# chmod 0770 /var/lib/mysql
slave# cp -av /var/lib/mysql-fragmented/mysql /var/lib/mysql

Also, if you run your `slave’ with read-only set, you should comment it out now.

Now, start up mysql. It will create fresh, empty InnoDB files (ibdata* and ib_logfile*). This can also take a long time, especially if you have a large-ish initial InnoDB datafile allocation (some of our servers pre-allocate 100GB of space for the InnoDB).

Now, create an empty trash database:

slave# mysqladmin -u root create trash

And reload your database:

slave# mysql -u root < /var/tmp/mysql-dump.sql

This will also take a very long time. At this point, `slave’ is fully loaded and defragged.

Now edit your my.cnf, and comment out skip-networking, and uncomment read-only.

Restart mysql.

Now, configure your slave:

slave# mysql -u root -e "CHANGE MASTER TO MASTER_HOST='master', MASTER_USER='replicator_user', MASTER_PASSWORD='s3cr3tp@ssw0rd', MASTER_LOG_FILE='Master_Log_File', MASTER_LOG_POS=Read_Master_Log_Pos"

where Master_Log_File and Read_Master_Log_Pos are the values you wrote down from before.

Now start your slave:

slave# mysql -u root -e "START SLAVE"

Verify that things are working:

slave# mysql -u root -e "SHOW SLAVE STATUS\G"

Now your slave is defragged assuming it replicates okay. If it doesn’t, it’s likely you’ve missed a database that you’re actually still using. Restore your backup (/var/lib/mysql-fragmented), let it catch up with your master, and then try again.

Do one final restart to make sure that replication starts up again automatically, and then you’re done. Well, almost. At this point, your master is still fragmented. So, the next step is to switch your MySQL master from master to slave, which is described in the yet to be written MySQL-Master-Slave-Switch-HOWTO.

[/config/mysql] permanent link

Fri, 26 Aug 2005

rng-tools

Many modern computers come with hardware random number generators that are supported by newer Linux kernels. They generally are available via a file in /dev, have varying speeds, and are occasionally buggy. However, when they work properly they can help out servers quite a bit by providing a steady source of entropy that would otherwise be lacking. If you’ve ever tried to generate a gpg key on a remote server and have it hang on you, you’ve run out of entropy.

rng-tools is a collection of tools that help make it easy to use the hardware random number generator safely. At the core of the toolkit is rngd, a daemon which reads from the hardware random device, tests the bits read for randomness, and then feeds them into the kernel’s normal entropy pool via /dev/random. Like all great utilities, it’s pretty much fire and forget; once installed, it will figure out if you have a useable hardware random device and start to use it or it will exit with a log message telling you that you are out of luck.

[/config/rng-tools] permanent link

Wed, 01 Jun 2005

irqbalance

Small but useful bit of info. By default, Linux handles all interrupts on the first CPU on SMP systems (you can verify this by looking at /proc/interrupts). irqbalance is a small daemon that periodically balances interrupts across all CPUs on the system. This provides a bit of a performance boost, especially if you are doing a lot of disk and network I/O.

It requires no configuration and consumes a tiny amount of memory; just install it and let it run.

[/config/irqbalance] permanent link

Mon, 24 Jan 2005

bind9 init script update

I’ve been re-visiting my bind9 init scripts now that some time has passed and I’ve had them in production for about 6 months or so. I’ve changed the chroot to use a tmpfs filesystem instead of an ext2 loopback; this makes some error cases easier to handle, as well as simplifying the script by quite a bit.

The downside to using tmpfs is that all changes made by named to the zone files are lost when the machine restarts; this is not a safe config to use if you are using Dynamic DNS.

The new init script and default file:

[/config/bind9] permanent link

Sun, 24 Oct 2004

smartmontools

smartd, from the smartmontools package, is one of those things that you don’t really want to think about, but you probably should (at least a little bit). It monitors the health of your computer’s disks using the S.M.A.R.T. system built into most modern ATA and SCSI hard disks. It can give you advanced warning of your disks failing, which can be just enough time to get all of your data off of them.

The Debian default config tells smartd to scan for all possible ATA and SCSI disks, and to monitor the ones that it finds. Despite the comment in the config file that “most users should… explicitly list the devices that they wish to monitor”, smartd seems to do a good job of not getting confused by CD-ROM drives, RAID controllers, etc., so letting it do a device scan just simplifies its configuration.

By default, smartd will just log what it finds into syslog, but it can do more. I like to have it email root if it predicts a drive failure, and to run a set of SMART self tests. The SMART tests are non-intrusive and can be safely run while the system is up.

The configuration file is sadly a bit cryptic, so you can can a copy from here:

The smartmontools package also includes a commandline utility, smartctl. The most useful commandline argument is --all, which will query and display all of the available SMART readouts from a device, like so:

smartctl --all /dev/hda

It will generate a few screens worth of output, so dumping it into a file or less is probably what you want.

[/config/smartmontools] permanent link

Tue, 19 Oct 2004

microcode.ctl

Intel CPUs have a feature which allows them to be updated with bug fixes on the fly. Now, most people probably could care less if they have a fully updated CPU or not, so long as it works. However, then it comes out that the CPU doesn’t always work (scroll down to the x86 CPU detection section).

The Debian [microcode.ctl] package installs the userland microcode loader and downloads the current microcode from Debian’s servers. It’s worth checking http://www.urbanmyth.com/microcode/ for updates every so often; they aren’t annnounced very widely. If there is a new one, just download it, bzunzip2 it, and put it in /usr/share/misc/microcode.dat. Then run /etc/init.d/microcode.ctl start, and your CPU will be fully patched.

[/config/microcode.ctl] permanent link

Wed, 13 Oct 2004

Watchdog

I had a server in our colocation facility crash a couple of days ago due to an out of memory condition. Sadly, Linux’s behavior when it runs out of memory appears to consist of putting fingers in ears and screaming “kernel: __alloc_pages: 0-order allocation failed” to the console over and over, as opposed to doing something useful like letting me log in via ssh.

Enter watchdog. All that watchdog does is open /dev/watchdog, and write a single byte into it every 10 seconds. So long as it does this, nothing happens. However, should the watchdog stop writing that byte for over a minute, the system is reset, hard (in other words, just the thing to hack around stupid memory behavior, and hopefully removing the need for me to go to the colo).

Unfortunately for me, I don’t have the money for server hardware that supports watchdog hardware, so I’m stuck with the Software Watchdog (supplied by the softdog kernel module). The softdog can help in some situations, but won’t do any good in the event of a true hardware lockup when interrupts are scrambled. Still, it’s better than nothing.

From my testing, the watchdog appears to do no harm, and consumes minimal system resources. I’m setting it up on all of my systems, Just In Case (tm).

[/config/watchdog] permanent link

Wed, 06 Oct 2004

Config

apt-spy is a nice utility to help automate configuration of your APT sources.list. It is pretty easy to use.

First, you need to update the mirrors list:

sudo apt-spy update

Once that completes, you then generate a sources.list, either via region:

apt-spy -d sarge -a North-America -o /tmp/sources.list

or by country code:

apt-spy -d sarge -s US -o /tmp/sources.list

You can get a list of known Region and Country Codes from /etc/apt-spy.conf.

Generating the sources.list will take a while; possibly over an hour. This is definately something that you want to run as a background task while you’re busy doing something else.

Once you’ve got your sources.list, you will probably want to edit it to add the appropriate security updates source:

deb http://security.debian.org/ sarge/updates main contrib non-free

You may also want to add the contrib and non-free sections to your generated apt sources, as apt-spy does not do this by default. Once you’re happy with your sources.list, copy it to /etc/apt/sources.list.

Because of the load generated by apt-spy on the Debian mirrors, you should not have this run automatically from cron to select the fastest mirror; instead, just re-run it if you notice that your current mirror is overloaded.

[/config/apt-spy] permanent link

Sun, 03 Oct 2004

portmap chroot

My chroot patch to portmap is complete! To use it, you can either download the patch and build your own portmap, or grab this deb, which is built for Debian sarge. The patch should apply cleanly to either the woody or sarge versions (and possibly others).

Once you’ve got a patched portmap installed, you can download the init and default files:

The init script sets up a tiny loopback chroot, and copies the only file that portmap needs (/etc/localtime) into it.

You can use the defaults file to change the chroot location or to disable the use of the loopback mount.

[/config/portmap] permanent link

Fri, 24 Sep 2004

Defensive fuckery

Let’s pretend that some fucko has setup a zone and pointed it at your nameserver without asking you first. Let’s say that it pisses you off. What to do?

Play defense.

First, setup a zonefile that looks like this:

    $TTL 7D
    @           SOA localhost hostmaster.localhost 42 3600 1800 604800 3600
                NS  localhost
    localhost   A   127.0.0.1
    *           MX  0 localhost
                A   127.0.0.1

and save it into db.smackdown.

Note that this zone is general purpose; it doesn’t matter what the zone that’s being pointed at you is called. Then, in the named.conf, add something like this:

    zone "fucko.com"
    {
        type master;
        file "master/db.smackdown";
    };

That’s all you have to do. Any query that anyone sends to your server for that zone will get bounces to their local machine. Eventually, the zone owner will realize that this is hurting them, and they will (hopefully) stop. If not, you can at least rest easy, knowing that you’re fucking with the foolish.

Again, this was Paul Vixie’s idea; I just wanted to keep a copy.

[/config/bind9] permanent link

Tue, 21 Sep 2004

TTLs

The BIND master Paul Vixie gave some good advice on NANOG about default TTLs for DNS records that seemed worth remembering here.

A default TTL of 1 or 2 hours is plenty for standard internet traffic; you’ll get the caching benefit for the immediate repeat visitor, but you probably won’t see much of a caching benefit beyond that. However, you should increase the TTLs of your NS records to 2 days. This helps reduce DNS load, since your NS records are what keeps people from chasing all the way up to the root servers.

People used to throw around a week or so as a default TTL, but since it doesn’t really gain you much performance-wise there’s little reason to leave the TTL that high. It only limits your options for changing that record in case of emergency.

[/config/bind9] permanent link

Thu, 19 Aug 2004

forward first

Found out yesterday that if you configure BIND9’s named to use forwarders, it will timeout and fail if it can’t reach any of the forwarders. I guess if you’re behind a firewall this may be what you want, but in general I’d imagine that you’d want your local named to try the lookup if it can’t get to the forwarders as a last resort. Adding a forward first; clause to your options stanza causes named to try the forwarders first, but if they don’t respond in time it will perform the DNS lookup itself.

I’ve modified /etc/bind/named.conf to do this, as I think it’s a better default behavior.

[/config/bind9] permanent link

Tue, 25 May 2004

mod_gzip with mod_ssl

Unfortunately, there’s an incompatibility with mod_ssl and mod_gzip when running on Apache 1.3. In order to get them to work, you need to setup the proxy hack.

The general idea is that you split your SSL host into two virtual hosts: the frontend does the SSL, and the backend virtual host that does the compression.

First, load the Apache proxy module:

LoadModule proxy_module /usr/lib/apache/1.3/libproxy.so

It should go after your auth modules, but prior to ssl or gzip.

Tell Apache to listen on a high port:

Listen 10443

Setup the proxied vhost. :

<VirtualHost _default_:10443>
    ServerName secure.example.com
    # Whatever other config you need for this host...
</VirtualHost>

Then, setup the front-end. You should apply any mod_rewrite rules here, not in the backend. :

<VirtualHost 10.0.0.1:443>
    ServerName secure.example.com
    SSLEngine On
    # Whatever other SSL config you need for this vhost.
    ProxyRequests Off
    ProxyPass / http://secure.example.com:10443/
    ProxyPassReverse / http://secure.example.com:10443/
    mod_gzip_on No
</VirtualHost>

It works pretty well, although you will see two entries in your access log per HTTPS request. You can strip the duplicates by removing entries from your webserver’s IP address prior to log processing.

Note that the backend virtual host should not be accessible externally, so you should use a per-host firewall or set Apache access restrictions on it to limit access to localhost only.

[/config/libapache-mod-gzip] permanent link

Fri, 21 May 2004

mod_gzip Config

mod_gzip is an Apache httpd module that compresses HTTP traffic on the fly. This can lead to significant bandwidth savings, and can make your web pages appear to load faster. On modern systems, it consumes a small amount of CPU and memory resources. In other words, it’s a win.

Unfortunately, due to various browser and Apache module bugs, you can’t just slap it in your server and call it a day.

To begin, first you must load the module. It should be loaded after all of your other modules. :

LoadModule gzip_module /usr/lib/apache/1.3/mod_gzip.so

You should then make a new LogFormat that contains the gzip compression stats (so that you can verify that it’s working, and so you can brag about how much bandwidth you’re saving). :

LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" 
\"%{User-Agent}i\" \"mod_gzip %{mod_gzip_result}n 
In:%{mod_gzip_input_size}n Out:%{mod_gzip_output_size}n\"" 
vhost-combined-mod_gzip

If your log processing software is cool enough to handle the additional fields, then you can use this for your access.log, otherwise you’ll need to make a new CustomLog.

You then need to add a global mod_gzip configuration section:

<IfModule mod_gzip.c>
    # Turn on mod_gzip processing by default.
    mod_gzip_on Yes

    # If a file is smaller than 300 bytes, just send it.
    mod_gzip_minimum_file_size 300

    # If a file is larger than 10MB, just send it.
    mod_gzip_maximum_file_size 10485760

    # If a file is smaller than 1MB, do the compress in memory.
    mod_gzip_maximum_inmem_size 102400

    # This is only useful for debugging
    mod_gzip_keep_workfiles No

    # Let mod_gzip check to see if there's already a static compressed
    # version of the resource.  If there is, just send that one instead of
    # re-compressing the uncompressed version.
    mod_gzip_can_negotiate Yes

    # De-chunk output from CGIs and other Apache modules so that we can
    # compress it.  It's better if the CGIs that you want to compress don't
    # chunk their output in the first place, though.
    mod_gzip_dechunk Yes

    # Now, we teach mod_gzip when to compress or not.

    # NO: broken browsers which request gzipped content but then can't
    # handle it.
    mod_gzip_item_exclude reqheader "User-agent: Mozilla/4.0[678]"

    # YES: HTML docs
    mod_gzip_item_include file \.html$
    mod_gzip_item_include file \.htm$

    # YES: CGI scripts
    mod_gzip_item_include file \.cgi$
    mod_gzip_item_include handler cgi-script

    # YES: text files, Apache directory listings
    mod_gzip_item_include mime httpd/unix-directory
    mod_gzip_item_include mime text/

    # NO: images are already compressed
    mod_gzip_item_exclude mime image/

    # NO: Due to browser bugs, external CSS and JavaScript documents can't
    # be compressed, ever.
    mod_gzip_item_exclude file \.css$
    mod_gzip_item_exclude file \.js$
</IfModule>

That’s it; your Apache will now gzip documents on the fly if the browser requests them.

[/config/libapache-mod-gzip] permanent link

Four servers

Apparently the methods that NTP implementation use to detect bad time sources want 4 servers as sources. I guess that means that an ideal NTP site config would consist of 4 hosts that refer to upstream time (either other servers, or GSM/radio clocks/GPS sources), and that are peer with each other. The broadcast hosts then refer to those four as the site upstream NTP servers.

Another useful tip:

ntpq -p | grep --silent ^\*

will tell you if your local ntpd has latched onto a good upstream time source.

[/config/ntp] permanent link

Wed, 19 May 2004

dhcpd configuration gotchas!

I’ve spent most of the day fighting with the ISC dhcpd, and I’ve finally figured out why things weren’t working, and it’s kinda funny/wierd, so I’m sharing.

For those who don’t remember/never knew, the config file looks sort of like this:

default options and stuff;

subnet 10.0.0.0 netmask 255.255.255.0
{
    range 10.0.0.10 10.0.0.20;
    option routers 10.0.0.1;

    host foo
    {
        hardware ethernet de:ad:be:ef:00:00;
        fixed-address foo.example.com;
    }   

    group
    {
        filename "/pxelinux.0";
        next-server 10.0.0.2;
        host netbooter
        {
            ...
        }
    }
}

Looks fine, right? It even works, parses, dhcpd doesn’t complain at all.

However, it’s wrong.

Let’s say you want to have your group of netbooters use a different router. No problem, you think. I’ll just put an “option routers” statement in the group, and call it a day.

If you try to do this, dhcpd will return the router specified in the subnet that the client is on, regardless of any other options that may be in closer scope. It does this, of course, to spite you. “Foolish network admin, you thought DHCP would save time”.

After about an hour, I suddenly realized that nowhere in the dhcpd.conf manpage does it suggest that you can nest a group inside of a subnet stanza. And lo, if you move the group to the top level, it all Works As It Should (tm).

In other words, the above should look like this:

default options and stuff;

subnet 10.0.0.0 netmask 255.255.255.0
{   
    range 10.0.0.10 10.0.0.20;
    option routers 10.0.0.1;

    host foo
    {   
        hardware ethernet de:ad:be:ef:00:00;
        fixed-address foo.example.com;
    }
}
group
{   
    filename "/pxelinux.0";
    next-server 10.0.0.2;
    host netbooter
    {   
        ...
    }
}

Now, I can appreciate that I read the manpage wrong, and wrote the config file wrong, and I accept that as my fault. But why does dhcpd mostly work when you nest a group within a subnet? Why doesn’t it explode, spewing error messages, or issue a warning? It’s the worst possible software behavior case: it just happens to work most of the time, but for reasons that nobody can explain.

[/config/dhcp3-server] permanent link

Sun, 25 Apr 2004

BIND9 Config

The Debian sarge bind9 package now defaults to running named as a non-root user. This is great, and the bind9 maintaner rocks for taking the time to do this.

If you are not using sarge or a sarge bind9 backport, you need to create a user for named: adduser —system —home /var/cache/bind —group —disabled-password bind

Then, download and install this script and default file:

You should change the defaults in /etc/default/bind9 if you don’t want the chroot to be in /srv/bind/chroot. The init script takes care of building the chroot loopback device, mounting it, and creating the chroot directory structure inside of the loopback filesystem. It also copies your config files from /etc/bind into your chroot on start and reload. This means you don’t have to change any habits to take advantage of the chroot!

The chroot on a loopback is due to Linux (and Unix in general) using filesystems as security boundries. Traditionally, a secure chroot must be located on its own partition and filesystem. Normally, running a loopback incurs an undesireable amount of overhead, but named has such low disk IO requirements that the increased security is worth while. If you don’t have the Linux loopback device, or if you don’t want to use it for this, set bind_use_loopback="no" in /etc/default/bind9.

If you are running Red Hat, Stephen Gildea has a version of this init script available.

Occasionally the root name servers change. A running named will notice, but the db.root still needs to be kept up to date so that named will know about changes after a restart. Putting this script in your monthly cron will take care of it for you. You will need to install the dnsutils package.

This config is designed for an infrastructures.org scenerio. Every host runs a local copy of bind which is used as the local nameserver. The local bind uses the organization’s primary name servers as forwarders. The local bind can optionally use TSIG to communicate with the primaries which prevents DNS poisoning attacks, which can be handy when you want to store your Kerberos information in DNS. Note that every machine can act as a caching nameserver for Win32 or MacOS clients; this makes it easier to recover from some disasters (since you can just reconfigure clients to point at any of your Debian hosts for name service immediately). Obviously, your authoritative nameservers will not be using this configuration, but they shouldn’t be the same as your caching nameservers anyway.

[/config/bind9] permanent link

Tue, 30 Mar 2004

sysvinit config

By default, /etc/inittab has six virtual terminals active, which waste memory that could be used for something else. Besides; if you want virtual terminals, you can (and should) install and use screen, which works better anyway.

Yes, I know that the memory saved by doing this is only a meg or so, but I still have a couple of machines in use where an extra meg is useful.

[/config/sysvinit] permanent link

monit config

monit is a daemon watching daemon. It’s designed to monitor the status of services on the local host and restart them automatically if they should crash. If you’ve ever accidentally nuked sshd on a remote host, you’ll probably dig monit. I also use monit to automatically restart daemons that have been updated using a cluster distribution tool like systemimager.

The monit configuration that I use. This config is designed to restart daemons that have failed, and to watch for changes to local binaries and configuration files, restarting daemons if appropriate.

[/config/monit] permanent link

NTP Config

The guys over at JuiceCo have a great page on NTP configuration. Rather than reproducing their excellent work here, I suggest that you go there and check them out instead.

[/config/ntp] permanent link

BIND9 policy

[/config/bind9] permanent link

NTP policy

[/config/ntp] permanent link