Just a quick post with the oneliner of the day.

When you are debugging catalog compilation issues or other puppet performance issues in general, it is good to know exactly which catalogs are slow to compile. Knowing which catalogs are substantially slower than others allows you to focus on those catalogs and the modules they contain.

Just a quick post with the oneliner of the day.

Scenario: after a bacula director restart a couple of jobs were stuck on the FD with message:

1
2
3
4
5
6

Running Jobs:
Writing: Incremental Backup job node.cluster.company.com JobId=8702 Volume=""
    pool="bacula.director.company.com:pool:default.incremental" device="DefaultFileStorage" (/mnt/bacula/default)
    spooling=0 despooling=0 despool_wait=0
    Files=0 Bytes=0 AveBytes/sec=0 LastBytes/sec=0
FDSocket closed

There were a couple of these jobs that were stuck, preventing all other jobs from running, because those were waiting for a free slot on the FD.

I’ve been using bacula for a couple of months now in conjunction with puppet to make automated backups of all servers that are managed by puppet. My bacula setup labels a volume for every job it runs with a unique name:

1

Label Format = "${Job}.${Year}${Month:p/2/0/r}${Day:p/2/0/r}.${Hour:p/2/0/r}${Minute:p/2/0/r}"

These volumes are automatically purged once the retention of all files contained on the volume expires (which is configured per-pool). Due to the unique names however, the volumes cannot be recycled. The result of this is that the volumes that have been marked as purged in the catalog remain as-is on the disk. After some time this ultimately resulted in a full disk, thus halting all backups performed on that pool. Not good. Not good at all.

Running a dedicated Minecraft server can be a challenging job. You have to find a balance between performance and usability using “server software” that doesn’t seem to be designed to provide for long running, resilient services.

Being a first-time Minecraft server operator I had to tackle various challenges in order to come up with a way to provide a stable and reliable service to my players. The following article is a recollection of the things I implemented and scripts I wrote in order to run a Minecraft 1.8 server. The scripts mentioned are specific to Gentoo Linux, but could also be used on most other Linux flavours, albeit with some modifications to match that platform’s init.d scripts.

If you’re using tmux or screen as a wrapper for your terminal sessions (such as when using the Text Triumvirate), chances are that you are presented with the following error when you invoke a command that uses 256 color mode (such as multitail or htop):

Error opening terminal: screen-256color.

The issue

I experienced a seemingly weird issue with a freshly installed CentOS server today.

SSH Public key authentication was correctly set up; The sshd_config was properly configured and a ~/.ssh/authorized_keys was present with the correct rights and verified correct contents (as the file was yanked from another, working, server with scp).

All attempts to connect to the machine using key authentication silently failed however.

Letting Ubuntu pick the apt-get mirror closest to you is pretty easy.

I suffered from a Munin version 2.0.10 installation that refused to update the majority of the graphs. Only the first two of a long list were being updated, the rest all ‘hung’ at the same moment.

After a little investigating, the problem surfaced:

1
2
3

$ su - munin --shell=/bin/bash munin-cron
File is not a perl storable at blib/lib/Storable.pm (autosplit into blib/lib/auto/Storable/fd_retrieve.al) line 398, at /usr/lib64/perl5/vendor_perl/5.12.4/Munin/Master/Utils.pm line 362
File is not a perl storable at blib/lib/Storable.pm (autosplit into blib/lib/auto/Storable/fd_retrieve.al) line 398, at /usr/lib64/perl5/vendor_perl/5.12.4/Munin/Master/Utils.pm line 362

I started out by fixing all items the munin-check script suggested, which is always a good starting point.

Keeping your Gentoo Linux server up to date isn’t as straightforward as let’s say an Ubuntu box, where you would just run $ apt-get update && apt-get upgrade && apt-get clean for example.

Gentoo is far too flexible for a one size fits all approach. The commands outlined below come pretty close for daily use though:

On my servers I use the nifty program Fail2Ban to perform logbased automatic firewalling of ‘bad’ ip’s.

The idea behind this is easy: Some IP performs an action I don’t approve of. This can be any number of things, e.g. requesting pages in Apache that are commonly accessed by bots and/or scanners, or trying to log in to SSH with accounts that do not exist on the system. This bad behavior gets logged, and Fail2Ban keeps tabs on those logs, and using a number of rules it determines if a host is ‘bad’ enough to temporarily or permanently ban all access to the server. It does so by adding a few chains to Iptables (one for each thing it checks for), and dynamically adding/removing IP’s to/from these chains.

This all works perfectly. However, there’s one issue; When Iptables gets reloaded, it restores its default rules, removing the Fail2Ban chains and all the rules they contain, even if the ip’s in the chain were marked as permanent.