Puppet: Calculating average catalog compilation times

 |  400 words — 2 minutes  |  puppet bash linux oneliner

Just a quick post with the oneliner of the day.

When you are debugging catalog compilation issues or other puppet performance issues in general, it is good to know exactly which catalogs are slow to compile. Knowing which catalogs are substantially slower than others allows you to focus on those catalogs and the modules they contain.

Bacula: Cancelling all jobs that are currently writing

 |  200 words — 1 minute  |  backup bacula linux oneliner

Just a quick post with the oneliner of the day.

Scenario: after a bacula director restart a couple of jobs were stuck on the FD with message:

Running Jobs:
Writing: Incremental Backup job JobId=8702 Volume=""
    pool="" device="DefaultFileStorage" (/mnt/bacula/default)
    spooling=0 despooling=0 despool_wait=0
    Files=0 Bytes=0 AveBytes/sec=0 LastBytes/sec=0
FDSocket closed

There were a couple of these jobs that were stuck, preventing all other jobs from running, because those were waiting for a free slot on the FD.

Bacula: Purging and deleting old volumes

 |  600 words — 3 minutes  |  bacula backup linux

I’ve been using bacula for a couple of months now in conjunction with puppet to make automated backups of all servers that are managed by puppet. My bacula setup labels a volume for every job it runs with a unique name:

Label Format = "${Job}.${Year}${Month:p/2/0/r}${Day:p/2/0/r}.${Hour:p/2/0/r}${Minute:p/2/0/r}"

These volumes are automatically purged once the retention of all files contained on the volume expires (which is configured per-pool). Due to the unique names however, the volumes cannot be recycled. The result of this is that the volumes that have been marked as purged in the catalog remain as-is on the disk. After some time this ultimately resulted in a full disk, thus halting all backups performed on that pool. Not good. Not good at all.

Gentoo: Running a Minecraft 1.8 server

 |  2300 words — 11 minutes  |  gentoo linux minecraft

Running a dedicated Minecraft server can be a challenging job. You have to find a balance between performance and usability using “server software” that doesn’t seem to be designed to provide for long running, resilient services.

Being a first-time Minecraft server operator I had to tackle various challenges in order to come up with a way to provide a stable and reliable service to my players. The following article is a recollection of the things I implemented and scripts I wrote in order to run a Minecraft 1.8 server. The scripts mentioned are specific to Gentoo Linux, but could also be used on most other Linux flavours, albeit with some modifications to match that platform’s init.d scripts.

SELinux: Allowing SSH public key authentication

 |  300 words — 2 minutes  |  SELinux SSH CentOS Linux

The issue

I experienced a seemingly weird issue with a freshly installed CentOS server today.

SSH Public key authentication was correctly set up; The sshd_config was properly configured and a ~/.ssh/authorized_keys was present with the correct rights and verified correct contents (as the file was yanked from another, working, server with scp).

All attempts to connect to the machine using key authentication silently failed however.

Munin: failing with Storable error

 |  300 words — 2 minutes  |  Linux Munin

I suffered from a Munin version 2.0.10 installation that refused to update the majority of the graphs. Only the first two of a long list were being updated, the rest all ‘hung’ at the same moment.

After a little investigating, the problem surfaced:

$ su - munin --shell=/bin/bash munin-cron
File is not a perl storable at blib/lib/ (autosplit into blib/lib/auto/Storable/ line 398, at /usr/lib64/perl5/vendor_perl/5.12.4/Munin/Master/ line 362
File is not a perl storable at blib/lib/ (autosplit into blib/lib/auto/Storable/ line 398, at /usr/lib64/perl5/vendor_perl/5.12.4/Munin/Master/ line 362

I started out by fixing all items the munin-check script suggested, which is always a good starting point.

Gentoo: Updating and Cleaning

 |  400 words — 2 minutes  |  Gentoo Linux

Keeping your Gentoo Linux server up to date isn’t as straightforward as let’s say an Ubuntu box, where you would just run $ apt-get update && apt-get upgrade && apt-get clean for example.

Gentoo is far too flexible for a one size fits all approach. The commands outlined below come pretty close for daily use though:

Iptables: Creating persistent bans from Fail2Ban

 |  700 words — 4 minutes  |  fail2ban firewall iptables linux php

On my servers I use the nifty program Fail2Ban to perform logbased automatic firewalling of ‘bad’ ip’s.

The idea behind this is easy: Some IP performs an action I don’t approve of. This can be any number of things, e.g. requesting pages in Apache that are commonly accessed by bots and/or scanners, or trying to log in to SSH with accounts that do not exist on the system. This bad behavior gets logged, and Fail2Ban keeps tabs on those logs, and using a number of rules it determines if a host is ‘bad’ enough to temporarily or permanently ban all access to the server. It does so by adding a few chains to Iptables (one for each thing it checks for), and dynamically adding/removing IP’s to/from these chains.

This all works perfectly. However, there’s one issue; When Iptables gets reloaded, it restores its default rules, removing the Fail2Ban chains and all the rules they contain, even if the ip’s in the chain were marked as permanent.