Red Hat RHCE course learnings

I recently did the RH300 fast-track Red Hat Certified Engineer course, and learnt a few things. Some of the things I learnt were specific to red hat, some were genuinely new, and some were things I knew but had forgotten. Here is a summary, in no particular order.

Red Hat specific

There is a free version of Satellite Server called Spacewalk that works for Fedora and Centos clients.  Satellite requires Oracle (suck), but Red Hat is working on a mysql/postgres solution.   Incidentally our instructor hinted that RHEL desktop might be dropped soon in favour of free Fedora desktops for enterprise customers.

Red Hat is looking at removing the --no-deps option from RPM because too many people break their install with it.

To update your yum repo with createrepo, you should delete the repodata directory.  Since creating a repo is slow with many packages, you should separate out your corporate packages that require regular changes into a separate repo.

You can do a kickstart from GRUB directly by adding some kernel params (the ksdevice removes a prompt on devices with multiple interfaces, and noipv6 cuts out a long ipv6 timeout):

ks=http://my.kickstart/box.cfg ksdevice=eth0 noipv6

The easy way to configure network services on Red Hat is to use the 'setup' ncurses UI, which calls out to the relevant Text User Interfaces (TUIs) such as 'system-config-network-tui'

Configure and manage LVM with system-config-lvm.  It is an impressive GUI, and makes resizing partitions and file systems really easy.

Install the setroubleshoot server and use the GUI to find selinux problems (it usually gives you a sensible solution):

sealert -b

To change a selinux context to a reference context, ie. give the file the same context as /home:

chcon --reference=/home /home/new

You also can change selinux context by specifying a context from the many pre-built ones available from the targeted policy in the /etc/selinux directory:

chcon -t public_content_t /shared

chcon -t samba_share_t /windowsshare

To check selinux status (e.g is it enforcing?):

sestatus

To see the selinux status of files/processes, add 'Z' to the usual tools:

ls -lZ

ps -auxZ  

You can also see status and change it with (also handles iptables):

system-config-securitylevel-tui

iptables rules are stored in /etc/sysconfig/iptables and can be edited directly, or rules applied and then saved with 'service iptables save'

If you have more than one mail agent installed (e.g. sendmail and postfix), you can switch between them with the 'system-switch-mail' ncurses gui, which is a frontend to the alternatives system.

Other Random Learnings

The kernel limits the number of partitions possible on IDE and SCSI/SATA disks in different ways (in addition to the MBR limitations of 3 primary and 1 extended).  For IDE (/dev/hd*), the max number of partitions is 63, for SCSI/SATA (/dev/sd*) it is 15.

To view and set disk labels for ext2/3:

dumpe2fs /dev/sda1

e2label /dev/sda1 label

dumpe2fs will also show you the mount defaults for the partition, which is handy to know when /etc/fstab just says 'defaults'.  You can change these defaults with 'tune2fs', this adds the 'acl' option:

tune2fs -o acl /dev/sda1

To get disk labels and UUIDs:

blkid

Then if you want to know which device has that label or UUID:

fuser UUID="sdfkshdfkjshdkjshdf"

Backup a disk partition table with:

sfdisk -d /dev/sda > sda.backup

To find out which process is holding a mount point:

fuser -m /mnt/point

and to kill all those processes (be careful):

fuser -km /mnt/point

To get serial numbers and other information from the system management BIOS (handy when you are in a different location to a server):

dmidecode

 x86info has a nicer formatted version of the /proc/cpuinfo CPU information

To see if a NIC has a link I usually use ethtool, but that isn't installed by default, so if it is unavailable:

ip link show eth0

You can get access to the bootloader of a xen machine with:

xm create -c xenname

Automagically install SSH public keys into authorized_keys on a remote machine

There is a handy little utility called 'ssh-copy-id' that simplifies getting your ssh public key into a remote machine's authorized_keys. The syntax is:

ssh-copy-id [-i [identity_file]] [user@]machine

It handles creating the ssh directory and authorized_keys file on the remote side (if necessary), and setting the right file permissions. Neat!

yum/rpm vs. apt/dpkg (apt wins)

A comparison of Yum (RHEL 5.4) vs. Apt (ubuntu lucid)

Yum/RPM Pros:

'rpm -V' is very cool. It allows you to verify files that were installed from RPMs, which includes comparing owner, group, mode, md5, size, major/minor device numbers, symlink strings, and modification times to what was originally installed by the rpm. Use 'rpm -V -a' to verify all installed packages. Despite some wishlist bugs being filed, the closest dpkg comes is 'debsums', which only does hash comparisons.

Yum has $releasever and $basearch macros that can be put into yum repo configs. It would be great if Debian could do the same so $releasever would expand to 'lucid' or whatever distro you are running. This would make distributing standard apt sources.list files across an organisation running multiple versions much simpler.

Yum has 'localinstall', which allows you to install an rpm but satisfy its dependencies from the yum repo - neat! I believe the same is possible with apt, but it would look like:

dpkg -i single.deb; apt-get -f install

which is not exactly intuitive.

Yum/RPM Cons:

Yum has no command-line (tab) completion, which is super annoying.  I found I had to either do a search for my package first, or use commands like:

yum install http-*

which resulted in the installation of heaps of stuff I didn't need.

The decision to replace or not replace config files during package upgrades is left to the individual package maintainers. This is a serious bad idea. The two strategies are:

• 'rpmsave': install the new config, back the old one up as '.rpmsave'; and
  
• 'rpmnew': leave the old config in place and put the new one in as '.rpmnew'.
  

The first is a recipe for disaster, and means you absolutely must test every upgrade to see if it will bring down your production server. In contrast, if an apt update is going to change a config file that has been modified from the original packaged version, it pauses the update to let you decide how to proceed. You can view a diff, and decide to keep yours or install the package version. Whatever you decide, a copy of the old version is left in the directory. For apt, this behaviour is consistent for every package, ie. an apache package maintainer can't accidentally deface your website by replacing your highly-customised config with the default one.

When you uninstall a package with yum, the removal of rpms that were installed as dependencies is left up to the individual package post scripts. This means that you might install 10 dependencies, but only uninstall 8.

In contrast, Apt will uninstall each package it installed and leave configuration files in place (unless you specify --purge). Apt also has 'autoremove' (there is no yum equivalent), which allows you to remove any packages that are no longer needed. Packages that fall into this category are usually libraries that were dependencies for multiple applications, all of which have since been removed.

If you are running a 64-bit OS and point to a repository that has both 64-bit and 32-bit packages in it, apt is smart enough to realise that you probably want to install the 64-bit package. Not so with yum:

yum install openssl

might get you either the 32 or the 64 version. Worse still, running yum install openssl might try to install the 32-bit version over the top of the already installed 64-bit version.

The redhat community seems to be claiming this isn't a bug. OK, it is just a design flaw then, one that is going to start hurting a lot when the masses move to 64bit.

Yum package groups have long annoying names containing spaces that need to be escaped on the commandline. They also have 'optional' listed packages, but lack any way to install them (apart from copy paste). Apt has the same concept except it just uses meta-packages. The metapackages look and behave exactly the same as regular packages, which means you don't need a groupinstall command and they turn up in search results.

apt-get install --install-recommends

will get you all the recommended packages (you can also set this permanently in /etc/apt/apt.conf). In addition Debian has the concept of 'suggested' packages; you can also change apt.conf to install these automatically.

'yum search ftp' will turn up packages with an ftp URL (e.g. libsoup, which has nothing to do with ftp). Whoops! Apt does the right thing and searches the package title and descriptions, not the URL.

Yum search doesn't seem to handle multiple terms very well, it does an OR where it should do an AND.

Apt is way faster - the speed discrepancy is particularly noticeable when using search commands.

Bash if statements

I'm constantly forgetting the bash if statement syntax, and it takes a while to find the right example on the web or to read the man page. Some examples below.

One liner to get the number of seconds since a file was modified:

if [ -f /var/lib/puppet/state/puppetdlock ] ; \
then echo $[`date +%s` - `stat -c %Y /var/lib/puppet/state/puppetdlock`] ; else echo 0 ; fi

The same if statement with line breaks:

if [ -f /var/lib/puppet/state/puppetdlock ]
then
echo $[`date +%s` - `stat -c %Y /var/lib/puppet/state/puppetdlock`]
else
echo 0
fi

Microsoft's attempt to fix windows packaging: The Common Opensource Application Publishing Platform (CoApp)

I read about Garrett Serack's project to fix the mess that is windows packaging (or lack of packaging) with The Common Opensource Application Publishing Platform (CoApp) and was pretty surprised. It seems Garrett is a long time linux user who has somehow found himself working at Microsoft, and has come up with this project to try and fix things. His post is a pretty good description of just how bad packaging is on windows, and provides good background on how a lack of standardisation has got us into dependency hell. In his words:

Frankly it's taken an extremely long time to convince the powers-that-be at Microsoft that Linux's package management is stellar compared to Windows.

and

My intent is to completely do away with the [current Windows application] practice of everybody shipping every damn shared library.

Reading some of his responses to the slashdot comments, some interesting tidbits came out. He is planning to use WiX to chain MSI's together to handle dependencies, and Windows SxS to handle multiple versions of libraries.

Will it work? Who knows, it is a massive undertaking. I imagine some open-source community members might have to do some soul searching to figure out if they want to contribute to a project that will make windows better, and presumably make Microsoft more money.

Django formset.as_table() displays vertically only

Like other people, I want to be able to display a formset as an ordinary 'horizontal' table, i.e. where all headings are in their own row, separate from the data:

<table>
<thead>
<tr><th>column1</th><th>column2</th></tr>
</thead>
<tbody>
<tr><td>form1.value1</td><td>form1.value2</td></tr>
...
</tbody>
</table>

Whereas the django formset.as_table() method displays as below, with the heading on the same row as the data:

<table>
<tr><th>column1</th><td>form1.value1</td></tr>
<tr><th>column2</th><td>form1.value2</td></tr>
</table>

The solution is a pretty nasty gob of template code, since to fix it you need to make all the error handling and form magic explicit. Django devs: Let's have a new method or parameter to display the formset horizontal rather than vertical.

OpenLDAP completely broken on install under Karmic

I hate OpenLDAP. Thanks to a stupid decision by OpenLDAP packagers, the server is completely unusable after install on Ubuntu. We went from a working debconf on jaunty to a completely broken install on karmic (and it doesn't look like it will be fixed in lucid either).

An OpenLDAP install on karmic no longer creates a database, installs schemas, or creates an admin user. There are basically no good HOWTOs for this initial configuration, and all the official documentation is wrong - it says to use dpkg-reconfigure, which no longer works. A thread on the ubuntu forums is the best you will get.

All I wanted to do was move my existing (working) ldap database to another server, which should have been a 5-minute job with slapcat, slapadd.

I finally got it working using the instructions in the thread to create a database, then added the users using 'slapadd -n1' as originally intended (I never got to import the original config '-n0' successfully). There is a whole lot of black magic involved: I don't understand what installing the schemas does, and I only vaguely understand what the ldif file does. The error messages provided by openldap might as well be in another language because they are completely uninformative.

Once I had a working setup I was getting a prompt with 'I have no name!'@box. Despite there being a lot of bullshit about this being caused by permissions on ldap config files in forums, it turns out if you install 'nscd' it magically goes away. I have no idea why, but nscd also seems to be the antidote to other ldap bugs, so you might as well have it :)

It's enough to make me want to use a windows DC and likewise....

Update: according to a bug report, this doco is now up-to-date, with the same information as in the thread.