It took me several weeks to track down this problem, one that dramatically impacted the speed of specific actions that require frequent Kerberos lookups. The symptom is slow Kerberos actions such as doing a “kinit”. The backend doesn’t matter I don’t think — we have both MIT Kerberos and Active Directory, and the service is hitting both. On a RHEL 5 machine with a similar configuration, such a lookup in our environment, which requires a few hops and DNS lookups and such, takes around 80ms. On the new RHEL 6 machines, the same lookup takes around 300ms. Most of the time this is barely noticeable, because Kerberos actions are infrequent and normally only need to occur once.

It so happens that an important service we run is Subversion for source code management. Our Subversion runs under Apache (using mod_dav_svn) with Kerberos for authentication. We allow both password-based authentication and ticket-based authentication. Apache handles these as negotiate requests using the mod_auth_kerb module.

When authenticating with the password dialog, you put in your password once, Apache takes it from there and performs the Kerberos lookup, and all further actions occur speedily. But when using a ticket, the preferred authentication method, actions are very slow. This is specially noticeable for large check-ins, but is annoying most of the time, even for small actions, because SVN has to perform several requests for a simple update or small check-in.

I eventually tracked down the problem as being related to the newer version of Kerberos on RHEL 6. (For a while I was convinced the culprit was SSSD, but not so!) Specifically, newer Kerberos RPMs are patched to load in SELinux label configurations and use them when creating temp files. Unfortunately the label configurations are very large files full of regexes, all of which need to be churned through and memory mapped — on every request! In our case that portion of the operation takes about 120ms, and happens twice per request.

The solution is to disable SELinux completely (not just set it to permissive mode) and restart, or, perhaps to recompile krb5 without the selinux patch. Of course, once I finally figured out what was going on, I discovered a previously filed bug that is languishing in Red Hat’s Bugzilla that outlines this exact issue.

So for the next person who has this problem, I hope this pops up earlier in the Googles, and saves you some aggravation.