toofishes.net

Where is my new kernel, Slicehost?

A while back, I posted about a long-needed Slicehost kernel upgrade from a 2.6.24 to a 2.6.31 build. Since I run Arch Linux on my slice, it is a lot easier on the maintenance if I am running a relatively up to date kernel. Given that 2.6.24 was released in January 2008, it was definitely time for something more recent.

Fast forward two and a half months. I’ve been running this 2.6.31-302-rs build of theirs the entire time, and I was a very early adopter. Given this, I found two non-trivial issues with the build and gave their support team a heads up. The first was an issue I noted in my earlier post regarding iptables and the recent connection tracking module changing names and thus no longer being included.

Ticket 14486, November 1, 2009

I upgraded to the 2.6.31 kernel and noticed my iptables rules didn’t get loaded correctly. It looks like it is due to the xt_recent kernel module not being available. In 2.6.24 (through I think 2.6.27), it was known as ipt_recent, which is probably why it got omitted in the new build. Can we get it back?

dmcgee@toofishes /etc
$ locate xt_recent.ko

dmcgee@toofishes /etc
$ locate ipt_recent.ko
/lib/modules/2.6.24-24-xen/kernel/net/ipv4/netfilter/ipt_recent.ko

And the “offending” rules in case you were curious:

$ cat /etc/iptables/iptables.rules | grep recent
#-A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 300 --hitcount 10 -j DROP
#-A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set

The second ticket was an issue I found soon after. Although my slice now has 321 MB of memory, it also has a ton more overhead than it used to due to a bunch of unnecessary kernel-level processes.

Ticket 14489, November 2, 2009

Another note about the new 2.6.31 kernel- it ends up having a ton more overhead because it looks like JFS and XFS are built-in, and all of their kernel processes get started even if one isn’t using either of these filesystems. Because they aren’t modules I can’t unload them and free up some memory.

Example 1: Process count spike, look at the thin line and the spike at the end of October

Example 2: Memory usage jump, note the application memory and swap in use

dmcgee@toofishes ~
$ ps -eLf | grep -E 'jfs|xfs'
root        42     2    42  0    1 Nov01 ?        00:00:00 [jfsIO]
root        43     2    43  0    1 Nov01 ?        00:00:00 [jfsCommit]
root        44     2    44  0    1 Nov01 ?        00:00:00 [jfsCommit]
root        45     2    45  0    1 Nov01 ?        00:00:00 [jfsCommit]
root        46     2    46  0    1 Nov01 ?        00:00:00 [jfsCommit]
root        47     2    47  0    1 Nov01 ?        00:00:00 [jfsSync]
root        48     2    48  0    1 Nov01 ?        00:00:00 [xfs_mru_cache]
root        49     2    49  0    1 Nov01 ?        00:00:00 [xfslogd/0]
root        50     2    50  0    1 Nov01 ?        00:00:00 [xfslogd/1]
root        51     2    51  0    1 Nov01 ?        00:00:00 [xfslogd/2]
root        52     2    52  0    1 Nov01 ?        00:00:00 [xfslogd/3]
root        53     2    53  0    1 Nov01 ?        00:00:00 [xfsdatad/0]
root        54     2    54  0    1 Nov01 ?        00:00:00 [xfsdatad/1]
root        55     2    55  0    1 Nov01 ?        00:00:00 [xfsdatad/2]
root        56     2    56  0    1 Nov01 ?        00:00:00 [xfsdatad/3]
root        57     2    57  0    1 Nov01 ?        00:00:00 [xfsconvertd/0]
root        58     2    58  0    1 Nov01 ?        00:00:00 [xfsconvertd/1]
root        59     2    59  0    1 Nov01 ?        00:00:00 [xfsconvertd/2]
root        60     2    60  0    1 Nov01 ?        00:00:00 [xfsconvertd/3]

I got very quick responses to both of these tickets. Both were along the lines of “thanks for letting us know, I’ll get this over to our kernel guys”. I got the impression there would be a new kernel in a week’s time or so, especially given this more detailed follow-up response I received a few days later.

Slicehost Support, November 5, 2009

Hello Dan,

Thank you for your messages about the new kernel. We’re working on a new kernel now that will take up much less resources and also provide the same amount of iptables functionality as the previous kernels. The new kernel is currently being tested in our testing environment and we hope to have it released very soon.

As soon as it is available, we will post an update on https://slicehost.com/blog/ and it will be available for you in the SliceManager. We apologize for the issues in the current kernel and we are working to fix the problems as quickly as possible.

Please let us know if we can assist you further at any time.

So what has happened since? Nothing. I sent them another follow-up email a month later asking on the status of the new kernel. I got a response telling me they are “working through some compatibility issues” and “we are in the final stages of the last round of testing”. Wow, final stages of the last round of testing? That sounds good, they really do some good regression.

Only problem? This amazing new kernel still is nowhere to be seen. Slicehost, your good will meter is shrinking faster than you know right now. It has been nearly 50 days since that conversation and I still see no new kernel available. Will this blog post help? I have no idea, but know that your customers are trying real hard to stay loyal but Linode is looking awfully good right now.

Tags

See Also