toofishes.net

The real story behind Arch Linux package signing

This is going to be a long but hopefully informative read. I’d encourage you to sit down and put on your reading glasses, or better yet, go brew yourself a nice hot cup of coffee.

We’ve taken a lot of unjustified heat in just the last month or so regarding package signing, and I’d like to clear the air as well as debunk some awful “journalism” encountered that provoked this blog post. Thank you, LWN, for providing the catalyst.

If you don’t want the backstory, skip down to “The Forbidden Subject”.

Background

For those of you that are reading my blog for the first time, I (Dan McGee, aka toofishes) am the current lead developer of pacman, and have been in that role since roughly May of 2007, when I stepped in to fill the shoes of Aaron Griffin, who is now the Arch Linux “overlord” since Judd stepped down. I have been contributing work to the pacman codebase since late 2006, so this piece of software is not new to me by any means.

Currently I am assisted by another great developer and maintainer, Allan McRae. He stepped into a role of working primarily with makepkg around May 2008, but now commits and reviews code changes all over the codebase. Did I mention he is also in charge of the Arch toolchain as well as several other [core] packages? I’m not sure how he does it.

The Story

FS#5331 : Signed packages

I know it is a shocker, but pacman has a bug tracker. One of the 5 oldest bugs in there is FS#5331 : Signed packages. Opened in September 2006, it had no serious comments until July 2007. Once the comments started, no one produced any actual patches, code, or anything to proceed with any sort of plan. The bug sat relatively silent until March 2010.

The first patch

Step forward to 2008. On June 1, 2008, a day that will live in infamy, the very first patch dealing with package signing showed up on the pacman-dev mailing list.

If you browse down a few messages, after a few revisions, to where I said the patch looked good, you will find this gem of a quote. I hate to quote myself, but I think it proves a point that has been lost in the recent furor.

Other than that the patch looks fine, I’ve started putting these changes in a local branch that will end up in master soon enough. Looking forward to seeing perfect PGP support in pacman/libalpm!

We’ll come back to that in due time.

Another thing to note is the issue that still persists to this day came up back then: why can’t we just sign the database? It was answered, everyone accepted the answer (at the time), and we moved forward trying to ensure the entire problem was solved.

Follow-up patches

What happened next was typical of both pacman development and OSS development in general- the original contributor of this work sent a few more patches, stopped responding to requests to fix issues in the work, and left it in our laps. For the maintainer of a project, being dumped on like this is never a great thing, but at least here the work was in good enough shape to fix up and commit to a gpg branch for later use.

As has been the case nearly every single time a fuss is raised about this stuff, we were prepping the 3.2 release in July 2008 right as this was all happening, so the main developers couldn’t work on it. But our original developer showed up just long enough to say he was still interested in finishing this work. Guess who we never heard from again?

The catch with these initial patches is they were doing the “easy” stuff. Signing a package as the last step of building is not very hard- it is a simple invocation of the gpg command line client. Adding this signature to a pacman database was not too hard either. But the patches from others stopped here, unfortunately. Looking at the authors of the follow-up patches that have since been committed on top of this original work, it is no surprise to see three names: mine, Allan McRae, and Xavier Chantry (another long time pacman contributor).

Until December 2008, the stage was silent. I had to speak up once a discussion started bikeshedding without producing any working code. There is a nice quotable bit in there from me (“First off- stop talking. Start coding.”), but the important bit is that the ground rules were laid on what would be an acceptable end result. The link I continue to show people also came into play at this time: Attacks on Package Managers.

We got a few more patches and contributions in December 2008. I did a good bit of work in this time to integrate reading signatures into libalpm, get it under pactest, and all that (not so) fun stuff.

The dormant period

Once again, no one was working on package signing. It was brought up briefly at the end of June 2009, and some discussion happened but once again no solid results were produced. People kept informing Allan and I it wasn’t clear where we were heading, so we pointed them at the wiki roadmap and asked them to help edit and clarify. Apparently that is too much work for most people, as they seemed to fade away as quickly as they spoke up. Once again, the primary developers were wrapping up the 3.3 release, so we didn’t have much free time on our hands.

You can imagine at this point, a year down the road from the first patches, none of the primary pacman developers are very interested in implementing this themselves. Perhaps this is true, with the ironic twist that more than half of the patches on our long-lived gpg branch are from the three main contributors. I think the most truthful statement is that no one wanted to take the lead on this and finish it by themselves. At this point, the work is nearly where it stands today, as most of the additional work I merged in the last few days was simply bitrot cleanups (aside from pacman-key). However, nowhere have you seen any sense of “even if you produce good work and get things finished we won’t take it” attitudes from Allan or I.

Xavier undertook a rebase and cleanup of the stale gpg branch in August 2009, merging in a few old patches from the mailing list.

And you guessed it- another silent period until April 2010.

Recent history

The thread that sums up the “all talk, no walk” part of this whole package signing thing started in April 2010. This is just the part that was on pacman-dev, but it started on arch-general, stretched into May, and accounts for 57 emails in one thread. The sad part? In the package signing work I pushed in the last few days, I see no patches that made it from this timeframe.

We finally got a contributor that stuck around with Denis A. Altoé Falqueto from June 2010 until now. His contributions weren’t huge or frequent, but he did write the pacman-key tool which is now merged into master and attempted to keep package signing on the list of features moving forward.

This is the first time period where I would say we failed those that wanted to work on package signing. We weren’t quick with responding to patches and giving feedback. Note that this holds true for all patches, not just these ones. I think all of us were quite busy and just didn’t have the time or energy we had in the past. When we did work on pacman, we wanted to work on things that were fun, rather than slogging through patch review.

The Forbidden Subject

I couldn’t help but steal the dramatic title. On February 18, 2011, Your Signature Please arrived in our mailbox. Keep in mind the following:

I happened to be skiing in Colorado this day (Friday) and was gone the entire weekend. Do you think I was going to waste time reading a novel? Not a chance. Poor Allan for trying to do so, as he has now been thrown under the bus for being the naysayer, and his words twisted and changed in multiple forms of publication.

Some memorable quotes from this thread that quickly went the wrong way:

Allan: Have you actually looked at the current implementation at all?

IgnorantGuru: I read some discussions of it, but I have not looked at it. Frankly it interests me far less than having signatures available at this point.

An attempt to let cooler heads prevail:

On Sat, Feb 19, 2011 at 12:00 AM, IgnorantGuru jgj7.pacmandev@mailnull.com wrote:

[…. lots of talk….]

Denis: There’s no political problems here. It’s just lack of manpower to make it. That’s it. This is a standard open source community, there’s just momentum when there is personal interest.

And a solo quote from Allan that is oft-repeated as him being anti-signing:

As I said, it really does not affect me. I use the master server for my repo db downloads and know exactly which package updates to expect given I see all commits to our svn repos. So the scope in which I could be attacked is very small and I am prepared to take that risk. So my priorities are clearly different to other peoples. The key difference is, I submit patches to implement what I consider a priority…

Chances of anyone going to over mailman and reading this entire thread are slim- I know this. But realize Allan had not only the first reply, but the last email in the thread. If you read it, you can see why some of his emails were perhaps filled with anger. But he never dismissed anyone, or told them to fuck off, or said things about their mother, or let Godwin’s Law prevail.

Once again, were left hanging on the promise of patches on the way from those raising the most trouble in this thread. They never showed up. Thankfully, at least Denis, mentioned earler, proposed a few new patches.

The Media Blitz

From here, shit hit the fan. The Arch Linux forum moderation team got caught up in the scuffle. IgnorantGuru started his crusade with this post on an existing thread. They closed IgnorantGuru’s forum rant post that looked like a blog post, which later did show up as a blog post. We were then the target of multiple sensationalist blog posts that he also tried to drum up on reddit.

Mr. IgnorantGuru filed this “flyspray”, FS#23103, asking to add sha256sums to our package databases. A reasonable request that quickly turned into a war of words, but I attempted to straighten it out by telling him the standard patch submission rules we use for pacman. I was treated to this, to which I did not respond:

Are you willing to add it if I take the time to submit a patch, or are you just wasting my time? I ask because thus far I have met nothing but unwillingness, so please don’t waste my time. I don’t really see why a patch is necessary as it is a trivial addition, but if you want one I will be happy to provide it. Thank you.

Surprise- no patch showed up in my inbox or on the bug report. I in fact did exactly what the request asked for a few days later, noting that the changes were not trivial at a +12/-5 diff.

The Deal Breaker

I was willing to let all of this slide and fade into darkness as it normally does, until someone showed me the LWN article Arch Linux and (the lack of) package signing. This forced me to write this post as it is full of lies, lies, and more lies.

First, shame on you Nathan Willis, Jonathan Corbet, and LWN for allowing this to be published. This is not journalism- this is propaganda fueled by a rogue blogger who you’ve decided to let create a story where there isn’t one. I’m going to address points in the article that are just flat out wrong.

The topic had come up before, but no one acted on it, and several of the core Arch developers dismissed the subject as an unimportant one that they were not willing to work on personally.

I challenge you to find any of us that said package signing is or was “unimportant”, and that we are not willing to work to get it into the core of the package manager. The only sound byte latched onto here was the one I previously quoted from Allan- it wasn’t important to him personally so he didn’t feel obliged to devote time to it. This is also a good time to point back to my original quote from the very first patch we received.

We are also not paid for our work on Arch. I do not know a single core developer that gets paid to regularly hack on the distro- we are nothing like a Red Hat or Canonical. I can guarantee you that both Allan and I would go straight to work on package signing if we were getting paid to do so and guaranteed a long term job furthering Arch Linux.

A few, he said, did take the issue seriously and had submitted patches to Pacman, but core developers refused to act on them.

Please show me in my detailed history above where this happened. Since you cannot, I congratulate you on perpetuating rumors further.

McRae … sought out every discussion of the topic and tried to quash others’ efforts to work on a solution.

Are you kidding me? Did you even read the mailing list thread he is referring to? You clearly did not or you would have observed what I did above, but doing research on a completely public conversation before publication must be optional these days.

In the second bug report, IgnorantGuru even suggested a lightweight solution that involved only signing the main server’s package database … McRae countered …

So if your links in your article are right, you mean FS#23103. The funny thing is, this bug neither deals with signatures (only checksums) nor has a single comment from the aforementioned Allan McRae. Busted.

(Editor’s Note: I gave LWN 12 hours to respond to this before I made it public. They have since fixed the article where it said “second” to read “first”, so the above doesn’t apply directly anymore. For the first bug, FS#23101, the article is correct in saying “IgnorantGuru even suggested a lightweight solution”. However, suggestions don’t produce working software, code produces working software, and not a single piece of code was provided on this bug report.)

He then describes patches sent by himself and other Arch contributors, and what McRae and other core developers did to prevent merging them. This second post covers similar ground as its predecessor, but the comment thread provides even more detail, as McRae eventually joins in.

Show me one patch he has ever sent us- just one. You won’t find anything.

Next, show us exactly what we did to prevent merging them. You won’t find anything.

Finally, you further your baseless libel of Allan by referring to this “second post”, but providing no link so your readers can independently verify the allegations.

(Editor’s Note: LWN corrected me here a bit, saying I may have misread the wording of this paragraph. The linked post and the “second post” are in fact one and the same- IgnorantGuru’s blog post. Finally, the “comment thread” refers to the comments on his blog, which I also did not understand. I never would have expected a magazine article to cite comments on a blog post as authoritative.)

That of course is the second level at which the core developers’ resistance is troubling: the fact that they would prevent security patches from going into the project.

Please continue the fear mongering and baseless allegations. Allan made one point- that this wasn’t very important to him- and it is now interpreted as a blockade against all attempts to introduce signing.

Congratulations, LWN, for dropping to a new low. You won’t be seeing my money anytime soon any more than the rags in the grocery store checkout do.

Where are we now

With pacman 3.5 out the door and 3.6 in development, package signing is not falling out of the spotlight. Instead, three different merges plus additional follow-up commits have already taken place of the code that in some cases is 2.75 years old.

Still, no one has stepped up in the last two days to tackle items from the Package Signing TODO list. I foresee Allan and I slogging through this with hopefully a little help from Denis, Xavier, and our newest regular contributor Dave Reisner. It will get done, but it all takes time since we are only volunteers.

Tags

See Also