Engineering

Fixing a YAML Vulnerability in Ruby

Heap overflow!

Like everyone else, we got this in our INBOX: Heap Overflow in YAML URI Escape Parsing (CVE-2014-2525). The problem with fixing it on our Ubuntu 12.04 LTS (precise) systems, was that there was no libyaml available from Canonical that was > 0.1.5

If you don’t care about how we solved it, and only want to know how you can solve it on your machines, click here

The vunerability was difficult to patch because it required our Ruby 1.9.3-p484 to be built against libyaml 0.1.6.

We verified our Ruby’s yaml version like this:

$ ruby -rpsych -e 'p Psych.libyaml_version 
[0, 1, 5]

Yaml in the My Ruby

Yaml is also linked into Ruby’s psych module.

$ ldd /usr/lib/ruby/1.9.1/x86_64-linux/psych.so
...
libyaml-0.so.2 => /usr/lib/x86_64-linux-gnu/libyaml-0.so.2 (0x00007f82c951b000)

That gets fixed when the .so file gets swapped out from underneath it.

Our Solution

We decided if Canonical wasn’t going to host the newest version of libyaml, we would.

Build it

The solution was to download yaml and build it from the patched source.

$ ./configure --prefix=/usr
$ make

It’s as simple as it sounds. We just downloaded the newest patched version, and built it.

Package the new library

To package the new library, we need to find the old library and create a new version so we can make a debian package out of it.

$ dpkg -l | grep yaml
ii  libyaml-0-2                      0.1.4-2ubuntu0.12.10.3              Fast YAML 1.1 parser and emitter library
ii  libyaml-dev                      0.1.4-2ubuntu0.12.10.3              Fast YAML 1.1 parser and emitter library (development)

We rebuilt both of these packages. The dev package contains a header file that was unchanged and depends upon libyaml.

Exploding the old package for its content

Now that we located the old packages, we exploded them so that we could replace the library file with the new 0.1.6 build. Deb’s have two parts, a “control” tree with the metadata and a “tree” portion with the archive of files. AFAICT, you need to extract them separately, the “control” with a -e and the tree with a -x.

So here’s how we unpacked one of the archives, and then put it back together after editing it:

$ mkdir -p libyaml-dev/DEBIAN
$ dpkg-deb -e /tmp/libyaml-dev_0.1.4-2ubuntu0.12.10.3_amd64.deb libyaml-dev/DEBIAN
$ dpkg-deb -x /tmp/libyaml-dev_0.1.4-2ubuntu0.12.10.3_amd64.deb libyaml-dev
$ tree
.
└── libyaml-dev
    ├── DEBIAN
    │   ├── control
    │   └── md5sums
    └── usr
        ├── include
        │   └── yaml.h
        ├── lib
        │   └── x86_64-linux-gnu
        │       ├── libyaml.a
        │       ├── libyaml.so -> libyaml-0.so.2.0.2
        │       └── pkgconfig
        │           └── yaml-0.1.pc
        └── share
            └── doc
                └── libyaml-dev
                    ├── changelog.Debian.gz -> ../libyaml-0-2/changelog.Debian.gz
                    ├── copyright
                    └── README -> ../libyaml-0-2/README
10 directories, 9 files
$ dpkg-deb -b libyaml-dev ./my-new-debian-archive.deb

Distribute the Debian

We used reprepro to package and sign the debian archive. To make a new archive, you start with a single file conf/distributions. You need to specify which distributions you support. We supported two, precise (12.04 LTS) and quantal (12.10). We cribbed some of our content from content like this. Reprepro allows you to sign the content in your archives. You’ll need to make a keypair with gpg and then put the key-id of the public key in the conf file as a SignWith attribute.

$ mkdir conf
$ cat > conf/distributions
Origin: Canonical
Label: Partner archive
Suite: precise
Version: 12.04
Codename: precise
Architectures: amd64 armel armhf i386 powerpc
Components: partner
Description: Ubuntu Precise 12.04
SignWith: 79269265

Origin: Canonical
Label: Partner archive
Suite: quantal
Version: 12.10
Codename: quantal
Architectures: amd64 armel armhf i386 powerpc
Components: partner
Description: Ubuntu Quantal 12.10
SignWith: 79269265

We then add the libyaml 0.1.6 Debian we created to the precise and quantal databases of the repository:

$ reprepro -Vb . includedeb precise /tmp/my-new-debian-archive.deb
Created directory "./db"
/tmp/foo.deb: component guessed as 'partner'
Created directory "./pool"
Created directory "./pool/partner"
Created directory "./pool/partner/liby"
Created directory "./pool/partner/liby/libyaml"
Exporting indices...
Created directory "./dists"
Created directory "./dists/precise"
Created directory "./dists/precise/partner"
Created directory "./dists/precise/partner/binary-amd64"
Created directory "./dists/precise/partner/binary-armel"
Created directory "./dists/precise/partner/binary-armhf"
Created directory "./dists/precise/partner/binary-i386"
Created directory "./dists/precise/partner/binary-powerpc"
Successfully created './dists/precise/Release.gpg.new'
Successfully created './dists/precise/InRelease.new'

$ reprepro -Vb . includedeb quantal /tmp/my-new-debian-archive.deb 
/tmp/foo.deb: component guessed as 'partner'
Exporting indices...
Created directory "./dists/quantal"
Created directory "./dists/quantal/partner"
Created directory "./dists/quantal/partner/binary-amd64"
Created directory "./dists/quantal/partner/binary-armel"
Created directory "./dists/quantal/partner/binary-armhf"
Created directory "./dists/quantal/partner/binary-i386"
Created directory "./dists/quantal/partner/binary-powerpc"
Successfully created './dists/quantal/Release.gpg.new'
Successfully created './dists/quantal/InRelease.new'

Host on AWS

We didn’t want to bring up a server, so we host our repository on AWS. First, we made a new bucket in S3. We then enabled the “Static Website Hosting” stuff:

We also made the entire bucket world-readable:

{
  "Version": "2008-10-17",
  "Statement": [
      {
        "Sid": "AllowPublicRead",
        "Effect": "Allow",
        "Principal": {
        "AWS": "*"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::grh-aptitude/*"
    }
  ]
}

Then we used s3cmd to sync our repository to S3

s3cmd sync . s3://foobar-bucket

Fixing the problem

So how do we now deploy these fixes? We’ve compiled a bug free version of libyaml and libyaml-dev. We’ve packaged everything up nicely and hosted it S3. So how do we get these files to our servers?

If you’re as strict as we are about NEVER touching production with your bare hands, you’ll have to use some sort of Configuration Management software. We use Puppet at Grand Rounds, and we crafted a custom class so we could add a custom source using the apt module

class deps {
  apt::source { 'grand-rounds':
    location          => 'http://grh-aptitude.s3-website-us-east-1.amazonaws.com/',
    release           => 'precise',
    repos             => 'partner',
    include_src       => false,
    key               => '79269265',
    key_server        => 'pgp.mit.edu',
    before            => Package["libyaml-dev"],
  }

Because, to use our repository, you have to add a new file to the /etc/apt/sources.list.d directory. Something like this:

# grand-rounds
deb http://grh-aptitude.s3-website-us-east-1.amazonaws.com/ precise partner

That makes the repository available to apt. Then, Puppet fetches our key from mit and adds it to the apt-key keyring. You can do this by hand like this:

sudo apt-key adv --keyserver pgp.mit.edu --recv-keys 79269265

Then all we need to do is update aptitude and then install the newest versions from our new repos:

$ sudo apt-get update
$ sudo apt-get install libyaml-dev
$ aptitude show libyaml-dev
Package: libyaml-dev              
State: installed
Automatically installed: no
Multi-Arch: same
Version: 0.1.6
Priority: optional
Section: libdevel
Maintainer: ken@grnds.com
Architecture: amd64
Uncompressed Size: 0 
Depends: libyaml-0-2 (= 0.1.6)
Breaks: libyaml-dev (!= 0.1.6)
Replaces: libyaml-dev (< 0.1.6)
Description: Fast YAML 1.1 parser and emitter library (development)
 LibYAML is a C library for parsing and emitting data in YAML 1.1, a human-readable data serialization format. 

 This package contains development headers and static libraries.

Done!

You’ve fixed it!