Hackage Security Alpha Release

Edsko de Vries – Wednesday, 08 July 2015

Well-Typed is very happy to announce the first alpha release of the Hackage Security library, along with integration into both cabal-install and the Hackage server and a tool for managing file-based secure repositories. This release is not yet ready for general use, but we would like to invite interested parties to download and experiment with the library and its integration. We expect a beta release running on the central Hackage server will soon follow.

Hackage Security and related infrastructure is a project funded by the Industrial Haskell Group to secure Hackage, the central Haskell package server. A direct consequence of this work is that we can have untrusted Hackage mirrors (mirror selection is directly supported by the library). A minor but important additional side goal is support for incremental updates of the central Hackage index (only downloading information about new packages, rather than all packages).

TL;DR: Hackage will be more secure, more reliable and faster, and cabal update should generally finish in seconds.

Security architecture

Security is notoriously difficult to get right, so rather than concocting something ad-hoc the Hackage Security framework is based on TUF, The Update Framework. TUF is a collaboration between security researches at the University of Arizona, the University of Berkeley and the Univerity of Washington, as well as various developers of the Tor project. It is a theory specifically designed for securing software update systems, and suits the needs of Hackage perfectly.

TUF covers both index signing and author signing. Index signing provides the means to verify that something we downloaded from a mirror is the same as what is available from the central server (along with some other security properties), thus making it possible to set up untrusted mirrors. It does not however deal with compromise of the central server. Author signing allows package authors to sign packages, providing a guarantee that the package you download is the one that the author uploaded. These two concerns are largely orthogonal, and the current project only adds support for index signing. Author signing will be the subject of a later project.

Very briefly, here is how it works. The bits in red refer to new features added as part of the Hackage Security work.

Hackage provides a file 00-index.tar.gz (known as “the index”) which contains the .cabal files for all versions of all packages on the server. It is this file that cabal update downloads, and it is this file that cabal install uses to find out which packages are available and what their dependencies are.

Hackage additionally provides a signed file /snapshot.json (“the snapshot”), containing a hash of the index. When cabal downloads the index it computes its hash and verifies it against the hash recorded in the snapshot. Since mirrors do not have the key necessary to sign the snapshot (“ the snapshot key”), if the hash matches we know that the index we downloaded, and hence all files within, was the same as on the central server.
When you cabal install one or more packages, the index provides cabal with a list of packages and their dependencies. However, the index does not contain the packages themselves.

The index does however contain package.json files for each version of each package, containing a hash for the .tar.gz of that package version. Since these files live in the index they are automatically protected by the snapshot key. When cabal downloads a package it computes the hash of the package and compares it against the hash recorded in the index; if it matches, we are guaranteed that the package is the same as the package on the central server, as the central server is the only one with access to the snapshot key.
The client does not have built-in knowledge of the snapshot key. Instead, it can download /root.json (“the root metadata”) from the server, which contains the public snapshot key. The root metadata itself is signed by the root keys, of which the client does have built-in knowledge. The private root keys must be kept very securely (e.g. encrypted and offline).

This description leaves out lots of details, but the purpose of this blog post is not to give a full overview of TUF. See the initial announcement or the website of The Update Framework for more details on TUF; the Hackage Security project README provides a very detailed discussion on how our implemention of TUF relates to the official TUF specification.

Software architecture

Most of the functionality is provided through a new library called hackage-security, available from github, designed to be used by clients and servers alike.

Client-side

Although we have integrated it in cabal-install, the hackage-security library is expressly designed to be useable by different clients as well. For example, it generalizes over the library to use for HTTP requests; cabal uses hackage-security-HTTP, a thin layer around the HTTP library. However, if a client such as stack wants to use the hackage-security library to talk to Hackage it may prefer to use hackage-security-http-client instead, a thin layer around the http-client library.

Using the library is very simple. After importing Hackage.Security.Client three functions become available, corresponding to points 1, 2 and 3 above:

checkForUpdates :: Repository -> CheckExpiry -> IO HasUpdates
downloadPackage :: Repository -> PackageId -> (TempPath -> IO a) -> IO a
bootstrap       :: Repository -> [KeyId] -> KeyThreshold -> IO ()

Some comments:

A Repository is an object describing a (local or remote) repository.
The CheckExpiry argument describes whether we should check expiry dates on metadata. Expiry dates are important to prevent attacks where a malicious mirror provides outdated data (see A Look In the Mirror: Attacks on Package Managers, Section 3, Threat Model) but we may occassionally want to accept expired data (for instance, when the central server is down for an extended period of time).

The [KeyId] and KeyThreshold arguments to bootstrap represent the client’s “built-in” knowledge of the root keys we alluded to above. In the case of cabal-install these come from the cabal config file, which may contain a section such as

remote-repo hackage.haskell.org
  url: http://hackage.haskell.org/
  secure: True
  root-keys: 2ae741f4c4a5f70ed6e6c48762e0d7a493d8dd265e9cbc6c4037dfc7ceaec70e
             32d3db5b4403935c0baf52a2bcb05031784a971ee2d43587288776f2e90609db
             eed36d2bb15f94628221cde558e99c4e1ad36fd243fe3748e1ee7ad00eb9d628
  key-threshold: 2

(this syntax for specifying repositories in the cabal config is new.)

We have written an example client that demonstrates how to use this API; the example client supports both local and remote repositories and can use HTTP, http-client or curl, and yet is only just over 100 lines of code.

Server-side

The server-side support provided by Hackage.Security.Server comes primarily in the form of datatypes corresponding to the TUF metadata, along with functions for constructing them.

It is important to realize that servers need not be running the Hackage software; mirrors of the central Hackage server may (and typically will) be simple HTTP file servers, and indeed company-internal package servers may choose not to use the Hackage software at all, using only file servers. We provide a hackage-security utility for managing such file-based repositories; see below for details.

Trying it out

There are various ways in which you can try out this alpha release, depending on what precisely you are interested in.

Resources at a glance

hackage-security library github tag “pre-release-alpha”

cabal-install github branch “using-hackage-security”

hackage github branch “using-hackage-security”

Secure Hackage snapshots

We provide two almost-complete secure (but mostly static) Hackage snapshots, located at

http://hackage.haskell.org/security-alpha/mirror1
http://hackage.haskell.org/security-alpha/mirror2

If you want to use cabal to talk to these repositories, you will need to download and build it from the using-hackage-security branch. Then change your cabal config and add a new section:

remote-repo alpha-snapshot
  url: http://hackage.haskell.org/security-alpha/mirror1
  secure: True
  root-keys: 89e692e45b53b575f79a02f82fe47359b0b577dec23b45d846d6e734ffcc887a
             dc4b6619e8ea2a0b72cad89a3803382f6acc8beda758be51660b5ce7c15e535b
             1035a452fd3ada87956f9e77595cfd5e41446781d7ba9ff9e58b94488ac0bad7
  key-threshold: 2

It suffices to point cabal to either mirror; TUF and the hackage-security library provide built-in support for providing clients with a list of mirrors. During the first check for updates cabal will download this list, and then use either mirror thereafter. Note that if you wish you can set the key-threshold to 0 and not specify any root keys; if you do this, the initial download of the root information will not be verified, but all access will be secure after that.

These mirrors are almost complete, because the first mirror has an intentional problem: the latest version of generics-sop does not match its signed hash (simulating an evil attempt from an attacker to replace the generics-sop library with DoublyEvil-0.3.142). If you attempt to cabal get this library cabal should notice something is amiss on this mirror, and automatically try again from the second mirror (which has not been “compromised”):

# cabal get generics-sop
Downloading generics-sop-0.1.1.2...
Selected mirror http://hackage.haskell.org/security-alpha/mirror1
Downloading package generics-sop-0.1.1.2
Exception Invalid hash for .../generics-sop-0.1.1.2.tar45887.gz
when using mirror http://hackage.haskell.org/security-alpha/mirror1
Selected mirror http://hackage.haskell.org/security-alpha/mirror2
Downloading package generics-sop-0.1.1.2
Unpacking to generics-sop-0.1.1.2/

(It is also possible to use the example client to talk to these mirrors, or indeed to a secure repo of your own.)

Setting up a secure file-based repo

If you want to experiment with setting up your own secure repository, the easiest way to do this is to set up a file based repository using the hackage-security utility. A file based repository (as opposed to one running the actual Hackage software) is much easier to set up and will suffice for many purposes.

Create a directory ~/my-secure-repo containing a single subdirectory ~/my-secure-repo/package. Put whatever packages you want to make available from your repo in this subdirectory. At this point your repository might look like

```
~/my-secure-repo/package/basic-sop-0.1.0.5.tar.gz
~/my-secure-repo/package/generics-sop-0.1.1.1.tar.gz
~/my-secure-repo/package/generics-sop-0.1.1.2.tar.gz
~/my-secure-repo/package/json-sop-0.1.0.4.tar.gz
~/my-secure-repo/package/lens-sop-0.1.0.2.tar.gz
~/my-secure-repo/package/pretty-sop-0.1.0.1.tar.gz
```

(because obviously the [generics-sop](http://hackage.haskell.org/package/generics-sop) packages are the first things to come to mind when thinking about which packages are important to secure.) Note the flat directory structure: different packages and different versions of those packages all live in the one directory.

Create public and private keys:

# hackage-security create-keys --keys ~/my-private-keys

This will create a directory structure such as

~/my-private-keys/mirrors/id01.private
~/my-private-keys/mirrors/..
~/my-private-keys/root/id04.private
~/my-private-keys/root/..
~/my-private-keys/snapshot/id07.private
~/my-private-keys/target/id08.private
~/my-private-keys/target/..
~/my-private-keys/timestamp/id11.private

containing keys for all the various TUF roles (proper key management is not part of this alpha release).

Note that these keys are stored outside of the repository proper.

Create the initial TUF metadata and construct an index using

# hackage-security bootstrap \
                   --repo ~/my-secure-repo \
                   --keys ~/my-private-keys

This will create a directory ~/my-secure-repo/index containing the .cabal files (extracted from the package tarballs) and TUF metadata for all packages

~/my-secure-repo/index/basic-sop/0.1.0.5/basic-sop.cabal
~/my-secure-repo/index/basic-sop/0.1.0.5/package.json
~/my-secure-repo/index/generics-sop/0.1.1.1/generics-sop.cabal
~/my-secure-repo/index/generics-sop/0.1.1.1/package.json
...

and package the contents of that directory up as the index tarball ~/my-secure-repo/00-index.tar.gz; it will also create the top-level metadata files

~/my-secure-repo/mirrors.json
~/my-secure-repo/root.json
~/my-secure-repo/snapshot.json
~/my-secure-repo/timestamp.json

The timestamp and snapshot are valid for three days, so you will need to resign these files regularly using
```
# hackage-security update \
                   --repo ~/my-secure-repo \
                   --keys ~/my-private-keys
```
You can use the same command whenever you add any additional packages to your repository.
If you now make this directory available (for instance, by pointing Apache at it) you should be able to use cabal to access it, in the same way as described above for accessing the secure Hackage snapshots. You can either set key-threshold to 0, or else copy in the root key IDs from the generated root.json file.

Setting up a secure Hackage server

If you are feeling adventurous you can also try to set up your own secure Hackage server. You will need to build Hackage from the using-secure-hackage branch.

You will need to create a subdirectory TUF inside Hackage’s datafiles/ directory, containing 4 files:

datafiles/TUF/mirrors.json
datafiles/TUF/root.json
datafiles/TUF/snapshot.private
datafiles/TUF/timestamp.private

containing the list of mirrors, the root metadata, and the private snapshot and timestamp keys. You can create these files using the hackage-security utility:

Use the create-keys as described above to create a directory with keys for all roles, and then copy over the snapshot and timestamp keys to the TUF directory.
Use the create-root and create-mirrors commands to create the root and mirrors metadata. The create-mirrors accepts an arbitrary list of mirrors to be added to the mirrors metadata, should you wish to do so.

Note that the root.json and mirrors.json files are served as-is by Hackage, they are not used internally; the snapshot and timestamp keys are of course used to sign the snapshot and the timestamp.

Once you have created and added these files, everything else should Just Work(™). When you start up your server it will create TUF metadata for any existing packages you may have (if you are migrating an existing database). It will create a snapshot and a timestamp file; create metadata for any new packages you upload and update the snapshot and timestamp; and resign the snapshot and timestamp nightly. You can talk to the repository using cabal as above.

If you a have Hackage server containing a lot of packages (a full mirror of the central Hackage server, for instance) then migration will be slow; it takes approximately an hour to compute hashes for all packages on Hackage. If this would lead to unacceptable downtime you can use the precompute-fileinfo tool to precompute hashes for all packages, given a recent backup. Copy the file created by this tool to datafiles/TUF/md5-to-sha256.map before doing the migration. If all hashes are precomputed migration only takes a few minutes for a full Hackage snapshot.

Integrating `hackage-security`

If you want to experiment with integrating the hackage-security library into your own software, the example client is probably the best starting point for integration in client software, and the hackage-security utility is probably a good starting point for integration in server software.

Bugs

Please report any bugs or comments you may have as GitHub issues.

Roadmap

This is an alpha release, intended for testing by people with a close interest in the Hackage Security work. The issue tracker contains a list of issues to be resolved before the beta release, at which point we will make the security features available on the central Hackage server and make a patched cabal available in a more convenient manner. Note that the changes to Hackage are entirely backwards compatible, so existing clients will not be affected.

After the beta release there are various other loose ends to tie up before the official release of Phase 1 of this project. After that Phase 2 will add author signing.