Recent Hackage improvements

You may or may not have noticed but over the last few months we’ve had a number of incremental improvements to the Hackage site, with patches contributed by numerous people.

I’m very pleased that we’ve had contributions from so many people recently. Apart from one patch that took a long time to merge we’ve generally been reasonably good at getting patches merged. Currently there’s just 1 outstanding pull request on hackage-server’s github site.

I gave a talk a couple months ago at the London Haskell User Group about getting started with hacking on Cabal and Hackage. Unfortunately the video isn’t yet available. (I’ll try and chase that up and link it here later).

An idea we floated at that talk was to run a couple hackathons dedicated to these and related infrastructure projects. If you want to help organise or have a venue in London, please get in touch. If you can’t get to London, fear not as we’d also welcome people attending online. Of course there’s also the upcoming ZuriHac where I expect there will be plenty of infrastructure work going on.

If you do want to get involved, the github site is the place to start. Discussion of features happens partly in issues on github and the #hackage channel on IRC. So those are good places to get feedback if you decide to start working on a bug or feature.

Recent changes

Visible changes

• Code to render README and changelog files as markdown has been merged, though this is not yet in its final form.

  The idea is that these days many packages have good README files but less good descriptions, partly because Haddock markup has never been very good for writing long prose descriptions while markdown is much better suited for that. So the goal is to display package READMEs in a useful form. Some people would like to be able to just write a README and not have to partially duplicate that in the .cabal file description, and we want to support that option.

  So while the code has been merged (which is good because it was big and had been suffering bitrot), there is still a question of exactly when to display the README. The initial patch would show it in place of the description whenever the README existed, while a later revision would show both the description and README. Unfortunately while this greatly improved things for some packages it made things greatly worse for others.

  We are now discussing when to show the README inline so that it benefits those packages that have good READMEs, without making other packages much worse. Suggestions include only showing the README when the description field is blank, and putting the README at the bottom of the package page like github does to cope better with very long READMEs, or using an expander widget. We very much welcome feedback on this question, and pull requests even more so.

  In the meantime while that is being sorted out, as a temporary solution, the package page just has a link to the README rendered as markdown. If nothing else this makes it easy to see what various packages READMEs would look like when included inline (including some rendering issues due to the variety of markdown dialects).

  Thanks to Christian Conkle and Matthew Pickering for the original patch, and to Matthew Pickering and Michael Snoyman for fixing the bitrot and pestering us to get it reviewed and merged. It’s all much appreciated.

• Changelogs are now also rendered as markdown. This doesn’t always work because some changelogs are not markdown and look bad when rendered as such, so we also provide a plain text link. See for example the changelog for zlib-0.6.1.0.

  The changelog link has been moved to the list of properties, immediately after the list of all versions. We’d had feedback that people had been overlooking the changelog link where it was previously.

  Future directions here include displaying the changelog inline, or parts or with an expander.

• The display of dependencies and version ranges has been improved. Since the early days of Hackage we used a disjunctive normal form presentation. This has never been particularly visually appealing or easy to understand as well as rather verbose. We now present a much shorter summary of the dependencies and their versions constraints. This display has been tweaked a few times to make it a more accurate summary.

  Thanks to Kristen Kozak for making several improvements and to Michael Snoyman for feedback.

• License files are now linked from the package properties. So for example rather than just saying “OtherLicense”, it now links directly to the license file inside the package tarball. See for example iconv-0.2.

• The modules list on the package page now only links to Haddock pages that actually exist. This solves the problem of libraries with “semi-public” modules that are technically exposed but where they have a haddock HIDDEN pragma. The haddock index is now also linked.

• Browsing the content of the package tarball is much improved. We now show a full directory listing, rather than just one directory at a time, and the presentation is generally clearer. See for example the contents of zlib-0.6.1.0.tar.gz.

• Improvements to the mime types of files served from the contents of package documentation or tarballs. We now declare more things to be UTF8 and text/plain, so they’re easier to browse online rather than triggering your browser to download the file.

  Thanks to Rabi Shanker and Adam Bergmark.

• The recent upload page now links to another page (and corresponding RSS feed) which lists all revisions to .cabal files from authors and trustees editing .cabal metadata. These feeds now always contains at minimum 48 hours worth of changes.

  Thanks to Gershom for this.

• A new audit log for Hackage administrators. All changes to user’s permissions (by adding and removing them from groups) are now logged, with an optional reason/explanation, and this is visible to the admin.

  Thanks to Gershom for contributing this feature and Herbert Valerio Riedel for agitating to get it included.

• The presentation of edits/diffs between .cabal file revisions has been improved to identify which component the change was in (since sometimes similar changes have to be made to multiple components in a package). See for example the revisions to fay-0.18.0.4.

  Thanks to Rabi Shanker.

A few boring but important ones

• Build fixes for newer GHC versions. People run hackage-server instances locally so it’s always useful to be able to build with the latest, even if the central community site is using an older stable GHC for its build.

• Potential DOS fix. Older versions of the aeson package had a problem with parsing large numbers that would cause it to take huge amounts of memory. We had been stuck with an old aeson package for a while due to incompatible changes in the JSON output for derived instances.

  Thanks to Michael Snoyman for reporting that and Daniel Gröber for fixing it.

• Updated to Cabal-1.22. This has a number of small effects, like support for new licenses, and compilers (HaskellSuite).

  One regression that we discovered is that Cabal-1.22 changed one of the distribution Q/A checks on the -Werror flags, which meant that Hackage started rejecting these, even when guarded by a manual “dev” flag.

  I have since sent in a patch for Cabal to allow -Werror and other development flags when guarded by a manual cabal configuration flag. This lets people use -Werror in development and in their CI systems but without us having on Hackage lots of packages that bitrot quickly due to GHC introducing new warnings and breaking packages that use -Werror by default. This Cabal change has been cherry-picked into the community hackage-server instance for the time being and is now deployed.

  Thanks to the many people who pointed out that this had changed as was a problem, and to Herbert Valerio Riedel for identifying the source of the problem.

• Related to the Cabal version update is a change in the way that AllRightsReserved packages are handled. Cabal used to complain about distributing AllRightsReserved packages, but that’s not appropriate at that level of the toolchain. Hackage has been updated to implement that check instead, but only for the community’s public instance. If you run your own local Hackage server then it will now accept AllRightsReserved packages.

Miscellaneous small changes

• In the administrators’ control panel, old account requests and resets are now cleaned up.

• Internally there is a new cron subsystem which will be useful for new features.

• More resources are available in JSON format. Thanks to Herbert Valerio Riedel for pointing out places where this was missing.

• Improved README instructions for the hackage-server itself. Contributed by Michael Snoyman.