Several people have been wondering recently what is up with the new hackage-server implementation. At the weekend at HacPDX-II we made quite a bit of progress in fixing things and getting more people involved.
The HacPDX-II hackathon took place this last weekend in Portland Oregon. Thomas DuBuisson contacted me a week or two before to see if we could make the new hackage-server a focus for the hackathon and get several people involved. This worked out quite well, we got several people who attended HacPDX-II helping out. Also, just the fact that Thomas declared that this project would be the focus meant that it energised some of the rest of us who are interested in the hackage-server but who could not attend HacPDX-II in person, including Matthew Gruen and Jeremy Shaw. So before getting into the technical details I'd really like to thank Thomas for organising things and to Matthew and Jeremy for giving up much of their weekends to help us out.
Jeremy Shaw, of Happstack fame amongst other things, sent in two excellent patches to shift hackage-server to using newer infrastructure. One patch updated everything to using the latest Happstack 6.x version, and the other converted from using happstack-data to the newer acid-state package.
David Lazar and Thomas DuBuisson got stuck in, tried things out and found and fixed various bugs. In particular, David fixed some things to do with the interface that package maintainers can use to mark packages as deprecated, and Thomas fixed up some stuff to do with package tags.
Matthew Gruen, who did his GSoC project with me on the hackage-server last summer, spent much of the weekend hacking, debugging and answering questions from the rest of us. In particular he and I spent quite a while thinking about HTTP authentication issues.
Originally, last summer, we had the crazy idea that we could offer either HTTP Basic or Digest authentication based on the user account. The reason we wanted to do this is because the current ~800 user accounts on the central hackage.haskell.org server all use HTTP basic auth (stored in Apache's standard htpasswd database format) but really we'd like to migrate to using HTTP digest auth because it's more secure than basic auth. Unfortunately, due to the way HTTP authentication works, having some user accounts using basic and some using digest authentication just will not work. That means we cannot do a smooth transition from basic to digest for the existing user accounts. So we had a think about what system to go for and how to clean up the mess in the current code that tries to offer different auth systems to different users.
For the moment what we've decided is to go for digest authentication exclusively and to think about how to do migration separately. This is partly so that we can get something working now, because several people want to use the hackage-server now, it's not just the central hackage.haskell.org that we have to think about.
One option for migration is to import all the old accounts and have a special re-registration page for existing users. That would authenticate them using their old passwords and then enable their new accounts with the new digest authentication system. The point is it would not be transparent, but it should not be too much of a bother.
Going for digest authentication means that we had to fix a few bugs in the client side HTTP implementation. Matthew tracked down the problem and I've sent the patches off to the maintainer of the HTTP package that cabal-install uses.
I spent a few days before the hackathon working on the hackage-mirror client. This is a program included in the hackage-server package that can synchronise packages between servers. In particular it can copy packages from the old hackage.haskell.org to your own hackage-server instance. Along with the HTTP digest work, on the final day of the hackathon I managed to get it to sync packages from hackage.haskell.org to my hackage-server running on localhost and to do the uploads authenticated using digest auth. So that was quite a good conclusion to things.
One thing that worked quite well for the hackathon was that we used a common darcs repository and gave everyone write access. We all just worked away and darcs pushed and pulled using the common repository. This is a model where darcs works really well. Now that the hackathon is over, I'll move the changes into the main repository.
So there's still plenty to do, but I think we've shown that the new hackage-server implementation is nearly ready to be used, for new server instances at least. I'm going to be doing more work on the mirroring, because a client of Well-Typed wants to use it, so I expect to post an update on that sometime soon.