Category Archives: Launchpad

Closing milestone bugs using Launchpad’s API

A few years ago I wrote a contrib script for Launchpad’s launchpadlib called ‘close-my-bugs.py’ which attempted to close (aka mark them ‘fix released’) all of your bugs in a project that were targeted to a particular milestone.

For various reasons it grew out of date and when I needed to use it recently, it didn’t work!  Long story short, I just fixed it up and added a couple of new features:

  • You can optionally close just your own bugs, or all the bugs in the milestone
  • You can search for bugtasks targeted against a series in your project (these are not normally picked up when searching in a project’s milestone)

You can grab the code here:

bzr branch lp:launchpadlib

contrib/close-my-bugs.py

Error handling in Go

There’s been a debate raging in some corners of the internet lately about how superior Go‘s error handling is to other languages.  I am going to address some of the points made, here:

Claim 1: It’s impossible to ignore errors in Go, they are “in your face”

This is patently false.  Take this example:

fmt.Println("Hello world")

Pretty innocuous wouldn’t you say?  Well let’s take a look at the language documentation for fmt.Println:

// Println formats using the default formats for its operands and writes to standard output.
// Spaces are always added between operands and a newline is appended.
// It returns the number of bytes written and any write error encountered.
func Println(a ...interface{}) (n int, err error)

So Println can return an error!  Where did we check it?  Well, we didn’t.  Any other claims that it’s OK to ignore it in this case further strengthen my argument.

Some will say that it’s a deliberate choice to ignore the error and I deserve all I get. Well, was it? I didn’t even know that Println returned an error until I looked at the documentation (and who is going to do that for Println?). And that’s the point, if I need to look at the documentation to see that it can return an error, then if I am using a language that raises exceptions I will have also seen its documentation about how it deals with errors.

You could even argue that an exception is superior in this case.  With Go, the code will march on regardless, oblivious to the fact that Println failed.  With exceptions, it’ll fail and show you exactly where it failed.

The language will error at compile time if you try to ignore an error returned as a second value and you only take the first.  But this is trivially bypassed by assigning it to _, which when reading code is easily missed compared to the exception style of “catching then dropping”, because Go itself encourages this style of assigning to _ with its own range statement as a deliberate way of ignoring things that the language is trying to force you to see.

So really in both cases, ignoring the error doesn’t really stand out as wrong.

Here’s a concrete example in Go I was recently shown:

w := bufio.NewWriter(os.Stdout)
for _, name := range ListAll(conf) {
    fmt.Fprintln(w, name)
}
w.Flush()
return

As you can see, the caller completely forgot to check the error returned from Fprintln and Flush and there would be no compiler warning about it.

Claim 2: Exceptions teach developers to not care about errors

Citing an example where someone didn’t catch an exception and the code consequently blew up is really not a good example of this claim.  It’s a bug, for sure and you get a full traceback of your error in the resulting exception, which is handy.  You go away and fix it quickly based on that info.

If I am in the same situation with Go and I ignore a returned error from a function, at some point (which is likely to be nowhere near the place where the error occurred) my code will blow up.  I’ll have to run up the debugger to try and find out where it really occurred though.

Because unused variables in Go are a compile-time error, it’s actively discouraging you from assigning the result of the function to a variable (or you can deal with it, of course).  For anyone who’s not read the full documentation for a function call or missed its return value (we’re all human) as I said above – you’re not even going to notice that you missed it.

Based on this, I can see no difference at all that suggests one way or the other teaches developers to not care about errors.  Developers do care about errors, really, but bugs creep in however careful you are.  And when they do, I’d rather have a decent indication of where the bug is.

Other parts of error handling that I dislike

When you look at the average Go program, you will see a lot of this:

if err != nil {
  return nil, err
} 

This is the recommended way of error handling in in Go.  But this is not error handling, it’s error propagation.  In nearly all languages there will arise situations where in well-factored code you have a low-level error that you need to pass right back up to the entry point for the caller. That means you need this error propagation code in every single place where you check for errors.  There’s no syntactic sugar, just the same three lines everywhere.

For me, this vastly decreases the readability of the code. This is where exceptions excel because inside my own library I can factor the bejeesus out of it into many small functions and if I need to return an error, I just catch a lower-level exception in the top-level function and return something else.  You can do this in Go with a panic(), but it seems to be discouraged.  Panic() feels almost exactly like using exceptions, only the syntax is worse. If Go’s style is to encourage people to handle errors like this, it needs the sugar.

Conclusion

Many people might think that I completely hate Go’s error handling from this post.  That’s not strictly true – I don’t hate it, I just think it can be improved.  I challenge assumptions that I see which state that Go’s error handling is superior in some way, when as far as I can see it’s not that different from other languages in terms of usefulness.

Go is clearly in its infancy.  Most languages will have started out with youthful enthusiasm and realised that some change was needed.  These languages are the successful ones where developers enjoy coding in it and feel productive.  I hope that Go embraces change as it matures and attracts more developers.

I welcome comments on this post – unlike some people I won’t censor them or delete ones I can’t argue with (unless they are outright abusive and use foul language, this is a family blog!).

A KDE user’s view of Unity – Day One

As most of my colleagues know, I am a KDE fan.  I love using Kubuntu, despite its lack of “official” attention from Canonical.

I’ve heard all this fuss about Unity and I’ve no real idea what it’s all about, so I thought I’d give it a try.  Today will be my first full day of using it (and the last for 2 weeks since I am heading out on vacation) and I will attempt to write up my experiences.  It’s a bit of a brain dump, but here follows day one!

First impressions

My initial impression is that it looks pretty slick, but not as pretty as KDE4.  I’m clicking around a bit to see what’s what and I’ve immediately noticed a few things that I will miss from KDE:

  • There doesn’t seem to be a way to put widgets on the panel – I want my CPU/MEM/SWAP meter! (Some time later I found “indicators” in the Software Centre, there’s a system load indicator, but after installing there’s no obvious way of using it)
  • I am used to ctrl-F<N> to switch virtual desktops, I can’t see a way of emulating that.  (Ok I found the setting an hour later, great)
I’m also not sure about the global menu thing, I’m finding it a lot more effort to mouse around to get to menus.
Right, so let’s try and configure things as I like them.  First, I am a focus-follows-mouse junkie (I first used X Windows in the early 1990s with twm!) so let’s poke around in the settings.  Hmmm, doesn’t seem like I can change that… Ok some quick Googling reveals I have to install a so-called compiz settings manager and run ccsm. Right, I can set it in there.  That’s bizarre that I have to install something to edit more settings!
Now, focus-follows-mouse  has also made the global menu even less desirable as it changes to whatever app I mouse through to get to it. Eugh.  (I realise this is a concious design decision by the Unity team, rather than a shortcoming though – doesn’t mean I have to like it!)

Visual differences

Ok so I’ve had some more time to get used to the layout now.  More thoughts:
  • The icons on the left have some weird triangles on them, some solid some not.  I’ve no idea what they mean, but I am sure I will find out at some point.
  • It seems like I have to run an application before I can add it to the left panel as a quick launcher (by setting “Keep in Launcher”)
  • The KDE file browser Dolphin is massively better than Nautilus (I’ll write more about that some other time)
  • The mail indicator is a nice idea at the top right, but it wants to use Evolution.  I don’t! How can I make it use kmail?
  • Having a music player integrated into the volume indicator is a nice touch. I’d like to use Amarok though; it lists it but there’s no controls for it.
  • Clicking on the time indicator shows a calendar, like KDE.  I suspect I can’t use an arbitrary calendar app though, it seems tied to evolution.
  • The logout button shows a load of seemingly arbitrary menu items. Odd.

Interaction

As a KDE user I love krunner (a bit like Gnome-Do). So one my reactions is to hit alt-f2 when I want to run something.  I see that it starts up a smiliar dialog as hitting the button at the top left.  I started typing the name of the program I wanted to run, “kmail”, and it found it quickly.  I hit “enter” to run it but nothing happened.  I noticed that the kmail icon was not on the list of icons found any more!  I then noticed that if I backspace my input one character, so it says “kmai”, then the kmail icon appears again.  This must be a bug I guess.
I decided to have a play with the menu system a bit.  It seems a bit more work to find stuff than simply mousing around the K menu but I’ll keep trying it out and see how I get on.  My initial impression is that it needs some work though as it feels as though it’s trying to hide results for no reason until you click on “See N more results →”.  I don’t understand why it doesn’t put a scroll bar up and show everything right away, I’m sure this would be quicker to navigate.
Dealing with windows:
  • The window controls have moved to the opposite corner.  Not a problem, but hard to get used to.
  • alt-tab selects windows that are not on the current virtual desktop.  This is *really* annoying 🙁
The final point to note today is that there appears to be no way to save my session and have it restored at login.  This is a feature I rely on extremely heavily in KDE and I am really sad to see it not available in Unity.
I hope to write ongoing reports over the next week or so.  I am actually away on holiday so it will be sporadic, but I’m sure I will find some moments.

Twisted is great!

I’ve been learning a lot of Twisted lately and I’m really enjoying it.  Back in the day when I did C++ coding I almost exclusively used event-driven programming, so getting back into it again with a Python twist (pun fully intended) is making my coding juices flow once more.

One of the drivers for me to learn more about it is to be able to maintain the Launchpad Build Farm manager, which is written mostly using Twisted events.  When it was first put together by Celso to replace the old slave scanner, it was quite a tough task to slot it into the existing Launchpad code base without too much disruption.  He did a pretty good job, but now the code is in need of an overhaul as it scales quite badly.

There’s three main reasons that it scales badly:

  1. It scans each builder in turn once per “cycle” that it wakes up to do its scan
  2. It synchronously uploads build results, blocking further progress until the upload completes
  3. It synchronously polls the builders before setting up Twisted Deferreds to do the actual dispatch in parallel.

Jelmer is currently tackling (2) above, and I plan on sprinting with Jono in a couple of weeks to tackle (3).

I started tackling (1) a while ago now as it was obvious there was a better way.  So now instead of polling each builder in turn, it sets up a separate Deferred event for each builder in the system and lets it fire independently of all the others.  Because the code is not really threaded, there are no race conditions or exclusivity to worry about and we get a couple of really big wins from this approach:

  • If there’s any kind of exception during the scan, it only takes out the current scan cycle on a single builder, not all of them.  Other builders continue to dispatch quite happily.
  • We can now overlap events better.  Previously, we did all the dispatching at the same time and that involves mostly sat waiting for the reset scripts to finish on the virtual builders (which sometimes take up to 30 seconds).  Now, we can kick that off and poll another builder or upload a build at the same time.  Throughput has increased slightly but the overall effect is dramatic on the build farm page as you no longer see builders idle for long periods even though there’s a big queue.

This code went live in production a month ago and we had a massive problem with it immediately that I had not thought of.  Because the builder reset events (which are a sub process) now happen in parallel with other non-Twisted events they get interrupted with SIGCHLD when the sub process finishes – this manifests itself as an EINTR exception in the middle of a comms operation with another builder.  We added some code to retry the operation in this case and it finally worked!  I’m really happy that this was the only problem. 🙂

However, the really, really big win in scalability will come with Jelmer’s fix which will land Real Soon Now.  This will remove the blocking that happens when uploading builds – a block that can last for over a minute for large uploads.  I can’t wait to see this branch land.

Here’s to more Twisted!

Launchpad SFTP upload service

We launched (forgive the pun) the SFTP service to upload packages securely to Ubuntu and PPAs a few months ago, but I’m not seeing much use of it.  I would really appreciate it if more people would give it a go!

It also has the bonus that it doesn’t suffer from the bug where uploads sometimes hang with 1k to go, as on the regular FTP server.

Configure your dput.cf with these changes for Ubuntu:

method = sftp
incoming = ubuntu/
login = <your LP account name here>

For PPAs it’s similar, just make sure your fqdn and incoming are set appropriately.

Let me know in the comments how you get on.

Feeling productive?

As the lead on the Launchpad Soyuz project, I have to deal with a lot of different tasks at the same time.  I hate this as much as the next engineer, but burying my head in the sand and hoping things will just be all right simply won’t work.

At the end of the day I was feeling really unproductive and anxious about time wasted – I’d sit back and think, what did I do today?  I felt really busy but I can’t remember what the heck I’ve been doing! Even worse, I’d leave my office at the end of the day in a bad mood and the 30 seconds it takes to get back into the house is not enough to ease the tensions, which made it hard to interact with my wife and kids.  I want to reclaim that precious hour between finishing work and the kids going to bed!

So, what can I do about this?

Three things:

1. I made a conscious effort to finish my major work items 30 minutes before leaving the office and to spend those 30 minutes doing all the simple tasks that you know you need to sort out but never get around to doing.  This has had a dramatic affect on not only how I feel when I finish, but also I get the warm fuzzy feeling of having cleared out stuff that was niggling me at the back of my brain.

2. I have eschewed technology in favour of the classic pen and paper!  I used to keep TODO lists electronically, but I found this hard to keep up with as they were always hidden by other windows on another virtual desktop somewhere.  So now,  each week I start on a fresh piece of paper and write down all the things I know I need to do, with a little box next to them.  When I’ve done that task, I can put a great big satisfying tick next to it.  At the end of each day I can see visible progress.  This is very uplifting!  I start a new sheet each week so that the TODO list is not cluttered, and it also forces me to think of anything I missed.

3. Take regular breaks.  I know, this is easier said than done.  Michael Nelson put me on to the Pomodoro Technique which while I don’t follow religiously, does give some effective tips on how to time-box your activities.

If you’ve got any more tips, or used mine and found them useful, I’d love to hear about it!  Please leave a comment.

Success in Wellington

One of the  things that Launchpad is working on is the daily builds project, where we take a source branch, a recipe and turn it into a source package that can be built.

It turns out that this is quite hard because you need a secure environment to run the Bazaar recipe builder in.  Fortunately we already have the Launchpad Build Farm!  So, on January 11th the Soyuz and Codehosting teams, plus Launchpad Strategist Jono Lange and community contributor William Grant all met at the West Plaza Hotel in Wellington, New Zealand for a week-long coding sprint with the goal of getting a recipe build job through the build farm before the week was out.

The team hard at work

The amount of work was pretty sizeable, but we’d already done quite a lot of preparatory work before leaving so that non-Soyuz developers could get involved from day one.

We split the task into a few areas:

  • Writing the code that runs on the build slave itself
  • Job dispatching
  • Job collection
  • Job candidate selection
  • Creating a generalised build infrastructure to make it easy to plug in new job types

The team split into coding pairs that mostly lasted the week and there were no bloodied noses!  We had a strict rule that no pair could contain two people from the same Launchpad team, so that we could spread the knowledge around better.  This worked out really well.

The local coffee shop certainly had plenty of custom that week!  But by the end of it, on Friday afternoon at about 5:45pm, we finally got a recipe build job to run through a local Launchpad instance on William’s laptop and eventually turned into a binary package.  Success!

We’ll be building on this work in the next couple of months so that it has a UI that will allow a user to click on a branch and get it built and uploaded to their PPA.  It’s all very cool!

Finally, huge thanks to everyone involved, including the hotel.  It was the best wifi/internet anyone had ever had at any hotel, ever.

Bjorn, William and Michael
The team celebrates after a long week

Soyuz 3.0 review

The Soyuz team was very, very busy over the last year fixing lots of bugs and adding plenty of new features.  These are the highlights that I’d like to mention!

New features

  • Multiple PPAs per person – split your packages into different repositories without the hassle of creating new LP users
  • Karma for uploads! Distro and PPA uploaders (and the package creator if different) will be recognised for their work and get karma.
  • Massively improved webservice APIs to control various operations, such as copying packages, manipulating builds, inspecting PPAs etc.  Allows script-based control of many soyuz-related operations.
  • Hugely faster build farm scanner, builds are dispatched a LOT quicker now.  That means there’s less waiting around for your packages to get built.
  • Private PPA subscriptions / tokens.  People can now control who is able to download their PPA software.
  • Package sets and their upload ACLs implemented for Ubuntu.  Karmic and onwards will be using package sets for upload permissioning.
  • Security in Soyuz. Complete support for the Canonical security team to use a private PPA and directly unembargo packages from it

3.0 UI

We also did a lot of complete page redesigns for 3.0:

  • PPA page split into two pages; one user-focused, one developer-focused
  • New build page, with a cleaner look
  • New global /builders page; sortable table data and a time-based estimate of the queue sizes
  • New distribution source package release page; much better presentation of data
  • New distribution source package page; makes more use of upstream package description/logo etc.
  • Distro series source package release page gets a new layout
  • Builder +admin and builder +edit collapsed into a single page
  • New builder page

But the work doesn’t stop here.  We’re already thinking about 4.0!

Soyuz template conversions all done!

Thanks to the hard work from everyone in the Soyuz team, the Soyuz effort to convert all our templates to the new 3.0 UI standard is all done!  If you’re a beta tester, the new pages can be seen on on the edge service already.

Many of the changes were complete page re-designs based on better knowledge of use-cases that has been gathered over the years.  There was certainly a lot of dust on some of them!

There are still some minor tweaks to be made, but all feedback is most welcome.

A foray into virtual machines

One of the things that’s frustrating about being a Launchpad Developer is waiting three hours for the test suite to complete.  It’s always irked me that my quad-core sat there with three idle cores, and parallelising the tests would be tough, so I thought I’d try and partition the tests across some virtual machines.

I know pretty much nothing about VMs so I was pretty pleased to come across this JeOS VMBuilder resource!  I got my first VM built pretty quick with it, but then, what next?  The instructions don’t tell me how to actually run up my new VM.

So, I had a little chuckle to myself when I saw this blog post from Dustin Kirkland just appear!  I’m going to try his suggestions out now and see how it works out, but it looks like just what I need.