Michael Brunton-Spall

Tech Weekly podcast: when books go social

bruntonspall — Mon, 09 Jan 2012 11:20:16 +0000

I blathered on the podcast for a bit again.

This article titled “Tech Weekly podcast: when books go social” was written by Presented by Charles Arthur and produced by Matt Hill, for guardian.co.uk on Tuesday 3rd January 2012 17.36 UTC

How do you discover new books? Matteo Berlucchi is tired of buying books from a certain online retailer only to be recommended very similar reads. He tells us why his aNobii.com is different.

Charles Arthur sits in the presenter hotseat for this week’s programme. He’s joined by Josh Halliday and Michael Brunton-Spall to discuss the week’s developments, including what Rupert Murdoch’s supposed Twitter feed tells us about the verification process and why those network crashes on New Year’s Eve look to be a thing of the past.

Plus, Aleks Krotoski talks to Chad Dickerson from vintage retailer Etsy about its rise and rise, and how to scale up a business without losing your customer focus.

Don’t forget to…

• Comment below
• Mail the producer tech@guardian.co.uk
• Get our Twitter feed for programme updates or follow our Twitter list
• Like our Facebook page
• See our pics on Flickr/Post your tech pics

guardian.co.uk © Guardian News & Media Limited 2010

 Published via the Guardian News Feed plugin for WordPress.

Map, map and flatMap in Scala

bruntonspall — Fri, 02 Dec 2011 10:56:39 +0000

Scala (stairs) by Paolo Campioni

One of the things I like about Scala is it’s collections framework. As a non CS graduate I only very lightly covered functional programming at university and I’d never come across it until Scala. One the benefits of Scala is that the functional programming concepts can be introduced slowly to the programmer. One of the first places you’ll start to use functional constructs is with the collections framework.

Chances are your first collection will be a list of items and we might want to apply a function to each item in the list in some way.

Map works by applying a function to each element in the list.

scala> val l = List(1,2,3,4,5)

scala> l.map( x => x*2 )
res60: List[Int] = List(2, 4, 6, 8, 10)

So there are some occasions where you want to return a sequence or list from the function, for example an Option

scala> def f(x: Int) = if (v > 2) Some(v) else None

scala> l.map(x => f(x))
res63: List[Option[Int]] = List(None, None, Some(3), Some(4), Some(5))

flatMap works applying a function that returns a sequence for each element in the list, and flattening the results into the original list. This is easier to show than to explain:

scala> def g(v:Int) = List(v-1, v, v+1)
g: (v: Int)List[Int]

scala> l.map(x => g(x))
res64: List[List[Int]] = List(List(0, 1, 2), List(1, 2, 3), List(2, 3, 4), List(3, 4, 5), List(4, 5, 6))

scala> l.flatMap(x => g(x))
res65: List[Int] = List(0, 1, 2, 1, 2, 3, 2, 3, 4, 3, 4, 5, 4, 5, 6)

This comes in really useful with the built in Option class because an option can be considered a sequence that is either empty or has 1 item.

scala> l.map(x => f(x))
res66: List[Option[Int]] = List(None, None, Some(3), Some(4), Some(5))

scala> l.flatMap(x => f(x))
res67: List[Int] = List(3, 4, 5)

So with that all covered, lets look at how you can apply those concepts to a Map. Now a map can be implemented a number of different ways, but regardless of how it is implemented it can be thought of as a sequence of Tuples, where a tuple is a pair of items, the key and the value.

scala> val m = Map(1 -> 2, 2 -> 4, 3 -> 6)
m: scala.collection.immutable.Map[Int,Int] = Map(1 -> 2, 2 -> 4, 3 -> 6)

scala> m.toList
res69: List[(Int, Int)] = List((1,2), (2,4), (3,6))

We can access a tuple by accessing the inner variables _1 and _2

scala> val t = (1,2)
t: (Int, Int) = (1,2)

scala> t._1
res70: Int = 1

scala> t._2
res71: Int = 2

So we want to think about using map and flatMap on our Map, but because of the way a map works it often doesn’t make quite the same sense, we probably don’t want to apply a function to the tuple, but to the value side of the tuple, leaving the key as is, so for example we might want to double all the values. Map provides us with a function to do exactly that.

scala> m.mapValues(v => v*2)
res73: scala.collection.immutable.Map[Int,Int] = Map(1 -> 4, 2 -> 8, 3 -> 12)

scala> m.mapValues(v => f(v))
res74: scala.collection.immutable.Map[Int,Option[Int]] = Map(1 -> None, 2 -> Some(4), 3 -> Some(6))

But in my case I wanted to do something more like flat map in this case, I want a map to come out that misses out the key 1 because it’s value is None. flatMap doesn’t work on maps like mapValues, it get’s passed the tuple and if it returns a List single items you’ll get a list back, but if you return a tuple you’ll get a Map back.

scala> m.flatMap(e => List(e._2))
res85: scala.collection.immutable.Iterable[Int] = List(2, 4, 6)

scala> m.flatMap(e => List(e))
res86: scala.collection.immutable.Map[Int,Int] = Map(1 -> 2, 2 -> 4, 3 -> 6)

Ok so we are pretty close to using options with flatMap, we need to filter out our None’s, we can do returning a list with just e => f(e._2) and we’ll get the list of values without the None’s, but that isn’t really what I want. What I need to do is return an Option containing a tuple. So here’s our updated function:

scala> def h(k:Int, v:Int) = if (v > 2) Some(k->v) else None
h: (k: Int, v: Int)Option[(Int, Int)]

and here’s how we might call it:

scala> m.flatMap ( e => h(e._1,e._2) )
res109: scala.collection.immutable.Map[Int,Int] = Map(2 -> 4, 3 -> 6)

but this is pretty ugly, all those _1 and _2′s make me sad. If only there was a nice way of unapplying the tuple into variables. Given that this works in python and in a number of places in scala I thought this code should work:

scala> m.flatMap ( (k,v) => h(k,v) )
:10: error: wrong number of parameters; expected = 1

I spent way too long today looking at this (in 5 minute chunks broken by meetings to be fair), before I gave in and asked a coworker what the hell I was missing. The answer is seems is that an unapply is normally only executed in a PartialFunction, which in scala is most easily defined as a case statement. So this is the code that works as expected:

scala> m.flatMap { case (k,v) => h(k,v) }
res108: scala.collection.immutable.Map[Int,Int] = Map(2 -> 4, 3 -> 6)

Note that we switch to using curly braces, indicating a function block rather than parameters, and the function is a case statement. This means that the function block we pass to flatMap is a partialFunction that is only invoked for items that match the case statement, and in the case statement the unapply method on tuple is called to extract the contents of the tuple into the variables. This form of variable extraction is very common, and you’ll see it used a lot.

There is of course another way of writing that code that doesn’t use flatMap. Since what we are doing is removing all members of the map that don’t match a predicate, this is a use for the filter method:

scala> m.filter( e => f(e._2) != None )
res114: scala.collection.immutable.Map[Int,Int] = Map(2 -> 4, 3 -> 6)

scala> m.filter { case (k,v) => f(v) != None }
res115: scala.collection.immutable.Map[Int,Int] = Map(2 -> 4, 3 -> 6)

scala> m.filter { case (k,v) => f(v).isDefined }
res116: scala.collection.immutable.Map[Int,Int] = Map(2 -> 4, 3 -> 6)

Annoyed by Guardian Facebook app?

bruntonspall — Sat, 12 Nov 2011 15:26:26 +0000

Are your friends sharing links to the Guardian Facebook app in their twitter feeds but you don’t use Facebook and want to see the original guardian page?

Unfortunately due to the way the web (and Facebook) works, if somebody copies the URL in their browser and posts it to twitter you’ll be looking at the same thing as them, and if that was a guardian Facebook app, but you haven’t logged into Facebook, Facebook wont tell us anything so we can’t redirect you back to the guardian.

However, most modern browsers support special bookmarks with javascript in them, called Bookmarklets, and since the original Facebook url has the important path data needed for the guardian site, it’s possible to manipulate the current url to take you back to the guardian site without logging into Facebook.

Basically, drag this link: to your bookmarks toolbar up above this page, then click this link: Are these the 10 heaviest albums ever made? and then click the new fb->g link on your toolbar and you should go to the guardian site.

This will only work if you are not logged into facebook.

How does it work?

The Bookmarklet is a link with a javascript protocol (before the colon). Your browser will execute the code when you click the button, and it will do so within the context of your current page.

It gets the current url, and looks for a parameter called cancel_url. That’s where the url for the original apps.facebook.com is. Next we replace apps.facebook.com/theguardian with www.guardian.co.uk, then we update window.location with the result. This causes your browser to load the new url, and you should be on the guardian site.

Note that it does no error checking, it does not handle unusual guardian urls (that might not start www.guardian.co.uk) nor does it check whether there is a cancel_url before acting.

You can view the source and repurpose for your own uses at jsfiddle.

Adding Google Plus redirect to your Nginx powered site

bruntonspall — Sat, 12 Nov 2011 12:55:02 +0000

A quick one, this morning I’ve added the plus url to my website, so http://www.brunton-spall.co.uk/+ now redirects to my Google+ profile.

This, it turns out, is really easy to implement if you run Nginx as the front to your website.

You simply need to update your server configuration to include the following snippet:

location ~ ^/?+$ {
  rewrite ^ https://plus.google.com/YOUR_NUMBER permanent;
}

This will ensure that your domain.tld/+ issues a redirect to your Google+ profile.

Hope this helps

Identifiers are not numbers

bruntonspall — Sat, 24 Sep 2011 12:13:14 +0000

“I am not a number, I am a free man”

So something bugs me (I know, big shock), and it’s API documentation like this:

The document will be returned as:
ID: Integer
Name: String
Published: Date/Time

My position on this is simple, in an API (we’ll come back to internal design later), an ID should always be of String type.

So why is this a big deal? I mean you implement this with a database table with autoincrement right, so it is a number underneath?

There’s a few issues with this and I’ll deal with them in reverse stupidity order.

Numbers are more than identifiers

A number represents a mathematical concept. It implies strict ordering (2 is always > than 1), it implies a set of operations (1+2=3) and it implies an infinite set. If you provide external developers with numbers they are going to assume that these implications are available and might make some form of sense.

You would hope that most people wouldn’t assume that something like twitter ids can be added or multiplied and expect anything sensible from them, but that comparison function, that’s a bit more tricky. It’s true that tweet 1 was sent before tweet 2, and even that tweet 1,000,000,000 was sent before tweet 2,000,000,000. But as twitter scales guaranteeing that ordering becomes harder and harder in a massively parallel system, and there is no value in a client ordering tweets this way. Each tweet has a date/time field on it indicating when it was sent, and ordering based on that field is far more semantically correct (and what the user expects).

Numbers aren’t equal in all languages

If you argue that representing something as a number is saving you memory, because an integer only occupies say 2 bytes, you are probably prematurely optimising. Number representation is a very complex subject, one which I know only a little, but I know enough to know that I should never make assumptions about how a number is represented internally.

I come from a C background, and despite it being fairly common knowledge how big various numbers are stored, we have a sizeof operator for a reason. Compiling your program on a slightly different architecture could change your integers from 2 bytes to 4 bytes to 8 bytes to anything inbetween. It’s a pretty common source of bugs for new developers to assume that unsigned int a is 4 bytes in size.

When you provide a number in an API, You have no idea who is consuming the API or what system they are running on. It might be an x86 EC2 instance, or it might be a mobile device, it might be a super computer, or a graphic compute unit.

It’s fine in your system when you create your 4 billionth item and exceed the 32bit MAXINT (4,294,967,295 for unsigned ints), but will it be fine for all your users? If your samples and current users get numbers in the low 100′s, but 2 years down the line you have a wildly successful product and your id’s are now into the billions upon billions? Do you want people writing API libraries to have to worry and think about this?

One fantastic issue that twitter discovered fairly recently is that javascript and therefore many json libraries are required to represent integers as a 53 bit number, that’s a pretty large number, but it’s several orders of magnitude less than most people would expect.

You make database scaling hard

Autoincrementing primary keys are fine if your datastore is a single logical machine (a highly connected cluster), but as you add parallelism, there are a number of situations where you might want to be able to allocate id’s from more than one cluster at the same time without synchronisation between them.

If your ID’s in your table are not autoincrementing integers, but are an identifier that maybe looks like “DC1-DB1-0001″ (or some other similar system), an identifier that takes into account the datacenter and database identifiers, reconciliation for tweet creation is perfectly possible.

You lock yourself into technologies

Autoincrementing primary keys are a feature of most relational databases. Some support it natively, some support it with triggers and stored procedures. However for scalability reasons, most non-relational databases don’t support them. Mongo for example provides primary keys that are OID’s or Object ID’s. These are a UUID4 type identifier that takes into account the host, the time and a few other things at creation time. BigTable provides horrible “keys” that are totally opaque, I have no idea how a key is generated (nor should I need to know).

Adding an autoincrementing key on top of these technologies is often possible, but it is often not ideal.

Conclusion

So there is a number of good reasons not to use numbers as an identifier in your API. Just use a string, trust me on this, you’ll use sightly more bytes, but you wont have to convert the integer into a string at API render time, and your client wont convert it back. Your string will be opaque to the user, meaning you can change primary identifier format willy nilly and providing the old and new format don’t conflict it won’t matter. Nobody will try to use false ordering on your ID’s when they should use a field instead, and it wont make me angry and ranty when I read your API documentation.

Addendum

I’m not saying don’t use a number in your database schema. It’s entirely possible for your ID’s that you know you’ll only ever generate a few hundred ID’s and that you wont have a date/time field, but order of creation is important and ordering the ID’s is the simplest way to do so. There are many valid reasons to use a numerical id internally for various things, but when you convert it into an API or external system, just turn that number into a string.

I strongly suspect that there will be people who will see your “{ ‘id’:’123′ }” and will turn it into a number themselves, but then later if you return a document “{ ‘id’: ’01-007′}” they can’t really complain, because you always told them it was an opaque string.

Scala, lazy collections, streams and recursion

bruntonspall — Thu, 01 Sep 2011 17:08:22 +0000

I’m currently rewriting the deployment system at the guardian in Scala, and although I’d say I know Scala, I’m learning lots of things as we go. I’m lucky enough to be pairing with Graham Tackley, our platform team lead and someone who knows Scala far better than I do, and this means that we often write a bit of code, then go back and improve it and so forth.

The bit of Scala code I’m particularly proud of is around waiting for a port to be opened. During our deployment scripts we restart our application servers, then we want the scripts to wait until the server is up and able to serve requests before we move on.

Writing this kind of code is not terribly difficult, but testing that it works can be very tricky. I decided to take the blackbox kind of approach to this, and my test spins up a second thread that listens on the port in question so that we can execute the real code that attempts to connect to the port. In java or most other languages this would be something fairly long winded, but in my Scala test it looks like this:

The spawn function is provided by concurrent.ops and is built into Scala and represents a throwaway thread that we aren’t terribly interested in. We don’t want to join on it, we just want it to start up and execute. The code inside creates a new java.net.ServerSocket, calls the blocking accept method which blocks until it gets a connection, closes the connection and then closes the original listening connection. We can then create and execute our port connection code and it should validly connect to the port 9998 on localhost and work beautifully.

The next test needed to be the failure case, if the waitForPort task cannot connect to a port after a specified timeout it should error, preferably by throwing an exception of some form. Testing this was also fairly easy, I wrote the traditional try { blah(); fail()} catch… stuff, then checked with Graham if there was a better way, and of course there was:

ScalaTest with the ShouldMatchers makes for a simply beautiful test that is clear to read and was easy to write.

However when it came to implementing the method, I struggled a little bit. I’ve never learnt functional programming, so closures and maps and so forth are still a bit alien to me, so I ended up writing some very java style code:

The isSocketOpen simply tries to open a socket and catches any exceptions to return false, and true if it opened successfully. Having written this, knowing there was a better solution, I asked again, and this prompted Phil Wills to upload his blogpost about the Stream class and lazy evaluation. I won’t cover the details here, but essentially using the return statement in Scala is frowned upon because of the way it works, and it’s more idiomatic Scala to use Options and flatMap, here’s the updated code:

Here we’ve changed isSocketOpen to checkSocketOpen and made it return an option. At this point I also made checkSocketOpen do the sleep, which we could have done before, but is necessary now.

To explain, this code takes the range 1 to 10, which is a Seq of ints (1,2,3…) and we flatMap across it with a function that returns an option. If you haven’t seen this before, here’s a quick overview of what the flatMap actually does:

map applies a function on each item in a collection and returns a transformed collection, essentially [a,b].map(f) is the same as calling [f(a), f(b)].

If our function f returns an Option, this is either Some(true) or None. An Option can really be thought of as a sequence that either contains 1 item or is empty, so Some(true) is the same as [true] and None is the same as [].

This means our assuming that our function f returns None for everything but c, the map on [a, b, c, d] will return something like [ [], [], [true], [] ]. That’s often not a lot of use for us, and what we are interested in is stuff that returned something.

flatMap is a function that essentially flattens the returned sequences catenated together, so if f(x) = [1,2] and f(y) = [4,5], then [x, y] map f == [ [1,2], [4,5] ] but [x, y] flatMap f == [ 1, 2, 3, 4 ]. This also has the handy shortcut that [] flattens into nothing, so essentially gets skipped, so our flatMap on [a, b, c, d] should return [true].

We then use headOption, since it’s possible that we returned either [true], [true, true...] or []. head would return us the first true, but on an empty list would die horribly, headOption returns an option that is Some(h) where the list is [h, t] and None where the list is []. We then match on the option and throw an error if we got None (and therefore not a single call returned a true).

Our function checkSocketOpen has a side effect of pausing for an amount of time though, and since flatMap executes the function once for each item in the original list, this means that we will always pause for the entire connection timeout no matter whether we successfully connect the first time or not. (kind of, actually if we only pause on a failed connection, and if we successfully connect each time, we wont pause, but we will create 10 connections rather than just the one we need to confirm the port is open).

Phil in his blog has a very simple fix for this, which is to wrap our original sequence in a Stream class. Stream is another Scala provided class, and in this case it is a sequence that is lazily evaluated. Essentially this means that the flatMap doesn’t call the checkSocketOpen on all items immediately but instead only calls it when you try to get to the first item. This means that headOption executes each function one after another until one of them returns true or it gets to the end of the sequence, it then returns the option based on what happened.

This means that if the first call returned none, it moved on and called f again over and over until it returned true, then the headOption stopped and never called the remaining calls, making this system stop as soon as the first connection was made.

Success! We’ve removed the return statement and made the code significantly harder to read and understand. I’ve just taken several hundred words explaining why it works and I’m still not entirely clear on the details.

So we started wondering if there was an alternative way of doing this, something that would still be idiomatic Scala but that would be obvious to junior or learning Scala programmers like myself. Graham and I wondered if what a recursive solution would look like, so we came up with this code:

We decided very early on to inline the checkSocketOpen function as recursive itself, so although this looks like more lines of code, it now combines both functions.

This function, when called at the bottom with checkOpen(0) initialises currentTry to 0, then tries to open a socket, if it is successful it just returns, if attempting to open a socket throws an exception it sleeps for a bit, then calls checkOpen with currentTry incremented by one.

All recursive functions have to have a guard, which is to say some statement that prevents infinite recursion (unless that’s what you want anyway), this function checks to find out if currentTry is greater than MAXCONNECTIONATTEMPTS and if it is, sys.error throws an exception which bubbles all the way back up through 10 copies of checkOpen before exiting properly.

If you know that your range is bounded fairly low, (such as 10 retry attempts) this is a perfectly acceptable way of writing this function. We suspect because of the way we’ve written the try catch that it’s not possible to make this function tail recursive, which would effectively allow infinite (or lots) recursive calls, but I’m sure it’s possible to adjust the try catch into a subfunction that might make it tail recursive if you really wanted to.

So there you go, that’s been a day of Scala learning for me, I now know the issues with the return statement, the Stream classes which might be useful later, and I’ve written my first truly recursive program in anger. I’m feeling a little bit proud.

Of course there are improvements to make, making the function tail recursive would be education as I’m aware of tail recursion but am not entirely sure how to actually make a function tail recursive. I’d also love to make that ServerSocket stuff into one line again, originally I had ServerSocket(port).accept().close() as one line, but I discovered in testing the stream stuff that the ServerSocket hangs around and keeps accepting connections if you don’t close it, which I can’t do in a single line without keeping a reference to it.

If you are interested in our new deploy system Magenta and the upcoming RifRaf web application for it, then I suggest you keep an eye out on our github repository and feel free to send patches, advice and help our way.

Google+ – A gentle introduction

bruntonspall — Fri, 01 Jul 2011 21:48:59 +0000

So a couple of days ago Google launched Google+, a product that did not slip out quietly it seems despite Google’s intentions.

You’ve probably read journalists who love it and journalists that hated it, but since the cat appears to be out of the bag and the invitations are flowing fast and furious I thought I’d write a quick few bits about how to get started using it.

First of all your invitation. If somebody you know has invited you, there’s a good chance that you didn’t realise it was an invitation. There are 2 forms of invitation.

As a “member of the press” (by association not directly) I got a real invitation from the Google PR team, which appeared to come from one of our developers. It looked a bit like this and it was a real invitation.

A real full invite

You can see clearly that it is an invitation and you can sign up from the mail easily.

During the first few hours, Google ‘accidentally’ left a send invitation feature into Google+, which was quickly noticed by the press and the number of people coming in massively increased. Google quickly disabled the send an invite functionality, and we don’t know when it’s coming back.

However there is another way to invite people to Google+ and they get an email that looks like this:

This is actually an invite

Firstly this is not an official way of inviting someone, and may not work. This email doesn’t say invitation except in tiny print at the bottom of the email, and instead looks like an opportunity to see the marketing bumph about Google+.

But clicking that button will result in your seeing one of two screens, if you are unlucky, it appears that Google are disabling signups globally for a few hours at a time, and you’ll get the “We are overloaded” page.

When Google is not allowing invites, this is what you see

If you get this try again in a few hours (or minutes) and you might get lucky and get the “Make an account” page (I don’t have a screenshot I’m afraid, you can’t miss it, it asks for your name and has a big signup button).

First steps in Google+

The first page you’ll see will contain a lot of welcome bumph. Go ahead and read it, watch the videos and come back after you’ve experimented. I’ll wait for you.If you are a popular kind of person with geeks, then there is a good chance that you will have a number of notifications that you are in peoples circles already.

People are sharing with me

If you unlucky and only know one or two geeks then it’s possible that the only person who has added you is the person who invited you. Never mind, you can find friends all on your own.

First of all, you can simply mine your gmail contact book. This is particularly easy to do, and Google have put a lot of thought into it.

A list of friends

In the section that looks like this under your circles, you can see all of your contacts who are not yet on Google+. At first you will probably instinctively start dragging contacts down to the relevant circle one at a time. This is a fairly inefficient way of sorting your contacts.

Click to select multiple friends

Instead simply click all of the contacts that you want in a single circle (say the 25 members of my family I wanted to select) and then drag one of them, you should see that they all come down and group together to be dropped into a circle. This makes life a lot easier when sorting out those tricky large groups.

Creating a new circle of friends

Create lots of circles, you can add people to multiple circles, and it’s very easy to rename, delete and even move people from one circle to another. I currently have circles for my church, my friends from Milton Keynes, my friends from School, people I’ve worked with, and people I currently work with at the Guardian.

Speaking of people you went to school with, I don’t really use GMail (especially not my gmail account itself) to email people, so I only had 60 − 80 addresses in there. For a number of my contacts, I instead use Facebook. At the time I’m writing this, adding your Facebook contacts requires a couple of steps. I expect this to change eventually, but for now, it turns out that Yahoo works as a very good middleman.

Login to Yahoo Mail for your Yahoo account (or create one if you don’t have one). Click the contacts link, and select the Import Contacts link from the Popular Tools that you should see.

You can now select Facebook, enter your Facebook details and Yahoo will happily slurp all of your Facebook friends into your Yahoo email list. Handy huh?

Find and invite from Yahoo

Now back in Google+, Under the circles tab, select the Find and Invite link and you should be able to add your Yahoo contacts. Authorise it from your Yahoo account by following the instructions and suddenly all of your Facebook friends should be available in the Find and Invite tab.

People who are in paul's circles

Finally, once you’ve got a few friends who are on Google+, you can start looking at their circles to see if there are any friends that you are missing. In pretty much any section of Google+ you can click somebodies name and get to their profile page. On the left is a quick view of a few people in their circle, and people who have added them to their circles (and people you’ve got in common).

People in pauls circles

If you hit the view more, you can get a nice view of all their contacts with an easy way to add them to your own circles. With this you can very easily start adding a large number of people who are already on Google circles.

Doing all of this should give you a fair number of people into your circles, but you might be missing a number of the less geeky people in your life. You might have them in your circles, but they don’t have a Google+ profile yet. How disappointing. So lets discuss invites.I cannot guarantee that this works, the only sure way to send invites is to wait for Google to fully enable invites. For now the following seems to send out the magic email.

Email your friends

Go to your stream and start sharing something. Remove the Public/All Circles users that are currently in the list and add a circle that has members you want to invite. You should see an option to email people not on Google+, tick it and confirm the pop up that comes up when you click share.

This will cause each of those people to get an email indicating that they’ve been mentioned, and they can click the link in the email and hopefully they’ll get in and sign up.

I hope you found this useful, and feel free to find me on Google+ and say hello or thanks.

Google’s Chrome browser hits 160m users – but what does it mean for the web?

bruntonspall — Tue, 14 Jun 2011 12:30:38 +0000

This article titled “Google’s Chrome browser hits 160m users – but what does it mean for the web?” was written by Michael Brunton-Spall, for guardian.co.uk on Wednesday 11th May 2011 19.38 UTC

We have decisively moved from a world of one browser provided by your operating system to a world where people install the browser of their choice. The increasing growth of Firefox usage has risen to the point where 31% of users on the Guardian website use Firefox and only 37% use Microsoft’s Internet Explorer (split into 22% IE8, 9% IE7 and approx 3% each for IE6 and IE9).

Google’s entry into the market has increased usage over time: one year ago at the previous Google IO, there were 70m Chrome users worldwide; today we were told there are over 160m. Unusually for a browser, Google has committed to a forced upgrade process and releases updates every six weeks.

Why? To combat the problem that more than 50% of Windows computers around the world are still running Windows XP – an operating system that is approaching 10 years old – and a browser that’s often the same age.

Google has been trumpeting its Chrome browser achievements at Google IO, showing off what a wonderful platform it is for the web. Over the last year, the team has improved the performance of the browser when running complex web applications by improving the performance of the underlying Javascript engine that web developers use to add logic to a simple web page. Additional improvements in security have made the “Google web” a safer place to be – which can only be good for the web, and for users.

The biggest announcements they have made is massive improvements to the graphics functionality in your browser. Implementations of the HTML5 Canvas technology that powers many of the graphical bits of the web are 10 times faster than they were in previous versions, and those improvements don’t require the developers to upgrade their application.

This is good news for game developers such as Rovio, who are able to build Angry Birds for the web, and on new builds of the Chrome browser it will run faster.

If developers want much, much faster graphics, the Chrome browser team has spent time vastly improving the WebGL graphics library available in Chrome. This is a subsystem of the browser, often called part of the HTML5 specification, but actually a separate specification in its own right, that allows developers to build 3D applications and utilise the high performance graphics hardware available in your machine.

Google’s improvements in Chrome means that WebGL applications can run up to 10 times faster than the equivalent Canvas implementation.

The dark side of all of these improvements is that we are likely to start seeing “Optimized for Google Chrome” badges on the web, as many of these features won’t work across all browsers. That wasn’t good with Internet Explorer in the bad old days of the Netscape wars, and wouldn’t be good today. Some things such as Canvas work in most browsers (but not in the most common versions of Internet Explorer), but WebGL’s implementation is inconsistent across all the browsers.

Google’s advancements are welcome for experimental applications, and developers building applications for the Chrome App Store, but their impact on the wider web will probably be negligible, since cross-browser implementation is considered so important by many organisations.

Published via the Guardian News Feed plugin for WordPress.

Clearing up some myths about AV

bruntonspall — Thu, 28 Apr 2011 00:10:24 +0000

I’ve been thinking about the AV question a lot recently, partly because it’s happening here and now, partly because it’s naturally been the talk of the office and partly because I find the whole area quite fascinating.

People seem to fall into one of 2 categories, either they have a strong opinion, or they aren’t interested. Those of a strong opinion appear to refuse to be convinced either way, and those who don’t really care probably wont vote anyway. Despite this there has been a lot of media coverage, political talking and leaflets that have tried to convince people one way or the other.

Personally, I’m all for AV and leaflets like the recent no to av leaflet made me even more convinced to vote yes to AV, but a discussion with one of my brighter friends who is opposed has made me realise that I need more than a gut feeling for why I’m voting yes.

Firstly, much of what the politicians are saying is complete rubbish, and that’s sadly from both sides. Both systems give each person one vote, both have possibilities to be unfair and I’m not really sure what that word “fair” even means for a voting system.

However there are a few things that I think are valid criticisms, and deserve to be addressed

1. AV will negate tactical voting

This is one from the pro-AV side, they claim that tactical voting wont happen under AV. For those who need a reminder, tactical voting is where you want to vote Lib-Dem, but in your area that’s a “wasted vote”, so you vote Conservative to prevent the Labour candidate from getting in.

The claim is that you can validly order your votes 1- Lib-Dem, 2-Conservative, 3-Labour which doesn’t waste your vote, but if Lib-Dems really have no chance of winning means your vote will be transfered to the Conservative candidate over the Labour one.

The problem is that lets imagine a far more realistic example, say Bennelong, Australia which has 11 candidates from One Nation, Christian Democratic Party, Australian Sex Party, Carers Alliance, Family First, The Climate Sceptics, The Greens, Labor, Building Australia, Liberal and Liberal Democrats. In 2010, the initial votes varied from 170 to 41,582 depending on the party.

Ignoring the smaller parties, the race was set between Liberal and Labor with 48.53% and 37.12% respectively (the next party had 7.95%). By the time we reach the 8th preference it’s a 4 horse race, only Liberal (49.55%), Labor(37.9%), Green(9.17%) and CDP(3.38%) are left in, the CDP is knocked out and it’s votes transfer about 50% to Liberal, 30% to green and 20% to Labor. That tips Liberal over the edge and into the 50% margin needed to win.

In this race, tactical voting on behalf of the green party if you support labour is worth it, your vote for green could have given labor the win if you’d picked them as a higher preference. I picked this constituency more or less at random (it had a large number of candidates and is early in the alphabet), but in even closer races you can imagine tactical voting still being needed.

2. It’s not good enough AV

This follows on from the above point. My vague understanding of the british AV proposal is that the voting and reducing stops as soon as someone reaches 50%. The australian system appears (and I could be wrong here, but the data seems to suggest that this is the case) to continue transfering votes until only 2 parties remain. In the above case, the data doesn’t stop at the 9th vote with 3 parties (one over 50%), it runs a 10th vote where the Green party is transfered, 80% (or 7000 votes) goes to Labor, but it’s not quite enough to beat out the Liberals who win with 53% of the vote.

In this case even if Labor had got 100% of Green votes it wouldn’t have changed the outcome, but you can imagine states where this might be the case, and running those last rounds could change the result of the election.

However, in those rather rare cases, I believe that first past the post, which is in essence taking the first round of AV and declaring the person with the most the winner, would also suffer from the same problem.

3. Not everybodies vote will count

The AV we use is also called Optional Preferential Voting, which means you don’t have to vote for everybody, you can just say put a single vote for the australian sex party and your vote won’t be transfered if they get knocked out.

In the australian system, most states use full preferential voting, where you must rank the candidates in order at all times, but Queensland and New South Wales use the same as we are suggesting.

This is actually the best objection I’ve heard against our form of AV, so I’ll explain the problem:

If we start with 5 parties, but lets say that everybody voting for Party A is fanatical and wont vote for any other party. Let’s also say that party A is 10% of the first round of voting and they get knocked out. In the next round of voting, the total number of votes counted is 10% less than in round 1, thus getting 50% of that vote does not mean you got 50% of the people who voted in the first round, and therefore the electorate as a whole.

It’s a pretty good argument, and it’s the thing that inspired me to go get the raw data from australia and start researching into this. Because lets face it, even if some people get knocked out, most races just aren’t that close that the knocked out ones who didn’t transfer are that relevant right?

Well for that I needed 2 core pieces of data, the total people in each round and the transfer votes, and I couldn’t find them (the australian data always seems to have the same total as ending number making me suspect I’ve got the wrong data source), but Antony Green for the London School of Economics has crunched the data and the results kind of surprised me.

In essence in Queensland of the 90 odd elections that happen each year, theres normally about 3 – 6 where the candidate didn’t have 50% of the original voters when they got their 50% at the end. That’s a small enough error margin for me. However for some reason New South Wales has a rather bad looking trendline, from 1981 through to now, it’s gone from 1-2 in 100 to around 20 in 100. Thats enough to potentially be significant, but without access to the raw stats, it’s difficult to calculate how disenfranchised the votes have become and posit any reason why this is happening

The fact remains that while it’s possible with our slightly limited AV to get results that might not be the true majority, it’s still in most cases going to produce a representative who has a majority backing to far more of an extent than FPTP will.

When it boils down to it my reason for Voting Yes to AV is that I believe that change is a necessary part of life, the act of changing will resolve some of these issues in some peoples minds, and will clear out some of the electoral cobwebs. Sure I’d rather have full preferential voting AV, or Proportional Representation than the AV we’re being offered, but I suspect that a No to AV result will be argued successfully that it’s a no to electoral reform of any sort, and I’d rather have AV electoral reform than nothing.

References:

Australia and AV

Guardian Focus podcast: The AV Referendum

Australian Voting Records and Data

Campaign for Electoral Reform

That No to AV leaflet explained

Do we want a fairer election system? Then the only answer is a Yes vote

AV vs FPTP as Beer vs Coffee

Failure at scale

bruntonspall — Fri, 04 Feb 2011 14:01:16 +0000

When you launch a high profile website, it sometimes will spectacularly fail for reasons of scale. Since this is an area of professional interest I thought I’d have a look to see whether there was anything obvious, and it was apparent that the developers didn’t appear to think at scale (and still haven’t fixed the issues).

When I visit the crime maps website, the very first thing I see is a landing page that I can type my postcode into. The HTML generated for this page must be the same for every user, so this page should be cached right? Unfortunately, not only is this page uncached but it doesn’t return any cache headers, last modified or expires headers of any form. The static files for this page do return a Last-Modified header, but I suspect that’s because they are hosted on S3. Of slight note is that the developers didn’t obey any of the website performance rules, there is no gzip and there is multiple CSS/JS files rather than a combined file.

Once I enter a postcode I get redirected to a dynamically generated page based on my post code. Again there are no cache headers here, and since it’s a dynamically generated page you might think it doesn’t matter, but since the page is entirely javascript driven, this HTML is could be cached easily. There are a few links that have the same query parameter passed to them, but again once you’ve decided to ban all non-javascript browsers you may as well rewrite those urls in the clients browser using javascript rather than server side.

Furthermore the data that is loaded by the javascript from /crime/radius which contains the crime data for my area, in monthly groups doesn’t have any new data since 2010-12, not a huge suprise I’d assume these are quartaly data sets, but that means that json response should also have cache headers, and of course it doesn’t.

In this case a cache wouldn’t help with scale for multiple users, but would for returning users, because the developer has specified the area they are interested in by a long/lat that is accurate to 7 decimal places – an accuracy that should be down to the millimeter of so I believe (4 places gives you accuracy to 6-10 meters or so). Since they’ve merely geolocated my postcode, accuracy that close is completely wasted and would destroy any cache that is there. My postcode is probably a hundred meters or so in size, so having a long/lat that is more accurate than 2 or 3 decimal places is probably wasted. Decreasing the accuracy of the center point could potentially give you less accurate data, but since what you are interested in is crimes within a 0.5 mile radius, you could simply increase the radius a bit and drop some of the outlying results in the clients browser, making for a much more scalable backend server.

At this point of the analysis I got just to depressed to even continue, it was clearly obvious that the developers had given very little thought as to how this website would scale. I suspect that since it is hosted on Amazon’s EC2 they figured they would scale by firing up new servers, but I contend that even at that 18m hits an hour (approx 4k a second) about 4 or 5 caches with good hit rates would have been able to serve most of the traffic without too much issue (I’d probably have suggested a cache warming for data for the major metropolises as well).

Remember kids, EC2 allows you to add new machines easily, but you still need to think about scale and performance when writing your application. EC2 doesn’t solve your scaling problems, it solves your capacity planning problems, you still need to build a site that scales without needed 1000′s of servers.