Build Pipelines

| Comments | featured, technology

Build pipelines, deployment, and immutable artifacts

What is the best way to build your code? How can you ensure repeatable deploys? What does build and deployment look like in a devops, continuous delivery kind of world?

I have seen many build and deployment systems that seem to make life hard for their developers and fail to deliver the value that they could.

The reason for this is that I don’t think the basic principles that help you get them right are well understood or explained. It’s equivalent to giving people screws and hammers and complaining that they hammer the screws into the wall.

I believe there are some fundamental principles around how a good build and deployment pipeline should work, that if done right will make the majority of things you want to do with your build and deploy pipeline really easy. In particular it will give you consistent results, work fast and scale as your team grows.

If you try to use many build pipeline tools without getting this stuff right, you’ll feel like you are constantly fighting your build tools and nothing will be as easy as it should be.

I’ve only learned these principles because over time I’ve done this wrong enough times to have struggled my way to doing it well. I hope I can save you from some of the mistakes I’ve made

Accountability in Agile

| Comments | featured

What happens when agile teams go wrong? How is it that in long running agile programs, time and time again, I see teams that don’t build software that meets the operational requirements, that doesn’t do security properly, that fails to engage with design or user research or analytics in quite the right way.

I think that one of the main issues that we see in long running agile teams is one of lack of accountability and division between accountability and responsibility.

As you may know from some of my previous posts, I’m a fan of developers understanding business practices and working out why the organisation does what it does. If you’ve read any good books on how to run a business and how to structure an organisation, you may be aware of what we mean by accountable and responsible, but many developers I speak to don’t seem to understand. They often confuse the two and they get very upset when they see managers who claim to be accountable but don’t seem to be responsible for anything, despite the fact that this is exactly the model that we use in Agile.

Should I Allow Different Languages or Runtimes in My Organisation?

| Comments | featured, technical

One of the much vaunted benefits of microservices is the claim of heterogeneous development environments. Because we agree that microservices should interact via well known or standardised protocols (like HTTP, Thrift, RPC), it means that different microservices can be written in completely different technology stacks.

This often seems to scare technology managers, heads of engineering and operations teams. If we diversify our technology stack it will make people rotation harder, it makes operating the services harder as there is less uniformity, and it means that monitoring, alerting, diagnostic tools and the various other tools and processes that the organisation has built up over time wont work any more.

I want to tell you about an experience I had at the Guardian, and how having a homogeneous environment hurt us as a development team and how moving towards heterogeneous environments improved the team culture and technology options.

The Real Benefit of Agile

| Comments | featured, technical

What is the point of doing agile development? We’re told that we will be more efficient, higher quality software that matches what the user wants, but is that really the best reason to do it?

I think that well executed “traditional waterfall” project management can also achieve the above aims. So what is the point of agile techniques and why do I continue to drink the agile kool-aid?

What Are Microservices and Why Are They Important

| Comments | featured, technical

Part of the problem with the debate around microservices is that we aren’t always arguing about the same definition of microservice, probably because not enough of them have read James Lewis’s defining post on microservices. So when two people disagree on the implementation details of the microservice, they can be speaking at cross purposes because they haven’t agreed what they are talking about.

Because I’m going to write more about microservices, why I think it’s such an important architectural movement, and why I think it ties into other movements, I feel like I need to set out my definition of a microservice, and what things are supporting practices that aren’t about microservices, but help to make microservices such a useful pattern

Dev, Ops and Business Value

| Comments | business, featured

I find myself increasingly being worried about the way that us technologists view our value to the organisations that we work in. Part of that is a strong lack of understanding of the purpose of the business and an over identification of technology and technology choices to the value of an organisation.

As an example, the other day, one of my friends tweeted the following mini rant on twitter:

Instead of Code Club, we should have Don’t Write New Code Until You’ve Fixed the Existing Code Club.

So what’s the problem with this quote?

Apart from coming across as a grumpy old system operator yelling “don’t change my system, you’ll break it” (which was a bit unfair, but Graham took it well), it reminds me that as developers and operators we think that the primary value we deliver is writing or maintaining code.

Scale Camp - a Brief History

| Comments | featured, technical

In organising the upcoming Scale Summit unconference, I’ve been reflecting on the history of Scale Camp, and what the purpose and point of running these events was to me.

Velocity Conference starts it all

Scale Camp was born out of a pair of frustrations. As a new developer, I was introduced to the Velocity Conference by Paul Nasrat, one of the Guardian’s system adminstrators. We watched a video by John Allspaw and Paul Hammond about 10+ deploys a day at Flickr.

Many of the velocity teachings were directly applicable to what we wanted to do at the Guardian, so I made a request to attend the conference to the management team.

2013 in Review

| Comments | featured, personal

It’s the new year, and as should be typical, it’s probably time to write an update on my blog about what I’ve done this year.

It’s been an interesting year. After 6 years of working at the Guardian, I felt it was time to move on to a job as an Architect at the Government Digital Service.

Prism and NSA Spying: Why I Don’t (Entirely) Believe It.

| Comments | featured, rants, technical

[EDIT: Note, I have absolutely no inside knowledge here whatsoever. I haven’t seen anything except via the stuff the Guardian has published publically.]

You may have read this morning that the Guardian and the Washington Post announced that they had an authenticated NSA training presentation on PRISM which claimed that they had access to multiple large companies servers and were able to spy on any and all communications.

In essence the presentation says that Gmail, Apple iMessage, Facebook and most of the other internet services are all actively monitored by the NSA.