Jekyll2020-04-28T11:40:41+00:00https://www.brunton-spall.co.uk/feed.xmlMichael Brunton-Spall on …The thoughts and musings of MBSUsing AWS Workspaces to control access to documents2020-04-28T10:00:00+00:002020-04-28T10:00:00+00:00https://www.brunton-spall.co.uk/post/2020/04/28/Using-AWS-Workspaces<p>I’ve recently worked on a project where we had to have some documents that needed to be kept reasonably secure, and on the clients computers for our project. We needed our developers to have some access to the documents, to visually inspect them, and to be able to run code on them, but we didn’t want the developers to have copies on their local laptops or computers.</p>
<p>We decided that <a href="https://aws.amazon.com/workspaces/">AWS Workspaces</a> would be a good fit for this usage. For those of you who haven’t used it, AWS Workspaces allows you to create desktop computers in AWS’s data centers which you can connect to via remote desktop protocol. It looks like your computer, but in fact your mouse and keyboard are attached to a remote computer. These are standard windows or linux desktop computers and the users can do anything on them that you’d expect to be able to do on a local computer, but of course nothing about the computer leaves the Amazon data center.</p>
<h3 id="locking-down-the-data">Locking down the data</h3>
<p>We had a smallish set of data that we wanted to do some data science on, that is running some scripts that might parse the files, look for commonalities and so on. Of course, in order to do that the developers need to be able to look at the files, so they know what they’re looking at. We could allow individual developers to see a handful of files by using a client laptop, but that laptop is restricted in running code, so we selected AWS Workspaces as a way of triaging access.</p>
<p>Our first step therefore is to migrate data into somewhere accessible by the Workspaces computers. We picked an S3 bucket for this. We were able to create the S3 bucket with ACL’s set to disallow public access.</p>
<p>Workspaces can be <a href="https://docs.aws.amazon.com/workspaces/latest/adminguide/amazon-workspaces-vpc.html#configure-vpc-nat-gateway">deployed inside an Amazon Virtual Private Cloud (VPC)</a>, which means that we can create a private subnet that cannot be accessed from the internet, create the workspaces in there, and then configure the AWS NAT Gateway to enable the amazon client to access the machine and to allow certain data transfer out.</p>
<p>This means that the developer can connect to our Workspaces client, and they get bought up on a machine inside the private subnet, totally isolated from the internet. However, using <a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpce-gateway.html">AWS Gateway Endpoint</a>, we can enable the private subnet to access the S3 bucket. On the S3 bucket, we can set an allow policy that allows access from the private subnet, and we can configure the <a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpc-endpoints-s3.html#vpc-endpoints-policies-s3">Endpoint Gateway with an endpoint policy</a> that allows access only to our specified bucket. This means that the devs can download files from the S3 endpoint, but cannot upload those to a new public bucket in any way.</p>
<p>This is pretty much all configured the way that the Workspaces documentation recommends, however we wanted a little extra security.</p>
<h3 id="authenticating-the-users">Authenticating the users</h3>
<p>While it would be nice to enable MFA, requiring developers to authenticate not just with a password, but with a token from their mobile phone, this requires configuring the Managed Active Directory to work with a RADIUS server, and that’s beyond my rather basic technical ability.</p>
<p>But we really don’t want anyone who tries to connect to our desktops to just try passwords in a password spraying attack. I really want some form of second factor. Luckily, AWS provides something called <a href="https://docs.aws.amazon.com/workspaces/latest/adminguide/trusted-devices.html">Trusted Devices within AWS Workspaces</a>. This means that a device requires a machine installed certificate that has been signed by a defined certificate authority.</p>
<p>This sounds pretty simple, we’ve got a couple of steps we need to take:</p>
<ol>
<li>Create a certificate authority</li>
<li>Create certificates per device/user</li>
<li>Get the certificates to the developers</li>
<li>Let them authenticate to AWS Workspaces</li>
<li>???</li>
<li>Profit</li>
</ol>
<p>Of course, this sounded easy, but it turned out to be much harder than it should be. The biggest problem here is that AWS Workspaces client doesn’t actually seem to provide any logging of any form. I spent a lot of time looking at an error screen telling me that it couldn’t connect with no explanation as to why.</p>
<h3 id="create-a-certificate-authority">Create a certificate authority</h3>
<p>If we were running a large estate or big corporate installation of this, we might want to use a full PKI process. In my case, we were looking for less than 5 developers having access to infrastructure that is going to be destroyed in a few months, so we can roll out own Certificate Authority.</p>
<p>Our first step is to create the Certificate Authority private key: <code class="language-plaintext highlighter-rouge">openssl genrsa -aes256 -out CA.key 2048</code>. This private key is going to be our root key, so give it a good password. However, you are going to be typing that password a lot, so make it something you can type! I recommend using three simple words, so something like <code class="language-plaintext highlighter-rouge">workspace signing easy</code> might work just fine.</p>
<p>Next we need to create the public version of the key, <code class="language-plaintext highlighter-rouge">openssl req -x509 -new -nodes -key CA.key -sha256 -days 1024 -out CA.pem -subj "/C=GB/ST=England/L=London/O=ORGHERE/OU=OUHERE/CN=CNHERE"</code>. You’ll want to update ORGHERE, OUHERE and CNHERE with relevant names. In reality, very little of this is going to matter, but the CN is used in a few places, so make it something memorable. <code class="language-plaintext highlighter-rouge">projectname.local</code> makes a good CN here.</p>
<p>Now you’ve got a CA.pem file, you can upload it to the AWS dashboard in AWS Workspaces and Workspaces will only allow clients that have a certificate signed by the CA to be allowed. Note that you only need to send the public portion of the key, <code class="language-plaintext highlighter-rouge">CA.pem</code>, to AWS. You still need to keep the private key, <code class="language-plaintext highlighter-rouge">CA.key</code>, secure on a local device somewhere.</p>
<h3 id="creating-a-client-certificate">Creating a client certificate</h3>
<p>You can do this one of two ways. The easiest is to generate the certificates on the CA machine and then copy both the private and public keys to the laptop or desktop that needs to use them. The harder way is to generate the private key on the laptop and only copy the intermediate files around. The second, harder way is marginally safer, but with decent passwords and a limited risk exposure, you may be willing to use the easier method.</p>
<p>You’ll need to replace the word CLIENT throughout with the name of the machine or some other identifier</p>
<p>First we need to generate a certificate on the local machine using <code class="language-plaintext highlighter-rouge">openssl genrsa -aes256 -out CLIENT.local.key 2048</code>. Again, you want to set a decent password here, as anyone who gets hold of this key can authenticate to your AWS Workspaces client.</p>
<p>Once you’ve got a private key, you need to generate a signing request using <code class="language-plaintext highlighter-rouge">openssl req -new -key CLIENT.local.key -out CLIENT.local.csr -subj "/C=GB/ST=MyCounty/L=MyTown/O=MyOrganisation/OU=MyOrganisationUnit/CN=CLIENT.local.client"</code>. This signing request, the csr file, is the thing that you need to get to your CA computer to be signed, so copy it over, or if you are lazy, do this all on the same machine!</p>
<p>Now we need to sign the certificate. This caused me a lot of pain in AWS Workspaces, I had to ask friends who had done similar things to get the exact options setup. The command is <code class="language-plaintext highlighter-rouge">openssl x509 -req -in CLIENT.local.csr -CA CA.pem -CAkey CA.key -CAcreateserial -out CLIENT.local.crt -days 365 -sha256 -extensions v3_req -extfile ext.txt</code>. This reads in the csr file, and the CA key, and it output a signed certificate in the crt file. It tells openssl to use the sha256 signing algorithm (needed by AWS Workspaces), and it tells it to use some v3_req extensions. This final part is really important, the whole thing wont work if you are missing this.</p>
<p>In order to generate the right extensions, you’ll need an ext.txt file, which should contain the following contents:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[v3_req]
keyUsage = digitalSignature
extendedKeyUsage = clientAuth
</code></pre></div></div>
<p>This tells the CA that it wants the certificate to be signed for client authentication and signature purposes.</p>
<p>You can now copy the certificate back over to your laptop.</p>
<h3 id="getting-osx-to-use-the-certificate">Getting OSX to use the certificate</h3>
<p>You’d think that this would be enough, but we also had problems getting OSX Keychain to actually import the CRT file. I’m not 100% sure that this step is necessary, but it’s what got it working for me, so I recommend it for you as well.</p>
<p>What we need for Keychain to easily import the certificate is to make a bundle of the key, the certificate and the public key for the CA. This bundle can be created with the pkcs12 tool as follows: <code class="language-plaintext highlighter-rouge">openssl pkcs12 -export -aes256 -out CLIENT.local.full.pfx -inkey CLIENT.local.key -in CLIENT.local.crt -certfile CA.pem</code>.</p>
<p>This will create a pfx file, which you can then double click on to import into your local keyring. If you do this, you’ll import the certificate into the users keychain, which means that whenever AWS Workspaces wants to access it, you’ll be prompted for your login password (or if you’ve got it setup, touchid). You can also import this certificate into the login keychain, which means it wont need your password to access, but that also means that anyone with access to your device can use it without your password.</p>
<p>Once you’ve got this in, you should be able to start the AWS Workspaces client, and it’ll authenticate properly and let you in.</p>
<p>I’ve <a href="https://github.com/bruntonspall/AWSWorkspacesCA">created some simple scripts to create CA and client keys</a> for you which you might find useful if you are doing the same thing.</p>
<p>I hope this helps you, I’m mostly impressed with Workspaces as a remote desktop solution, it does what it says on the tin, and ensures that my cloud data stays in the cloud, and not on the laptop that can be left in a pub, stolen on the train or otherwise lost.</p>
<h3 id="addendum-a-better-approach-for-macos">Addendum: A better approach for MacOS</h3>
<p>Some feedback on my approach outlined here has a more scalable approach if you are deploying to a larger number of client devices and want to run the CA slightly more securely. In this case, you’ll need to generate the CA certificates as above, but you can ask your developers to generate their certificates in a far more user friendly way.</p>
<p>Generating the certificate can now be done by using the Mac OS built in Certificate Assistant. Simply open up the keychain access app, and from the menu, choose Certificate Assistant > Request a certificate from a certificate authority. You need to enter the email address of you as a developer, the email of the certificate authority, and make sure that the keypair is generated as an RSA keypair (it should be the default). This will send a CSR (or certSigningRequest) file to your CA email address.</p>
<p>Now, as the CA, when you recieve this file, you can download it to a directory, and then sign the certificate request using <code class="language-plaintext highlighter-rouge">openssl x509 -req -in CLIENT.local.certSigningRequest -CA CA.pem -CAkey CA.key -CAcreateserial -out CLIENT.local.crt -days 365 -sha256 -extensions v3_req -extfile ext.txt</code>. This will generate the signed `crt’ file which can be emailed back to the developer.</p>
<p>Once you get your crt file back, simply double click it and your Mac will import it into your keyring and the next time you use the AWS Workspaces client, it should connect perfectly.</p>
<p>You’ll note that you don’t need to create the PFX file. I had a lot of problems originally with the certificates, and had to create a pfx file to get the Mac to import it, but if you do it this way, it doesn’t seem to need it. I think that’s because the key file is already imported into your local keychain, so the certificate has a match and is therefore trusted.</p>
<p>I’ve updated the github scripts to have a CA directory and a client directory, and there is a sign_request.sh in the CA directory to perform the above signing for you.</p>I’ve recently worked on a project where we had to have some documents that needed to be kept reasonably secure, and on the clients computers for our project. We needed our developers to have some access to the documents, to visually inspect them, and to be able to run code on them, but we didn’t want the developers to have copies on their local laptops or computers.What is legal basis under GDPR?2019-02-25T12:39:45+00:002019-02-25T12:39:45+00:00https://www.brunton-spall.co.uk/post/2019/02/25/What-is-legal-basis-under-GDPR--7e8442c658b9<p>In a conversation the other day, I was trying to explain why some data couldn’t be collected and processed under “legitimate interests”. I wrote the following to try to outline the different types of legal basis that data can be collected or processed.</p>
<p>I thought these examples might be useful to help you decide how you are collecting and processing data. This is not legal advice, and you should consult professional advice if you have concerns. You should also read <a href="https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/lawful-basis-for-processing/">the formal guidance for the UK</a></p>
<p>Also massive props to <a href="https://twitter.com/ChackoSunitha">Sunitha</a> for being my guinea pig for all the examples. She’s a lovely human and doesn’t really deserve all these things to happen to her!</p>
<h3 id="consent">Consent</h3>
<p>Sunitha goes to a website and puts her name and email address in to get access to a report on “data protection for you”, That was clearly active consent, and is covered under consent.</p>
<h3 id="contract">Contract</h3>
<p>Sunitha’s data is passed by her employer to a HR firm so that they can process the HR data in order to pay her. Her employer has signed a contract with the HR firm, and that is a contractual exchange of data. Note that Sunitha doesn’t have to consent knowingly to the HR firm having the data, although she should be bound under a contract to her employer to let them do that.</p>
<h3 id="legal-obligation">Legal obligation</h3>
<p>Sunitha’s employer is required to take a photocopy of her passport (personal data) to prove to the Home Office that they believe that she has a legitimate right to work in the country. They are required by law to do this, and therefore they don’t need her consent or a contract to do so.</p>
<h3 id="vital-interests">Vital interests</h3>
<p>Sunitha is found unconscious at the side of the road. The security guard who finds her looks in her wallet and discovers where she lives and her name to give to an ambulance when it arrives. The guard believes that her life or wellbeing might be in danger, and as such he has her “vital interests”, and can access that data without consent, a contract, or legal obligation.</p>
<h3 id="public-task">Public Task</h3>
<p>The Local Authority is inspecting Sunitha’s kitchen because she sells sandwiches on the side. They record that she is the owner of the kitchen and that it was clean. Her name and place of work are personal data, but the local authority can process that data because they are the official authority for maintaining food standards (as appointed by the Food Standards Authority). They don’t need consent, a contract, vital interest or legal obligation to do so.</p>
<h3 id="legitimate-interests">Legitimate Interests</h3>
<p>Finally, Sunitha is an MP who has had her expenses sent to a major newspaper. The newspaper has scanned and put all of the expenses online to allow citizens to look for illegitimate expenses and flag them. The newspaper considers that as an investigatory journalistic organisation interested in the misuse of public funds, that there is value in processing the data and making it available online, and declares that it has a “legitimate interest” in Sunitha’s expense receipts without consent, a contract, vital interest, legal obligation or public task to do so.</p>In a conversation the other day, I was trying to explain why some data couldn’t be collected and processed under “legitimate interests”. I wrote the following to try to outline the different types of legal basis that data can be collected or processed.Brexit and data transfers2019-02-21T16:10:15+00:002019-02-21T16:10:15+00:00https://www.brunton-spall.co.uk/post/2019/02/21/Brexit-and-data-transfers-a56eac4a2776<p>Post brexit, there may or may not be a problem with data being transferred across borders. But all of the guidance and people talking about this seem to have some very confused concepts of terms and the processes involved, making it really hard to get clear guidance for organisations.</p>
<h3 id="an-example-organisation">An example organisation</h3>
<p>To make this clearer, let us think about a simple example of an organisational problem and see what comes out.</p>
<p>Imagine that we are running MBS Ltd, a news company. We scrape the internet for news articles and then put together topic pages with recent articles. As a user, you can browse the website for free, or you can create an account with your email address, and we will track what articles you read, and create you personalised pages that recommend articles that we think you’ll like.</p>
<p>In this case, the personal data that we collect (and we put in our privacy policy) is:</p>
<ul>
<li>Your email address</li>
<li>A list of articles you clicked through to, and “thumbs up and thumbs down” ratings you give to articles</li>
</ul>
<p>We also use cookies to track anonymous users and build lists of articles that they view or like, so we can data mine that information to determine that “people who liked article X on topic A also liked article Y on topic B”.</p>
<p>We built the system as a simple Ruby on Rails web application, hosted on AWS in the EU-West-1 region. We have an AWS RDS instance that stores all of the data that we use.</p>
<h3 id="what-is-a-data-controller-and-data-processor">What is a data controller and data processor?</h3>
<p>If you are running a company that accepts personal data, such as email addresses, from end users, and you store that information, then you are a data controller.</p>
<p>The data controller is the person or organisation responsible for the data, and responsible for caring for the data and doing to it what they told the user that they would do to it.</p>
<p>In this case, we collect personal data about the majority of users, the cookie information and logs keep IP address and preferences, as well as the details of our signed up users. We are therefore the data controller for this data.</p>
<p>A data processor is anybody who stores data or processes data on behalf of the data controller. In this case, for our customers, we put the data into AWS, who acts as a data processor. We remain responsible for the data while it is in AWS, but AWS processes it on our behalf according to the standard contract that we have with AWS (those Ts&Cs that our CTO clicked through without reading when we first setup the startup).</p>
<h3 id="offshoring">Offshoring</h3>
<p>Offshoring is a tricky subject, and <a href="https://medium.com/@joelgsamuel/offshoring-the-8th-frontier-b1ce6cf38461">Joel has written more about this</a>, based on some comments of mine. But essentially, if we are a British company, then we have the following legal agreements.</p>
<p>We have a contract with AWS EMEA SARL, a company registered in Luxembourg that says that we allow them to process our data on our behalf. AWS says that <a href="https://d1.awsstatic.com/legal/aws-gdpr/AWS_GDPR_DPA.pdf">they are processor who will only act on our instructions</a></p>
<p>In essence, AWS says that it’s our responsibility to ensure that we have a legal right to give the data to AWS, and that they may reproduce the data as needed to move it around the internals of AWS.</p>
<p>We’ve selected the EU region, which means that our data is being held in servers in Ireland. AWS says that it has staff all round the world who potentially help administer their services on their behalf.</p>
<h3 id="current-legalstatus">Current legal status</h3>
<p>Currently we have a privacy policy and consent from our users to hold their data. Because the UK is within the EEA, the data is being held in the EEA, but an EEA company and not being transferred in or out of the EEA, so everything is fairly simple.</p>
<p>What happens if we leave without a deal?</p>
<p>In the event of the UK leaving the EU without a deal, things start to get a little tricky.</p>
<p>Firstly, <a href="https://www.gov.uk/government/publications/data-protection-law-eu-exit/amendments-to-uk-data-protection-law-in-the-event-the-uk-leaves-the-eu-without-a-deal-on-29-march-2019">the UK has already said that they will create a “finding of adequacy”</a> for the EEA that says that it is safe and acceptable for UK companies to transfer UK citizen data to an EEA company to be processed. This means that our basic system is fine, we can still transfer data into AWS Europe under the same process as we did before.</p>
<p>The EU has not said that they will necessarily create a finding of adequacy in the other direction, so it may not be possible for an EU company to transfer data into the UK without consent from the users. In our case this is probably fine, because we don’t receive information as a data processor from another company in europe.</p>
<p>End users are still allowed to make their own decisions about transferring data, so an EU citizen who choses to sign up to our service is allowed to hand their data to a UK company, providing our privacy policy and consent clauses are clear to them that they are doing so. This is a lawful basis of processing based on consent.</p>
<p>The problematic part is that data transfers from the EU to the UK might be forbidden by the EU GDPR since the UK will be outside the EEA and won’t have a finding of adequacy.</p>
<h3 id="what-is-a-data-transfer">What is a data transfer?</h3>
<p>The key question here is about what constitutes a data transfer. The <a href="https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/international-transfers/">guidance on international transfers</a> is still quite vague, and has holes you can drive a truck through, for example</p>
<blockquote>
<p>Personal data is transferred from a controller in France to a controller in Ireland (both countries in the EEA) via a server in Australia. There is no intention that the personal data will be accessed or manipulated while it is in Australia. Therefore the transfer is only to Ireland.</p>
</blockquote>
<p>This use of the words “server” or “intention” don’t make any sense to me. How one measures the intent for data to be accessed or manipulated by a “server”, given that any computer data transfer must involve some level of packet technology that will manipulate the data, store it on a drive, and then access the data to serve it back up. Additionally, storing data on a server outside the EEA that you “don’t intend to be accessed” doesn’t sound like a good defense if the said server is hacked somehow! It certainly doesn’t match up with the ICO’s definition of “processing”.</p>
<p>It’s also worth noting that you can transfer data internationally under a number of different arrangements. What we are worrying about here is the specific transfer of data without consent, without a “legal instrument” and without any of the other exemptions to international transfers. Since using a cloud service provider is commonly perceived to be one of these restricted transfers, that’s what we are worrying about here.</p>
<p>Anyway, if your data is stored in AWS Europe, and your staff in the UK are accessing that data to do their jobs, it’s slightly questionable whether this is a transfer. Reading the guidance, this is clearly not the intention, transfers are clearly meant to be “a transfer for the purposes of processing by a third party”, and access by the original data controller is not a transfer. However, this isn’t made clear and nobody is making it any clearer.</p>
<p>Secondly, it’s entirely unclear whether data provided by EU citizens, who thought it would be stored by a company based inside the EU, are content that when the UK leaves the EU without a deal, that their data can continue to be accessed by the company. If your original privacy policy and consent made clear that this was stored by a UK company, you might be fine, but if you had general terms that just stated that it was a EU organisation that kept data inside the EEA, it <em>might</em> be arguable that the UK should no longer have access post brexit.</p>
<h3 id="what-about-theus">What about the US?</h3>
<p>The EU made a finding of adequacy around US based companies providing:</p>
<ol>
<li>They are certified under the EU-US privacy shield</li>
<li>They provide a method of legal redress</li>
</ol>
<p>The first is pretty simple, since Privacy Shield is a self certification process from 2016 onwards, almost any US company that wants to do business with EU citizens or process data on behalf of EU companies has self certified.</p>
<p>The legal redress is a bit harder, because privacy shield says that the organisations <a href="https://www.privacyshield.gov/servlet/servlet.FileDownload?file=015t0000000QJdg">must comply with the EU data protection authorities</a>, and if the UK leaves the EU without a deal, then there may be no access for UK citizens to the EU data protection authorities.</p>
<p>Arguably, the terms at the time clearly meant that US companies should comply with the UK ICO (who is the UK’s data protection authority), but this still is legally unclear, and legally unclear is not a very happy place to be.</p>
<h3 id="so-known-unknowns">So known unknowns?</h3>
<p>As it stands right now, there are two major unknowns that we don’t have clarity on.</p>
<ol>
<li>Can a UK company access data held on EU citizens, that was until now held in the EU once the UK leaves the EU? Will it be different if that data is transfered to the UK now vs if you access the data across the border post brexit?</li>
<li>Can a UK company use the EU-US Privacy Shield agreement post brexit without additional modification?</li>
</ol>
<h3 id="what-if-i-move-my-data-to-aws-ukregion">What if I move my data to AWS UK region?</h3>
<p>Insourcing your data to a different processor based on UK soil before the brexit date would currently be legal, it’s a transfer within the EEA, and a perfectly acceptable form of restricted transfer. Once it’s in the UK, it seems to remain legally acceptable to access it after a no deal exit, and while transfers might be forbidden after an exit, EU citizens can still choose to give their data to you. So if you buy data from an EU company that contains personal data, you can’t conduct a restricted transfer post-brexit (although as before there are other clauses such as consent, contracts and others that you may be able to transfer it under).</p>
<p>However, using AWS UK (or other major cloud providers for that matter) may not solve the problem. Why? Because your contract is with AWS EMEA SARL, a Luxembourg, and thus European country, not with a UK company. While the data might rest on servers in the UK, it’s still definitely under the control of an EU company, and therefore accessing the data could still constitute a transfer from an EU company to a UK company.</p>
<p>The only way you could do this successfully right now would be to host the data with a UK based company who owns servers in the UK, rather than a US/EU company with servers in the UK.</p>
<p>I suspect that a UK company could manage servers in the EU, and this would be far more acceptable than the other way round, as per the transfer example, providing the data was never intended to be accessed or processed in the EU, it would not have been transferred.</p>
<p>This really does tell you that where the data is physically located is of almost no relevance in the ICO’s eyes, it’s entirely about where the organisation is legally based who does the processing, and what the intent is.</p>Post brexit, there may or may not be a problem with data being transferred across borders. But all of the guidance and people talking about this seem to have some very confused concepts of terms and the processes involved, making it really hard to get clear guidance for organisations.Discovery, Alpha, Beta, Live… Part 22018-12-09T12:00:42+00:002018-12-09T12:00:42+00:00https://www.brunton-spall.co.uk/post/2018/12/09/Discovery--Alpha--Beta--Live--Part-2-cb7bb844d140<p>This seems to have come up again, <a href="https://medium.com/notbinary/an-alpha-alpha-approach-5d62fb68429a">with discussions about what the purpose of a discovery, alpha beta actually is, and when you should build your MVP</a>.</p>
<h4 id="tldr"><strong>TLDR</strong></h4>
<p>Discoveries help you identify problems. There are no products, no MVP’s, and almost nothing resembling software development or delivery as typically imagined in this phase.</p>
<p>Once you have a known problem, you need to understand the potential solution space. GDS calls this an Alpha, I prefer to call it prototyping. There are no real products, in fact in many cases, you are rapidly experimenting. Development if it happens at all is rapid hypothesis driven development design purely to get the answer to a single hypothesis. “Does a user know their identification number?”, “If we ask for monthly salary, does that make sense to people paid weekly”. You iterate on hypothesis here, trying to derisk the final product and have a real proposal for what to build</p>
<p>Once you have a desired and tested solution (even if lightly), you can move onto starting to build a product. GDS calls this a Beta, but I think that covers a swathe of development. I’d say the first 2–6 weeks are building the alpha MVP. At this stage you are building a team, building the production capability (hosting, security, build pipelines etc). You are also planning what features are minimum, and it’s the chance to work out what order you’ll approach those in. I’d have called this an “Alpha” phase, but that concept is taken, so I think of this as MVP phase. For very large projects (Universal Credit size), this could be months, but I find you can’t keep teams in this early place for long, they want to get on a deliver, so next you move onto</p>
<p>Building and releasing your MVP. This for me is the now descoped “Private Beta”. This is the phase where you work out just how minimal your product can be to launch, and whether your users will consider that minimal feature set to be viable. Once you have that MVP, you need to get it in front of real users and start getting feedback, so you should start with 1 to 5 invited users, and then maybe roll up to maybe a few hundred.</p>
<p>Once this phase is done and you have users who are using your MVP, you are now into the more complex phase of product delivery, because you have two streams of work, a set of findings from your real users on the ways that they use your product or service, and a set of epics or further features that you always planned to build. You are now building and delivering features to a live audience, and probably growing that audience as well. Your team needs to change the way it operates because breaking the system will bother users. Note you might be in a private beta phase of your product, but live to real users (If only GDS had used different words to avoid this confusion 😭)</p>
<p>Finally you are ready for any number of users to join your service, so you throw the doors open. You are now in a public beta, you still may not have the entire feature set, but anyone can use this service. You are still in active development, and you are still building features from the backlog and gathering user feedback.</p>
<p>At some point, the feedback starts to dry up, you have built all the features you originally promised, and the need for an active development team starts to go away. You need to continue to operate the service, and you need to continue to make changes. There will be security patches, there will still be points of user frustration and of course, there will always be stakeholders with opinions on their favourite feature that they insist on for no discernable data backed reason. Your team is going to look very different now, but your service is Live, the beta banner can go away and the rate of change should drop considerably.</p>
<p>Finally, something else comes along to replace your service, and you migrate users over to it and tear down your service, having a requirements document of backlog burning ceremony and all going on to new and better things.</p>
<h3 id="the-phases-team-and-budget-structures">The Phases, Team and Budget structures</h3>
<h4 id="discovery">Discovery</h4>
<p>During a discovery, the aim for the team is to understand whether a problem hypothesis is actually a problem. What are the problems that users face, who already exists to solve that problem, and why are they not meeting user needs.</p>
<p>A critical question during the discovery should be “Is this a problem that Government should solve?” because there are many problems that exist, that Government is not the right place to solve the problem directly. Instead recommendations might be made to change policies, adjust funding, or share research and data with the charitable sector</p>
<p><strong>Funding</strong></p>
<p>A discovery pot for <em>n</em> discoveries per year</p>
<p><strong>Team Structure</strong></p>
<p>Small fast moving teams of around 3 -5 people. Skills will include delivery management, business analysis, user research and traditional research and analysis at least. Data science may come in handy here depending on the data available to you.</p>
<p><strong>How does work come in</strong></p>
<p>A backlog of potential problems, prioritised by the business in terms of where user needs are not being met, number of complaints and identified waste. Discoveries are selected from the backlog and completed</p>
<p><strong>What happens after</strong></p>
<p>Discovery outputs should be put into a discovery library. A recommendation should be written about whether there are potential solutions and placed into the Prototyping backlog</p>
<h4 id="prototyping">Prototyping</h4>
<p>Carrying out a set of prototypes to explore the potential solution space for problems identified at discovery. You could combine Discovery and Prototyping, especially if you have only a small team, which would increase learning, speed and efficiency, but make it far more likely that people get invested in the problem/solution. Fresh eyes will take a more critical look at the problem space and might determine solutions that the discovery team didn’t recognise</p>
<p><strong>Funding</strong></p>
<p>Via the discovery pot or a prototyping pot, funding for <em>x</em> prototypes per year</p>
<p><strong>Team Structure</strong></p>
<p>Small fast moving teams that can explore the solution space. A product manager, an interaction designer, a user researcher are all key to attempting to come up with prototypes. A developer or two at this stage can help build more high fidelity prototypes, but must be careful not to over complicate the prototype.</p>
<p><strong>How does work come in</strong></p>
<p>Outputs from discoveries should recommend prototyping and testing some solutions with users. These should be prioritised by the organisation based on the identified need, the ability of the organisation to meet the need and the capacity of the organisation to build something after. There’s no point starting a prototype if the organisation just cannot possibly make that prototype a reality later</p>
<p><strong>What happens after</strong></p>
<p>Prototypes should prove whether an idea is possible and whether it can meet the user need. Do we have the data, the technical capability to build this, and if we do will it meet the user need.</p>
<p>At the end of the prototyping phase, the collected learnings should be documented and stored in an archive. A rough outline of an MVP type program could be created, along with estimates for time, money and skills needed. This should be enough to build an outline business case for an MVP</p>
<h4 id="mvp">MVP</h4>
<p>The MVP is the first attempt to actually build a production version of a prototype. Some teams may want to get started fast by taking the prototype and modifying it, but it’s important to remember that to be viable, the code built in this phase has to meet all the robustness and reliability requirements of production software. That might not be possible with the prototype code, and I personally encourage throwing it away and starting again.</p>
<p>Teams will need to consider technical details like hosting, code sharing, developer access, analytics, logging, failure detection, as well as getting designers, user researchers and product managers to produce a working product.</p>
<p><strong>Funding</strong></p>
<p>The outline business case from the prototype phase should recommend a project funding structure. This should include the breakdown of whats needed for the MVP to get to a private beta.</p>
<p><strong>How does work come in</strong></p>
<p>The prototypes generated at the end of prototyping should be demo’d in show and tells and the outline business case considered by the organisations funding models. If the business case is approved, then the project will start.</p>
<p>Internally the team should have an “iteration-0” or “inception” process whereby they discuss the features and use a process such as MoSCoW to find the minimum feature set. This initial work should produce a backlog of “epics”, and the team can start estimating those, and breaking them down into actionable stories for the team.</p>
<p><strong>What happens after</strong></p>
<p>An MVP could be found to be unviable. When put in front of users during user testing, one of the fundamental hypothesis of the discovery and prototype could be found to be fundamentally flawed. If this is the case, the project should be stopped and the discovery or prototype documentation should be updated with this information.</p>
<p>If not, if the MVP is proven successful, then the full business case to roll the MVP out as a full service should be built and approved. This should happen while the MVP is being built, and if prioritised, the team should continue on from the MVP.</p>
<h4 id="build">Build</h4>
<p>This is about building the full service. The MVP may well have entered a private beta, but if not, it should at this point, so that the team can gather feedback on what is working.</p>
<p>The team will continue building and iterating on the product while simultaneously operating the product and ensuring that it is stable, secure and successful</p>
<p><strong>Funding</strong></p>
<p>This should be project funding, for larger projects over several years, all covered with a full business case.</p>
<p><strong>Team Structure</strong></p>
<p>This is your typical delivery team, so all the skills are probably necessary</p>
<p><strong>How does work come in</strong></p>
<p>The Epics that were deprioritised during the MVP should all be on the backlog and for the primary product delivery process. However the team should be gathering feedback from the users using the service and creating new work to iterate on what they have built. That learning should be shared with the team and the backlog should be regularly re-examined and prioritised.</p>
<p><strong>What happens after</strong></p>
<p>The service goes live, the crowd erupts with applause and everybody gets a drink…</p>
<p>No, seriously, the service is live, so the estimate is that the bulk of the delivery is done. The project should end and leave in place a service and a service management team of some form (more next)</p>
<h4 id="live">Live</h4>
<p>During live, the service is running, and you need to be able continue to support it during operational incidents, but also to maintain it, improve it and continue to review feedback</p>
<p><strong>Funding</strong></p>
<p>Continual operation fund and money to pay for the continual improvement</p>
<p><strong>Team Structure</strong></p>
<p>This is the real “it depends”. It really depends on the size of your service, and the size of your organisation.</p>
<p>There are two primary models for in-house maintenance teams.</p>
<p>Firstly you can have a team in existence to continually maintain the service. This is similar to the GOV.UK model, the CMS is never done, there will always be more than enough work to fund an entire team to maintain the system. What you save on RFQ’s and change request charges with your outsourced model you replace with salary costs for keeping this team in place.</p>
<p>If your service is small and the user need doesn’t change very much, then your service might be “done”. It will need maintenance, and there will be small feature requests, but probably not enough to require an entire team. Instead in this structure, you build a maintenance team who maintain a number of services in their portfolio. This team will need to be able to review the analytics, respond to user requests and patch and maintain the software, infrastructure and system in perpetuity.</p>
<p><strong>How does work come in</strong></p>
<p>The service will continue to produce analytics, user feedback and should be subject to user research still to ensure that it is meeting the users needs. Their context changes over time, and the service may need to change with it.</p>
<p>Additionally, the software will begin to age immediately, and will need continuous refreshing to ensure that it is maintained and healthy</p>
<p><strong>What happens after</strong></p>
<p>If the service is no longer needed, you can kill it, salt the ground and start all over again 😂</p>This seems to have come up again, with discussions about what the purpose of a discovery, alpha beta actually is, and when you should build your MVP.Getting started in programming with Advent of Code and Python2018-12-03T10:47:29+00:002018-12-03T10:47:29+00:00https://www.brunton-spall.co.uk/post/2018/12/03/Getting-started-in-programming-with-Advent-of-Code-and-Python-a51dce7db5ed<p>Have you always wanted to program? Have you been interested in the dark and mysterious ways of development? Maybe you’ve done some reading, done a bit of practice, but haven’t been able to find the motivation or the right kind of thing?</p>
<p>For me, development and programming is all about problem solving. I love a good problem, and I like thinking of ways to solve them. One of the reasons that I don’t program enough, apart from being a manager these days, is that finding arbitrary problems to solve isn’t very easy.</p>
<p>That’s why I love Advent of Code. It’s an annual challenge that pits you against a set of programming problems, but each one is scripted against a Christmas themed story. There’s a community of people who approach these problems for a variety of different reasons, from wanting to be first on the scoreboard, to wanting to learn how to approach problems in a brand new language.</p>
<p>I like to do them because the problems are somewhat abstract, but also grounded in a fun story. They are often slightly mathematical, in that there are some very good ways to solve them efficiently, but most of them can be solved in very naive ways as well, which is great for learning and practicing.</p>
<p>So, assuming you’d like a go, I’m going to walk through how you might get started this year. I personally like to use the programming language Python to solve these kinds of problems. It’s simple to learn, it has built in most of the functionality you will need, and it’s preinstalled on a Mac, so there’s very little fiddling around to do to get started.</p>
<p>So lets go</p>
<h3 id="install-anything-extra-youneed">Install anything extra you need</h3>
<p>You are going to need 3 things to start day 1.</p>
<ol>
<li>A text editor</li>
<li>Python installed</li>
<li>A github account</li>
</ol>
<h4 id="a-texteditor">A text editor</h4>
<p>You can choose a text editor of your liking. On a mac, you can press Apple-Space and type ‘textedit’ and you’ll get the built in text editor. On Windows you can use notepad.</p>
<p>While textedit or notepad will do the job, you’ll find life a lot easier if you install a programmers text editor. Why? Because they understand that you are trying to program and that’s different to just typing normal words.</p>
<p><img src="/images/uploads/1__XlA7HO62DBfgIAuP__PwapA.png" alt="Python code in TextEdit on the Mac" />
Python code in TextEdit on the Mac</p>
<p>I’d recommend using something like <a href="https://atom.io/">Atom</a> or <a href="https://code.visualstudio.com/">Microsofts VSCode</a>. Both are completely free, simple to install and both come out of the box with something called “Syntax Highlighting”. That means that it will put any reserved words or symbols into a different colour.</p>
<p>The primary help for a beginner with syntax highlighting is that it will warn you if you forget to close a bracket or a quote mark somewhere, because all the following text will be a single colour. It’s very helpful, so go to one of those sites and install that software, it’s simple and easy.</p>
<p><img src="/images/uploads/1__BqWA5HEsALHquHcIz4bi__A.png" alt="The same code in Atom. See the colours showing strings, special words etc" />
The same code in Atom. See the colours showing strings, special words etc</p>
<p>Both should install just fine on windows or Mac (I use a mac, so the windows instructions in here won’t be as good I’m afraid)</p>
<h4 id="installing-python">Installing Python</h4>
<p>Next we need python installed. Luckily for Mac users, python comes pre-installed on a mac. Press Apple-Space and type “terminal” and you should get a terminal window come up. In that window type “python” and you should see something like this. (This is called the REPL)</p>
<p><img src="/images/uploads/1__TlpxgyLPIFP__JHIweAEp0w.png" alt="Python on the mac, ignore the /usr/bin bit" />
Python on the mac, ignore the /usr/bin bit</p>
<p>At this prompt you can just type your code, and see what works. Type 2+3 and it should print 5 for example.</p>
<p>Hold down the ctrl key and press D to quit the REPL when you are done</p>
<p>If you are using windows, go to the <a href="https://www.python.org">Python website</a> and follow the instructions</p>
<p>While you can use the Python REPL, it’s not recommended. What we do most of the time is type our commands into a text file, and then run the file using python. This makes it easy to correct the code if you make a mistake, which you will do, and run it again.</p>
<p>Every developer is different on how they arrange their code, so you need to work out what works for you. I personally have a directory in my home directory called “work” and in there is a subdirectory for each project I work on. I created a directory for the advent of code 2018, and then I create a directory for each day.</p>
<p>But before we get there, we should discuss version control.</p>
<h4 id="a-githubaccount">A github account</h4>
<p>In order to get into the advent of code you need to sign in. They let you sign in with Gmail, Twitter and your Microsoft account if you want, but you should take this opportunity to get yourself a github account if you don’t have one already.</p>
<p>Github is where almost all developers in the world store their code. There are alternatives, but it’s a good start. If you ever want a job as a developer, a lot of places will look to see if you have a Github account as part of the screening process.</p>
<p>To sign up, go to <a href="https://github.com">Github</a> and hit the sign up button. Come up with a name for yourself. Remember that this might be your professional identity in future, so keep it clean. For years I was MIB because I only wore black, but as I realised that this would be professional, I changed my online presence to be bruntonspall everywhere. It doesn’t have to be your name, and for most people probably shouldn’t be.</p>
<p>Now, we have a choice at this point. I would recommend keeping all of your code on Github and regularly pushing and storing it there as you go. But I don’t want to cover using Git and GitHub in this tutorial, so if you want to do that, go read <a href="https://guides.github.com/activities/hello-world/">something like this</a> or get a friend to help.</p>
<p>If you are using github, then create a repository called AdventOfCode2018 and then clone it into your work directory. If not, from that terminal, type <code class="language-plaintext highlighter-rouge">mkdir work</code> to create the work directory, and then <code class="language-plaintext highlighter-rouge">cd work</code> and <code class="language-plaintext highlighter-rouge">mkdir AdventOfCode2018</code></p>
<p>Also, go to the <a href="https://www.adventofcode.com">Advent of Code website</a>, sign in with Github and click day one and you should be able to read the problem</p>
<h3 id="lets-get-started-withpython">Lets get started with Python</h3>
<p>Right, we’re going to program now. Before we start properly, I’m going to suggest the way that I structure my code. There are other ways of doing this, but this works well for me for this kind of coding problem.</p>
<p>Firstly, I use something called unit testing to explore the problem. Python comes with a module of code called <code class="language-plaintext highlighter-rouge">unittest</code> which does most of the hard work for you, and makes it easy to describe tests.</p>
<p>I write tests for my code before I implement them, and then I run the tests to be sure they fail for the right reason. As I go through the problem, everytime I encounter a bug or weirdness, I create a new test to document what I think should happen.</p>
<p>Lets take day one as an example.</p>
<p>We have been given a set of numbers to sum up. These look like <code class="language-plaintext highlighter-rouge">+1, -2, +1</code> and we need to add them all together and get the correct number.</p>
<p>So I created my first test to look like this</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def test_basic_solve(self):
data = [1, 1, -2]
self.assertEqual(0, puzzle.solve(data))
</code></pre></div></div>
<p>Ignore the syntax and how it looks, the critical thing here is to see how the test is structured.</p>
<p>First I create the data in the form I want. (We’ll cover the syntax in a bit)</p>
<p>Then I use a method called assertEqual to compare my expected result <code class="language-plaintext highlighter-rouge">0</code>, with the result of calling my code <code class="language-plaintext highlighter-rouge">puzzle.solve(data)</code>.</p>
<p>If my code didn’t account for negative numbers correctly, then this would fail my test. If it didn’t count correctly, it would fail the test, and so on.</p>
<p>For my first try, I made the puzzle.solve just simply always return -1, so my test failed. I can now write the code in puzzle.solve and know that when I get it right, the test will pass</p>
<p>I run the tests by typing <code class="language-plaintext highlighter-rouge">python test_puzzle.py</code> in my terminal, and it shows me which ones have passed or failed.</p>
<h4 id="structuring-yourcode">Structuring your code</h4>
<p>Python allows you to structure your code into 3 different concepts, modules, classes and functions. I use all three in my template code, and I’m going to explain how and why that works.</p>
<p>Firstly, in python a file contains code into a module. If I create a file called <code class="language-plaintext highlighter-rouge">myfoo.py</code> that has some code in it, then in another file, say <code class="language-plaintext highlighter-rouge">mybar.py</code> I need to bring that code into scope to be able to call it. I do this by putting an import statement at the top of my code. <code class="language-plaintext highlighter-rouge">import myfoo</code> will bring that code into scope for my file.</p>
<p>Inside a file, code in there is run when the file is either called by typing <code class="language-plaintext highlighter-rouge">python filename</code> or when the code in a file that is run that way says <code class="language-plaintext highlighter-rouge">import myfoo</code>.</p>
<p>Normally we don’t want to do that, so we bundle our code up into functions. A function is a piece of code that takes some arguments and normally returns a value. It’s a good way to divide up your code into neat segments that do different things.</p>
<p>In python we define a function using the <code class="language-plaintext highlighter-rouge">def</code> keyword, and the format looks a bit like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def my_func(a, b):
return a+b
</code></pre></div></div>
<p>In this case we’ve defined a function called my_func. We’ve said it takes two arguments, which we’ve called a and b.</p>
<p>When the function is called, you can essentially assume that we replace a throughout the indented code with the first argument and b with the second argument. So if we call the function by writing elsewhere <code class="language-plaintext highlighter-rouge">my_func(2,3)</code> then a would be replaced with 2 and b replaced 3.</p>
<p>Return is a special keyword that says stop processing the function and return back the value on the right. In this case, the sum of a and b together (or 5).</p>
<p>Finally classes. I don’t use classes in Python very much for this kind of coding because we tend not to be building systems that are complex enough to need them.</p>
<p>But the testing framework requires one to exist, but I’m not going to explain it, just show you what I do.</p>
<p>Here’s puzzle.py</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def parse(lines):
return None
def solve(data):
return -1
</code></pre></div></div>
<p>Here’s the test_puzzle.py file</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import unittest
import puzzle
class TestBasic(unittest.TestCase):
def test_pass(self):
data = puzzle.parse(file("input.txt").readlines())
answer = puzzle.solve(data)
self.assertEqual(0, answer)
if __name__ == '__main__':
unittest.main()
</code></pre></div></div>
<p>This is my basic template, I start each days puzzle with this code.</p>
<p>The puzzle.py defines two functions, because I split my solutions into two main parts. The first is to parse (that’s read and understand) the input file into a form or structure that I can use.</p>
<p>The second is to solve the puzzle with the input structure.</p>
<p>In the test code we do 3 things.</p>
<p>Firstly we import the unittest module and our puzzle file into scope. Where we say puzzle.something that will call a function in the puzzle file</p>
<p>Next we create a class to hold our tests. This can just be copied as is, but essentially it declares a class that inherits from the unittest.TestCase class, which the unittest framework uses to provide some features such as the asserts, and also to magically find our tests</p>
<p>Inside the class, we create a function called test_pass. It takes a single argument, which you can ignore, but that “self” is a standard python thing, we’ll use it in a minute</p>
<p>First we call the parse function in the puzzle module, and we use some of pythons built in modules to read the file and divide it into lines. If you create a file called input.txt and put some lines of stuff there, then that’s what would get sent to the parse function.</p>
<p>We take the return value of the parse function and call it data.</p>
<p>We then pass the data to the solve function and take that return value and call it answer.</p>
<p>We then compare the answer to 0 to see what it is.</p>
<p>Finally at the bottom of the file, there’s some code that says if this file is being run by calling <code class="language-plaintext highlighter-rouge">python <filename></code> then run the unittest frameworks main function. It’s this that finds and runs all my tests</p>
<p>If you have those files, and create a file called input.txt (which can be empty) and type <code class="language-plaintext highlighter-rouge">python test_puzzle.py</code> it will run the tests and should fail, as below</p>
<p><img src="/images/uploads/1__kw__c0PtywvjJ__4IERzj3ww.png" alt="" /></p>
<p>This failed because we said we wanted to get a 0, but the puzzle.solve function returns a -1</p>
<p>I store all this code in a folder called ‘template’ and I simply copy it into a folder called ‘day1’ to try to solve day 1, and repeat each day. This makes it nice and easy to start again each day</p>
<h3 id="solving-day-1-part1">Solving Day 1, Part 1</h3>
<p>Ok, so the device starts with a frequency of 0, and we get given a list of numbers +1, -2 etc, and we want to know what frequency we end up on.</p>
<p>We could just model this exactly as described, take each number and apply it to our current frequency, and that’s probably the easiest way of doing so.</p>
<p>Mathematically, hopefully you remember that adding a negative number to something is the same as taking away the number? That means that (2)+(-1) should equal 1. That means we don’t need to do anything different for the + or -letters, we just need to turn them into the right kind of number.</p>
<p>This is what I mean by separating the puzzle into parse and solve parts. Our first problem is to take a text file filled with “+1, -3” and turn those into numbers that we can do maths on.</p>
<p>Python has a set of handy functions for this stuff, called the <a href="https://docs.python.org/2.7/library/">Python Standard Library</a>, and in this case if you click the link to “numeric types”, you should see that there is a function called int(x), which is defined as “<em>x</em> converted to integer”.</p>
<p>Lets’ test this theory with a simple test,</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def test_basic_parse(self):
data = puzzle.parse('''+1
+3
+2'''.split())
self.assertEqual([1, 3, 2], data)
</code></pre></div></div>
<p>If we add this code into our testcase (being careful around indentation, it should be indented one step from the class, the same as the other def test… code.</p>
<p>This is going to try using the parse function on the string</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>+1
+3
+2
</code></pre></div></div>
<p>Note that the ‘’’ characters start a “multiline text string”, which is important as it remembers the line breaks in between each character. The function expects a list of lines, so we need to call the split function before passing it into the parse function.</p>
<p>We pass that information to the parse function and expect back a list of the numbers <code class="language-plaintext highlighter-rouge">[1, 3, 2]</code></p>
<p>If we run this test, it should fail and tell us that parse returned None, which is right, because we haven’t written that yet!</p>
<p>So let’s write that code shall we?</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def parse(lines):
ans = \[\]
for line in lines:
ans.append(int(line))
return ans
</code></pre></div></div>
<p>We’re going to do this nice and simple. First we create an empty list. We can add things to the list using the append function.</p>
<p>Next we step through each line in the lines of data passed to us, and we turn it into a number and add the number to the list.</p>
<p>Let’s try running that test.</p>
<p><img src="/images/uploads/1__EOOnRLKDDz6m8ofQmPcHBA.png" alt="" /></p>
<p>If we did this right, you should find that the test passed, but the original test is still failing. In unittests, you’ll get an output at the top, with passing tests represented by a <code class="language-plaintext highlighter-rouge">.</code> and failing tests with an <code class="language-plaintext highlighter-rouge">F</code>, so <code class="language-plaintext highlighter-rouge">.F</code> means that one passed and one failed.</p>
<p>So now we can turn the file into numbers, we can start doing maths on them!</p>
<p>Our solve function is going to be passed the list of numbers and needs to return the end frequency.</p>
<p>We could do this by running the real one, but let’s go slow. The AdventOfCode gives us some samples with known answers, so let’s try one. Add the following code to the test_puzzle.py file</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def test_basic_solve(self):
data = [1, 1, -2]
self.assertEqual(0, puzzle.solve(data))
</code></pre></div></div>
<p>Re-running the tests should give you 3 failing tests, so let’s see if we can write the solve function.</p>
<p>We want to step through each number and add it to the total so far. The simplest possible code that can solve this is very similar the bit we just wrote:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def solve(data):
total = 0
for num in data:
total = total + num
return total
</code></pre></div></div>
<p><img src="/images/uploads/1____Hct__ncywwGgfZ3ppYzhGQ.png" alt="" /></p>
<p>Now we’ve got a passing sample test, and our final test is showing something interesting. What’s that 599 answer? Well it’s the answer for my version of the code problem. Your number will be different because the Advent Of Code wants you to solve your own problems, not just look up the answers.</p>
<p>If you type your number into the Advent of Code website, you’ll have solved the first day!</p>
<p>Also, make that test pass, by writing the number 599 into the test, and you are good to go.</p>
<h3 id="part-2">Part 2</h3>
<p>I’m not going to solve day 1 part 2 for you, but it’s a more complex problem to solve.</p>
<p>I recommend starting by using a piece of paper and seeing how you would detect that you’ve hit the frequency a second time. I think the most likely thing is to keep a record of each frequency that you get to. A python dictionary with the key being the frequency and the value just being a simple “True” value should enable you to detect if you’ve seen that frequency before and promptly stop processing.</p>
<p>You’ll also need to have a go with the samples and make sure that your code will loop round the values more than once.</p>
<p>Best of luck, and I hope you enjoy it</p>
<p>You can get the code I’ve written over at <a href="https://github.com/bruntonspall/AdventOfCode2018">my github page</a>, although don’t look ahead to problems you haven’t solved. You can see people discussing the problems on the <a href="https://www.reddit.com/r/adventofcode/">Advent of Code reddit page</a>, and ask anyone you know to help</p>Have you always wanted to program? Have you been interested in the dark and mysterious ways of development? Maybe you’ve done some reading, done a bit of practice, but haven’t been able to find the motivation or the right kind of thing?The basics are sometimes the hardest things2018-11-07T23:06:44+00:002018-11-07T23:06:44+00:00https://www.brunton-spall.co.uk/post/2018/11/07/The-basics-are-sometimes-the-hardest-things-c87cee5dc7e5<p>If only we could apply patches, then we could do more interesting security work.</p>
<p>I’ve as guilty as the next person of over-simplifying how easy the basics actually are. We exhort and condemn organisations and people for not getting the basics right. “Having a company vision is just table stakes”, but we forget how how hard it can actually be to execute the basics well.</p>
<p>If we take patching, it kind of sounds easy to just apply patches to our software. But to do that we need to know what software we have deployed, and the moment you scale up from one team to 5 teams, you are going to lose that visibility.</p>
<p>Even if I can keep some visibility of what systems I have, I also need to pay attention to what software I’m running, what version it is, and what patches are actually available.</p>
<p>With multiple service delivery teams, each using a slightly different technology stack, we now need to track whether we are patching ruby, node.js, linux, windows, vmWare, Xen and a whole host of other software.</p>
<p>If we add in the complexity of a typical corporate estate, desktops, email, productivity tools, mail clients, instant messenger tools, collaboration tools as well as whatever else has been installed to help our service teams deliver, patching turns out to be a lot harder in practice than it sounds as a trite soundbite.</p>
<p>We should patch early, and patch often, and we should aim to get the basics right before we start investing in the machine learning quantum advanced threat protection that the vendor wants to sell us, but we shouldn’t deride people for finding it hard.</p>
<p><em>[This blogpost is part of an attempt to blog once a day for the entirety of November (#NaBloPoMo), inspired by</em> <a href="https://medium.com/u/1b3280d2beeb"><em>Terence Eden</em></a><em>. This series of blogposts are therefore unedited and written on the day. Errors and Omissions Excepted]</em></p>If only we could apply patches, then we could do more interesting security work.Nudge or strategy?2018-11-06T20:49:23+00:002018-11-06T20:49:23+00:00https://www.brunton-spall.co.uk/post/2018/11/06/Nudge-or-strategy--e7e5893bb3b8<p>Should you nudge your users into better behaviour, a small microaction at a time or do you need large sweeping changes to change behaviours?</p>
<p>This debate seems to have gone on for decades, in all manners of locations, from healthcare to technical architecture, economics to the adoption of devops.</p>
<p>I tend to favour the former, preferring to use nudge economics to slowly change direction for teams, but recognise the need for the latter sometimes.</p>
<p>But it seems like sometimes people are zealous adherants of a certain method and refuse to acknowledge that the other way might work.</p>
<p>In the conversation around password complexity, nudge zealots argue that by reducing the complexity of passwords, you increase the ability of users to remember them, and create better user behaviours for password hygiene. Whereas some people yell that decreasing password complexity even with increased length decreases the overall entropy in the system.</p>
<p>There are times where you need both methods in your toolbox, where you need to know when to apply which strategy.</p>
<p>Due to my background, I’ve always likened this to the problem of Path-finding in AI in computer games, and what’s called the hill climbing problem.</p>
<p>In this system, we have a map which has varying elevations, green being lowest and red as the highest.</p>
<p><img src="/images/uploads/1__9ZfT6lU6E__fwCIrcE9vhTQ.png" alt="A map with some hills" />
A map with some hills</p>
<p>A very simple algorithm is to look at the neighbouring cells and if you find cells that are higher than you, you can go there, if not, pick one of the cells at random and go that way.</p>
<p>However this will result in a problem on the map above, which is that anybody who starts on the right of the small hill will climb to the top of the small hill and go no further.</p>
<p><img src="/images/uploads/1__M9dIAyGzNYcxYVGsR1r5ZQ.png" alt="" /></p>
<p>A better algorithm might be to search the map for the highest point, and flow down hill as much as possible while reducing the distance from the current point to the start point. This would result in this map</p>
<p><img src="/images/uploads/1__e3__b5NQV5a5z__R__GLDhIBA.png" alt="" /></p>
<p>I view nudge actions as very similar to the first algorithm. They’ll get you so far, and will get you moving. They’re easy, efficient and simple. But they wont get you to your ultimate goal.</p>
<p>The strategic route takes more effort, more time and more thought, but it will get you to your ultimate goal in the end.</p>
<p>But really you need both practices in your toolbox, you need to be able to action immediately, and nudge your team to better practices, but you also need an eye on the strategic options open to you and know when to take more severe action.</p>
<p><em>[This blogpost is part of an attempt to blog once a day for the entirety of November (#NaBloPoMo), inspired by</em> <a href="https://medium.com/u/1b3280d2beeb"><em>Terence Eden</em></a><em>. This series of blogposts are therefore unedited and written on the day. Errors and Omissions Excepted]</em></p>Should you nudge your users into better behaviour, a small microaction at a time or do you need large sweeping changes to change behaviours?Language forms mental furrows2018-11-06T01:14:10+00:002018-11-06T01:14:10+00:00https://www.brunton-spall.co.uk/post/2018/11/06/Language-forms-mental-furrows-682ec3a0492d<p>When I worked at GDS, I worked with a lot of people who got very specific about their language. We talked about users, not customers; user needs not requirements and clear plain english where possible.</p>
<p>I was never very good at this, I have a tendency to use 20 words where one will do, and I’ve never been terribly concise or consistent about my language.</p>
<p>But the language that we use on a daily basis really matters. Why?Because language forms the habitual furrows into which our thoughts get organised.</p>
<p>If I am asked to write a resource utilisation plan for my development team, I’m likely to think of my staff as fungible resources that can be re-allocated at will. If I talk about the requirements for a system, then I get focused on producing the best requirements of the system rather than asking what the users actually need from the system.</p>
<p>This isn’t always true, there are many gifted and talented people that I’ve met, who are perfectly capable of talking about a resource plan, meaning their staff, and keeping in mind that Karen is a member of their team with opinions, hopes and dreams and not simply a Java programmer. But the reality is that when faced with a problem, many of us try to solve it, and if the problem is phrased in a certain way, we’ll tend to solve it with a certain mental headspace.</p>
<p>Even worse, this becomes habit over time, so this language forms habits for us. Asking “what is the user need?” becomes a knee jerk response to someone who says the word requirements, regardless of their intention or meaning.</p>
<p>If you want to break yourself of this habit, then the first approach to many problems has to be to rephrase the problem. Instead of solving the problem set in front of you, ask yourself how to rephrase the question and see if that helps shift your thinking</p>
<p><em>[This blogpost is part of an attempt to blog once a day for the entirety of November (#NaBloPoMo), inspired by</em> <a href="https://medium.com/u/1b3280d2beeb"><em>Terence Eden</em></a><em>. This series of blogposts are therefore unedited and written on the day. Errors and Omissions Excepted]</em></p>When I worked at GDS, I worked with a lot of people who got very specific about their language. We talked about users, not customers; user needs not requirements and clear plain english where possible.Organisations are made of people2018-11-04T22:23:01+00:002018-11-04T22:23:01+00:00https://www.brunton-spall.co.uk/post/2018/11/04/Organisations-are-made-of-people-92ae80241707<p>Does Google value your privacy? How about Facebook? Your bank cares about you we are told.</p>
<p>We often forget that organisations are not entities in themselves but are a contingent of people acting in broadly the same way.</p>
<p>When I worked for GDS I was often asked what “GDS thought” about a specific topic. What is GDS’s position on serverless? How about single page applications?</p>
<p>GDS as an organisation produced a service manual, which was the result of much internal discussion, but was also in many sentences, the result of a single person expressing their views and waiting for challenge.</p>
<p>An organisation cannot really hold a view, and cannot have a position. It’s leaders might consider that the organisation stands for something, but in reality, organisations are made up of highly varying people all with different views.</p>
<p>When you next talk to someone claiming to express their organisations view, or know what an organisation would say, you need to ask yourself whether they are simply expressing their own views, reinforced through their perception of the organisation.</p>Does Google value your privacy? How about Facebook? Your bank cares about you we are told.Build and Deploy are different concerns2018-11-03T23:39:41+00:002018-11-03T23:39:41+00:00https://www.brunton-spall.co.uk/post/2018/11/03/Build-and-Deploy-are-different-concerns-7902e2f85a32<p>You’ve spent days crafting the perfect bit of code, and you are ready to put it in front of real users.</p>
<p>You hit the build button, and some process takes your code and all the tests and makes sure that it works. In the same place, you hit the deploy button and the code is compiled and shipped to a production machine and everything works just fine right?</p>
<p>It’s 2am, and you are on call. Someone is ringing. You pick up and are told that the site search system isn’t working anymore. Nobody has changed this in months, but something isn’t working. You log in and look at the graphs. Memory use and disk use has been steadily climbing for months, and now the service can’t run. You restart the service but it doesn’t help, it just reloads the indexes from disk. In a fit of pique, you look at the changelog, and yup the last deploy matches with the steadily climbing memory usage, so a rollback will fix your problem.</p>
<p>You hit the deploy button to redeploy the app, but the deploy fails. During the deployment process, the system tried to download an old dependency that doesn’t exist anymore. You search fruitlessly, but trying to roll forward the dependency means branching the code, and the updated version of the library has compatibility issues.</p>
<p>This shouldn’t ever happen. A deployment shouldn’t require rebuilding the thing you are deploying. It should be constant and require no external dependencies.</p>
<p>A build process is exactly what it says, designed to build a relocatable binary package of some form. In the C and C++ days, this meant doing the compile and the link phase, to produce an executable.</p>
<p>But in Python or Ruby, JVM’s and .NET, the blurring of the boundary between compile time dependency and runtime dependency has been blurred. Do you need libssh installed by the operating system or do you do pip install -r requirements.txt at deploy time?</p>
<p>I’d argue that building a relocatable, redeployable artifact is the job of the build process. The only communication between the build process and the deployment process should be the creation of the artifact, and the registration of the artifact metadata.</p>
<p>At deploy time, the only action should be to take the artifact and put it into the running system. It doesn’t matter whether this is to rsync over the php and NOHUP the server, or transfer a self executing jar and run it on the server. From docker containers to go executables, elf binaries to zip files of code, the deployable artifact should remain inviolate for its entire life.</p>
<p>The reason for this is that deployments should be repeatable, consistent and reliable each and every time. Any build step that gets put into that deployment tends to break that model, and cause deployments that cannot be relied upon.</p>
<p><em>[This blogpost is part of an attempt to blog once a day for the entirety of November (#NaBloPoMo), inspired by</em> <a href="https://medium.com/u/1b3280d2beeb"><em>Terence Eden</em></a><em>. This series of blogposts are therefore unedited and written on the day. Errors and Omissions Excepted]</em></p>You’ve spent days crafting the perfect bit of code, and you are ready to put it in front of real users.