qone.io

different sides of devops in recrutment and engineering

Tue, 20 Dec 2016 00:00:00 +0000

The question is what is devops? How do you recruit a devops engineer. The short answer is you don’t. a devops engineer does not exist. The thing about recruiting a so called devops engineer is all a bit undefined and often miss understood and is often used by head hunters and recruiters to find software engineers/ops engineers.

If we look at it a bit in short first, what is devops! devops is a way of working in a engineering organisation. The way of working generally means a few thing. You start by moving the organisation from a itil or waterfall to a more agile way of working. What this means is that you strive to have a culture that allows people freedom under responsibility, short development cycles, allowing quick adoption and changes, to move fast. The one big part here is trust in the engineers.

simple prometheus exporter in python

Sun, 28 Aug 2016 00:00:00 +0000

writing a simple prometheus exporter to collect metrics from a external system that needs monitoring. I will use the python official prometheus_client package for python and falcon to serve the exporter.

notise the following example is written with the assumtion that you are collection metrics from some other systems when multiple services is written in the same framwork and have a generic set of metrics. That is why there is a service label included. Normaly you should get the service name, from your service discovery.

lessons learned by running docker on mesos and aurora in production

Sun, 21 Aug 2016 00:00:00 +0000

Short about how we run docker. The way we are running docker using aurora to scedule on mesos currently both on aws and on on premise. Every service have a aurora job defined. When a job is sceduled the service allocates resources from mesos and starts a job. That job is a set of one or multiple docker containers. This is all good and works ok.

The difficulties with running docker is not the running part. When we are working with life cycle of a service. This means that the amount of stopped containers, old version of images and unused images, and volumes, will start to eat up disk space. This is something that docker have not solved nicely yet. There is a lot of efforts from the community to come up with suggested solutions to purge junk that is building up while running it in production.

killing the traditional quality assurance way of testing

Sun, 07 Aug 2016 00:00:00 +0000

After years in testing roles it’s obvious that the traditional way of doing testing will no longer hold up. What i mena with the traditional way of testing is. Quality assurance, you have separate development and testing teams i.e QA teams. When the development team is done they deliver to QA and they do there work. QA follows a long and usually static test plan and process, your not suppose to step outside the boundaries of the test plan or the process. This way of working is not only slow it disconnects the guys working with testing and development.

Docker alpine smaller image footprint

Wed, 20 Jul 2016 00:00:00 +0000

Working with docker images to minimise the footprint i.e the size if a image. There is a few things that you can do to get smaller images. I will show some examples for a small go and python3 service built in a Debian and alpine linux based image to compare the result and the footprint that a image.

The key to building small docker images is only use one RUN step in the Dockerfile. Why you might think. Every RUN in docker is a layer. The layer will contain what you do in that layer like adding package cache. If you then remove the cache in a later RUN you will still have it in the parent layers. So what you do is use al lot of && in the same RUN and in the end remove the files and cache you don’t need. Selecting the base image will affect you the most when it comes to footprint. I will look on Debian and alpine based images.

Fabric dynamic hosts tasks

Thu, 30 Jun 2016 00:00:00 +0000

Using fabric with dynamic hosts. In the fabric documentation most of it is using examples and is assuming that you you have a static set of hosts. That is no longer the case when we are working with cloud and a dynamic amout of hosts. The way the fabric want’s you to run tasks is to use the fabric.api.env function to set the hosts you have. If you now want to do this on the fly you have one option a function called execute avalible in fabric.api that takes a key hosts this key will be used to tell fabric to run on this hosts as if you had set env.hosts .

why do we do need unit tests

Sun, 05 Jun 2016 00:00:00 +0000

You might be thinking well unit tests is kind of a obvious thing that you have when you are writhing code. If you do you are wrong. Not all developers want and can write unit tests. This is kind of interesting for me.

Short about me. I am a software developer in test, i have been spending years on designing, test automation and writing tooling to help other developers to to be more efficient in finding issues in the code they deliver. And strategies around quality and testing. Mostly on the backend and infrastructure area. Also being part of the design phase of a service to help point out risky areas.

python click bash zsh auto complete same files

Sun, 24 Apr 2016 00:00:00 +0000

Sharing the same auto completion for zsh and bash. In zsh there is somthing called bashcompinit that can be used to share the same completion for bash and zsh. here is a small example project that will fix completion for you when you are not using a nested command structure. see next post about nested completion for bash/zsh

project structur

├── MANIFEST.in
├── Makefile
├── auto_compleate_install.sh
├── ecli
├── ecli_lib
│ ├── __init__.py
│ └── main.py
├── requirements.txt
└── setup.py

click example cli command

python click bash zsh auto complete same files nested command groups

Sun, 24 Apr 2016 00:00:00 +0000

project structur

├── MANIFEST.in
├── Makefile
├── auto_compleate_install.sh
├── ecli-complete-nested.sh
├── ecli-nested
├── ecli_nested_lib
│ ├── __init__.py
│ └── main.py
├── requirements.txt
└── setup.py

click example cli command

python protobuf over http

Mon, 11 Jan 2016 00:00:00 +0000

The point of this is a smal example on how you could use protobuf to send data over http. In this example i will be using Python and the packet falcon for the server. A command line tool as the client. The example will just be a simple ping/pong containing a message, channel, and PING or PONG sent to the server. The server will respond with the same message and channel and a PONG.

jenkins test result dashboard

Sat, 14 Nov 2015 00:00:00 +0000

Minimal Jenkins test result dashboard

The background to making the dashboard is as part of the on boarding at work we are able to do a small projekt. We are using Jenkins as the tool for our continuous integration tests like many others. Since most of the Jenkins plugins that exists for build information is not that pretty, it made a lot of since to do a minimalistic dashboard for the test results. And why not do something in React as the frontend rendering engine and python as the backend. The dashboard is also using redis cache for the result data, and in the docker stack haproxy to terminate http on port 80. The frontend polls the backend every 15 sec for new data.

backup of linux vps using gsutil

Sun, 30 Aug 2015 00:00:00 +0000

Backing up linux servers to google clound storage with gsutil using the storage option nearline. What is needed to do this is a google cloud accound. Start by creating a new project for backup. After that use that Project ID to configure gsutil in the next step.

start by configuring gsutil.

$ gsutil config

Create a nearline bucket on google clound storage. The mb flag means create bucket. the -l flag means location options is ASIA, EU, US. For more options take a look in the gsutil mb doc

selenium as a service in osx

Sat, 15 Aug 2015 00:00:00 +0000

Setting up selenium server as a service in os x that starts on boot. What is needed is a plist service file to start and take stdout and stderr and log the output. I will be using homebrew for install of the selenium server. in this example i have used a vagrant image of Yosemite you can find it here vagrant image.

A NOTE about the OS X Licensing. Apple’s EULA states that you can install your copy on your actual Apple-hardware, plus up to two VMs running on your Apple-hardware. So using this box on another hardware is may be illigal and you should do it on your own risk.

docker machine and aws in combination with ansible

Sat, 13 Jun 2015 00:00:00 +0000

Initial setup with ansible and docker-machine. In this case i have been using aws to run a docker host for me created with docker-machine. I had a goal to provision a ec2 box and have a few docker containers running in that ec2 vm. A note here is that the same way should work on any of the docker-machine driver there is nothing unique with aws in the way i have done this. From what i know you cant NOT use ansible on boot2docker, i have assumed that there is a full linux vm running.

sqlite3 in memory db for testing db functions in python

Sat, 06 Jun 2015 00:00:00 +0000

designing a app to be as testable as posible. When writing a application that you like to, write unit tests the best options it to always on everything written for a application to not be dependent on that the application is running. Since if we can import a function and muck or just test the isolated function. One example on this is a database function, and a database. In this example i will use sqlite3 as a example. sqlite have a option to start the db in memory. This is something is very useful since you like to have a clean state of the database for every test.

encrypted backup with arq amazon and s3 glacier

Sat, 16 May 2015 00:00:00 +0000

As most that have any knowledge of IT and tech. Backup is a big topic also encryption now in the later days when there have been a lot of talk about that our data can be read if we are not using encryption everywhere. Now when most of os have most of our life in a digital form the information that we store in our phones, computers are more then ever a target for criminals and the data is important to us. Not only that electronics to break and when that day comes none will be happy if all there personal data, files, photos are lost due to a broken phone or computer. This is why backup is such a important thing. Doing backup right is not that hard. running around with a external drive or usb memory is not a god option, as a main backup. You will forget to do it, the best is to have one or multiple incremental backups that is running in the background so you don’t have to think about all the time. You should of course try to do restores from the backups every now and then to be sure that they are working and running as expected.

my travel gear

Sun, 03 May 2015 00:00:00 +0000

Travel with a Minaal. I am using packing cubes to organize my clothes. When i am packing clean clothes i am roling them and packing them in the packing cudes. This stops somewhat wrinkled clothes. The Minaal it self is greate to have some weight in. Backpacks i have used before in a Mission Workshop Vandal. That one is grate but it’s heavy. Load in the Minaal works grate with a heavy macboook pro retina 15, since it’s close to the back the for some resoce it’s not fealing that heavy. Also having lot’s of small pockets to packing in is grate, Even some hidden ones like the one in the bottom of the main compartment. When going trough the airport security the laptop pocket makes it easy to just take out and put back again.

docker compose with selenium

Fri, 24 Apr 2015 00:00:00 +0000

Running a local setup with selenium hub with Firefox and Google Chrome nodes using docker-compose . Using docker-compose makes the setup even more convenient the using the docker --link commands. Starting and stoping the setup is just easy. Scaling up and down the number of Chrome and Firefox nodes is just a command to.

docker-compose builds a setup based on a compose file that is a yaml file. Creating the setup file for a selenium hub with two nodes can look like this.

a theory of productivity

Sat, 18 Apr 2015 00:00:00 +0000

How to think about productivity. Lest take a example a math test. The test contains 20 problems, the total assumed time with no interruption to finish the test with all correct answers is 60 minutes. That is all god. Now lest assume that we mix in interruption in to the factor. At 5 times at random times you will get interrupted. The interruption will be to answer a random question, that will take 1-2 min to answer. You are then to return to the test. If we now assume that the interruption will be done so that you get interrupted when you are close to a answer. This interruption will result in 3 of 5 tests that you got interrupted no will be incorrect. For every interruption you also had to restart the current question you were on since you forgot the answer you almost had.

html email with aws ses and boto

Sat, 11 Apr 2015 00:00:00 +0000

Sending html email with amazon ses simple email service using the python module boto.

to be able to start using the ses service you need to verify two emails. The first you what is the address that you like to use to send from. The second is the one you like to receive it in. To be able to verify that addresses you need a valid address since a validation email will be sent to the address. after that is done, you can start yo use the mail addresses. At this stage you will only have a sandbox version of ses, you will only be able to send/receive from the verified addressed. at a later state you can request production that will open to send email to any one. When in the sandbox state of ses you will only have a limit of 200 emails every 24h.

bulding a debian docker base build for a flask app

Fri, 03 Apr 2015 00:00:00 +0000

Start by pulling down a debian base build. docker pull debian:8.0 .Creating a Dockerfile for a base build to a python flask app. based on debian 8

FROM debian:8.0

RUN apt-get -y update \
	&& apt-get upgrade -y \
	&& apt-get install -y \
		python-setuptools \
		python-pip

RUN pip install flask \
				flask-restful \
				pymongo

build the docker base image. the -t gives the build a name, the --rm=true means “Remove intermediate containers after a successful build”, the --no-cache=false means “Do not use cache when building the image”. the dot means look for the Dockerfile in the current folder. using the flags together means that all of the steps in the docker build is done every time since i want to make the full docker build. Not just from a commit a few steps in to the build.

python flask rest api auth option storing in mongoDB 3.0 running on docker

Sat, 28 Mar 2015 00:00:00 +0000

I have been looking on a way to auth rest api endpoints that needs to have auth. I will be using Python with flask, flask-restful, yaml, pymongo and passlib. passlib will be used to salt the password that will be stored in mondoDB. A note here is that i will be using mongoDB 3.0 which means that you have to install the un released pymongo 3.0. here is the packets and links you need.

productivity with Pomodoro and micro checklist

Wed, 25 Mar 2015 00:00:00 +0000

I have recently started to use a productivity theory called Pomodoro. The principe with Pomodoro is that you work focused on something for 25 min, having a micro break for 5 min and then continue. After 4 sessions you have a longer break for 15 min. I have been using this in combination with a micro checklist. What i mean with micro checklist is to have a check list like this.

pitfalls when writing commandline tools

Sat, 21 Mar 2015 00:00:00 +0000

Common pitfalls when using template files/support files in the working dir of the script, is easy to miss when you are writing command line tools. In this case using python. Some of the more annoying mistakes is when you have template files stores and referred to in a script, you assume that the person that will use you script stands in the folder were the script is. lest assume that we are using a file template.html in a folder named foobar. of corse this will not work if you are running the script from somewhere else. I normally stand in the same folder as the script when i am developing a tool. When you then stand in some other folder everything will fail since you are not giving the absolute path to the template files.

dynamic test generation

Sun, 15 Mar 2015 00:00:00 +0000

I what to show a api template i did to generate api post tests with a random varible lenght hex value as the payload in a http post request. In this case i am using python, in compination with nose, to run my tests. Note you can run this without nose but you will get all prints from the support libs. Nose will supress the prints if the test passes. To change to varible number of tests that you can generate you just change the random_data input integer. The purpose of this is to be able to check that a REST api can take the random input length of the hex payload. in the testmap you gets a dict that contains a random, int, float, hex, password.

stay up to date with tech

Mon, 09 Mar 2015 00:00:00 +0000

Keeping up with tech. I like read and be aware of what is going on in tech, security, apple, linux, development, testing, and some devops related topics. I have attached a OPML file from my feedly where i get most of my news if your interested. I am frequently listening to a few podcasts, I’m using a awesome podcast iPhone` app overcast to litsen. The smart speed saves some time, voice boost helps to boost the voice those is the features i think it best for me.

cold brew coffee

Sun, 08 Mar 2015 00:00:00 +0000

Brew some easy and good cold brew coffee. This can be used for just plain ice coffee, ice latte, and many more. A note it that it will be high in caffeine. What you need is.

a one liter jar
coffee beans newly roasted preferred
water filtered preferred
a french press

My favourite coffee roastery is called dropcoffee. it’s a small roastery in Stockholm. They always have something interesting to offer and always has some new coffee roast to try. Most of the roasts is lighter roasts look them up here is you like to try. dropcoffee

control temptation @ work

Wed, 04 Mar 2015 00:00:00 +0000

So what do i mean with control temptation @ work. When working with development you have to get it to a problem and try to solve it. Doing that requires that you can stay focused on the task in hand to get the most out of it. Lets say you take 10 - 15 min to get it to the right flow. During your day you have email, facebook, online newspapers, chat systems like IRC, HipChat, Skype, colleagues and many others things that can make you get out of your productive flow. The result of this will mean for every time you lose the 10 - 15 min to get in to the productive flow, over and over. And you may end up at the end of that day and realizing that you did not get done with what you wanted. That is not fun for anyone.

about

Mon, 01 Jan 0001 00:00:00 +0000

This is a developer blog by a software engineer working cloud infrastructure and PaaS. It will be about things that is related to, software development, test automation, devops automation using docker, continuous delivery/integration using docker, and open source. In i write about things that i encounter at work, things i use, and find interesting.

Most of my current interessts is in docker, prometheus, monitoring, ansible, python, go, AWS, GCE. that is most likely what will be in the comming posts for some time in the future.