31 Dec 2015
For the last couple of weeks I have been working on a puppet manifest to enable more resilience and build of vertica nodes on AWS.
Using automation tools has allowed the fast prototyping and development of many software solutions. Using puppet and Jenkins, a full environment is provided for the Data Analysts to use within about 1 hour.
Part of what I've been working on is to build vertica nodes whilst retaining the data. In addition to this, building in the logic to enable a node to fail, recover and join the cluster with next to no manual work.
Monitoring and detection is useful in this phase, so the database admins can double check the health of the data during and after the failure. Part of the solution includes automatically configuring the Vertica Management Console ready for use.
More progress is expected in the next month as I build auto scaling on top of this resiliency.
(Read more...)
30 Dec 2015
Within a project team you require a number of skills usually found across a number of team members.
Developer - This is the programming and design skill required for the end product this project is sponsored to deliver. For a web application, this might be Javascript, PHP, Paintshop etc. For a data analysis BI team, these skills will be more like R, tableaux or Qlikview.
Testing - These skills are about ensuring the best experience for the end user or customer of the deliverables. Performance (the speed of the product), usability (if it actually works as designed) and load testing (how many users or how much data can be provided) are important measurements in this area.
Operations - Providing the IT bits that the project and application rely on to function and serve. These are concerned with the servers, networks and other dependencies.
In terms of what DevOps engineers do, well that spans the developer and operations skills. The reason it is so specialist, is the blend of these skills. Someone who understand programming and infrastructure.
My team and I spend the majority of our time, writing and refining puppet modules. These are definitions of what a server needs to do, in order to provide a server, service or piece of infrastructure. They are programs in their own right.
We also spend time on writing tools to help the developers and testers move through the deployment processes. We don't want to slow the development or project down, so providing tools to enable the developers and testers to 'self serve' allows that flexibility.
Automation of the project steps and providing tools, buttons, one-click actions, to move the project and help the developers write, test and deploy code as easily as possible.
(Read more...)
21 Dec 2015
What is DevOps?
The best description I have seen is from Kurt Bittner;
"DevOps optimizes the software delivery pipeline, all the steps that you have to go through between when you have an idea and when a customer starts benefiting from that idea. In the traditional delivery processes, you have lots of hand-offs, lots of stops and starts. You have relatively inefficient processes, and it can take months -- and sometimes years -- to go from idea to having somebody get a benefit."
Optimising, automating and streamlining that process is what we are all about. Embracing Agile with open communications to help your team deliver value to your customers, faster and with higher quality. From Idea to usable and valuable product.
Infrastructure -> Development -> Testing -> Delivery
How do I get started?
Unfortunately it's not any one product or training course you can buy. It's an idea and ethos, which requires executive management across silos. You can however start with a few steps.
Step one; Environments.
Changing the way that environments are provisioned. That includes getting environments provisioned on-demand, using techniques like infrastructure-as-code using technologies such as Puppet to automatically generate environments based on configuration settings so that you can have an environment anytime you need it. This needs a virtualised infrastructure such as VMWare or Amazon Web Services. That removes a lot of friction and a lot of delays.
Step two; CI and Testing
Implementation of techniques like continuous integration and then, after that, test automation, based on APIs. There's a shift to APIs on an integrated architecture for the applications, and then usually deployment automation comes after that. Once you have environments provisioned in code that you can put into those environments, you need a way to move that code between environments.
At this point you may find Silo inside the organisation can slow or even stop some tasks from taking place. Resource contention and task priorities can get in the way of delivering the release to your customers. Spotify offer an approach to this called Tribes, Squads, Chapters and Guilds.
There's a shift in team structure to become more product-oriented with dedicated resources to a product, so that you can release, and do release after release most effectively. That tends to break the organization silos down and start shifting to a more product-centric organization and away from a functionally oriented organization.
Step three; Continuous Improvement
We are very much evolving within the DevOps space, and your organisation will too. There are many products to help and many tools your teams could create, adopt or improve upon.
For more information how we could help you become more agile and proactive, contact us now.
(Read more...)
