Friday, 1 November 2013

Dirty, dirty devops

No this isn't an opinionated rant about devops culture or anything. Its just a hacky knife plugin that lets me play with Sensu quickly.

I want to play with as an alternative to Nagios and Zabbix etc. I've already got a nice git repo full of our cookbooks etc, but I don't want to go near them. I don't want to even risk messing up master etc.

Note: This assumes you have a chef workstation set up with knife available etc.

Anyway, I wrote this tiny little knife plugin that just calls a couple of bash commands:

require 'chef/knife'
# other require attributes, as needed

module Sandbox
  class SandboxInit < Chef::Knife

    deps do

    banner "knife sandbox init (options)"

    def run
      puts "Creating chef repo as chef-repo"
      system("git clone git://")
      puts "Creating .chef dir"
      system("mkdir chef-repo/.chef")
      puts "Creating knife config file"
      system("echo 'current_dir = File.dirname(__FILE__)\ncookbook_path [\"\#{current_dir}/../cookbooks\"]' > chef-repo/.chef/knife.rb")
      puts "Done. You can know cd into chef-repo, run knife cookbook site install BLAH and use vagrant etc"

See what I mean with hacky? Eww. Anyway, stick this file in ~/.chef/plugins/knife/ then you can use it as follows:

$ mkdir playing-around
$ cd playing-around
$ knife sandbox init
WARNING: No knife configuration file found
Creating chef repo as chef-repo
Cloning into 'chef-repo'...
remote: Counting objects: 223, done.
remote: Compressing objects: 100% (139/139), done.
remote: Total 223 (delta 80), reused 178 (delta 49)
Receiving objects: 100% (223/223), 38.14 KiB | 0 bytes/s, done.
Resolving deltas: 100% (80/80), done.
Checking connectivity... done
Creating .chef dir
Creating knife config file
Done. You can know cd into chef-repo, run knife cookbook site install BLAH and use vagrant etc
$ cd chef-repo

Now you can follow the instructions here

$ knife cookbook site install sensu
$ cd cookbooks/sensu/examples
$ bundle install
$ librarian-chef install
$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
[default] Importing base box 'opscode-ubuntu-13.04'...
[default] Matching MAC address for NAT networking...
[default] Clearing any previously set forwarded ports...
[default] Creating shared folders metadata...
[default] Clearing any previously set network interfaces...
[default] Preparing network interfaces based on configuration...
[default] Forwarding ports...
[default] -- 22 => 2222 (adapter 1)
[default] -- 8080 => 8080 (adapter 1)
[default] Running 'pre-boot' VM customizations...
[default] Booting VM...
[default] Waiting for machine to boot. This may take a few minutes...
[default] Machine booted and ready!
[2013-11-01T17:16:35+00:00] INFO: Chef Run complete in 227.664146883 seconds
[2013-11-01T17:16:35+00:00] INFO: Running report handlers
[2013-11-01T17:16:35+00:00] INFO: Report handlers complete
[2013-11-01T17:12:47+00:00] INFO: Forking chef instance to converge...

We now have a machine that should have Sensu on it:

$ vagrant ssh
Welcome to Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64)

 * Documentation:

94 packages can be updated.
55 updates are security updates.

New release '13.10' available.
Run 'do-release-upgrade' to upgrade to it.

Last login: Sun May 12 22:41:29 2013 from
vagrant@vagrant:~$ ps -ef | grep sens
sensu     6097     1  0 17:16 ?        00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-server -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/ -l /var/log/sensu/sensu-server.log -L info
sensu     6154     1  0 17:16 ?        00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-api -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/ -l /var/log/sensu/sensu-api.log -L info
sensu     6211     1  0 17:16 ?        00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-dashboard -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/ -l /var/log/sensu/sensu-dashboard.log -L info
sensu     6276     1  0 17:16 ?        00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-client -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/ -l /var/log/sensu/sensu-client.log -L info
vagrant   6740  6651  0 17:21 pts/0    00:00:00 grep --color=auto sens

Woot. You can even browse the dashboard on http://localhost:8080 (username is admin and the password is secret).

When you're done, you can quickly clean up after yourself:

$ vagrant destroy
$ cd
$ rm -rf playing-around

Friday, 25 October 2013

How to ask about connectivity issues

So you're trying to connect one system to another using this newfangled TCP/IP stuff the kids are using these days. But what's this? It doesn't work? You need to ask someone to look into it. You even know who. But what should you ask and what information should you provide so that they can investigate quickly and effectively?

As well as a bit of background to the problem (applications involved, environments etc), you should provide at least the following:

  • Has it ever worked?
  • Have you changed anything recently?
  • Source IP
  • Destination IP
  • Protocol (TCP or UDP etc) if it isn't obvious from the context
  • Port if it isn't obvious from the context

Common causes are:
  • Wrong information - are you connecting to the right host?
  • Routing
  • Firewalls
  • Network or system outages

Have you tried the following?

  • Doing a host, dig or nslookup if dealing with a domain
  • Pinging the remote host
  • Connecting using netcat or telnet
  • Traceroute
It is probably worth providing the output to the above commands if possible.

Friday, 16 August 2013

How do I report a vulnerability?

Hanging out in the ##security IRC channel on Freenode you see a number of reoccurring topics. One of them is when someone says they've found a vulnerability, often in some website, and want suggestions on what to do next. So, I present to you:

Edit: The site has had some major updates following feedback from Jericho. Feel free to leave feedback in the comments below.

Thursday, 16 May 2013

A little python vs ruby

Just a little comparison of python vs ruby which nicely sums up why I prefer ruby to python.

Coming from a perl background, parsing strings and manipulating data structures is the bread and butter of sysadmin scripting. Here we're going to split a string, add to the array, then join into a string again.

First, in python:

a = "a b c".split()
>>> a
['a', 'b', 'c']
>>> len(a)
>>> a.append("d")
>>> ",".join(a)

Ok, fair enough I guess. Now lets look at ruby:

a = "a b c".split
=> ["a", "b", "c"] 
=> 3 
a.push "d" 
=> ["a", "b", "c", "d"] 
a.join "," 
=> "a,b,c,d" 

Notice the difference? I'll try to summarise it in text.

For python, we split the string to get an array. We then run a global method to find the size of the array. We then call a method on array to append the new element. Finally we run a join method on a string and pass it the array.

For ruby, we split the string to get an array. We then run a method on the array to get the size (.length would also have worked). We then run a method on the array to add the element. We then run a method on the array to join to get our final string.

To me ruby is far more consistent. I'm not saying python is a bad language, I just find ruby to be particularly enjoyable to use.

Criticisms, suggestions and improvements welcome :)

Tuesday, 16 April 2013

Passphrase Generators Part II

A while back I wrote a post about using bash as a passphrase generator (here). The idea being that it would be better to pick "randomly" from a list of common words than it would be to try to mentally choose four words at random. Humans are not very good a being random [1]. Derren Brown, if you're reading this, I'd love to see an episode where you plant a password into someone's head for something they believe is of utmost importance. This post addresses some security concerns about how the words are chosen at random.

Speaking of passphrases and passwords, I should probably clarify what I mean by the two terms. A password I take to mean a private string which more or less resembles a "word" in the traditional sense. Typical and poor examples would be "password", "Password1", "aksdfjslfjdf" etc. Something like "8yDcBPAllEFOLNNI89ZN4nq5" being a good example. A passphrase I take to mean a private string composed of multiple "words". For example "That's me in the corner, that's me in the spot light", "slgl sdf mcr asrj" are poor examples, and "Cupertino teenage fell Paradox" is a good example. The analogy being to word and phrase in spoken languages. So generally, a good passphrase will be better than a good password because of length, assuming you havn't done something silly like using the lyrics of your favorite song. The passphrase would still need to be random, but easier to remember.

Anyway, having spent some time chatting on IRC about the flaw of using % 10000, using Bash's $RANDOM variable and a few other things, I thought I would update the script to be more secure. The entropy lost from using % 10000 isn't particularly significant, but the way $RANDOM works means you could probably reduce the choice of possible words significantly if you knew when approximately the passphrase was being generated. Basically, $RANDOM seeds off gettimeofday's seconds, useconds and the bash PID. I haven't done the maths for exactly what the impact is, but I thought it would be worth this blog post.

I am actually going to provide two new scripts. The previous method used a dictionary file with 10000 common words downloaded from the internet, and embedded them in the Bash script. Now, I still think this is a good approach because 10000^4 still gives enough possible combinations and because there are only 10000 words, many words will be familiar to most people and should therefore be easier to remember. I will first present an alternative approach.


while true; do
    for i in $(seq $words); do
        word=$(/usr/bin/shuf -n1 --random-source=/dev/urandom /usr/share/dict/words)
        echo -n "$word "
    echo ""
    read -p "Accept (y/n) [n]: " answer

    if [ "$answer" == "y" ]; then
        exit 0


You will probably notice two things. Firstly, it only takes one argument; the number of words to show, or 4 by default. The previous version of the script printed X number of words on Y lines so that you could choose a passphrase that suited you. However, there was always the temptation to create a new passphrase from a combination of what was shown on the screen. This is not ideal. You are better off rejecting a number of whole passphrases and then accepting one as-is, rather than combining several passphrases into one. The reason being that the process of combining passphrases is no longer random and there will probably be a bias towards a structure such as "adjective noun verb adjective". Or something like that anyway.

So, instead, we just keep looping until you accept the passphrase and clear the screen if you reject it (the default). By clearing the screen I hope to lessen the chance of you combining several passphrase - out of sight, out of mind as they say.

The second major change here is that instead of using 10000 common words downloaded from the Internet, we're using the OS's provided dictionary file. If your distribution doesn't have that file, see here

We use the shuf command X number of times to select X words for the passphrase. On one of my machines there are 98326 words in /usr/share/dict/words. I think I saw many more on my Ubuntu system. So you can probably expect some pretty obscure passphrases. You'll also note that we tell shuf to use /dev/urandom as the source of randomness as the built in source isn't secure.

Right, now onto the other option. In this case we're going to just modify the previous script and fix the source of randomness. We'll also introduce the while true loop and hide previous rejected passphrases.



chars="a-zA-Z0-9 "
size=$(cat $dict | wc -l)
echo "#!/bin/bash" > $script
( echo -n "dict=( "; i=0; cat $dict | while read word; do echo -n "$word " | tr -c -d "$chars"; if [ "$i" -eq "10" ]; then i=1; echo ""; fi; ((i++)); done; echo ")" ) >> $script

echo "size=$size" >> $script
cat << 'EOF' >> $script

while true; do
    for i in $(seq $words); do
        index=$(cat /dev/urandom | tr -d -c '0-9' | fold -w 4 | head -1 | sed 's/^0*//')
        echo -n "$word "
    echo ""
    read -p "Accept (y/n) [n]: " answer
    if [ "$answer" == "y" ]; then
        exit 0
chmod +x $script

Other than the loop change, the only difference is our source of randomness. Instead of using $RANDOM we read /dev/urandom directly. We strip out anything other than digits using the tr command and fold it to 4 characters (0000 to 9999). We then just take the first result and strip off any zeros from the front. We do this because bash doesn't treat 0080 as 80 and isn't happy using 0080 as an index. There is probably a better way of stripping of the zeros using bash's string manipulation, but this will do for now.

Running this script is the same as the previous version, you give it the dictionary and an output file name. E.g.

$ wget
$ ./ top10000en.txt

Unfortunately the dictionary isn't provided over HTTPS, so it is possible that someone is going to MITM your download and change all the words to suit themselves.

Perhaps something like this might help:

$ wc -l top10000en.txt
10000 top10000en.txt

 $ sort < top10000en.txt | uniq | wc -l

That way at least you know there are 10000 unique words. Either that or find a new dictionary available over HTTPS (and let me know where you found it if possible).

Hopefully you'll find either of these approaches useful.


Thursday, 14 March 2013

Ultra Light Graphs (ULG)

Ultra Light Graphs, or ULG for short, is a tool that allows you to create basic graphs using nothing more than a few lines of text. The text format is designed to be meaningful in itself, which makes ULG a sort of markup language for graphs. It isn't intended to replace something more comprehensive like graphml or dot, but it does allow you to quickly write something that is readable in your favourite text editor and can be turned into an image that you might want to insert into documentation etc.


Installation can do be done using ruby gems:

$ gem install ulg

Using ULG

ULG is very easy to use. Just give it a file with the text in it and it will generate a png by default.

$ ulg -h
Usage: ulg.rb [-o OUTPUT] [-s FILE] [-d] FILE FILE...

Reads from stdin if no input FILE given

Other options:

  -o OUTPUT  Output format (png, dot, svg). Default png
  -s FILE    Save to file
  -d         Debug mode
  -h         This help

For example

$ ulg basic.ulg
Reading basic.ulg
Saving output to basic.ulg.png

Basic example

Right, lets say you want to draw a basic graph. You open up your favourite text editor such as vim. You want two nodes and an arrow, so you might intuitively type something like this:

internet ==> web dmz

Now, this is what ULG would make of just that one line:

Nice and simple.

Node shapes

Maybe you don't want two boxes. Perhaps you want one of them to be oval. That's easy too:

(internet) ==> [web dmz]

That would give you:

Notice also how we've also explicitly marked "web dmz" as a box.

Edge label

You've probably noticed that the arrow, or edge in graph-speak, doesn't have a label. We can give it one by just inserting some text somewhere inside the arrow:

(internet) ==http==> [web dmz]

This would give us:


You can also control the shape of the arrow head, for example:

(internet) ==http==<> [web dmz]

Gives us:

Edge line

Of course, you might like your lines to be dashed instead of solid. No problem:

(internet) --http--> [web dmz]

Here we go:

Interesting example

Now, lets make things a little bit more interesting. I'll show you the text and image first, then I'll explain a few things afterwards:

option layout dot
option overlap false
option splines true

include backoffice.ulg

Internet        ==http==>       (web dmz|orange)

web dmz         ==8080==>       [app stack 1|blue]
web dmz         ==8000==>       [app stack 2|purple]

app stack 1     ==25==>         internet
app stack 2     ==5432|red==>   [database|green]

Where backoffice.ulg is

(back office) ==5432|red==> database
(back office) ==80==> (internet|red)

This would result in this rather pretty graph:

Quite a few things have happened here. Looking at the text, you will have noticed three lines starting with the word "option". These are basically just option names and values that get passed directly to Graphviz as global options.

We also include a ULG file called "backoffice.ulg", the contents of which is shown afterwards. You'll see that backoffice defines an "internet" node and the main text refers to an "Internet" and an "internet" node and that the graph only shows one "internet" node. This is because ULG ignores case and spaces when determining what the node is. So "Internet", "internet" and "inter net" would all be the same thing. This allows you to be a little be messy and inconsistent with your typing ;)

The "internet" node was labelled "internet" and not "Internet". This is because ULG defines styles based on the first occurrence of the node or edge. Because we included backoffice.ulg first, it defined the style for the internet node. This also means that once we have defined a node, such as the "(web dmz|orange)" we ignore any style characters and simply use the label. The next two occurrences were referred to as just "web dmz".

Another thing you will have noticed is the use of colour. This is achieved by simply adding "|colour" to the node or edge label. Any colours Graphviz understands should be fine.

Its probably also worth mentioning at this point that ULG doesn't care about whitespace between nodes and arrows, so you can use spaces or tabs - whatever is easiest to read. It does however require that an arrow is made up of at least two line characters (such as == or --)

Other things you can do

Finally, this example shows the different combinations of supported node and arrow styles.

# This line is ignored

# Include a file
include common-opts.ulg

# An option
option rankdir TB

# Graph
[box]   = normal ==>    (oval|blue)
oval    =dot|red=o      <diamond>
oval      box
box     --diamond--<>   diamond

another box     === no arrow ==     {hexagon}

This gives us:

Hopefully this example speaks for itself.

Monday, 11 March 2013

OpSec Basics

This is a simple little list of OpSec ideas that should be followed in most situations. Nothing ground breaking, just a gentle reminder :)

Take regular backups

Regularly backup systems and data using a combination of incremental and full backups.

Encrypt transmitted data whenever possible

Even on local networks. You may have perimeter firewalls and a switched network, but you have to assume someone is listening.

Minimize software to minimize vulnerability

Reduce the attack surface for remote access and local privilege escalation by only having essential software installed and services running.

Run different network services on separate systems

Not only does this simplify diagnosing operational issues, but means a compromise of one service doesn't directly affect other services. It also reduces the possibility of another service being used in the next stage of an attack.

Least privilege

This should apply at all levels, from system accounts to application logins. Don't use shared credentials and use systems that have granular enough permissions. Someone might not want to do something malicious, but if their system has been compromised the attacker might.

Patch your sh!#

Unpatched software is a liability. Every unpatched system is low-hanging fruit for an attacker.

Be informed

You should have a clear insight into the standard operational and opsec behaviour of your environment. This means having good operational monitoring and metrics, security systems such as HIDS and NIDS and a hardened centralized log collector for system logs and SIEM. Once you have that, take the time to review logs, events and set up alerts if possible.

Segment your network

Compromise is inevitable  so the trick is to limit damage once it happens. Network segmentation stops the attack from moving around freely and thus adds complexity and delay to the next stages of their attack.

Use good crypto and don't roll your own

This applies to everything from choosing a password, to storing passwords and encrypting backups. Don't rely on home-made encryption or obfuscation tricks.

Have a response

It is a question of when you get hacked, not if. You should have an incident response plan well in advance of anything happening. Basics should include how to limit damage, how to collect information for forensics or the authorities and details of who to contact.

Default configuration

When you buy a new piece of kit, build a new system or install an application, or should remember to check and adjust any default configuration. Most things try to be easy, not secure, so leaving in default configuration can be dangerous. Default users are a classic example, but other more subtle configuration might give too much information away or might be permissive in terms of access or actions.