Python library fabric is a great way to automate command line tasks. If you use command line, know Python, and don't know bash-scripting (or prefer using Python to bash-scripting) then you are likely to find fabric extremely useful in automating repetitive tasks. In this "live" post I will record different ways in which I use fabric.

Using fabric to run a local command (example: rsync)

The following is a simple code snippet that shows how you can use fabric to fetch files from a remote folder (say, /home/fabricuser/projects/myproject) to a local folder (say, ~/localProjects/myproject) using rsync.

Of course, replace the pathnames as necessary. But for the code snippet to work:

(1) you must save the code snippet locally with the name fabfile.py. A fabric script must be named fabfile.py;

(2) you must save the code snippet in the folder that contains the local folder that you want to sync. In the example here we are syncing the folder myproject that is contained within ~/localProjects folder; so the code snippet must be saved as fabfile.py in ~/localProjects.

Also, replace fabricuser by your username and 123.456.78.90 by the ip address of your server.

from fabric.api import local

# absolute remote path
remotePath = '/home/fabricuser/projects/myproject' 

# relative path from ~/localProjects
localPath = './' 


def getFiles():                
    local( 'rsync -avz fabricuser@123.456.78.90:{0} {1}'\
                        .format(remotePath, localPath) )

Once you have saved the above code snippet in ~/localProjects as fabfile.py, you can get files from the remote server by navigating to ~/localProjects using the terminal, and running fab getFiles. This is so much easier that typing out the entire rsync -avz <source> <destination> command repeatedly.

The only fabric specific command in the above snippet is the local() command. local() says that "run the shell command in the parantheses on the local machine". In the above snippet we are running the standard rsync command on the local machine.

A fabric script can contain multiple functions, and you can then use fab <function name> to execute the tasks specified by that function. The code snippet below expands the above script with two additional functions:

from fabric.api import local


def getFiles():        
    # absolute remote path
    remotePath = '/home/fabricuser/projects/myproject' 

    # relative path from ~/localProjects
    localPath = './' 

    local( 'rsync -avz fabricuser@123.456.78.90:{0} {1}'\
                        .format(remotePath, localPath) )


def sendFiles():
    # absolute remote path
    remotePath = '/home/fabricuser/projects' 

    # relative path from ~/localProjects
    localPath = './myproject' 

    local( 'rsync -avz {0} fabricuser@123.456.78.90:{1}'\
                        .format(localPath, remotePath) )


def syncBothWays():
    getFiles()
    sendFiles()

Function sendFiles sends files from the local folder to the remote folder. To use this we need to run fab sendFiles at the command prompt. To get files and send files using a single command, you can run fab syncBothWays. In each case, you must be in the folder where fabfile.py is saved.

Suppose you are using the pdflatex command to generate a PDF from your LaTeX document, and you need to insert pages from an external PDF document in the PDF document that you are creating. You can do this quite easily using the pdfpages package by Andreas Matthias. I learnt about this package from this post on Stackoverflow. The documentation accompanying the package is very good and quite easy to understand. Nevertheless, I find it helpful to have some ready examples for common use cases. The following code snippets provide some such examples. They build on the answer in the linked Stackoverflow post.

First, add \usepackage{pdfpages} to the preamble of your LaTeX document. All the snippets below assume that you have done this.

The command syntax is:

\includepdf[key1=val1, key2=val2, key3=val3]{pathToFile}

where:

\includepdf is the name of the command
key=val is a comma separated list of options
pathToFile is path to the external PDF file that you want to include.

For this post we will assume that the external PDF file is in the same directory where the LaTeX file is, and is called external.pdf.

Most basic use (no options are specified)

%--------------
%% Snippet 1
%--------------
\includepdf{external.pdf} 

This will insert the first page of external.pdf without numbering the inserted page.

The pagecommand option

If you want to the inserted page to have a page number add the option pagecommand={}. With this option, the command will be:

%--------------
%% Snippet 2
%--------------
\includepdf[pagecommand={}]{external.pdf}

Now you will have the first page of the external.pdf file and it will numbered as if it were part of your LaTeX document. The default value of the pagecommand option is { \thispagestyle{empty} }, which is why page number was missing in the output of Snippet 1. By including pagecommand = {} you override this option. In general, according to the documentation, "[pagecommand] declares LaTeX commands, which are executed on each sheet of paper".

Including specific pages

Snippets 1 and 2 insert only the first page of external.pdf document. What if you wanted to insert more pages or other pages? For this you would use the pages option. I have left the pagecommand option in there so that you can see how different options can be combined. It is not required for including specific pages.

To insert all pages from external.pdf, use:

%--------------
%% Snippet 3
%--------------
\includepdf[pages=-, pagecommand={}]{external.pdf}   

To insert specific pages, say pages 2, 4, and 7, use:

%--------------
%% Snippet 4
%--------------
\includepdf[pages={2, 4, 7}, pagecommand={}]{external.pdf}

To insert a range of pages, say pages 9 through 13, use:

%--------------
%% Snippet 5
%--------------
\includepdf[pages={9-13}, pagecommand={}]{external.pdf}

If you want to include specific pages and a range of pages, you can combine snippets 4 and 5 as:

%--------------
%% Snippet 6
%--------------
\includepdf[pages={2, 4, 7, 9-13}, pagecommand={}]{external.pdf}

Scaling the pages

According to the documentation, pages are scaled automatically as they are inserted. This can be prevented by including noautoscale = true option. However, for a more fine-grained control on scaling you can use the scale option provided by the graphicx package. For example, you can use:

%--------------
%% Snippet 7
%--------------
\includepdf[pages=-, scale=0.9, pagecommand={}]{external.pdf}

Value of scale option should be between 0 and 1.

Rubinstein: How game theory will solve the problems of the Euro Bloc and stop Iranian nukes

The article assumes that you have root access to a remote Ubuntu 12.04 server and know its IP address. For the sake of this post let us assume that the IP address of the server is 111.222.333.444

Step 1: SSH into the server as root user.

Type the following and hit enter

ssh root@111.222.333.444

You will be shown a RSA key fingerprint and asked whether you want to continue connecting. Choose yes and hit enter. You will be prompted for your password. Enter the password and you will be logged into the server.

Step 2: Create a new user

Create a new user on this server. For security reasons it is best to access your server as a different user than the root user. Let us call the new user trialuser. You create this user using the command

adduser trialuser

and answering the questions that follow that command.

Step 3: Add the new user to the sudo group

Add the newly created user to the sudo group so that the user will have sudo privilege. This is done using the command:

usermod -a -G sudo trialuser

Note that since we are still logged in as the root user, we don't need to add sudo in front of the above commands.

Step 3: Logout from the root account and login as the trialuser

Logout of the root account and log back in as the trialuser using

ssh trialuser@111.222.333.444

Step 4: Install vim

We will need to make changes to various configuration files. For this I prefer to use the vim editor. If you don't want to use vim you can skip this step. Before installing vim run:

sudo apt-get update
sudo apt-get upgrade

After those two commands, run:

sudo apt-get install vim

Step 4: Disable root login and allow only trialuser to login

The next step is to secure your server by disabling the root login.

Open /etc/ssh/sshd_config using vim (sudo vim /etc/ssh/sshd_config). Find line PermitRootLogin and change from

PermitRootLogin yes

to

PermitRootLogin no

Also add the line

AllowUsers trialuser

at the bottom of sshd_config file. This will ensure that only you, trialuser, (remember to replace trialuser with your username) can login into the server.

The changes you made above will not take effect until you reload ssh. To do so, enter the command:

sudo reload ssh

Step 5: Install fail2ban

fail2ban is an awesome utility which (amongst other things) bans IP addresses with repeated failed login attempts. This helps against brute force attacks on your server. To install fail2ban run:

sudo apt-get install fail2ban

According to the fail2ban installation guide on Digital Ocean, to configure fail2ban we must copy /etc/fail2ban/jail.conf to /etc/fail2ban/jail.local and make any configuration changes to jail.local file.

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

You can make configuration changes to the /etc/fail2ban/jail.local file such as changing bantime (time for which an IP address is banned). You can find more information in the fail2ban article on Digital Ocean.

Once you have made the changes in the /etc/fail2ban/jail.local file restart fail2ban for those changes to take effect by running:

sudo /etc/init.d/fail2ban restart

If you have a static ip address, you can add it to the ignoreip line in file /etc/fail2ban/jail.local, so that fail2ban does not accidentally lock you out. If you have more than one IP address, separate them by space.

A couple of hours after your server is online take a look at /var/log/fail2ban.log to see pesky IP addresses that fail2ban has banned for attempting to break in into your server.

Step 6: Setup SSH keys

On your local computer, navigate to ~/.ssh folder. In this folder see if there exist two files id_rsa and id_rsa.pub. If so, you already have a ssh key and don't need to generate a new one.

If you don't see those two files, then you need to generate an ssh key. You can do so using the command:

ssh-keygen -t rsa

Enter the command and follow the instructions. If you accepted the default location for saving the key, you will find files id_rsa and id_rsa.pub in your ~/.ssh folder.

Once you have your ssh key, to use it with the server add the public key (id_rsa.pub) to the ~/.ssh/authorized_keys file on your server. As noted in the Digital Ocean article here you can use either of the following two commands to do this:

ssh-copy-id trialuser@111.222.333.444

or

cat ~/.ssh/id_rsa.pub | ssh user@123.45.56.78 "cat >> ~/.ssh/authorized_keys"

Your server is now setup with some basic security. You can certainly make it more secure. I will leave it to you to explore those options.

Import the libraries and specify the type of the output file.

The first step is to import the python libraries that we will use.

## numpy is used for creating fake data
import numpy as np 
import matplotlib as mpl 

## agg backend is used to create plot as a .png file
mpl.use('agg')

import matplotlib.pyplot as plt 

Convert the data to an appropriate format

The second step is to ensure that your data is in an appropriate format. We need to provide a collection of values for each box in the boxplot. A collection can be expressed as a python list, tuple, or as a numpy array. Once you have the different collections, one for each box, you combine all these collections together in a list, tuple or a numpy array. That is, the data for the boxplot is in the form of a list of lists, or list of arrays, or a tuple of arrays etc.

Let us create the data for the boxplots. We use numpy.random.normal() to create the fake data. It takes three arguments, mean and standard deviation of the normal distribution, and the number of values desired.

## Create data
np.random.seed(10)
collectn_1 = np.random.normal(100, 10, 200)
collectn_2 = np.random.normal(80, 30, 200)
collectn_3 = np.random.normal(90, 20, 200)
collectn_4 = np.random.normal(70, 25, 200)

## combine these different collections into a list    
data_to_plot = [collectn_1, collectn_2, collectn_3, collectn_4]

Create the boxplot

The list of arrays that we created above is the only required input for creating the boxplot. Using data_to_plot we can create the boxplot with the following code:

# Create a figure instance
fig = plt.figure(1, figsize=(9, 6))

# Create an axes instance
ax = fig.add_subplot(111)

# Create the boxplot
bp = ax.boxplot(data_to_plot)

# Save the figure
fig.savefig('fig1.png', bbox_inches='tight')

This gives:

That was easy. If you are satisfied with the default choices for the axes limits, labels and the look of the plot there is nothing more to do. Most likely that is not the case and you want to customize these features. Matplotlib provides endless customization possibilities.

Change color of the boxes, whiskers, caps and the median

The bp variable in the code snippet above is a python dictionary with keys boxes, whiskers, caps, fliers and the median. The values in this dictionary are the geometric instances that we see in the plot. For example, the values bp['boxes'] are the polygon instances that we see as boxes; values bp['whiskers'] are the line instances that we see as whiskers. To change the colors of boxes, whiskers etc., we need to access these instances from the bp dictionary and change their properties. The code below shows how to do this. In addition to changing the colors we also increase the line width to make the differences in color easier to see. Add the following code before the fig.savefig('fig1.png', bbox_inches='tight') line in the above snippet.

## add patch_artist=True option to ax.boxplot() 
## to get fill color
bp = ax.boxplot(data_to_plot, patch_artist=True)

## change outline color, fill color and linewidth of the boxes
for box in bp['boxes']:
    # change outline color
    box.set( color='#7570b3', linewidth=2)
    # change fill color
    box.set( facecolor = '#1b9e77' )

## change color and linewidth of the whiskers
for whisker in bp['whiskers']:
    whisker.set(color='#7570b3', linewidth=2)

## change color and linewidth of the caps
for cap in bp['caps']:
    cap.set(color='#7570b3', linewidth=2)

## change color and linewidth of the medians
for median in bp['medians']:
    median.set(color='#b2df8a', linewidth=2)

## change the style of fliers and their fill
for flier in bp['fliers']:
    flier.set(marker='o', color='#e7298a', alpha=0.5)

This gives:

Change x-axis labels and remove tick marks from the top and right axes.

This is easy. To do this pass a list of custom labels to ax.set_xticklabels() function. The list should have the same length as the number of boxes in the boxplot. For example, suppose instead of the default x-axis labels that we see in the plots above, we want labels 'Sample1', 'Sample2', 'Sample3' and 'Sample4'. We can modify the labels using the following line (add it before the line where the figure is saved):

## Custom x-axis labels
ax.set_xticklabels(['Sample1', 'Sample2', 'Sample3', 'Sample4'])

To remove the tick-marks from top and right spines, or rather to keep ticks only on the bottom and left axes, add the following lines to the code.

## Remove top axes and right axes ticks
ax.get_xaxis().tick_bottom()
ax.get_yaxis().tick_left()

With the custom x-axis labels and removal of top and right axes ticks, the boxplot now looks like the following:

If you are curious to learn more about creating boxplots with matplotlib, you may find the following links helpful.

1. Official matplotlib documentation on boxplots

2. Boxplot example on matplotlib website