Deploying Locally to Mac OS X with Capistrano

Lately I’ve wanted to deploy locally to my mac using Capistrano. Not for actual deployment locally, but to test some custom tasks I was working on. Its a lot quicker and safer to run custom tasks against a local sandbox then some real server somewhere. Writing real tests for custom tasks is basically too difficult to be worth it, so testing locally is probably the least crummy option right now. Another side benefit of testing locally is the speed and removing the need for a net connection.

So out of the box, this won’t work:

  role :app, "localhost"
  role :web, "localhost"
  role :db, "localhost"

  cap my_custom_task

For one thing, mac’s don’t have sshd (the ssh server) running by default. So first do that – go to System Preferences -> Sharing -> and enable “Remote Login”. That should start the ssh daemon.

Now if you deploy, Capistrano should be able to login but you may run into path issues, especially if you want to use gems in a custom task. If you just want to use a very basic task locally, you might be okay here — but if you see ‘command not found’ or missing libraries, you probably need to keep reading.

Capistrano runs SSH over a non-interactive shell, and by default you have an empty environment when logged into a OS X non-interactively via SSH. To fix that, open up /etc/sshd_config in your favorite text editor. Change the following:

PermitUserEnvironment yes

This will allow you to set the environment in your home .ssh folder. Before you leave that file, also find these directives and make the following changes for security:

PermitRootLogin no
Protocol 2
AllowUsers [username here]

This will turn off root login, allow only the more recent version of ssh, and explicitly allow only the one username you’ll be using. If you want to lock it down even more you should set the port to something much higher then 22, turn off password auth, and setup keys with good pass phrases. My mac is behind a router most of the time anyways, so I’m not going to worry about it. If you have a password of ‘password’ or something similiar for the account you are using, you should definitely make that stronger.

Now, create a file for you local ssh environment.

mate ~/.ssh/environment # or vi, or nano, etc...

Now add a PATH statement to that file like the following:

PATH=/usr/local/bin:/usr/local/sbin:/usr/local/mysql/bin:/usr/local/lib:opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
RUBYOPT=rubygems

You’ll probably want the /usr/local and /opt/local (for MacPorts) stuff ahead of /usr/bin so that if you use Ruby or Rubygems in your tasks, the non-broken version you installed yourself will get used. After saving that file, go back to the system preferences and turn off ssh (remote login) and turn it back on again to pick up all changes. Now you can run any custom task against your localhost, and it should work pretty much like a typical linux box, assuming you’ve installed everything locally that your have remote.

If you still have issues, create a simple task like so:

task(:debug_env) { run "env" }

Run that locally and then run it remotely to see the differences in your environment, and hopefully that will reveal something missing from your path or maybe some env variable you need.

This was all done with Capistrano 1.4.1, as I haven’t taken the plunge into the new version 2 yet.

• Introduction
• How to Install Capistrano
• Prepare your Project for Capistrano
• Working with Capistrano Recipes
• Tips & Tricks
• Further Reading

gem install capistrano

gem install capistrano-ext

capify .

set :application, "fancy_shoes"

set :scm, :git
set :repository, "git@account.beanstalkapp.com:/repository.git"
set :scm_passphrase, ""

set :scm, :subversion
set :repository, "https://account.svn.beanstalkapp.com/repository"

set :user, "server-user-name"

require 'capistrano/ext/multistage'

set :stages, ["staging", "production"]
set :default_stage, "staging"

server "my_fancy_server.com", :app, :web, :db, :primary => true
set :deploy_to, "/var/www/fancy_shoes"

server "my_fancy_server.com", :app, :web, :db, :primary => true
set :deploy_to, "/var/www/fancy_shoes_staging"

cap deploy:setup

cap deploy:check

cap deploy

cap production deploy

set :deploy_via, :remote_cache

namespace :deploy do
  task :restart, :roles => :web do
    run "touch #{ current_path }/tmp/restart.txt"
  end

  task :restart_daemons, :roles => :app do
    sudo "monit restart all -g daemons"
  end
end

after "deploy", "deploy:restart_daemons" 

set :branch, 'production'

set :branch, 'staging'

This isn’t -entirely- necessary, but it’s a good idea for security reasons. To add a user, go to:

System Preferences > Accounts

Click the “+” button near the bottom of the account list. You may need to unlock this ability by hitting the lock icon at the bottom corner and entering the admin username and password.

When the New account window comes out enter a name, as short name and a password. I entered the following:

Name: hadoop
Short name: Hadoop
Password: MyPassword (well you get the idea)

Once you are done, hit “create account”. Now, log in as the hadoop user. You are ready to set up everything!

Step 2: Install/Configure Preliminary Software

Before installing Hadoop, there are a couple things that you need make sure you have on your system.

1. Java, and the latest version of the JDK
2. SSH

Because OS X is awesome, you actually don’t have to install these things. However, you will have to enable and update what you have. Let’s start with Java:

Updating Java

Open up the Terminal application. If it’s not already on your dock, you can access it through

Applications > Utilities > Terminal

Next check to see the version of Java that’s currently available on the system:

$:~ java -version
java version "1.5.0_13"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_13-b05-237)
Java HotSpot(TM) Client VM (build 1.5.0_13-119, mixed mode, sharing)

You may want to update this to Java Sun 6, which is available as an update for OS X 10.5 (Update 1). It’s currently only available for 64-bit machines though. You can download it here.

After you download and install the update, you are going to need to configure Java on your system so the default points to this new update.Go to:

Applications > Utilities > Java > Java Preferences

Under “Java Version” hit the radio button next to “Java SE 6″ Down by “Java Application Runtime Settings” change the order so Java SE 6 (64 bit) is first, followed by Java SE 5 (64 bit) and so on. Hit “Save” and close this window.

Now, when you go to the terminal, and type in “java -version” you should get the following:

$:~ java -version

java version "1.6.0_43"

Java(TM) SE Runtime Environment (build 1.6.0_43-b01-447-10M4203)
Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01-447, mixed mode)

Onto ssh!

SSH: Setting up Remote Desktop and Enabling Self-Login

SSH also comes installed on your Mac. However, you need to enable access to your own machine (so hadoop doesn’t ask you for a password at inconvenient times). To do this, go to

System Preferences > Sharing (under Internet & Network)

Under the list of services, check “Remote Login”. For extra security, you can hit the radio button for “Only these Users” and select hadoop

Now, we’re going to configure things so we can log into localhost without being asked for a password. Type the following into the terminal:

$:~ mkdir .ssh
$:~ ssh-keygen -t rsa -P ""
$:~ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

Now try:

$:~ ssh localhost

You should be able to log in without a problem.

You are now ready to install Hadoop. Let’s go to step 3!

Step 3: Downloading and Installing Hadoop

So this actually involves a few smaller steps:

1. Downloading and Unpacking Hadoop
2. Configuring Hadoop

After we finish these, you should be ready to go! So let’s get started:

Downloading and Unpacking Hadoop

Download Hadoop. Make sure you download the latest version (as of this post, 1.0.4 are the latest versions). We call our generic version of hadoop hadoop-* in this tutorial.

Unpack the hadoop-*.tar.gz in the directory of your choice. I placed mine in /Users/hadoop. You may also want to set ownership permissions for the directory:

$:~ tar -xzvf hadoop-*.tar.gz 
$:~ chown -R hadoop hadoop-*

Configuring Hadoop

There are two files that we want to modify when we configure Hadoop. The first is conf/hadoop-env.sh . Open this in nano or your favorite text editor and do the following:

- uncomment the export JAVA_HOME line and set it to /Library/Java/Home

- uncomment the export HADOOP_HEAPSIZE line and keep it at 2000

You may want to change other settings as well, but I chose to leave the rest of hadoop-env.sh the same. Here is an idea of what part of mine looks like:

# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use.  Required.
 export JAVA_HOME=/Library/Java/Home

# Extra Java CLASSPATH elements.  Optional.
# export HADOOP_CLASSPATH=

# The maximum amount of heap to use, in MB. Default is 1000.
 export HADOOP_HEAPSIZE=2000

The next part that we need to set up is hadoop-site.xml. The most important parts to set here are hadoop.tmp.dir (which should be set to the directory of your choice) and to add mapred.tasktracker.maximum property to the file. This will effectively set the maximum number of tasks that can simulataneously run by a task tracker. You should also set dfs.replication ‘s value to one.

Below is a sample core-site.xml file:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>
  <name>hadoop.tmp.dir</name>
  <value>/Users/hadoop/hadoop-1.0.4/hadoop-${user.name}</value>
  <description>A base for other temporary directories.</description>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:9000</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>

<property>
  <name>mapred.job.tracker</name>
  <value>localhost:9001</value>
  <description>The host and port that the MapReduce job tracker runs
  at. If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>

<property>
<name>mapred.tasktracker.tasks.maximum</name>
<value>8</value>
<description>The maximum number of tasks that will be run simultaneously by a
a task tracker
</description>
</property>

<property>
  <name>dfs.replication</name>
  <value>1</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
</property>

</configuration>

Now to our last step!

Step 4: Formatting and Running Hadoop

Our last step involves formatting the namenode and testing our system.

$:~ cd ~
$:~ hadoop-*/bin/hadoop namenode -format

This will give you output along the lines of

$:~ hadoop-*/bin/hadoop namenode -format
 08/09/14 21:22:14 INFO dfs.NameNode: STARTUP_MSG:
 /***********************************************************
 STARTUP_MSG: Starting NameNode
 STARTUP_MSG:   host = loteria/127.0.0.1
 STARTUP_MSG:   args = [-format]
 ***********************************************************/
 08/09/14 21:22:14 INFO dfs.Storage: Storage directory [...] has been successfully formatted.
 09/09/14 21:22:14 INFO dfs.NameNode: SHUTDOWN_MSG:
 /***********************************************************
 SHUTDOWN_MSG: Shutting down NameNode at loteria/127.0.0.1
 ***********************************************************/

Once this is done, we are ready to test our program.

Running Hadoop

First, start up the DFS. This will start up a TaskTracker, JobTracker, and DataNode on the machine.

$:~ hadoop-*/bin/start-all.sh

As input for our test, we are going to copy the conf folder up to our DFS.

$:~ hadoop-*/bin/hadoop dfs -copyFromLocal hadoop-*/conf input

You can check to see if this actually worked by doing an ls on the dfs as follows:

$:~ hadoop-*/bin/hadoop dfs -ls
Found 1 item

Now, we need to compile the code. cd into the hadoop-*/ directory and do:

$:~ cd hadoop-1.0.4
$:~ ant examples

This will compile the example programs found in hadoop-*/src/examples

Now, we will run the example distributed grep program on the conf program as input.

$:~ hadoop-*/bin/hadoop jar /Users/hadoop/hadoop-1.0.4/build/hadoop-examples-1.0.4-SNAPSHOT.jar grep input output 'dfs[a-z.]+'

If this works, you’ll see something like this pop up on your screen:

13/04/25 15:53:59 INFO mapred.JobClient: Map input bytes=94
13/04/25 15:53:59 INFO mapred.JobClient: SPLIT_RAW_BYTES=118
13/04/25 15:53:59 INFO mapred.JobClient: Combine input records=0
13/04/25 15:53:59 INFO mapred.JobClient: Reduce input records=3
13/04/25 15:53:59 INFO mapred.JobClient: Reduce input groups=1
13/04/25 15:53:59 INFO mapred.JobClient: Combine output records=0
13/04/25 15:53:59 INFO mapred.JobClient: Reduce output records=3
13/04/25 15:53:59 INFO mapred.JobClient: Map output records=3

The last step is to check if you have output!

You can do this by doing:

$:~ hadoop-*/bin/hadoop dfs -ls output
Found 2 items

Found 2 items-rw-r–r– 3 hadoop supergroup 0 2013-04-25 15:53 /user/hadoop/output/_SUCCESS
-rw-r–r– 3 hadoop supergroup 52 2013-04-25 15:53 /user/hadoop/output/part-00000

The most important part is that the number next to the <r 1> should not be 0.

To check the actual contents of the output do:

$:~ hadoop-*/bin/hadoop dfs -cat output/*

Alternatively, you can copy it to local disk and check/modify it:

$:~ hadoop-*/bin/hadoop dfs -copyToLocal output myoutput
$:~ cat myoutput/*

Stopping the Hadoop DFS

When you’re done running jobs on the dfs, run the stop-all.sh command.

$:~ hadoop-*/bin/stop-all.sh

That’s all! Happy Map Reducing!

public class SetBits {

	/**
	 * @param args
	 */
	public static void main(String[] args) {

		int N = 1024;
		int M = 21;
		System.out.println(Integer.toBinaryString(M));
		System.out.println(Integer.toBinaryString(N));
		int j = updateBits(N, M, 2, 6);
		System.out.println(Integer.toBinaryString(j));
		System.out.println(j);
	}

	public static int updateBits(int n, int m, int i, int j) {
		int max = -1; /* All 1's */
		System.out.println("Max:       " + Integer.toBinaryString(max));

		// 1's through position j, then 0's
		int left = max - ((1 << j) - 1);
		System.out.println("Left:      " + Integer.toBinaryString(left));

		// 1's after position i
		int right = ((1 << i) - 1);
		System.out.println("Right:     " + Integer.toBinaryString(right));

		// 1's with 0s between i and j
		int mask = left | right;
		System.out.println("Mask:      " + Integer.toBinaryString(mask));
		System.out.println("n & mask:  " + Integer.toBinaryString(n & mask));
		System.out.println("m << i:    " + Integer.toBinaryString(m << i));

		// Clear i through j, then put m in there
		return (n & mask) | (m << i);
	}
}