Table Of Contents

Previous topic

Writing Full-Featured Applications

Next topic

Pydoop Script User Guide

Get Pydoop


Pydoop is developed by: CRS4


Supported Platforms


Pydoop has been tested on Gentoo, Ubuntu and CentOS. Although we currently have no information regarding other Linux distributions, we expect Pydoop to work (possibly with some tweaking) on them as well.

Apple OS X

Pydoop has been tested on OS X 10.9 (Maverick) and OS X 10.10 (Yosemite). Install the Homebrew version of Python, then follow the instructions below.


We have included a patch by trtrmitya that adds FreeBSD support, but we have not tested it.

Get Pydoop

Source Distribution

We recommend installing Pydoop via pip:

pip install pydoop

To get the source code, clone our Git repository:

git clone

Where the master branch corresponds to the latest release, while the develop branch contains code under active development.


In order to build and install Pydoop, you need the following software:

  • Python version 2.7
  • setuptools version 3.3 or higher
  • either of the following:
    • Apache Hadoop version 1.0.4, 1.1.2, 1.2.1, 2.2.0, 2.4.1, 2.5.2 or 2.6.0
    • CDH version 4 or 5 installed from dist-specific packages or Cloudera Manager parcels (no tarball)
    • HDP 2.2
  • OpenSSL


  • JPype to build the alternate HDFS backend
  • Avro Python implementation to enable Avro I/O

These are also runtime requirements for all cluster nodes. Note that installing Pydoop and your MapReduce application to all cluster nodes (or to an NFS share) is not required: see Installation-free Usage for additional info. Moreover, being based on Pipes, Pydoop cannot be used with Hadoop standalone installations.

Other versions of Hadoop may or may not work depending on how different they are from the ones listed above.


Before compiling and installing Pydoop, install all missing dependencies.

In addition, if your distribution does not include them by default, install basic development tools (such as a C/C++ compiler) and Python header files. On Ubuntu, for instance, you can do that as follows:

sudo apt-get install build-essential python-dev

Set the JAVA_HOME environment variable to your JDK installation directory, e.g.:

export JAVA_HOME=/usr/local/java/jdk


If you don’t know where your Java home is, try finding the actual path of the java executable and stripping the trailing /jre/bin/java:

$ readlink -f $(which java)
$ export JAVA_HOME=/usr/lib/jvm/java-6-oracle

If you have installed Hadoop from a tarball, set the HADOOP_HOME environment variable so that it points to where the tarball was extracted, e.g.:

export HADOOP_HOME=/opt/hadoop-1.0.4

The above step is not necessary if you installed CDH from dist-specific packages. Build Pydoop with:

python build

This builds Pydoop with the “native” HDFS backend. To build the (experimental) JPype backend instead, run:

python build --hdfs-core-impl=jpype-bridged

For a system-wide installation, run the following:

sudo python install --skip-build

For a user-local installation:

python install --skip-build --user

The latter installs Pydoop in ~/.local/lib/python2.X/site-packages. This may be a particularly handy solution if your home directory is accessible on the entire cluster.

To install to an arbitrary path:

python install --skip-build --home <PATH>


  1. “java home not found” error, with JAVA_HOME properly exported: try setting JAVA_HOME in

  2. “ not found” error: try the following:

    export LD_LIBRARY_PATH="${JAVA_HOME}/jre/lib/amd64/server:${LD_LIBRARY_PATH}"
  3. non-standard include/lib directories: the setup script looks for includes and libraries in standard places – read for details. If some of the requirements are stored in different locations, you need to add them to the search path. Example:

    python build_ext -L/my/lib/path -I/my/include/path -R/my/lib/path
    python build
    python install --skip-build

    Alternatively, you can write a small setup.cfg file for distutils:


    and then run python install.

    Finally, you can achieve the same result by manipulating the environment. This is particularly useful in the case of automatic download and install with pip:

    export CPATH="/my/include/path:${CPATH}"
    export LD_LIBRARY_PATH="/my/lib/path:${LD_LIBRARY_PATH}"
    pip install pydoop
  4. Hadoop version issues. The Hadoop version selected at compile time is automatically detected based on the output of running hadoop version. If this fails for any reason, you can provide the correct version string through the HADOOP_VERSION environment variable, e.g.:

    export HADOOP_VERSION="1.0.4"

Testing your Installation

After Pydoop has been successfully installed, you might want to run unit tests to verify that everything works fine.

IMPORTANT NOTICE: in order to run HDFS tests you must:

  1. make sure that Pydoop is able to detect your Hadoop home and configuration directories. If auto-detection fails, try setting the HADOOP_HOME and HADOOP_CONF_DIR environment variables to the appropriate locations;

  2. since one of the test cases tests the connection to an HDFS instance with explicitly set host and port, if in your case these are different from, respectively, “localhost” and 9000 (8020 for package-based CDH), you must set the HDFS_HOST and HDFS_PORT environment variables accordingly;

  3. start HDFS:

  4. wait until HDFS exits from safe mode:

    ${HADOOP_HOME}/bin/hadoop dfsadmin -safemode wait

To run the unit tests, move to the test subdirectory and run as the cluster superuser (see below):


Superuser Privileges

The following HDFS tests may fail if not run by the cluster superuser: capacity, chown and used. To get superuser privileges, you can either:

  • start the cluster with your own user account;
  • edit hdfs-site.xml in your configuration and set the dfs.permissions.supergroup (dfs.permissions.superusergroup in Hadoop 2) property to one of your unix groups (type groups at the command prompt to see to which groups your account belongs), then restart the Hadoop daemons:

If you can’t acquire superuser privileges to run the tests, just keep in mind that the failures reported may be due to this reason.

Hadoop2 / CDH4

With Apache Hadoop 2 / CDH 4, before running the unit tests, edit hdfs-site.xml and set dfs.namenode.fs-limits.min-block-size to a low value:


then restart Hadoop daemons.