pyftpdlib Tutorial

1.0 - Introduction

pyftpdlib implements the server side of the FTP protocol as defined in RFC 959. pyftpdlib consist of a single file, ftpserver.py, which contains a hierarchy of classes, functions and variables which implement the backend functionality for the ftpd. This document is intended to serve as a simple API reference of most important classes and functions. Also included is an introduction to customization through the use of some example scripts.

If you have written a customized configuration you think could be useful to the community feel free to share it by adding a comment at the end of this document.

2.0 - API reference

2.1 - class AuthorizerError

AuthorizerError()

Base class for authorizers exceptions.

2.2 - class DummyAuthorizer

DummyAuthorizer()

Basic "dummy" authorizer class, suitable for subclassing to create your own custom authorizers. An "authorizer" is a class handling authentications and permissions of the FTP server. It is used inside FTPHandler class for verifying user's password, getting users home directory and checking user permissions when a filesystem read/write event occurs. DummyAuthorizer is the base authorizer, providing a platform independent interface for managing "virtual" FTP users. Typically the first thing you have to do is create an instance of this class and start adding ftp users:
>>> from pyftpdlib import ftpserver
>>> authorizer = ftpserver.DummyAuthorizer()
>>> authorizer.add_user('username', 'password', 'home/directory', perm=('r', 'w'))
>>> authorizer.add_anonymous('/home/nobody')

Relevant methods defined in this class:

add_user(username, password, homedir[, perm=("r")[, msg_login="Login successful."[, msg_quit="Goodbye."]]]) Add a user to the virtual users table. AuthorizerError exceptions raised on error conditions such as insufficient permissions or duplicate usernames. Optional perm argument is a tuple defaulting to ("r") referencing user's permissions. Valid values are "r" (read access), "w" (write access) or an empty string "" (no access). Optional msg_login and msg_quit arguments can be specified to provide customized response strings when user log-in and quit.
add_anonymous(homedir[, **kwargs]) Add an anonymous user to the virtual users table. AuthorizerError exception raised on error conditions such as insufficient permissions, missing home directory, or duplicate anonymous users. The keyword arguments in kwargs are the same expected by add_user method: perm, msg_login and msg_quit. The optional perm keyword argument is a tuple defaulting to ("r") referencing "read-only" anonymous user's permission. Using a "w" (write access) value results in a warning message printed to stderr.
remove_user(username) Remove a user from the virtual user table.
validate_authentication(username, password) Return True if the supplied username and password match the stored credentials.
has_user(username) Whether the username exists in the virtual users table.
get_home_dir(username) Return the user's home directory.
r_perm(username[, obj=None]) Whether the user has read permissions for obj (an absolute pathname of a file or a directory).
w_perm(username, [obj=None]) Whether the user has write permissions for obj (an absolute pathname of a file or a directory).

2.3 - class FTPHandler

FTPHandler(conn, ftpd_instance)

This class implements the FTP server Protocol Interpreter (see RFC 959), handling commands received from the client on the control channel by calling the command's corresponding method (e.g. for received command "MKD pathname", ftp_MKD() method is called with "pathname" as the argument). All relevant session information are stored in instance variables. conn and ftpd_instance parameters are automatically passed by FTPServer class instance. Basic usage simply requires creating an instance of FTPHandler class and specify which authorizer it will going to use:
>>> ftp_handler = ftpserver.FTPHandler
>>> ftp_handler.authorizer = authorizer

Relevant methods and attributes defined in this class:

banner String returned when client connects.
max_login_attempts Maximum number of wrong authentications before disconnecting (defaulting to 3).
permit_foreign_addresses Wether enable FXP feature (defaulting to False).
permit_privileged_ports Set to True if you want to permit PORTing over privileged ports (not recommended, defaulting to False).
masquerade_address The "masqueraded" IP address to provide along PASV reply when pyftpdlib is running behind a NAT or other types of gateways. When configured pyftpdlib will hide its local address and instead use the public address of your NAT (defaulting to None).
passive_ports What ports ftpd will use for its passive data transfers. Value expected is a list of integers (e.g. range(60000, 65535)). When configured pyftpdlib will no longer use kernel-assigned random ports (defaulting to None).
close() Close the current channel disconnecting the client.

2.4 - class FTPServer

FTPServer(address, handler)

This class is an asyncore.dispatcher subclass. It creates a FTP socket listening on address, dispatching the requests to a handler (typically FTPHandler class). It is typically used for starting asyncore polling loop:
>>> address = ('127.0.0.1', 21)
>>> ftpd = ftpserver.FTPServer(address, ftp_handler)
>>> ftpd.serve_forever()

Relevant methods and attributes defined in this class:

max_cons Number of maximum simultaneous connections accepted (defaulting to 0 == no limit).
max_cons_per_ip Number of maximum connections accepted for the same IP address (defaulting to 0 == no limit).
serve_forever([, **kwargs]) Starts the asyncore polling loop. The keyword arguments in kwargs are the same expected by asyncore.loop() function: timeout, use_poll, map and count.
close_all([map=None[, ignore_all=False]]) Stop serving; close all existent connections disconnecting clients. The map parameter is a dictionary whose items are the channels to close. If map is omitted, the default asyncore.socket_map is used. Having ignore_all parameter set to False results in raising exception in case of unexpected errors.

2.5 - class DTPHandler

DTPHandler(sock_obj, cmd_channel)

This class handles the server-data-transfer-process (server-DTP, see RFC 959) managing all transfer operations regarding the data channel. sock_obj is the underlying socket used for the connection, cmd_channel is the FTPHandler class instance. Unless you want to add extra functionalities like bandwidth throttling you shouldn't be interested in putting hands on this class.

Relevant methods defined in this class:

get_transmitted_bytes() Return the number of transmitted bytes.
transfer_in_progress() Return True if a transfer is in progress, else False.
enable_receiving(type) Enable receiving of data over the channel. Depending on the type currently in use it creates an appropriate wrapper for the incoming data.
push(data) Push a bufferable data object (e.g. a string) onto the deque and initiate send.
push_with_producer(producer) Push data using a producer and initiate send.
close() Close the data channel, first attempting to close any remaining file handles.

2.6 - class AbstractedFS

AbstractedFS()

A class used to interact with the file system, providing a high level, cross-platform interface compatible with both Windows and UNIX style filesystems. It provides some utility methods and some wraps around operations involved in file object creation and file system operations like moving files or removing directories.

Relevant methods and attributes defined in this class:

root User's absolute home directory.
cwd User's relative current working directory.
normalize(path) Translate a relative FTP path into an absolute virtual FTP path.
translate(path) Translate a FTP path into equivalent filesystem absolute path.
format_list(basedir, listing) Return a directory listing emulating "/bin/ls -lgA" UNIX command output. basedir is the absolute dirname, listing is a list of files contained in that directory. For portability reasons permissions, hard links numbers, owners and groups listed are static and unreliable but it shouldn't represent a problem for most ftp clients around. If you want reliable values on unix systems override this method and use other attributes provided by os.stat().

2.7 - class FileProducer

FileProducer(file, type)

Producer wrapper for file[-like] objects. Depending on the type it creates an appropriate wrapper for the outgoing data.

Relevant methods and attributes defined in this class:

out_buffer_size Number of bytes to read from file (defaulting to 65536).

2.8 - Functions

log(msg)

Log messages intended for the end user.

logline(msg)

Log commands and responses passing through the command channel.

logerror(msg)

Log traceback outputs occurring in case of errors.

debug(msg)

Log function/method calls (disabled by default).

3.0 - Customizing your FTP server

Below is a set of example scripts showing some of the possible customizations that can be done with pyftpdlib. Some of them are included in demo directory of pyftpdlib source distribution.

3.1 - Building a Base FTP server

The script below is a basic configuration, and it's probably the best starting point for understanding how things work. It uses the base DummyAuthorizer for adding a bunch of "virtual" users.

It also sets a limit for connections by overriding FTPServer max_cons and max_cons_per_ip attributes which are intended to set limits for maximum connections to handle simultaneously and maximum connections from the same IP address. Overriding these variables is always a good idea (they default to 0, or "no limit") since they are a good workaround for avoiding DoS attacks.

#!/usr/bin/env python
# basic_ftpd.py

"""A basic FTP server which uses a DummyAuthorizer for managing 'virtual
users', setting a limit for incoming connections.
"""

import os

from pyftpdlib import ftpserver


if __name__ == "__main__":

    # Import a dummy authorizer for managing 'virtual users'
    authorizer = ftpserver.DummyAuthorizer()
    authorizer.add_user('user', '12345', os.getcwd(), perm=('r', 'w'))
    authorizer.add_anonymous(os.getcwd())

    # Instantiate FTP handler class
    ftp_handler = ftpserver.FTPHandler
    ftp_handler.authorizer = authorizer

    # Define a customized banner (string returned when client connects)
    ftp_handler.banner = "pyftpdlib %s based ftpd ready." %ftpserver.__ver__

    # Instantiate FTP server class and listen to 0.0.0.0:21
    address = ('', 21)
    ftpd = ftpserver.FTPServer(address, ftp_handler)

    # set a limit for connections
    ftpd.max_cons = 256
    ftpd.max_cons_per_ip = 5

    # start ftp server
    ftpd.serve_forever()

3.2 - Logging management

As mentioned, ftpserver.py comes with 4 different functions intended for a separate logging system: log(), logline(), logerror() and debug(). Let's suppose you don't want to print FTPd messages on screen but you want to write them into different files: "/var/log/ftpd.log" will be main log file, "/var/log/ftpd.lines.log" the one where you'll want to store commands and responses passing through the control connection.

Here's one method this could be implemented:

#!/usr/bin/env python
# logging_management.py

import os
import time

from pyftpdlib import ftpserver


def get_time():
    return time.strftime("[%Y-%b-%d %H:%M:%S] ")

def standard_logger(msg):
    f = open('/var/log/ftpd.log', 'a')
    f.write(get_time() + msg + '\n')
    f.close()

def line_logger(msg):
    f = open('/var/log/ftpd.lines.log', 'a')
    f.write(get_time() + msg + '\n')
    f.close()

if __name__ == "__main__":
    ftpserver.log = standard_logger
    ftpserver.logline = line_logger

    authorizer = ftpserver.DummyAuthorizer()
    authorizer.add_anonymous(os.getcwd())
    ftp_handler = ftpserver.FTPHandler
    ftp_handler.authorizer = authorizer
    address = ('', 21)
    ftpd = ftpserver.FTPServer(address, ftp_handler)
    ftpd.serve_forever()

3.3 - Storing passwords as hash digests

Using FTP server library with the default DummyAuthorizer means that password will be stored in clear-text. An end-user ftpd using the default dummy authorizer would typically require a configuration file for authenticating users and their passwords but storing clear-text passwords is of course undesirable.

The most common way to do things in such case would be first creating new users and then storing their usernames + passwords as hash digests into a file or wherever you find it convenient.

The example below shows how to easily create an encrypted account storage system by storing passwords as one-way hashes by using md5 algorithm. This could be easily done by using the md5 module included with Python stdlib and by sub-classing the original DummyAuthorizer class overriding its validate_authentication method:

#!/usr/bin/env python
# md5_ftpd.py

"""A basic ftpd storing passwords as hash digests (platform independent).
"""

import md5
import os

from pyftpdlib import ftpserver


class DummyMD5Authorizer(ftpserver.DummyAuthorizer):

    def validate_authentication(self, username, password):
        hash = md5.new(password).hexdigest()
        return self.user_table[username]['pwd'] == hash

if __name__ == "__main__":
    # get a hash digest from a clear-text password
    hash = md5.new('12345').hexdigest()
    authorizer = DummyMD5Authorizer()
    authorizer.add_user('user', hash, os.getcwd(), perm=('r', 'w'))
    authorizer.add_anonymous(os.getcwd())    
    ftp_handler = ftpserver.FTPHandler
    ftp_handler.authorizer = authorizer
    address = ('', 21)
    ftpd = ftpserver.FTPServer(address, ftp_handler)
    ftpd.serve_forever()

3.4 - Unix FTP Server

If you're running a Unix system you may want to configure your ftpd to include support for 'real' users existing on the system.

The example below shows how to use pwd and spwd modules available in Python 2.5 or greater (UNIX systems only) to interact with UNIX user account and shadow password database. This basic authorizer also gets the user's home directory.

Note that users must already exist on the system.

#!/usr/bin/env python
# unix_ftpd.py

"""A ftpd using local unix account database to authenticate users
(users must already exist).
"""

import os
import pwd, spwd, crypt

from pyftpdlib import ftpserver


class UnixAuthorizer(ftpserver.DummyAuthorizer):

    def add_user(self, username, home=None, **kwargs):
        """Add a 'real' system user to the virtual users table.
        If no home argument is specified the user's home directory will be used.
        The keyword arguments in kwargs are the same expected by add_user
        method: 'perm', 'msg_login' and 'msg_quit'.
        """
        # get the list of all available users on the system and check if
        # username provided exists
        users = [entry.pw_name for entry in pwd.getpwall()]
        if not username in users:
            raise ftpserver.AuthorizerError('No such user "%s".' %username)
        if not home:
            home = pwd.getpwnam(username).pw_dir
        ftpserver.DummyAuthorizer.add_user(self, username, '', home, **kwargs)

    def validate_authentication(self, username, password):
        pw1 = spwd.getspnam(username).sp_pwd
        pw2 = crypt.crypt(password, pw1)
        return pw1 == pw2

if __name__ == "__main__":
    authorizer = UnixAuthorizer()
    # add a user (note: user must already exists)
    authorizer.add_user('user', perm=('r', 'w'))
    authorizer.add_anonymous(os.getcwd())
    ftp_handler = ftpserver.FTPHandler
    ftp_handler.authorizer = authorizer
    address = ('', 21)
    ftpd = ftpserver.FTPServer(address, ftp_handler)
    ftpd.serve_forever()

3.5 - Windows NT FTP Server

This next code shows how to implement a basic authorizer for a Windows NT workstation (windows NT, 2000, XP, 2003 server and so on...) to authenticate against existing Windows user accounts. This code uses Mark Hammond's pywin32 extension so the PyWin32 extensions must also be installed.

Note that, as for unix authorizer, users must be already created on the system.

#!/usr/bin/env python
# winnt_ftpd.py

"""A ftpd using local Windows NT account database to authenticate users
(users must already exist).
"""

import os
import win32security, win32net, pywintypes

from pyftpdlib import ftpserver


class WinNtAuthorizer(ftpserver.DummyAuthorizer):

    def add_user(self, username, home=None, **kwargs):
        """Add a 'real' system user to the virtual users table.
        If no home argument is specified the user's home directory will be used.
        The keyword arguments in kwargs are the same expected by add_user
        method: 'perm', 'msg_login' and 'msg_quit'.
        """
        # get the list of all available users on the system and check if
        # username provided exists
        users = [entry['name'] for entry in win32net.NetUserEnum(None, 0)[0]]
        if not username in users:
            raise ftpserver.AuthorizerError('No such user "%s".' %username)
        if not home:
            home = os.environ['USERPROFILE']
        ftpserver.DummyAuthorizer.add_user(self, username, '', home, **kwargs)

    def validate_authentication(self, username, password):
        try:
            win32security.LogonUser(username, None, password,
                win32security.LOGON32_LOGON_NETWORK,
                win32security.LOGON32_PROVIDER_DEFAULT)
            return 1
        except pywintypes.error:
            return 0

if __name__ == "__main__":
    authorizer = WinNtAuthorizer()
    # add a user (note: user must already exists)
    authorizer.add_user('user', perm=('r', 'w'))
    authorizer.add_anonymous(os.getcwd())
    ftp_handler = ftpserver.FTPHandler
    ftp_handler.authorizer = authorizer
    address = ('', 21)
    ftpd = ftpserver.FTPServer(address, ftp_handler)
    ftpd.serve_forever()

4.0 - Advanced usages

Here is a list of "hacks" / advanced features which could be useful to include in your own FTP server.

4.1 - Adding bandwidth throttling capabilities to asyncore

An important feature for an ftpd is limiting the speed for downloads and uploads affecting the data channel.

The basic idea behind this script is to wrap sending and receiving in a data counter and sleep loop so that you burst to no more than x Kb/sec average. Such sleep must be "asynchronous" since we want to avoid the polling loop from blocking.

To accomplish such behaviour I used a brand new socket_map for asyncore and I've overrided the original DTPHandler class. When the overrided DTPHandler realizes that more than x Kb in a second are being transmitted it temporary blocks the transfer by calling sleep method which removes the channel from the socket_map for a certain number of seconds. When such seconds have passed the channel is re-added in the socket_map.

#!/usr/bin/env python
# throttled_ftpd.py

"""ftpd supporting bandwidth throttling capabilities for data channel.
"""

import os
import asyncore
import time

from pyftpdlib import ftpserver


class socket_map_w_sleep(dict):
    """A modified socket_map for asyncore supporting asynchronous 'sleeps' for
    connected channels.
    """
    sleep_map = {}

    def items(self):
        for fd, params in self.sleep_map.items():
            obj, wakeup = params
            if time.time() >= wakeup:
                dict.update(self, {fd : obj})
                del self.sleep_map[fd]
        return dict.items(self)

    def __len__(self):
        return len(dict(self)) + len(self.sleep_map)

class ThrottledDTPHandler(ftpserver.DTPHandler):
    """A DTPHandler which wraps sending and receiving in a data counter and
    sleep loop so that you burst to no more than x Kb/sec average.

    It is an DTPHandler subclass which overrides some methods of
    asyncore.disptacher class (del_channel, recv and send), using a modified
    socket_map.
    """

    # smaller the buffers, the less bursty and smoother the throughput
    in_buffer_size = 2024
    out_buffer_size  = 2024

    # maximum number of bytes to transmit in a second (0 == no limit)
    max_send_speed = 0
    max_recv_speed = 0

    def __init__(self, sock_obj, cmd_channel):
        ftpserver.DTPHandler.__init__(self, sock_obj, cmd_channel)

    # --- overridden asyncore methods

    def del_channel(self, map=None):
        fd = self._fileno
        if map is None:
            map = self._map
        if map.has_key(fd):
            del map[fd]
        self._fileno = None
        # include sleep_map for key removing
        if map.sleep_map.has_key(fd):
            del map.sleep_map[fd]

    def recv(self, buffer_size):
        chunk = asyncore.dispatcher.recv(self, buffer_size)
        if self.max_recv_speed:
            self.throttle_bandwidth(len(chunk), self.max_recv_speed)
        return chunk

    def send(self, data):
        num_sent = asyncore.dispatcher.send(self, data)
        if self.max_send_speed:
            self.throttle_bandwidth(num_sent, self.max_send_speed)
        return num_sent

    # --- new methods

    def sleep(self, secs):
        """Remove current channel from the asyncore socket_map for the given
        number of seconds.
        """
        # remove channel from the "main_map" and put it into "sleep_map"
        obj = self._map.pop(self._fileno)
        self._map.sleep_map[self._fileno] = (self, (time.time() + secs))

    time_next = 0
    data_count = 0

    def throttle_bandwidth(self, len_chunk, max_speed):
        """A method which count data transmitted and call self.sleep(secs)
        so that you burst to no more than x Kb/sec average."""
        self.data_count += len_chunk
        if self.data_count >= max_speed:
            self.data_count = 0
            sleep_for = self.time_next - time.time()
            if sleep_for > 0:
                self.sleep(sleep_for * 2)
            self.time_next = time.time() + 1


if __name__ == '__main__':
    # set a modified socket_map for asyncore
    asyncore.socket_map = socket_map_w_sleep()

    authorizer = ftpserver.DummyAuthorizer()
    authorizer.add_user('user', '12345', os.getcwd(), perm=('r', 'w'))

    # use the modified DTPHandler class
    dtp_handler = ThrottledDTPHandler
    dtp_handler.max_send_speed = 51200  # 50 Kb/sec (50 * 1024)
    dtp_handler.max_recv_speed = 51200  # 50 Kb/sec (50 * 1024)

    ftp_handler = ftpserver.FTPHandler
    ftp_handler.authorizer = authorizer
    ftp_handler.dtp_handler = dtp_handler

    ftpd = ftpserver.FTPServer(('127.0.0.1', 21), ftp_handler)
    # Start asyncore loop by using the modified socket_map and set a small
    # timeout value to get a more precise throughput.
    ftpd.serve_forever(timeout=0.001, map=asyncore.socket_map)