Chapter 4

Extensions and csi -setup

To simplify the management of extension libraries, a simple mechanism based on shared object files and dynamic loading is used to allow the loading of extension libraries at run-time. Currently this mechanism is only available on platforms that support dynamic loading of shared object files, at the time of writing this is Linux, FreeBSD, HP/UX, Windows (Cygwin) and Solaris. The macro chicken-setup is used to make packaging, building and installation as simple as possible for the user.

An extension is a compressed archive containing the source-code, compilation parameters and additional information. Technically an extension is a gzipped tar-archive, containing the source file (or a directory hierarchy with source-files, in the case extensions containing multiple modules) and a setup script with additional information.

The CHICKEN interpreter provides an option for performing basic operations on extensions. To build and install an extension, enter

% csi -setup EXTENSION

(Note that you have to have write-access for the directory where system-wide extensions are installed. Normally this directory is /usr/local/lib/chicken. You can create this directory by entering csi -setup without additional parameters.)

The registry will be created automatically on its first use. Do create the registry explicitly, enter

csi -setup -init

This will overwrite any existing registry (after asking the user for confirmation). This may also be needed should the extension-registry be damaged.

If you want to have your extensions installed at a different location than the default, you have two choices: either set the environment-variable CHICKEN_REGISTRY to the name of your desired registry-directory, or create a directory named $HOME/.chicken-registry.

To ``wrap'' up an extension, that is, to create a packaged archive ready for distribution, enter

% csi -setup EXTENSION -wrap

The -setup option loads the setup-script EXTENSION.setup and passes any arguments that follow the extension-name as command-line arguments.

A setup-script is a normal Scheme-program, which should contain an invocation of the chicken-setup macro, like this:

(chicken-setup (command-line-arguments) SPEC ...)

[syntax] (chicken-setup ARGUMENTS SPEC ...)
Installs or wraps extension libraries with the specifications SPEC .... ARGUMENTS should be a list of strings, where the string may be one of the following:

If no argument is given, then the extension will be extracted, built and installed in one step.

SPEC is a setup-specification that describes the structure and attributes of the extension:

(EXTENSION-NAME
  (ATTRIBUTE ...)
  SUBMODULE ...) )

The following attributes are allowed:

(author STRING)
The name of the author of the module.

(license STRING)
The type of license this module is distributed under.

(comment STRING ...)
Additional comments.

(syntax)
(source)
Marks this module as source, that is, it is not compiled. Use this attribute for syntax/macro modules.

(file STRING)
If the module (or sub-module) has a different filename than the module-name, then the actual filename is declared using this property.

(make STRING-OR-EXP)
Defines the command used to build the module. This defaults to an invocation of the compiler-script csc with the appropriate arguments and options to build the shared library object. The argument may also be a Scheme expression that is evaluated to build the current module.

(when EXP)
If the evaulation of EXP returns #f, then this module will not be built.

(options STRING ...)
Sets the compiler options for Scheme files that are to be compiled using csc.

(c-options STRING ...)
Sets the C-compiler options for Scheme files that are to be compiled using csc.

(link-options STRING ...)
Sets the linker options for Scheme files that are to be compiled using csc.

(require EXTENSION ...)
Lists extensions that are required by the current module.

(require-for-syntax EXTENSION ...)
Lists extensions that are required at compile-time by the current module.

(require-at-runtime EXTENSION ...)
Lists extensions that are required at runtime to support a syntax-extension. This means that an expansion of

(require-for-syntax EXT)

where EXT is defined with a specification that contains an attribute like

(require-at-runtime EXTR1 EXTR2)

will result in this code

(require 'EXTR1 'EXTR2)

(provide EXTENSION)
Declares the given extensions as aliases for the current extension.

As an example, we take two simple extensions: an implementation of red-black trees and a simple ``functor''-like syntax that encapsulates the operations on the rb-tree data-type. The files for an extension should reside in a separate directory, so in the case of our simple functor-implementations we have the following directory-structure:

functor.scm
functor/
  utils.scm

Here we put the macro-definition for define-functor in a separate file, this simplifies the use in compiled code. The code follows:

;;; functor.scm

(require 'srfi-1)
(require-for-syntax 'match)

(define-macro (define-functor name imports . body)
  (let ([exports 
         (filter-map
          (match-lambda
            [('define (name . llist) . body) name]
            [('define name val) name] 
            [_ #f] )
          body) ] )
    `(define (,name ,@imports)
       ,@body
       (values ,@exports) ) ) )

and

;;; functor/utils.scm

(define (instantiate-functor f . imports)
  (apply f imports) )

One more thing is needed: a setup specification that contains information about the files and settings of the extension. In this case we have:

;;; functor.setup

(chicken-setup (command-line-arguments)
  (functor ((syntax)
            (require-at-runtime (functor utils)) )
    (utils ()) ) )

Now we create an extension package:

% csi -setup functor -wrap
wrapping extension `functor' ...
functor.setup
functor.scm
functor/
functor/utils.scm

This will create the compressed archive functor.egg.

A user of this package can now use csi in combination with the -setup option to extract the contents from the archive, to build its components and install it in his Chicken system for easy use.

% ls
functor.egg
% csi -setup functor -build
extracting files from extension `functor' ...
functor.setup
functor.scm
functor/
functor/utils.scm
building extension `functor' ...
backing up registry...
adding entry for `/home/felix/chicken/lib/functor.setup' ...
compiling: csc -s ./functor/utils.scm -o ./functor/utils.so -O1 
removing temporary extension `/home/felix/chicken/lib/functor.setup' ...
restoring registry

Note that functor.scm is not compiled, because it is explicitly marked as syntax, which means it is only meant to be used at compile-time, in source-form.

The extension is now installed and ready for use.

% csi -quiet
>>> (require-for-syntax 'functor)
; loading /home/felix/chicken/lib/functor.scm ...
; loading /home/felix/chicken/lib/functor/utils.so ...
>>> (define-functor adder (+) 
      (define (make-adder init)
        (lambda (x)
          (set! init (+ init x))
          init) ) )
>>> (define ma (instantiate-functor adder +))
>>> (define a1 (ma 99))
>>> (a1 1)
100
>>> (a1 22)
122
>>>

The extension for red/black trees is created in much the same way:

;;; rbtree.scm
;
; taken directly from Chris Okasaki's ``Purely functional Data Structures''

(require-for-syntax 'match)
(require-for-syntax 'functor)

(define-functor rb-tree (rb<?)

  (define (member? x t)
    (match t
      [() #f]
      [(_ a y b) 
       (cond [(rb<? x y) (member? x a)]
             [(rb<? y x) (member? x b)]
             [else #t] ) ] ))

  (define balance
    (match-lambda
      [(or ('black ('red ('red a x b) y c) z d)
           ('black ('red a x ('red b y c)) z d)
           ('black a x ('red ('red b y c) z d))
           ('black a x ('red b y ('red c z d))) )
       `(red (black ,a ,x ,b) ,y (black ,c ,z ,d)) ]
      [body body] ) )

  (define (insert x s)
    (define ins
      (match-lambda
        [() `(red () ,x ())]
        [(and s (color a y b))
         (cond [(rb<? x y) (balance (list color (ins a) y b))]
               [(rb<? y x) (balance (list color a y (ins b)))]
               [else s] ) ] ) )
    (match (ins s)
      [(_ a y b) `(black ,a ,y ,b)] ) ) )

The setup-specification looks like this. Since we only have a single file, we don't need to put it in a separate directory:

(chicken-setup (command-line-arguments)
  (rb-tree ((options "-O2")
            (file "rbtree.scm")
            (require-for-syntax functor match) ) ) )

Note the properties used: (options "-O2") overrides the default compiler-options ("-O1") and we use a different name for the source-file2. This extension requires the define-functor extension, we have to use require-for-syntax to make sure that the macro is included at compile-time.

The wrapping, unwrapping, building and installation works identical to the previously given example. Here is some code that uses both extensions:

% cat testrb.scm
(require 'rb-tree 'srfi-1 'extras)
(require '(functor utils))

(define-values (num-rb-member? num-rb-balance num-rb-insert)
  (instantiate-functor rb-tree <) )

(randomize)
(print "building list...")
(define vals (list-tabulate 10000 (lambda (_) (random 100000))))

(print "building rb-tree...")
(define t1 '())
(for-each (lambda (n) (set! t1 (num-rb-insert n t1))) vals)

(print "building hash-table...")
(define ht (make-hash-table))
(for-each (lambda (n) (hash-table-set! ht n #t)) vals)

(print "linear list:")
(time (every (lambda (n) (memq n vals)) vals))

(print "hash-table:")
(time (every (lambda (n) (hash-table-ref ht n)) vals))

(print "rb:")
(time (every (lambda (n) (num-rb-member? n t1)) vals))
% csc testrb.scm -O2
% testrb
building list...
building rb-tree...
building hash-table...
linear list:
    1.74 seconds elapsed
       0 seconds in GC
       0 mutations
     161 minor GCs
       0 major GCs
hash-table:
    0.03 seconds elapsed
       0 seconds in GC
       0 mutations
     779 minor GCs
       0 major GCs
rb:
    0.27 seconds elapsed
    0.02 seconds in GC
       0 mutations
    4684 minor GCs
       0 major GCs

A special case of extensions is one that consists only of a single source file and which doesn't provide any additional syntax definitions. In this case, we can build and install the extensions simply by entering:

% csi -setup foo

where foo.scm is the name of the source file containing the code.

It should be noted that the extension-mechanism does not address any namespace-issues. It is recommended to use the export declaration- specifier to export toplevel names that are to be used by client code. This will hide module-local definitions. Prefixing exported names with something like <extension-name>:<definition> avoids namespace-collisions with other libraries and names of client code.


2 just for the heck of it