For Yacas developers


A crash course in Yacas maintenance for developers

This document intends to give a concise description of the way Yacas is maintained. There are a few parts to maintenance to take into account:

http://www.xs4all.nl/~apinkus/backups/


The autoconf/automake system

The important thing to remember with the autoconf/automake system is that there are files enumerating the files and directories that are part of the distribution. The Makefile.am files contain the lists of files that are part of the distribution, and are in the respective directory. If a file is added by a developer, but it is not added to the list in the file Makefile.am, it will not be added to the tarball yacas-*.tar.gz which is uploaded to the backup repository. It has the nice side effect that you can have local files which don't automatically get added to the distribution.

Adding a directory requires adding the directory name to various places also, in configure scripts etc. To make sure that the tar.gz distribution is complete, you can run the command
make distcheck
This may take a little while, as it needs to rebuild and test the whole system from the tar.gz tarball.


Maintaining Yacas through a cvs repository

CVS provides an efficient way for developers to work together, automatically merging changes various developers make, and at the same tile is a back up system (uploading your changes to another computer from which you can easily obtain it at a later time). After a little effort setting it up it becomes very easy to use. This section describes the few commands needed for keeping your version and the version in the Yacas repository up to date.


How does cvs work?

CVS has a copy of the files in the repository somewhere in a directory on some system, possibly your computer. Then there is such a thing as a cvs server which you can talk to to synchronize your version of the source code with the version on the server.

CVS uses a diff-like scheme for merging differences: it looks at two text files, determines the different lines, and merges accordingly. It discovers the changes you made by looking at the version you checked out last and the version you have now, to discover which lines changed (it maintains an automatic version number for each file).

If the version of a file on your system and the version in the cvs repository has a line that has been changed by both you and some one else, the cvs repository will obviously not know what to do with that, and it will signal a 'collision' which you will have to solve by hand (don't worry, this rarely happens). More on that later.

In commands to be described in this document are in short:


Checking out an initial version of Yacas

There are two ways to check out a version of Yacas: as anonymous user and as maintainer. Anonymous users don't need to log in, but also have no right to commit changes. Maintainers first need to get an account (at sourceforge), and their account needs to be enabled so they are allowed by the maintainer to make changes. A maintainer needs to log in with every command. To be able to log in, you need ssh1 installed (ssh2 will not work). You can find this at http://www.ssh.org/download.html.

To check out Yacas as anonymous user, type:

cvs -d:pserver:anonymous@cvs.yacas.
  sourceforge.net:/cvsroot/yacas login
cvs -z3 -d:pserver:anonymous@cvs.yacas.
  sourceforge.net:/cvsroot/yacas co yacas

To check out as a maintainer, type:

export CVS_RSH=ssh1

This will tell CVS to use ssh1 for communication. Then, in order to download the yacas source tree, type

cvs -d:ext:loginname@cvs.yacas.sourceforge.
  net:/cvsroot/yacas co yacas
where loginname is your name on the sourceforge system. This creates a directory yacas/ with the full most recent distribution. You need to enter your password there, but other than that, that's it!

Those lines typed above are long and obscure, but it is also the last time you need to type them. From now on, if you want to do anything with cvs, just go into the yacas/ directory you just checked out, and type the cvs command without the -d:... flag. This flag just tells cvs where to find the repository. But future cvs commands will know where to find them, which is why you don't need that flag.


Use case scenario 1 : getting the latest version of Yacas

You haven't looked at Yacas for a while (shame on you!) and want to check out the latest version. Just type

cvs update -d

on the command line in the yacas directory, and that should essentially download the latest version for you in that directory (just the changes). The -d option here states that you are also interested in new directories that were added to the repository. Oddly enough, cvs will only get you changed and added files, not added directories, by default.

A command

cvs -q update -d

will print messages only about changed files.


Use case scenario 2 : you made changes to Yacas

You got the latest version, but saw this huge, glaring omission in Yacas, and start hacking away to add it yourself. After a while, after playing with the code you wrote, and if you think you are finished with it, you decide you like to add it to the cvs repository.

First, you should test the new Yacas system:
make test
If there are any failed tests, you need to fix them.

Now you can start entering your changes to the CVS. If you created some new files, you need to tell CVS to add them to the source tree:

cvs add [list of file names of ascii text files]

This adds ascii text files. If you added binary files (GIF images in the documentation directory, or something like that), you can add it to the CVS with

cvs add -kb [list of file names of binary files]

Note that, when adding files to the CVS, you should normally also add them to the Yacas tar.gz distribution. This is done by adding the file name to the EXTRA_DIST variable in the file Makefile.am in the directory where you were adding the file.

In case files need to be removed, there are two options:

There seems to be no easy way to rename or move files; you would have to remove them at their old location and add them at a new location.

Now, when finished with that, you might want to 'commit' all changes with

cvs commit

If the commit succeeds, an email is sent out to the maintainers, who can then scan the diff files for changes, to see if they agree with the changes, and perhaps fix mistakes made (if any).

If there is a collision, the commit fails (it will tell you so). This might happen because someone else also edited the same place in a file and their changes cannot be automatically merged with yours. In case of a collision, you need to invoke cvs update twice. The cvs update outputs a list of file names with a character in front of them. The important ones are the files with a 'C' before them. They have a collision. You can go into the file, and see the collision, which the cvs system conveniently marks as:

<<<<<<
old version
===========
new version
>>>>>>

You can edit the file by merging the two versions by hand. This happens very rarely, but it can happen. Use cvs commit afterwards to commit.

The commit and update commands can be performed in specific directories, and on specific files, if necessary, by stating them on the command line. Or you can go into a sub directory and do a cvs commit or cvs update there, if you are confident that is the only place that changed or whose changes you are interested in.

That is basically it, a quick crash course cvs. It is actually very convenient in that usually all that is needed is a cvs commit to fix small bugs. You type that in, and your version gets merged with the changes others made, and they get your changes, and you backed up your changes at the same time (all with that little command!).

You can find more information about cvs at http://cvsbook.red-bean.com/ .


Preparing and maintaining Yacas documentation


Introduction

Yacas documentation in HTML and PS/PDF formats is generated by Yacas scripts from Yacas source files. However, it is cumbersome to write those source files in the Yacas language. The scripts txt2yacasdoc.pl, book2TeX.sh, book2ys.sh, ytxt2tex makes it possible to create and maintain the documentation in an easy-to-read form.

The "source" form of all documentation is maintained in a special plain text format. The format is such that it is clearly readable without any processing and is easy to edit. To compile the documents, the system processes the plain text docs with a script to prepare Yacas-language files and then runs other scripts to produce the final documentation in HTML and other formats.

The source text must be formatted in a certain fashion to delimit sections, code examples, and so on, but the format is easy enough to enter in a plain text editor. Text is marked up mostly by TAB characters, spaces, and asterisks "*" at the beginning of a line. The script txt2yacasdoc.pl converts this markup into Yacas code. This Yacas code is executed using the Yacas interpreter to generate the final documentation in TeX and HTML formats.


Organization of the Yacas documentation

All documentation source files are kept in the subdirectory manualmaker/in. During compilation, Yacas language files as well as HTML, LaTeX and PS/PDF files are automatically generated in the manualmaker subdirectory. Contributors should only need to edit files in manualmaker/in.

Currently, documentation consists of five "books" (introductory tutorial, programming tutorial, miscellaneous essays, user's reference manual, and programmer's reference manual). The first three books are meant as stand-alone texts, while the last two are "reference" books and are meant to be used together. The Yacas Help() command will show a reference article from either of the two reference books.

Books may be divided into "chapters", "sections" and "subsections". Reference documentation contains each command description in a separate "section" which is marked by a special *CMD label (see below).

Stand-alone books are free-form, but reference books must be written with a certain template that allows online hyperlinking. The file manualmaker/in/dummies is an example template for a reference manual section.

At the beginning of each book there must be a book title and a short book description (labeled *BLURB) which does not appear in printed documentation but only in the HTML top-level book index.

At the beginning of each chapter there may be a "chapter introduction" labeled *INTRO which is also a short description of the contents of that chapter. It may be one paragraph only. Note that the chapter introduction is only really useful for the HTML reference manual because it is the text that appears at the very top of a reference manual section. In the HTML version of reference docs, each chapter contains a list of all functions described in it, and this list goes right after the first paragraph of "chapter intro". For the printed (PS/PDF) documentation the "chapter intro" is indistinguishable from any other text.


Formatting of source text files

Formatting of source text files uses TAB symbols; if your editor does not support them and converts them to spaces, you should convert the results back to contain real TAB symbols using the standard Unix unexpand utility or a custom perl script.

You may want to examine the source of this file (manualmaker/ in/ YacasDocs.chapt.txt) to see how various features of the markup are used. Currently the following markup is implemented:

*	Item
*	Another item

Note that the text of an item continues until the next itemized line is given or until end of paragraph (does not have to be all in one line).

Nesting of enumerated or itemized environments is not supported, except for fringe cases of nesting just one itemized list at the very end of an enumerated list or vice versa.

Enumerated environment is currently only implemented in LaTeX docs; HTML docs render them as itemized.

There is a special feature for displayed equations: Any punctuation immediately following the second pair of dollar signs will be displayed on the same line. (This is used to get around the limitation of mathematical expressions that cannot end with a comma or a period or with another punctuation mark.) For example, the formula "$$x^2/2$$," will produce

x^2/2,

with the comma on the same line. A formula such as "$x+1;$" will generate an error; the semicolon should be moved out of the dollar signs.

As special exceptions, one can enter the symbols " TeX" and " LaTeX" as if they are Yacas symbols, i.e. "$TeX$" produces " TeX". One can also enter the name Yacas as "{Yacas}".

Mathematical expressions must be valid Yacas expressions, with no unbalanced parentheses, no undefined infix operators, no hanging periods and so on, or else the Yacas script that formats the docs will fail! (This limits the scope of mathematical formulae but is hopefully not critical.)

Currently, when creating online HTML documentation, mathematics is kept in Yacas notation and set in boldface font. (This may change in the future.) Of course, LaTeX typesets printed documentation with correct mathematical symbols.

Another feature of the LaTeX exporter is that it will try to show all functions and infix operators according to their mathematical meaning, and if no such meaning is defined in Yacas, then it will show them exactly as they are written in Yacas. For infix operators to work, they have to be declared in the standard library, or else an error will occur when processing the manual.

For example, Yacas operators = and == are both represented by and equals sign "=", the operator := becomes "identically equal" ( a:=b), and the cosmetic operators <> and <=> become a<>b and a<=>b. But you cannot use an undefined infix operator such as ":=*".

*INCLUDE ../essays/howto.chapt

Note that the included document must be a Yacas-language file, not a .txt file. (This will become the IncludeFile() call in the Yacas document -- an alias to Load().)

For example, the text
*FOOT This is an example footnote
generates the footnote

This is an example footnote.

For example,
*EVAL "Yacas version: " : Version()
will insert the string ` Yacas version: 1.0.53rev1 ' into the manual.


Formatting text for the reference manual

The formatting explained in the previous section is enough to create most of the user guide and tutorial documentation. The script txt2yacasdoc.pl also implements some additional markup features to help create the reference manual.

A typical reference manual subsection documenting a certain function may look like this in plain text:

*CMD PrintList --- print list with padding
*STD
*CALL
	{PrintList}(list)
	{PrintList}(list, padding);

*PARMS
{list} -- a list to be printed

{padding} -- (optional) a string

*DESC
Prints {list} and inserts the {padding} ...

*E.G.

	In> PrintList({a,b,{c, d}}, " .. ")
	Out> " a ..  b .. { c ..  d}";

*SEE Write, WriteString
Compare this with the reference manual section on the function PrintList to see how this plain text markup is rendered in the finished documentation.

Notes:

*CMD Sin, Cos, Tan --- Trigonometric ...

In addition to these labels, there are the following tags:

Usage of the *A tag currently does not directly affect the appearance of the HTML docs, since the anchor tags <a></a> are invisible. In the printed LaTeX docs, this tag can be used to manually add index entries. The *CMD tag generates all necessary HTML anchors for commands in the reference manual, so only non-command index entries need to be manually entered.

The *INTRO and *BLURB tags only work for one paragraph. There must be no empty line between *INTRO/*BLURB and that paragraph. Also, there must be no empty line between the "blurb" and the book title (for technical reasons). There must be one and only one "blurb" paragraph in a "book" and no more than one "chapter intro" paragraph per chapter.

This markup should be sufficient for creating reference documentation in plain text.


Indexing the documentation books

It is not difficult to automatically generate an alphabetically sorted index for the books. An "index entry" is a piece of text that does not appear in the book where it is entered, but instead is printed in the alphabetical list at the end of the text with the relevant page number.

Currently the following facilities are provided for indexing:

After LaTeX generates a "raw" index file *.idx, the makeindex utility is used to post-process and sort the index into the .ind file. If you do not have makeindex on your system, the book indices will not be generated.

Note that makeindex is not always friendly to special (non-alphanumeric) characters. For example, it uses the symbol ! to separate index topics, which may conflict with Yacas commands. In other words, document index must be tested and sometimes debugged.

In the HTML docs, the index is currently not generated on a separate page, although HTML anchors are inserted in the text. The reference manual uses the HTML anchors to provide online help through the ? command.

An index entry may be a "topic" with "subtopics", which usually appears in book indices like this:
gnus, 51
   tame, 51
   wild, 52-341
This effect can be achieved with the ! topic separator:
*A gnus
*A gnus!tame
*A gnus!wild
This is a special feature of makeindex.

Currently, it is possible to include a command or an equation into an index entry, for example,
*A {InterestingCommand}
*A calculation of $Sqrt(x)$
But this may sometimes conflict with the topic separator.


Summary of mark-up labels

Mark-up labels must appear as first characters on a line. The following mark-up labels are currently defined:

Labels that affect only the current line:

Labels that affect a subsequent paragraph (and must be separated by an empty line from what follows):

Special labels for the reference manual that accept several arguments on the same line:

Special labels without arguments that generate headings for the reference manual:


Debugging the manual

Sometimes the manual compilation make or make texdocs will break after you edit the plaintext manual sources. This can happen for one of these reasons:

In case of a math syntax error, the documentation exporter cannot print the paragraph where the error occurred, but it usually prints the preceding paragraph. Currently, the easiest way to locate the error is to generate the .tex output and look at it, e.g.:
make ref.book.tex; less ref.book.tex
The last line in the .tex file must be \end{document}. If it is not, then the last portion of the text you see in the .tex file is the text directly before the paragraph where the error occurred. Most probably, there is a malformatted math formula in the next paragraph of your plaintext source.

If the last line is \end{document} but latex does not finish, you will have to run latex by hand, e.g.
latex ref.book.tex
and look at the error message(s) it prints.


Using the script txt2yacasdoc.pl

The script txt2yacasdoc.pl is used to transform plain text markup into the Yacas language. The script acts as a stream filter:

perl txt2yacasdoc.pl < file.txt \
  > file.chapt

In this example, file.txt contains some formatted plain text (source text) and the resulting file file.chapt will be produced in Yacas-language documentation format.

There is a single option for txt2yacasdoc:

perl txt2yacasdoc.pl -debug < file.txt \
  > file.chapt
This option is to be used for debugging, i.e. when the resulting file does not compile in Yacas. The effect of this option is to introduce more breaks between text strings in the generated file, so that the Text() function is called more often. It is then easier to locate the source of the problem in the Yacas-language file (Yacas will tell you the last line in the Yacas-language file at which a syntax error occurred). This option is largely obsolete because the Text() function is called frequently enough by default. See below for hints about finding syntax errors in documentation when the manual does not compile.


book2TeX: preparing typeset documentation

The script book2TeX.sh prepares a TeX file out of a Yacas-language documentation book. Usage is similar to book2txt.sh, except that only one file is processed at a time and the file must be a "book", not just a "chapter". For example:
book2TeX.sh intro.book intro.book.tex
will create a LaTeX-formatted version of the introductory tutorial. The LaTeX file can be processed with standard tools, for example
latex intro.book.tex
dvips -o intro.book.ps intro.book dvi
will prepare a Postscript version, while
pdflatex intro.book.tex
will prepare a PDF version.

To generate printed docs, it is necessary to run latex (at least) three times in a row. This is because at first latex does not know how much space will be taken by the table of contents and the index, so the page numbers are all off by a few pages. Only on the second run latex generates correct page numbers for the TOC (the .aux file) and for the index (the .idx file). After this the index file has to be processed by the makeindex routine to sort it, and the third latex run is needed to actually insert the correct TOC and the processed index into the final document.

The shell commands in book2txt.sh execute the following Yacas commands:
Use("book2TeX.ys");
ToFile("file.chapt.tex")
  Load("file.chapt");
This requires that the Yacas script book2TeX.ys be available in the current directory. The shell script book2TeX.sh assumes that book2TeX.ys is stored in the same directory as book2TeX.sh and that the Yacas executable is available in the directory ../src/. Alternatively, the command line of the Yacas executable can be specified by the -run option. For example, the Makefile runs book2TeX.sh like this:
book2TeX.sh -run "yacas-dir/src/yacas --rootdir
  yacas-dir/scripts" file.book file.book.tex
Note that the entire Yacas command line is given in quotes.

Not all features of Yacas documentation are compatible with TeX typesetting. To prevent errors, documentation source should avoid certain things. In general, it is a good idea to check the typeset appearance of documentation, since it helps detect errors.

For example, the symbols %, { }, < >, #, \, _ and & are special to TeX. They should not normally be used in plain text; it is okay to use them in "typerwriter" text (within braces {}) or code samples -- but not in section or chapter heads, because it makes it difficult to export to TeX correctly. TeX commands may be entered but will not be correctly rendered in HTML online documentation.

Sometimes fixed-font text will hang over the right edge of the printed page. A workaround is to break the fixed-font text into shorter fragments or to rephrase the text.

Another concern is that code examples (TAB-indented blocks) are typeset in a fixed-width font and may not fit into the width of the page. To avoid this, the lines in the code examples should not be longer than about 50 characters.

The current implementation uses a "report" style which allows chapters, sections, subsections and includes an automatically generated table of contents and an index. The standard 10 point font and two-column format are used to save space (and trees).

Some complicated mathematical expressions may not correctly render in TeX. This is because Yacas uses its internal function TeXForm() to transform Yacas expressions to TeX. Mathematical expressions are entered in the plain text documentation source using Yacas syntax, then transformed to a special non-evaluating call TeXMath() in the Yacas-language documentation, which formats into HTML using a Write() call or into TeX using a TeXForm() call, as necessary. Testing should be performed on documentation before releasing it. The most stringent limitation is that the expression between dollar signs should evaluate in Yacas (preferably to itself) and not cause syntax errors. In case of doubt, check that the expression evaluates without errors and then try to use TeXForm on that expression and see if that evaluates without errors as well. For example, expressions such as x=+1 will cause a syntax error and this will break the compilation of the manual (both HTML and TeX).


book2ys: extracting Yacas code from books

book2ys.sh is a shell script that extracts Yacas code examples from a documentation chapter into a separate file. Usage is similar to book2TeX.sh. For example, the benchmarking test code wester.yts can be automatically extracted from the corresponding essay chapter by the command
sh ../manmake/book2ys.sh wester-1994.chapt
After this command, the file wester-1994.ys is created. Note that wester-1994.chapt is in Yacas language and is itself a generated file.

Using this script, one can write documentation and code together, a la "literate programming".

To prepare a documentation chapter in such a way that code extraction is possible, one only needs to make sure that all code examples in the chapter will become a correct sequence of Yacas expressions when cut out and written sequentially into a file. So, for instance, semicolons at the end of each statement are required. The script book2ys will not export example Yacas session code with "In>" and "Out>" prompts but it will export all other example code.

See the source file wester-1994.chapt.txt to get a feeling of how the source documentation is formatted to allow completely automatic code extraction. Note that the printed documentation will be in twocolumn format, and therefore it is necessary to split lines that are too long.


ytxt2tex: Conversion of plain text documentation to LaTeX

An auxiliary script ytxt2tex converts plain text documentation to LaTeX. This is a standalone script; it is installed into /usr/local/bin and requires the Yacas executable also on the path, as well as the script files book2TeX.* and txt2yacasdoc.pl in the /manmake subdirectory of the Yacas installation tree /usr/local/share/yacas/. It also requires perl and the Unix shell sh.

The script ytxt2tex can be used outside of the Yacas source tree to convert individual documents to LaTeX. This is useful if you would like to produce TeX documents and if you find the plain text format of the Yacas documentation more comfortable. Therefore, ytxt2tex is a kind of a special-purpose TeX preprocessor designed for producing Yacas documentation.

Limitations of this script are:

These limitations may be easily overcome by editing the resulting TeX file (but you need to know at least some TeX to do that).

The general usage pattern is
ytxt2tex [-o outputfile] file1.txt [file2.txt] ...
All source files must have extension ".txt". The command-line option -o specifies the name of the output TeX file. If the -o option is not given, the output file will be file1.tex (i.e. the name of the first .txt file with the .tex extension). If several .txt files are given, the first one must *INCLUDE all others.

To illustrate the usage of the script ytxt2tex, consider two examples.

The first example is just one plaintext file example1.txt. This file will have to be a "book" in itself, i.e. it will have to include a book title indented by four TAB symbols. For example:

*REM file: example1.txt
                Example Document Title
*REM this is a section title:
		Numbers and letters
*REM here are some index entries:
*A numbers
*REM simple index entries like this are OK
*A letters

Numbers and letters are very important, etc.

This file example1.txt can be converted to a LaTeX file example1.tex by the following simple command:
ytxt2tex example1.txt
If the resulting file should be named something other than example1.tex, say output1.tex, then the command is
ytxt2tex -o output1.tex example1.txt

The second example is a longer "book" consisting of several plaintext files. One of these files is a "master file" and it should include all other files using the *INCLUDE label. The *INCLUDE label should contain file names without the .txt extension.

Suppose we have prepared the files book1.txt, chapter1.txt, and chapter2.txt containing the preamble text and two chapters. For example:

*REM this is the file "book1.txt"
*BLURB or, The Multitudinous Attempts
to Avoid Counterproductivity
                 Relationships of Entities
*INCLUDE chapter1
*INCLUDE chapter2

The chapter files might be:

*REM this is the file "chapter1.txt"
		Entities and Epiphenomena
The history of the ambiguous question of
epiphenomenological discourse can be
traced to the pre-postmodern period...

*REM this is the file "chapter2.txt"
		Substrates and Superficiality
In the preceding chapter, we have thoroughly
investigated the metaphilosophical aspects of
the trans-homocentric considerations...

The command to create a final LaTeX file book1.tex is
ytxt2tex book1.txt chapter1.txt chapter2.txt
The "master file" book1.txt that includes all other text files must be given first. The -o option can be used if the final LaTeX file should be named something else than book1.tex. For example,
ytxt2tex -o MyBook.tex book1.txt chapter*.txt

By default, both table of contents and the index are generated. The commands to create a PostScript file out of the LaTeX file might be:
latex MyBook.tex
latex MyBook.tex
makeindex MyBook.idx -o MyBook.ind
latex MyBook.tex

Note that the resulting LaTeX file needs to be processed three times if the table of contents or index are to be used.


book2txt: Conversion of existing documentation to plain text

(Note: as of version 1.0.49, all Yacas documentation is converted to plaintext format. This section is left for reference only.)

Currently, most but not all of the Yacas documentation markup functionality is implemented in the simple plaintext filter; also, documentation includes some extra HTML files. However, almost all of the reasonable markup needed to write documentation is present. Therefore it is possible to maintain most of the documentation in the plain text format described above. To convert existing Yacas documentation back to the plain text format, a script book2txt.ys/book2txt.sh can be used.

By using a command such as
book2txt.sh file.chapt
one can create a source text file file.chapt.txt corresponding to the Yacas documentation file file.chapt. For example:

12:51pm scriabin> book2txt.sh intro.book
[editvi.ys] [gnuplot.ys] 
True;
Out> True;
Quitting...
File 'intro.book.txt' was created.
12:51pm scriabin>

In the above example, the shell commands in book2txt.sh executed the following Yacas commands,
Use("book2txt.ys");
ToFile("file.chapt.txt")
  Load("file.chapt");
This requires that the Yacas script book2txt.ys be available in the current directory. The shell script book2txt.sh assumes that book2txt.ys is stored in the same directory as book2txt.sh.

Of course, it is possible that some features of Yacas documentation were not implemented in the script and in that case the resulting file must be edited by hand. But the purpose of the book2txt script is exactly this: to make a plain text source file to be edited and maintained.

Several files can be converted at once, for example:
book2txt.sh f1.chapt f2.chapt file3.book
Each file is processed by an independent Yacas session. Any errors of processing are printed on the screen.