http://invisible-island.net/ncurses/
Copyright © 2015-2018,2023 by Thomas E. Dickey

Synopsis

This is a summary of my experience with symbol versioning, mostly for ncurses, but also with Cdk and dialog.

Background

Initial

Starting in 1997, Red Hat was in the habit of incorporating my weekly patches without mentioning the ncurses patch date in the RPM changelog. Rather, they labeled their ncurses packages citing only their package release number on top of the ncurses major/minor version numbers (e.g., ncurses-4.2-18.i386). Although the changelog mentions the ncurses patches, the package version did not incorporate these patch dates until 2006, e.g., ncurses-5.5-24.20060715.x86_64. Unlike Debian, Red Hat does not routinely install the changelogs as part of the package, requiring the interested user to find the source-RPM to get this information.

However starting February 20, 1999, I made a series of changes which modified the binary interface of ncurses. Besides adding new functions (and renaming one to make it “private”) I changed the layout of the terminal data block used for direct access to an in-memory terminfo description. The changelog notes this as pre-release, i.e., a caveat to other developers that a new release is contemplated, but treat it as volatile until the actual release is announced.

I released the resulting ncurses 5.0 on October 23, 1999. The Red Hat packagers disagreed, as shown in their package changelog entries for 1999

* Wed Dec 22 1999 Cristian Gafton <gafton@redhat.com>
  - revert to the old major number - because the ABI is not changed (and we
    should be handling the changes via symbol versioning anyway)
* Fri Nov 12 1999 Bernhard Rosenkraenzer <bero@redhat.com>
  - Fix a typo in spec
  - Add the 19991006 patch, fixing some C++ STL compatibility problems.
  - get rid of profiling and debugging versions - we need to save space...
* Thu Nov 04 1999 Bernhard Rosenkraenzer <bero@redhat.com>
  - 5.0
  - some spec cleanups to make updating easier
  - add links *.so.5 to *.so.4 - they are fully binary compatible.
    (Why did they change the invocation number???)
* Thu Sep 23 1999 Cristian Gafton <gafton@redhat.com>
  - make clean in the test dir - don't ship any binaries at all.
* Tue Sep 14 1999 Preston Brown <pbrown@redhat.com>
  - fixed stripping of test programs.
* Mon Aug 30 1999 Preston Brown <pbrown@redhat.com>
  - removed 'flash' capability for xterm; see bug #2820 for details.
* Sat Aug 28 1999 Cristian Gafton <gafton@redhat.com>
  - add the resetall script from Marc Merlin <marc@merlins.org>
* Sat Aug 28 1999 Preston Brown <pbrown@redhat.com>
  - added iris-ansi-net as alias for iris-ansi (bug #2561)
* Sat Jul 31 1999 Michael K. Johnson <johnsonm@redhat.com>
  - added ncurses-intro.hmtl and hackguide.html to -devel package [bug #3929]
* Wed Apr 07 1999 Preston Brown <pbrown@redhat.com>
  - make sure ALL binaries are stripped (incl. test binaries)
* Thu Mar 25 1999 Preston Brown <pbrown@redhat.com>
  - made xterm terminfo stuff MUCH better.
* Sun Mar 21 1999 Cristian Gafton <gafton@redhat.com>
  - auto rebuild in the new build environment (release 16)
* Sat Mar 13 1999 Cristian Gafton <gafton@redhat.com>
  - fixed header for C++ compiles
* Fri Mar 12 1999 Jeff Johnson <jbj@redhat.com>
  - add terminfo entries for linux/linux-m on sparc (obsolete termfile_sparc).
* Thu Feb 18 1999 Cristian Gafton <gafton@redhat.com>
  - updated patchset from original site

There is more to the story than the changelog. I (and Florian LaRoche) discussed the ABI change at length with Cristian Gafton in April 1999. He did not mention symbol versioning at that time, making only these points (repetitively):

The discussion was inconclusive, aside from an addition to the ncurses FAQ.

Of course, changing the ABI early will not solve the problem of a packager that insists on treating development snapshots as releases. My practice (reinforced by packagers who were eager to be the first to provide a given release) was to change the ABI when preparing the actual release. I did that in the changes for 1999/02/20.

Gafton did not mention symbol versioning in our discussion. The only mention of versioning during 1999 in my email was in a discussion with Tim Mooney on how to embed the version number (and dependencies) on Digital Unix.

In any case, ncurses 5.0 differed from the 1999/02/23 snapshot by its support (through added functions and the change to the terminal data block) for user-defined terminal capabilities. The reason for the release did not change after February, though it took longer than expected (e.g., July 1999) because I lost contact with Florian (one response in mid-May). RMS sent mail in September, and — a month passing without getting a response either — replied on October 17:

    >     > I sent Florian mail.  I will wait a month for a response.
    >     >
    >     it's been a month.  have you gotten a response from Florian?
    >
    > No response yet.  I sent another message yesterday and I will wait
    > another week.

    it's another week.

Ok, I think we have to appoint you as the ncurses maintainer.

Would people please record Thomas E. Dickey <dickey@clark.net> as the
maintainer of ncurses, add him to the usual lists, and send him
maintain.text?

Followup

I did not ignore symbol versioning, but this was one of many areas to investigate while developing ncurses. For example:

my comments on the Cygwin mailing list (after the ncurses 5.0 release): 
On Sun, Nov 19, 2000 at 03:52:33PM -0500, Charles S. Wilson wrote:
> > (I thought all of that overly-precise specification was discarded
> > long ago since it's too cumbersome for practical use -- except of course on
> > win32...)
>
> yep. but we're on win32.

;-)

> > why isn't 'extern' part of the macro, btw?
>
> because I also had to edit the various scripts, like
> ncurses/tinfo/MKnames.awk which defines macros based on other macros:
>
>   #define DCL(it) NCURSES_EXPORT_VAR(IT) it[]

ok (I recall running into that one).

> Yes, I had to munge up lots of the scripts to get things right.  *That*
> is only part I'm worried about whether it still works on Linux &etc.  (I
> *think* it should be okay -- since, in this context,
> Linux=Cygwin_static_build)
>
> Scripts affected:

well, (of course) I'm interested.  It's a shame the solution has to be so
cumbersome.  I've been contemplating adapting the headers in a similar
way (not so regular ;-) for symbol versioning with glibc (but won't get into
that til next month, since there's too many things that I have to do ahead
of that).

or this, a little later:

Date: Sat, 28 Jul 2001 08:52:07 -0400
From: Thomas Dickey <dickey@herndon4.his.com>
To: Pradeep Padala <p_padala@yahoo.com>
Subject: Re: Correction of HOWTO

they're not simple bugs, but things that take some study.  (so it's
hard to "jump" in).  These are (offhand) the issues I have in mind:

        + report by user that resizeterm doesn't treat parent windows properly
          (says that there are cases where the resizing applies to parent
          and child windows out of order, causing a mismatch in window limits)

        + how to setup and use "symbol versioning" as in glibc (this is a
          Linux-only issue).

        + request by user (with partial, undocumented implementation) to add
          interface to support 'select' file-descriptors

        + review wide-character implementation, see what parts are missing,
          etc.  (I have in mind keeping the libncursesw configuration for
          a while - it can't be binary-compatible with the narrow version).

At the time, there was little information about symbol versioning — only a few comments here and there about it. I had an impression that it was done using compiler directives, which would (somehow) transform names into versioned symbols.

Now (in 2016), you can see something like documentation:

That (in particular the reference to Solaris) would have been interesting. But until around 2005, the interested reader could only find things like this:

That is, at the outset

That it was a feature of the loader and required a script was not apparent. Finding the loader manual is easier now than it was then. Finding the binutils source repository is easier as well. Its changelog does not mention Solaris, symbol versioning or loader scripts. The comment about Solaris appeared in the initial revision in May 1999, in the info version of the loader documentation, not in the manual page.

Today, you might find this information using a web search. Web searches in 2000 were less productive. The first mention I find in my email for Google is in 2000, a half-dozen times (out of several thousand items), versus fifty-five for AltaVista. Seeing the comment by Gafton in early 2000, I would have looked in the manual page, and not found anything. It took a while for a web-accessible version of the loader documentation to be available and be noticeable.

Gafton commented occasionally on symbol versioning, but without giving useful information, e.g.,

Likewise, Frequently Asked Questions about the GNU C Library (2003) which mentions symbol versioning provides no useful information on how to provide symbol versioning.

Much later (after 2010), when I began to research this area for ncurses, I noticed a couple of mailing list comments by Gafton to Ulrich Drepper. Those gave me the impression that the former was more of an advocate of the technique, possibly some influence on the paper which Drepper wrote about shared libraries in 2006. Here are some relevant links

Finding information long afterwards is usually easier than doing it at the time. At the time, you can record information which may be useful later, but putting the pieces together is a different matter.

Motivation

What (eventually) caused things to change were two changes that I made to ncurses starting early in 2005:

Extended colors

SVr4 curses supported 8 colors. ncurses improved on that, supporting 16 colors (since version 1.8.1 in November 1993, although none of the terminal descriptions used more than 8). Either way, the color pair was encoded in a 32-bit chtype value. That also held an 8-bit character, as well as bits to encode video attributes such as bold and underline.

Implementations of XPG4 Curses starting in the mid-1990s (led by DEC's OSF/1 with noticeable lagging by other vendors) used a structure cchar_t. While chtype and its interfaces were unchanged, the new data type had its own new interfaces. X/Open Curses Issue 4 (December 1994) and Version 2 (July 1996) described it:

cchar_t
A type that can reference a string of wide characters of up to an implementation- dependent length, a colour-pair, and zero or more attributes from the set of all attributes defined in this document. A null cchar_t object is an object that references a empty wide-character string. Arrays of cchar_t objects are terminated by a null cchar_t object.

In preparing to develop the wide-character ncurses library, I took that into account, but did not notice that the Unix vendors' cchar_t structure had added a field for color – and that it could support more than just 8 colors.

Interestingly enough, none of the Unix vendors used this extra space for colors, nor were there any terminals that could use the extra space. The extra space provided was apparently just a side-effect of moving the information away from the character value.

Eric Raymond had initially added cchar_t to ncurses (marking all of the related functions as missing) based on the December 1994 X/Open document. Here is the declaration in ncurses' C language header curses.h:

typedef chtype  attr_t;         /* ...must be at least as wide as chtype */
 
#ifdef XPG4_EXTENDED
#ifndef _WCHAR_T
typedef wchar_t unsigned long;
#endif /* _WCHAR_T */
#ifndef _WINT_T
typedef wint_t  long int;
#endif /* _WINT_T */
 
#define CCHARW_MAX      5
typedef 
{
    attr_t      attr;
    wchar_t     chars[CCHARW_MAX];
}
cchar_t;
#endif /* XPG4_EXTENDED */

Shortly after ncurses 4.2, I used that to simply move the color and video attributes to a copy of the data from chtype, leaving the 8-bit character code unused in the resulting attr_t type:

    + correct macros for wattr_set, wattr_get, separate wattrset macro from
      these to preserve behavior that allows attributes to be combined with
      color pair numbers.

Although ncurses supported 16 colors (since version 1.8.1 in November 1993), it was not until early 1995 when Eric Raymond copied SCO's terminfo file (in particular, the hp+color building block) that anything used 16 colors. That building block uses nonstandard escape sequences (given a choice between Tektronix and HP, the standards committees chose the former). Later, in May 1997, when I incorporated the IBM aixterm escape sequences into xterm there was a standards-based terminfo description using 16 colors: xterm-16color. The Linux console has a 16-color palette, but can use those only by combining the 8 ANSI colors with the bold attribute.

The xterm-16color entry was not widely used. But late in 1999, I added xterm-256color (see FAQ Why not make "xterm" equated to "xterm-256color"?).

That got more attention. When I started development early in 2005 to support extended colors (i.e., xterm's 256-color feature), I added a _color member to the cchar_t structure.

Extended mouse

Eric Raymond added definitions to the curses.h header for the mouse interface in September 1995. The initial version used six bits per button (four buttons), plus four bits for modifiers (control, shift, alt) and reporting position. Using that scheme, five buttons would require 34 bits.

At that moment, it was not a problem. But

Compatibility vs ABI

Following ncurses 5.0, I had been able to guarantee (or at least promise) good compatibility of the binary applications with the library. These features changed that:

Some packagers were flexible about this, e.g., Charles Wilson added the mouse feature for Cygwin early in 2009. I discussed that, and the extended colors with him several times.

Debian was less flexible, though it was clearly to their advantage:

The Debian bug report mentioned (#230990) gives a good summary of the position taken by Debian regarding these extensions: they involved a change to the application binary interface (ABI), and would be a lot of work to migrate to that new interface. His attempts had no effect (see the thread Headsup: ncurses soname bump 5 to 6 in September 2008 to get an appreciation for the non-technical obstacles presented).

Baumann was not the first Debian developer to mention symbol versioning. The issue was relevant to further bug reports, in particular how to manage versions of the low-level terminfo library (tinfo). I added a configure option --with-termlib late in 1997, but it took several years for the packagers to adopt it. In 2011, more than one Debian bug report was related to this issue, e.g.,

Debian #631592 dealt directly with this topic, but nothing came of that for some time. An attachment shows that the packager's proposed changes to construct a termlib involved moving the related symbol definitions to the new library's configuration. Here is a small portion of the packager's patch to illustrate:

...
diff -Nru ncurses-5.9/debian/libncurses5.symbols ncurses-5.9/debian/libncurses5.symbols
--- ncurses-5.9/debian/libncurses5.symbols      2011-03-03 22:25:04.000000000 +0100
 +++ ncurses-5.9/debian/libncurses5.symbols      2011-08-25 15:41:06.000000000 +0200
 @@ -176,74 +176,20 @@
   top_row@Base 5.5-5~
  unpost_menu@Base 5.5-5~
 libncurses.so.5 #PACKAGE# #MINVER#
- BC@Base 5.5-5~
   COLORS@Base 5.5-5~
  COLOR_PAIR@Base 5.5-5~
  COLOR_PAIRS@Base 5.5-5~
- COLS@Base 5.5-5~
   ESCDELAY@Base 5.5-5~
- LINES@Base 5.5-5~
   PAIR_NUMBER@Base 5.5-5~
- PC@Base 5.5-5~
 - SP@Base 5.5-5~
 - TABSIZE@Base 5.5-5~
 - UP@Base 5.5-5~
 - (optional)_nc_access@Base 5.5-5~
 - (optional)_nc_add_to_try@Base 5.5-5~
...
 diff -Nru ncurses-5.9/debian/libtinfo5.symbols ncurses-5.9/debian/libtinfo5.symbols
--- ncurses-5.9/debian/libtinfo5.symbols        1970-01-01 01:00:00.000000000 +0100
 +++ ncurses-5.9/debian/libtinfo5.symbols        2011-08-25 15:23:05.000000000 +0200
 @@ -0,0 +1,179 @@
 +libtinfo.so.5 #PACKAGE# #MINVER#
 + BC@Base 5.6+20070908
 + COLS@Base 5.6+20070908
 + LINES@Base 5.6+20070908
 + PC@Base 5.6+20070908
 + SP@Base 5.6+20070908
 + TABSIZE@Base 5.6+20070908
 + UP@Base 5.6+20070908
 + (optional)_nc_access@Base 5.6+20070908
 + (optional)_nc_add_to_try@Base 5.6+20070908
 ...

If those symbols had been versioned, then there would be something more informative than “Base” to tell others which library (and version) is expected to provide that symbol. However, as I pointed out, there was no concrete plan of action for me to do in the upstream ncurses;

My point of view here is that versioned symbols are essentially a 
Linux-specific feature which for quite a while was at best poorly 
documented.  When it's been discussed before, I've suggested that people 
interested in the feature might send a patch, but recall being told to not 
bother - it's their problem, not mine.

For instance, it was only later that I learned (without advice from the Debian developers) what that “Base” in the patch referred to. The documentation really was (and is) that poor.

Sven Joachim's comment in January 2012 was the one that got me to thinking how it could be done:

Unfortunately, switching SONAME seems to be out of the question since
even after the introduction of libtinfo5 we still have dozens of
libraries linking against libncurses.  My feeble attempt¹ to fix a few
of them has not changed anything yet.

The only safe way to do this seems to be to introduce versioned symbols
in ncurses, and I don't know at all how to do that in a way which

a) is acceptable for upstream;

b) works correctly for all the bazillion configuration options that
   change the ABI, for instance "--enable-widec", "--enable-ext-colors",
      "--enable-ext-mouse", "--with-termlib=...", "--with-ticlib=..." etc.

Process

Sven Joachim was referring to the configure script's large number of optional features (more on that later), but there are several pieces needed to make a useful symbol versioning scheme:

Analyze

First off, there were 126 configuration options in 2015 when I began developing a solution. But only a few give different symbols. Most turn features on and off.

Symbol versioning lets the developer choose to hide certain symbols (preventing applications from linking to private entrypoints), and to label all symbols (preventing applications from loading the wrong library at runtime).

The available symbols for hiding or labeling are determined by the configuration options. There are only a few basic configurations of interest:

Variant ABI=5 ABI=6
  ncurses5  
pthread ncursest5  
widec ncursesw5 ncursesw6
pthread + widec ncursestw5 ncursestw6

There are a few other options (such as broken-linker) which change symbols, but those are not relevant to the analysis because they are platform-specific.

Some of the symbols in the ncurses library are intended to be private. They are visible because the library is made up of many files, which share these private symbols. As a rule, these private symbols — beginning with “_nc_” — are not part of the application programming interface (API) but may be part of the application binary interface (ABI):

To limit the analysis:

For a given configuration such as “ncurses5” an entrypoint could be further configured using opaque or broken-linker, but a given library could supply only one of the possible (whether prefixed with “_nc_” or not) symbols.

Following this line of reasoning, I decided to

Starting in November 2014, I wrote scripts to build configurations (ncu-symbols) and collect data (ncu-mapsyms, (needed-syms). The scripts know about the different library names, and create (or update) the data files needed to label the library symbols.

Construct

Constructing the symbol version information and applying it was the first area of development. I looked for examples of libraries using symbol versioning, and came across a trivial example in whiptail:

Format: 1.7
Date: Thu,  8 Jul 2004 20:49:22 +0100
Source: newt
Binary: libnewt-dev libnewt-pic libnewt0.51 newt-tcl whiptail python-newt
Architecture: source i386
Version: 0.51.6-7
Distribution: unstable
Urgency: low
Maintainer: Alastair McKinstry <mckinstry@debian.org>
Changed-By: Alastair McKinstry <mckinstry@debian.org>
Description:
 libnewt-dev - Developer's toolkit for newt windowing library
 libnewt-pic - Not Erik's Windowing Toolkit, shared library subset kit
 libnewt0.51 - Not Erik's Windowing Toolkit - text mode windowing with slang
 newt-tcl   - A newt module for Tcl
 python-newt - A NEWT module for Python
 whiptail   - Displays user-friendly dialog boxes from shell scripts
Closes: 257807
Changes:
 newt (0.51.6-7) unstable; urgency=low
 .
   * Fix display problem with 'hidden' checkbox entries in whiptail being
     shown. Closes: #257807.
   * Used versioned symbols in libnewt.

On investigation, I found that the packager had simply marked all of the exported symbols which had the same prefix using a short data file:

NEWT_0.52 {
        global: newt*;
                _newt_wstrlen;
        local: *;
};
and applied the data using a linker option: 
SHLIBFLAGS= -Wl,-O1 -Wl,--version-script,newt.0.52.ver

About a year later it was incorporated in the upstream developer's sources.

Seeing that change was helpful:

I chose to not follow the simple “mark all” approach, because it would not be useful to pretend that all of the symbols were defined, say, in 2015. Instead, I made my scripts determine the first release version at which a symbol was added, and use that for the label. In any case, newt's developer also had to determine what versions added symbols after that point (see current version information).

I also develop two other applications which provide development libraries: Cdk and dialog. They share the same set of configure script macros:

I made similar changes in Cdk and dialog to support symbol versioning.

I created the files for Cdk manually, using my exports and externs scripts to get the symbol names, and comparing successive releases. That made a useful test-case for the makefile changes. Developing ncu-mapsyms (and ncu-symbols) took much longer since

While ncu-mapsyms is written for the ncurses libraries, it is reusable for the dialog library, which has none of these special cases.

Iterate

There are (as of December 2016) 944 different symbols listed in the map/sym files for ncurses. Starting with ncurses 5.0, it takes a couple of hours to generate the map/sym files. There are of course four configurations. But there are the variations due to the configure options. The ncu-mapsyms script records the options used within each release, allowing one to count the number of builds needed. Here is a summary of the number of releases and builds done for each configuration

Configuration Releases Builds
ncurses 12 44
ncursest 5 37
ncursestw 5 37
ncursesw 11 40

i.e., a total of 158 builds.

Produce

Since I began developing the symbol versioning scripts in 2014, I have updated the map/sym files when adding symbols. Rather than use the patch-date for each change, my practice is to use "current" for the development of new versioned symbols, and as part of the preparation for a new release then change the "current" to the release date.

Here are links to the map/sym files for each library:

Map file Sym file
cdk.map cdk.map
dialog.map dialog.sym
ncurses.map ncurses.sym
ncursest.map ncursest.sym
ncursestw.map ncursestw.sym
ncursesw.map ncursesw.sym

Maintain

While developing these map/sym files, I produced them directly using the scripts. Although doing more than a hundred builds for ncurses to regenerate the files took hours, I used them in this fashion for a few years with both Debian 5 and 6, and may have continued indefinitely except that the builds took their toll on the build machine, whose disk failed in 2019. I considered doing this on my Debian 8 machine, but found this would require additional work to build ncurses 5.0 on that platform. Also (because the scripts rely upon creating auxiliary files), the amount of work increases with time.

It takes only a few minutes with a text editor to add a new symbol. The reason for the scripts is that they made it possible to produce the set of 8 files with 944 symbols, digesting nearly 20 years of development.

Report

You can use objdump to show the symbol version information, e.g.,

objdump -axhT foo

Here are a few samples:

Here is a more useful script:

#!/bin/sh
# $­Id: list-versioned-symbols,v 1.6 2018/03/02 16:03:56 tom Exp $
unset LANG
unset LC_ALL
unset LC_CTYPE
for name in "$@"
do
        file "$name"
        [ -L "$name" ] && continue
        [ -f "$name" ] || continue
        objdump -CT "$name" | awk '
function after(text) {
        value = substr($0, index($0,text) + length(text));
        sub("^[         ]*[0-9a-f][0-9a-f]*[    ][      ]*", "", value);
        if ( split(value, values) == 2 )
                value = values[2] "@@" values[1];
        return value;
}
/DF [ ]*\*UND\*/        { printf "U %s\n", after("*UND*"); next; }
/DO [ ]*\*UND\*/        { printf "U %s\n", after("*UND*"); next; }
/DF [ ]*\.text/         { printf "F %s\n", after(".text"); next; }
/DO [ ]*\.data\.rel\.ro/ { printf "R %s\n", after(".data.rel.ro"); next; }
/DO [ ]*\.data/         { printf "D %s\n", after(".data"); next; }
        { next; }
        ' | sort -u -k2
done

Problems

Symbol versioning does not solve all problems related to matching a program with a library. It is a way to label symbols in a given version of a library:

Not all systems support versioned symbols. The page Host/Target specific installation notes for GCC has these comments:

The HP dynamic loader does not support GNU symbol versioning, so symbol versioning is not supported. It may be necessary to disable symbol versioning with --disable-symvers when using GNU ld.

and

To enable symbol versioning in ‘libstdc++’ with the Solaris linker, you need to have any version of GNU c++filt, which is part of GNU binutils. ‘libstdc++’ symbol versioning will be disabled if no appropriate version is found. Solaris c++filt from the Solaris Studio compilers does not work.

I have not ported the versioned symbols for ncurses to Solaris. Its linker expects that all of the symbols named in the “.map” files exist in each shared library. That is not true for the ncurses files; they list symbols which might be in a particular set of libraries, i.e., ncurses, tinfo, form, menu and panel. The files could be adapted for Solaris by a script which

Interestingly, FreeBSD supports versioned symbols; I was able to build ncurses using mapfiles on my FreeBSD 8 (and later) machines. That was because Daniel Eischen did the necessary work in 2006. Likewise, NetBSD and OpenBSD work for this configuration.

Gafton's remark about symbol versioning was probably misdirected. Symbol versions do not directly relate to data types; they relate to functions and data instances. I had changed the TERMTYPE data type without any need for changing the functions that used the corresponding pointers. Using symbol versioning to “fix;” the problem would mean that every function that used the changed type should get a new version — whether or not it used the structure member. Bumping the ABI (as I did for ncurses 5.0 and 6.0) is simpler and more direct.

While Debian uses versioned symbols, they are not currently as popular with Red Hat. While glibc in Fedora 38 uses them, other libraries may not. I have noticed a few bug reports, including this one:

Bug 1875587 - ncurses: Please enable symbol versioning

Here are a few interesting pages for further reading: