http://invisible-island.net/
Copyright © 1997-2021,2022 by Thomas E. Dickey


Synopsis

As of August 2014, I had nearly 2000 scripts in my bin-directory, totalling more than 100,000 lines. The list continues to grow (see screenshot). As of May 2021, it has grown to 2300 scripts, 140,000 lines. Most are POSIX shell, most of the remainder are Perl. Those total about the same number of lines:

Metrics by language:
                                Files    Lines    Chars    Notes
bash                              130     3192    70802      495
csh                                 3       27      766        6
expectk                             3     2650    80820     1047
gawk                                1     3575   131616     1449
ksh                                 1        8       87        1
mawk                                1       39     1032        2
perl                              243    69001  1980511     6841
python                              7      633    19592       24
ruby                                2     1190    33235      121
sh                               1952    60571  1451262     9011
tcl                                 1       16      323        3
unknown                            15       38     3990       11
                             -------- -------- -------- --------
Totals                           2359   140940  3774036    19011
Average                                     60     1600        8
 

Those scripts are distinct from scripts which I bundle with various programs. A few are modified versions of scripts written by others. Some of the scripts may be of interest to others.

General-purpose

There are several general-purpose scripts:

archive

Construct a gzip'd (or compress'd) tar file.
Run as unarchive, it reverses the process.

archive was originally written by Charles Van Gale, a system administrator at the Software Productivity Consortium in 1989 or 1990. I extended it to make the resulting tarball have a timestamp matching the newest file (using cpd), as well as using gzip or bzip2 compression:

the script...

diff-patch

A script that combines diffstat and makepatch.

exports

script to show which symbols are exported from a collection of object files

externs

display all external symbols used by a collection of object files

make-out

run "make", redirecting output to make.out.
If linked or renamed to make-log, it writes to make.log.

makepatch

script (originally by Johan Vromans) used for making patch-files.
I have modified it to sort the old/new lists to avoid confusing '/' with filename stuff.

path

like which, but provides the dual use of showing the path and showing details about the arguments. The conflict program is related.

run-out

run a command, redirecting output to run.out.
If linked or renamed to run-log, it writes to run.log.

Build-Scripts

The original intent of this page was to provide sample scripts for building programs. Since then, I have written scripts for routine builds of the programs which I develop, as well as for managing releases and updating this website.

These are some of the scripts that I use in configuring, building and saving the logfiles from building programs on multiple platforms. A complete set of configure- and make-logs is useful for bug reporting.

Build-framework

Each program has its own set of build-scripts. Sample scripts for the larger programs are included in the download. I use scripts named for the various programs (XXX in the following sections).

Though the oldest of these scripts (for saving build-logs from dbmalloc) dates back to 1995, they have evolved in stages. Initially, I would do test-builds and, after verifying the results, possibly discard the logs. However, for several years I have collected the logs, adding them to my source repository to compare.

As of January 2016, I had 5488 log-files (with 383 marked to denote systems where I could no longer build). My collection has grown to 10,717 files as of May 2021, with 61% marked as obsolete. Some builds produce multiple log-files, and I have build-scripts for about 50 programs. There are also a lot of machines to build with. The log files are checked into RCS, of course, so that I can compare successive builds and fix problems.

build-XXX

Build the program (within its working directory), with the most common configuration(s). As part of the process, install and uninstall the program (to a location in the build-tree).

build-all-XXX

Repeat the build-XXX script for a given program, using different compilers and/or warning levels. Depending on the platform, this may execute other scripts to build “.deb” (Debian), RPM packages, NSIS installers or Solaris packages.

build-opt-XXX

Build a given program, use the options from the corresponding all-opts-XXX script, one at a time, to build the program.

clean-XXX

These remove the files from a build within the given program's working directory.

save-XXX

I use these scripts to save logfiles after doing a build. They all assume that there is a parallel directory of the same name, e.g., ../save-vile, next to the current directory. The current directory is the top of the build tree.

In particular (making them useful for bug-reports), they save the results from the configure script, along with log-files that are created using the cfg-XXX and build-XXX scripts for the corresponding programs.

Where possible, I parameterized these scripts to make a reusable core. These scripts provide the reusable parts:

Those build-scripts are part of a larger set, which includes

As of May 2021, here is a summary of these build-scripts:

Metrics by phase:
                                Files    Lines    Chars    Notes
Bug report                         31      547    15582      155
Build phase (PART)                397    20620   470933     1570
Build support (HELP)              298    28061   752788     3305
Configure                         201     3421    76007      479
Connections                        99      474    13937      205
Development                        83    14238   421288     1549
Environment                       143     1776    39342      355
File-movement                      54      731    14487       84
Metrics                            19     3253    92463      349
Miscellaneous                     498    31202   882008     5577
Source-control                    112     7795   196881      905
Testing                           424    28822   798320     4478
                             -------- -------- -------- --------
Totals                           2359   140940  3774036    19011
Average                                     60     1600        8

Metrics for build phases:
                                Files    Lines    Chars    Notes
PART: build-*                      48     3080    57744      131
PART: build-all-*                  48      430     6268       96
PART: build-my-*                    3      348     7367       15
PART: bump-*                       36     7606   191397      153
PART: clean-*                      48      725    13702      111
PART: clobber-*                     9       46     1156       18
PART: release-*                    54     3531    83176      414
PART: release-*-docs               24     1265    27447      117
PART: save-*                       50     1409    28472      329
PART: tag-*                        30     1407    39696       82
PART: upload-*                     38      582    11296       77
PART: upload-*-docs                 9      191     3212       27
                             -------- -------- -------- --------
Totals                            397    20620   470933     1570
Average                                     52     1186        4

The matrix of scripts is not completely filled in; that depends on the program. Most are programs that I maintain. Some (such as PDCurses) are built because I use those in programs which I maintain, but have no reliable package provider:

Build-phases:
 build-*
 ^  build-all-*
 ---^  build-my-*
 ------^  bump-*
 ---------^  clean-*
 ------------^  clobber-*
 ---------------^  release-*
 ------------------^  release-*-docs
 ---------------------^  save-*
 ------------------------^  tag-*
 ---------------------------^  upload-*
 ------------------------------^  upload-*-docs
 ---------------------------------^  
 *  *  -  -  *  -  *  -  *  -  -  -   AdaCurses
 *  *  -  *  *  -  *  -  *  -  -  -   PDCurses
 *  *  -  -  *  -  *  -  *  -  -  -   XawPlus
 *  *  -  -  *  -  *  -  *  -  -  -   ac213
 *  *  -  *  *  -  *  -  *  *  *  -   ac252
 *  *  -  *  *  -  *  *  *  *  *  -   add
 *  *  -  -  *  -  *  -  *  -  *  -   atac
 *  *  -  *  *  -  *  -  *  *  *  -   bcpp
 *  *  -  *  *  -  *  -  *  *  *  -   blame
 -  -  -  -  -  -  *  -  -  -  *  -   buildscripts
 *  *  -  *  *  *  *  *  *  *  *  *   byacc
 *  *  -  *  *  -  *  *  *  -  *  -   c_count
 *  *  -  *  *  -  *  *  *  *  *  *   cdk
 *  *  -  *  *  -  *  -  *  *  *  -   cdk-perl
 *  *  -  *  *  -  *  *  *  *  *  -   cm_tools
 *  *  -  *  *  *  *  *  *  *  *  -   conflict
 *  *  -  *  *  -  *  *  *  *  *  -   copyrite
 *  *  -  *  *  *  *  *  *  -  *  -   cproto
 -  -  -  -  -  -  -  -  *  -  -  -   dbmalloc
 *  *  -  *  *  -  *  *  *  *  *  -   ded
 *  *  -  *  *  *  *  *  *  *  *  *   dialog
 *  *  -  *  *  -  *  *  *  -  *  *   diffstat
 -  -  -  -  -  -  *  -  -  -  *  -   gpl-scripts
 *  *  -  *  *  *  *  -  *  *  *  -   indent
 *  *  -  -  *  -  *  -  *  -  -  -   libX11
 *  *  -  *  *  -  *  -  *  -  -  -   libXaw
 *  *  -  -  *  -  *  -  *  -  -  -   libXaw3d
 *  *  -  *  *  -  *  *  *  -  -  -   libXcursor
 *  *  -  *  *  -  *  -  *  -  -  -   libXft
 *  *  -  *  *  -  *  -  *  -  -  -   libXpm
 *  *  -  *  *  -  *  *  *  -  -  -   libXt
 *  *  -  *  *  -  *  *  *  *  *  -   luit
 *  *  *  *  *  *  *  *  *  *  *  *   lynx
 -  -  -  -  -  -  *  -  -  -  *  -   mapsym-scripts
 *  *  -  *  *  *  *  *  *  *  *  *   mawk
 -  -  -  -  -  -  *  -  -  -  *  -   misc-scripts
 *  *  -  *  *  -  *  -  *  *  *  -   misc_tools
 -  -  -  -  -  -  *  -  -  *  *  -   my-autoconf
 *  *  *  *  *  *  *  *  *  *  *  *   ncurses
 *  *  -  -  *  -  *  -  *  -  -  -   ncurses-examples
 *  *  -  *  *  -  *  -  *  -  *  -   neXtaw
 *  *  -  -  *  -  *  -  *  -  -  -   prcs
 *  *  -  *  *  -  *  -  *  *  *  -   rcshist
 *  *  -  *  *  *  *  *  *  *  *  -   reflex
 *  *  -  *  *  -  *  *  *  *  *  -   sccs_tools
 *  *  -  *  *  -  *  *  *  *  *  -   tack
 *  *  -  -  *  -  *  *  *  *  *  -   tctest
 *  *  -  *  *  -  *  -  *  *  *  -   td_lib
 -  -  -  -  -  -  -  -  *  -  -  -   tdlib
 -  -  -  -  -  -  *  -  -  *  *  -   terminfo
 *  *  -  -  *  -  *  -  *  -  -  -   tin
 -  -  -  -  -  -  -  -  -  *  -  -   tin-beta
 *  *  -  -  *  -  *  -  *  -  -  -   twm
 *  *  *  *  *  -  *  *  *  *  *  *   vile
 *  *  -  *  *  -  *  *  *  *  *  -   vttest
 *  *  -  -  *  -  *  -  *  -  -  -   xfonts-encodings
 *  *  -  *  *  -  *  *  *  *  *  *   xterm

My script which reports this information (report-build-scripts) matches program names against a pattern. That works well enough where the scripts are named consistently. The support scripts are less regular: some follow a pattern, while others do not. The script uses a table of exceptions to help with its categorization of scripts.

Compiler wrappers

Along with the build-scripts, I use wrappers for gcc to turn on compiler warnings, as well as scripts for other compilers. They are included in the build-scripts, since that is where they are most used.

gcc-normal

This provides what should be the minimal options for gcc to ensure that C functions are prototyped. I added it in 1995 at the same time that I added gcc-strict. Before that, I was likely to just edit the makefile, not thinking that it would be simpler to make a wrapper script.

the script...

gcc-normal-64

This script adds gcc options to compile for 64-bits. When I added it in 2009, it was still common to have compilers on a nominally 64-bit platform such as Solaris default to compiling 32-bit code.

gcc-strict

I made this script, incorporating some posting by Fergus Henderson, e.g., this one to gnu.gcc.help from 1994.

the script...

gcc-stricter

Later, I was told about the gcc "fortify" option. Rather than modify the options in gcc-strict, I added this script.

the script...

Configuration scripts

Configure script wrappers containing configure options for specific programs are (usually) named for those programs, e.g.,

I am able to cross-compile to MinGW on several machines. There is no consensus on what directories those cross-compilers use. These scripts help:

Reporting Problems

If you have a machine that I've not ported to, and have problems building one of the programs that I maintain, I will require the relevant information:

A uuencoded or mime'd gzip'd/tar file is preferred, because the logfiles can be awkward to email. Mailing uncompressed log-files also has the drawback that they may be rejected on mailing lists.

Special-Purpose

These scripts are for more specialized uses.

Regress

remake an object file, reporting differences. The reference (first compile) is saved in the OBJ subdirectory.

I use this script when making structural changes to a C program that “should not” change its content, e.g., converting from K&R C or changing comments/formatting.

RegressMan

Like Regress, I use this for checking changes to manual page formatting.

check-manpage

Scan a directory tree, looking for nroff (man/ms) files, to check their syntax as well as verify that their macros are consistent.

If checknr were portable and handled manpages, it would be useful. This started as a wrapper for groff's checks, but has grown a lot.

classpath

Like path, show information about the user's CLASSPATH environment variable (which controls linking and loading of Java classes.

If no arguments given, this script displays the names of the directories in the class-path. If argument is given, it is assumed to be either a class name (e.g., "String"), or a jar-file.

All matching names are displayed; the first is the one that will be loaded.

Options (arguments beginning with "-") are passed to 'ls'.

compare-terminfo

Use ncurses' infocmp to compare two terminfo files. Its output is close, but is not sorted in a predictable manner that would allow diff'ing output from different comparisons.

count-nroff

Scan directory-tree or file(s) counting nroff files, adjusting for comments.

diffstat2css

Process one or more patch files, using diffstat's -C (color) option to markup the files (with SGR). We want to modify those so they all use reverse-video (since colored text is too hard to read), and produce html. To make the result a little more flexible, use the diffstat.css stylesheet.

The script expects one of two parameter conventions:

  1. patch-file, simply process with diffstat

  2. two directories as parameters, to pass those to diffstat using -S/-D

grep-bad-ifdefs

Search for patterns that (might be) broken defines

find-xterm-fonts

Find the X bitmap font-files which are used in xterm/uxterm app-defaults files, and for some systems (where the packaging system is useful) followup with the packages containing those files.

See my answer using this script.

grep-cast

Search for patterns that (might be) casts.

grep-date

Search for patterns that might be dates.

grep-ifdefs

Search for patterns that (might be) ifdefs.

grep-knr

Search for patterns that (might be) K&R function declarations.

grep-numbers

Search for patterns that (might be) numbers.

indent-html

Run tidy to indent one or more HTML files, optionally showing the differences made (-v) and with dry-run (-n) feature.

indent-perl

Run perltidy to indent one or more Perl files, optionally showing the differences made (-v) and with dry-run (-n) feature.

indent-tty

Format the output of stty -a into a single column, allowing diff to show just the single feature(s) which may change.

ldd-path

Edit the output of ldd for the given parameters, yielding only the absolute pathnames.

lessclose.sh

I use this script with less to remove temporary files.

lessopen.sh

I use this script with less to open special files.

libpath

Displays information about the user's LD_LIBRARY_PATH environment variable (which controls linking and loading of dynamic libraries.

If no arguments given, this script displays the names of the directories in the library-path. If argument is given, it is assumed to be a library name (e.g., "c" for -lc). All matching names are displayed; the first is the one that will be loaded.

Options (arguments beginning with "-") are passed to 'ls'.

manifest

Construct a manifest from the RCS description lines for the selected (symbolic) version. The script assumes that the following files are automatically generated from scripts:

MANIFEST – from this script
configure – from autoconf (configure.in, aclocal.m4)
config.h – from configure

Other files are all managed via RCS, and are present in a release only if they have had symbolic revisions added for that release.

nm_cmp

Compares the elements of an archive file, showing which are different or missing from one archive.

no-local

Run command without /usr/local in environment.

nodot

Strip leading "." from PATH to execute a command.

noenv

Strip environment, e.g., as if running in cron, and run a command.

nolocale

Strip locale environment-variables, i.e., leaving only POSIX, and run a command.

noterm

Strip all terminfo/termcap variables from environment, run command.

remake

Remove makefile target (e.g., an object file) and run make to recreate it.

report-linux-consolefonts

For each "psf" font listed in /usr/share/consolefonts, make a report showing which ISO-8859-x variants are covered, as well as the line-drawing used by ncurses.

report-terminfo

Analyze one or more terminfo descriptions to get some useful statistics. This assumes that you have tic, toe and infocmp.

rpm-deps

For a given package, ask the question: what packages satisfy its dependencies, and is there a “provides” which can be used as an alias for those.

script2log

Strips ANSI escape sequences and reduces backspace/overstrikes.
The script uses POSIX sed, which means that only BREs (Basic Regular Expressions) are permitted.
I have tested this with FreeBSD 10, OpenBSD 5.3, Solaris 10 as well as Debian 6 and later:

Here are two examples, using "cat -v" to make the nonprinting characters visible:

show-dircolors

Read the database printed by “dircolors -p” and show the different patterns with the indicated color by (as done by GNU ls) using hardcoded SGR sequences.

The output can be viewed with “less -R”.

See my answer using this script.

See these alternatives which use the LS_COLORS variable.

show-linux-consolefonts

Successively load and display console fonts using either showconsolefonts or a grid for the different ISO-8859-x locale settings. Like the ncurses test-programs, provide for reading a key for single-stepping or proceeding with a time-delay. It does not use the perl Curses module, since that would interfere with sending raw bytes to the screen.

sort-sources

Sort the list of parameters as if they are filenames containing a version. I use this to sort the names of tar-balls for my programs in builds as well as website maintenance.

By the way, I noticed a comment in remove the low version number of file indicating that GNU ls has a similar feature. However, the rules for parsing the version are a little different, and in my script the version number follows the program name.

touch-dirs

Recursively touch (change timestamps) on directories to reflect the newest file inside the directory. I use this for tidying up source-trees, e.g., as in the “-t” option of archive (which uses cpd).

the script...

Here are a few links which I found when looking for comparable scripts or programs:

Oddly, most applications in this area simply update the directories (and files) to the current date.

with-locale

Run a command with the given locale. The first parameter is the locale to use. The remaining parameters are the command to run.

Git(hub) Scripts

I wrote these scripts to provide a way to export information from my RCS archives to Git, starting in July 2016 (see Git exports):

bundle2github

Upload a Git bundle to github in my "testing" repository. That can be renamed, to provide a "readonly" snapshot.

compare-manifests

For a given project, use release2git -R to find its labeled releases, and generate a manifest-file for each of these. Discount the first line, and compare successive files to identify changes from one release to the next.

gimme-git

For the given directories, fetch current Git. The script also calls git-utimes to set file timestamps to correspond to their last commit-date.

Although Git is “distributed” that ignores performance. For useful results, you need a copy of each repository on a local machine (just like RCS!). I use this script with about 200 repositories that are related to my work.

git-stats

Make a summary report for files stored in Git, showing the number of commits, and amount of change for each committer.

This script uses the Perl module Text::CharWidth, because author names in Git are in UTF-8 (not US-ASCII). Interestingly enough, Perl does not appear to have an equivalent to wcwidth in its main distribution.

Here is part of the script's output (20 of 513 lines) for the gnome-settings-daemon:

Author                              Files  Commits  Inserts  Deletes
A S Alam                                1       22    10784     7787
Aaron Plattner                          2        2      205       10
Abderrahim Kitouni                      1        1        2        1
Abduxukur Abdurixit                     1        2     3074     2333
Adam Dingle                             1        1        0        1
Adam Jackson                            1        1       18        0
Adam Matoušek                           1        1     1341     1657
Adel Gadllah                            2        1       22        7
...
vasudeven                               1        2      225      141
Žygimantas Beručka                      1        2     2268     1837
Γιάννης Κουτσούκος                      1        1      652      741
Γιώργος Στεφανάνης                      1        1      620      188
Марко Костић                            1        4      769      859
Милош Поповић                           2        4     1814     1489
Мирослав Николић                        2       22    27259    23656
Роман Донченко                          2        3       16       15
િિશાલ ભલાણી                             1        1      623      788
甘露(Gan  Lu)                           1        1      918     1035
TOTAL                                 990     6507  1550730  1162857

Your browser probably does not render this properly (expect problems with fixed-pitch Unicode), but it displays correctly in xterm:

git-stats – screenshot in xterm

git-utimes

Update file timestamps to their last commit date/time. This is modified from the version written by Tom Christiansen: the initial import of some Git's doesn't fit pattern.

rcs2git

Initial script, using my modifications to rcs-fast-export to export snapshots of mawk.

release2git

Rather than attempt to export an RCS archive to Git using rcs-fast-export.rb (and have to do this repeatedly), analyze an RCS archive and apply release snapshots to a Git-ball incrementally. That loses most metadata (such as the individual file timestamps, comments, etc), but should work even with my larger archives.

resync-git

I use this for merging changes from a separate upstream repository, into my repository's master branch.

tag-cutoff

Use release2git to obtain a list of versions and files for a given cutoff date, and apply a label to that based on the cutoff date.

Symbol-versioning

I wrote these scripts to support generating map- and sym-files for ncurses6 symbol versioning:

analyze-curses-symbols

Count symbol usage for the given executables, and summarize their usage with categories for these types:

compare-mapsyms

Compare two “.map” files, generate a report showing the number of symbols added, deleted or renamed. A “renamed” symbol is one whose scope differs or is in a different version (the numeric plus “.” part on the end of the label).

Optionally show only the number of symbols changed.

dlg-symbols

Extract symbols from a given patch-date for dialog.

needed-syms

Inspect the given (build-)tree for libraries and programs, determining which symbols exported by the libraries are needed by other libraries or the found programs. Print a list showing the symbols, where they are declared, and their needed status (global or local).

ncu-mapsyms

Using needed-syms, construct “.map” and “.sym” files for the current build-tree's configuration of ncurses.

The “.sym” files are trivial – a comment header telling what options were used to generate the configuration, and a sorted list of symbols.

The “.map” files require special effort, because they are constructed as a series of version-nodes. So... to generate a new one requires us to read the existing “.map” file to relate the symbols' version and scope to the previous version.

To allow for symbols being in termlib/ticlib/ncurses, always generate the names as if they were in the separate library. Doing this allows splitting the map-file into parts and (possibly) making it work with Solaris, which does not accept a map-file declaring a symbol not in the library.

ncu-symbols

Successively build ncurses for release-dates with several combinations of configure options and merge the result to generate usable map- and sym-files.

report-mapsyms

Read one or more “.map” files, generate a report showing

Optionally show only the number of symbols in each scope.

Other Scripts

Here are some scripts written by others which I have modified.
In particular, those using a restrictive license (e.g., GPL) are not bundled with the others:

altchars

This is a slightly-modified copy of altchars by John H. Dubois III (from here). The original script carries forward a documentation error from AT&T, for the lantern pseudo-graphics character which was fixed in ncurses in 1997 as a result of work on lynx.

gnathtml

Starting in 2000 (for ncurses Ada95 binding), I have modified this script to make the generated HTML validate, as well as to use CSS. (The script is originally from gnat, but unmaintained).

linklint

I use linklint, but (see this) it has been unmaintained since around 2001. For example, see the Debian bug-reporting system. As of early 2016, I have fixed most of the problems which affect me:

man2html

This is an improved version of Earl Hood's man2html Perl script (see my expanded page on the topic).

rcs2log

I modified this script in 2003 (it was originally from emacs, but unmaintained):

unrpm

This is a wrapper for rpm2cpio which makes it easier to use. There actually is more than one script named “unrpm” — this is the best-known, and was seen in a posting to the fedora-legacy mailing list in 2004, stating that it came from the fedora-rpmdevtools package. According to the package's changelog, it was removed in 2005.

xcutselprint

This is useful for xterm testing, found in Debian #588785 (from http://lindi.iki.fi/lindi/xcutselprint).

Changes

The scripts on this page are a small subset of my collection. I use (predictably) another script to put together the tarballs as well as creating the changelogs on this page (with the help of rcs2log).

The "RCS_BASE" comment denotes the initial check-in using my RCS wrappers. Those preserve the modification time of the file, which in some cases predates the system where I keep these files.

In a few cases (such as path), I had created the script several years before. However, when I left the Software Productivity Consortium, I did not make a copy of the RCS archive for my general-purpose scripts (unlike ded and its friends). When moving to different projects, path is something that I need immediately; the source-controlled version tends to lag.

Download

Current

Archives

Older versions (archives)