http://invisible-island.net/
Copyright © 1997-2021,2022 by Thomas E. Dickey
As of August 2014, I had nearly 2000 scripts in my
bin-directory, totalling more than 100,000 lines. The list
continues to grow (see screenshot). As of May 2021, it has
grown to 2300 scripts, 140,000 lines. Most are POSIX shell, most
of the remainder are Perl. Those total about the same number of
lines:
Metrics by language:
Files Lines Chars Notes
bash 130 3192 70802 495
csh 3 27 766 6
expectk 3 2650 80820 1047
gawk 1 3575 131616 1449
ksh 1 8 87 1
mawk 1 39 1032 2
perl 243 69001 1980511 6841
python 7 633 19592 24
ruby 2 1190 33235 121
sh 1952 60571 1451262 9011
tcl 1 16 323 3
unknown 15 38 3990 11
-------- -------- -------- --------
Totals 2359 140940 3774036 19011
Average 60 1600 8
Those scripts are distinct from scripts which I bundle with
various programs. A few are modified versions of scripts written
by others. Some of the scripts may be of interest to others.
There are several general-purpose scripts:
Construct a gzip'd (or compress'd) tar file.
Run as unarchive, it reverses the process.
archive was originally written by
Charles Van Gale, a system administrator at the Software
Productivity Consortium in 1989 or 1990. I extended it to make
the resulting tarball have a timestamp matching the newest file
(using cpd), as
well as using gzip or bzip2 compression:
the script...
A script that combines diffstat and makepatch.
script to show which symbols are exported from a collection
of object files
display all external symbols used by a collection of object
files
run "make", redirecting output to make.out.
If linked or renamed to make-log, it writes to
make.log.
script (originally by Johan Vromans) used for making
patch-files.
I have modified it to sort the old/new lists to avoid confusing
'/' with filename stuff.
like which, but provides the dual use of
showing the path and showing details about the arguments. The
conflict program is
related.
run a command, redirecting output to
run.out.
If linked or renamed to run-log, it writes to
run.log.
The original intent of this page was to provide sample scripts
for building programs. Since then, I have written scripts for
routine builds of the programs which I develop, as well as for
managing releases and updating this website.
These are some of the scripts that I use in
configuring, building and saving the logfiles from building
programs on multiple platforms. A complete set of configure- and
make-logs is useful for bug reporting.
Each program has its own set of build-scripts. Sample scripts
for the larger programs are included in the download. I use
scripts named for the various programs (XXX
in the following sections).
Though the oldest of these scripts (for saving build-logs from
dbmalloc) dates back to
1995, they have evolved in stages. Initially, I would do
test-builds and, after verifying the results, possibly discard
the logs. However, for several years I have collected the logs,
adding them to my source repository to compare.
As of January 2016, I had 5488 log-files (with 383 marked to
denote systems where I could no longer build). My collection has
grown to 10,717 files as of May 2021, with 61% marked as
obsolete. Some builds produce multiple log-files, and I have
build-scripts for about 50 programs. There are also a lot of
machines to build with. The log files are checked into RCS, of
course, so that I can compare successive builds and fix
problems.
Build the program (within its working directory), with the
most common configuration(s). As part of the process, install
and uninstall the program (to a location in the
build-tree).
Repeat the build-XXX script for a
given program, using different compilers and/or warning levels.
Depending on the platform, this may execute other scripts to
build “.deb” (Debian), RPM packages, NSIS
installers or Solaris packages.
Build a given program, use the options from the
corresponding all-opts-XXX script, one at
a time, to build the program.
These remove the files from a build within the given
program's working directory.
I use these scripts to save logfiles after doing a build.
They all assume that there is a parallel directory of the same
name, e.g., ../save-vile, next to the current
directory. The current directory is the top of the build
tree.
In particular (making them useful for bug-reports), they
save the results from the configure script, along with
log-files that are created using the
cfg-XXX and
build-XXX scripts for the corresponding
programs.
Where possible, I parameterized these scripts to make a
reusable core. These scripts provide the reusable parts:
Those build-scripts are part of a larger set, which
includes
release-XXX, to prepare a tarball and
patch.debbuild-XXX,
rpmbuild-XXX,
mkpkg-XXX, using the tarballs for package
builds.tag-XXX, to label a program in
RCSupload-XXX, to put the prepared
tarball and change history on my website.
As of May 2021, here is a summary of these build-scripts:
Metrics by phase:
Files Lines Chars Notes
Bug report 31 547 15582 155
Build phase (PART) 397 20620 470933 1570
Build support (HELP) 298 28061 752788 3305
Configure 201 3421 76007 479
Connections 99 474 13937 205
Development 83 14238 421288 1549
Environment 143 1776 39342 355
File-movement 54 731 14487 84
Metrics 19 3253 92463 349
Miscellaneous 498 31202 882008 5577
Source-control 112 7795 196881 905
Testing 424 28822 798320 4478
-------- -------- -------- --------
Totals 2359 140940 3774036 19011
Average 60 1600 8
Metrics for build phases:
Files Lines Chars Notes
PART: build-* 48 3080 57744 131
PART: build-all-* 48 430 6268 96
PART: build-my-* 3 348 7367 15
PART: bump-* 36 7606 191397 153
PART: clean-* 48 725 13702 111
PART: clobber-* 9 46 1156 18
PART: release-* 54 3531 83176 414
PART: release-*-docs 24 1265 27447 117
PART: save-* 50 1409 28472 329
PART: tag-* 30 1407 39696 82
PART: upload-* 38 582 11296 77
PART: upload-*-docs 9 191 3212 27
-------- -------- -------- --------
Totals 397 20620 470933 1570
Average 52 1186 4
The matrix of scripts is not completely filled in; that
depends on the program. Most are programs that I maintain. Some
(such as PDCurses) are built because I use those in programs
which I maintain, but have no reliable package provider:
Build-phases:
build-*
^ build-all-*
---^ build-my-*
------^ bump-*
---------^ clean-*
------------^ clobber-*
---------------^ release-*
------------------^ release-*-docs
---------------------^ save-*
------------------------^ tag-*
---------------------------^ upload-*
------------------------------^ upload-*-docs
---------------------------------^
* * - - * - * - * - - - AdaCurses
* * - * * - * - * - - - PDCurses
* * - - * - * - * - - - XawPlus
* * - - * - * - * - - - ac213
* * - * * - * - * * * - ac252
* * - * * - * * * * * - add
* * - - * - * - * - * - atac
* * - * * - * - * * * - bcpp
* * - * * - * - * * * - blame
- - - - - - * - - - * - buildscripts
* * - * * * * * * * * * byacc
* * - * * - * * * - * - c_count
* * - * * - * * * * * * cdk
* * - * * - * - * * * - cdk-perl
* * - * * - * * * * * - cm_tools
* * - * * * * * * * * - conflict
* * - * * - * * * * * - copyrite
* * - * * * * * * - * - cproto
- - - - - - - - * - - - dbmalloc
* * - * * - * * * * * - ded
* * - * * * * * * * * * dialog
* * - * * - * * * - * * diffstat
- - - - - - * - - - * - gpl-scripts
* * - * * * * - * * * - indent
* * - - * - * - * - - - libX11
* * - * * - * - * - - - libXaw
* * - - * - * - * - - - libXaw3d
* * - * * - * * * - - - libXcursor
* * - * * - * - * - - - libXft
* * - * * - * - * - - - libXpm
* * - * * - * * * - - - libXt
* * - * * - * * * * * - luit
* * * * * * * * * * * * lynx
- - - - - - * - - - * - mapsym-scripts
* * - * * * * * * * * * mawk
- - - - - - * - - - * - misc-scripts
* * - * * - * - * * * - misc_tools
- - - - - - * - - * * - my-autoconf
* * * * * * * * * * * * ncurses
* * - - * - * - * - - - ncurses-examples
* * - * * - * - * - * - neXtaw
* * - - * - * - * - - - prcs
* * - * * - * - * * * - rcshist
* * - * * * * * * * * - reflex
* * - * * - * * * * * - sccs_tools
* * - * * - * * * * * - tack
* * - - * - * * * * * - tctest
* * - * * - * - * * * - td_lib
- - - - - - - - * - - - tdlib
- - - - - - * - - * * - terminfo
* * - - * - * - * - - - tin
- - - - - - - - - * - - tin-beta
* * - - * - * - * - - - twm
* * * * * - * * * * * * vile
* * - * * - * * * * * - vttest
* * - - * - * - * - - - xfonts-encodings
* * - * * - * * * * * * xterm
My script which reports this information
(report-build-scripts) matches program names against
a pattern. That works well enough where the scripts are named
consistently. The support scripts are less regular: some follow a
pattern, while others do not. The script uses a table of
exceptions to help with its categorization of scripts.
Along with the build-scripts, I use wrappers for
gcc to turn on compiler warnings, as well as
scripts for other compilers. They are included in the
build-scripts, since that is where they are most used.
This provides what should be the minimal options
for gcc to ensure that C functions are prototyped.
I added it in 1995 at the same time that I added
gcc-strict. Before that, I was likely to just edit
the makefile, not thinking that it would be simpler to make a
wrapper script.
the script...
This script adds gcc options to compile for
64-bits. When I added it in 2009, it was still common to have
compilers on a nominally 64-bit platform such as Solaris
default to compiling 32-bit code.
I made this script, incorporating some posting by Fergus
Henderson, e.g., this one to
gnu.gcc.help from 1994.
the script...
Later, I was told about the gcc "fortify" option. Rather
than modify the options in gcc-strict, I added
this script.
the
script...
Configure script wrappers containing configure options for
specific programs are (usually) named for those programs,
e.g.,
I am able to cross-compile to MinGW on several machines. There
is no consensus on what directories those cross-compilers use.
These scripts help:
If you have a machine that I've not ported to, and have
problems building one of the programs that I maintain, I will
require the relevant information:
- log from running
configure, with options
I recommend that you use run-out to
capture this.
config.cache (if available—"new"
versions of autoconf suppress this, but for example lynx preserves this file).config.log
config.status
- the generated headers (e.g.,
config.h)
- log from running
make, with options
I recommend that you use make-out
to capture this.
A uuencoded or mime'd gzip'd/tar file is preferred, because
the logfiles can be awkward to email. Mailing uncompressed
log-files also has the drawback that they may be rejected on
mailing lists.
These scripts are for more specialized uses.
remake an object file, reporting
differences. The reference (first compile) is saved in the
OBJ subdirectory.
I use this script when making structural changes to a C
program that “should not” change its content, e.g.,
converting from
K&R C or changing comments/formatting.
Like Regress, I use this for
checking changes to manual page formatting.
Scan a directory tree, looking for nroff (man/ms) files, to
check their syntax as well as verify that their macros are
consistent.
If
checknr were portable and handled manpages, it
would be useful. This started as a wrapper for
groff's checks, but has grown a lot.
Like path, show information about
the user's CLASSPATH environment variable (which
controls linking and loading of Java classes.
If no arguments given, this script displays the names of the
directories in the class-path. If argument is given, it is
assumed to be either a class name (e.g., "String"), or a
jar-file.
All matching names are displayed; the first is the one that
will be loaded.
Options (arguments beginning with "-") are passed to
'ls'.
Use ncurses' infocmp to compare two terminfo files. Its
output is close, but is not sorted in a predictable manner that
would allow diff'ing output from
different comparisons.
Scan directory-tree or file(s) counting nroff files,
adjusting for comments.
Process one or more patch files, using diffstat's -C (color) option to
markup the files (with SGR). We want to modify those so they
all use reverse-video (since colored text is too hard to read),
and produce html. To make the result a little more flexible,
use the diffstat.css stylesheet.
The script expects one of two parameter conventions:
-
patch-file, simply process with
diffstat
-
two directories as parameters, to pass those to
diffstat using -S/-D
Search for patterns that (might be) broken defines
Find the X bitmap font-files which are used in xterm/uxterm
app-defaults files, and for some systems (where the packaging
system is useful) followup with the packages containing those
files.
See
my answer using this script.
Search for patterns that (might be) casts.
Search for patterns that might be dates.
Search for patterns that (might be) ifdefs.
Search for patterns that (might be) K&R function
declarations.
Search for patterns that (might be) numbers.
Run tidy to indent one or more HTML files,
optionally showing the differences made (-v) and
with dry-run (-n) feature.
Run perltidy to indent one or more Perl files,
optionally showing the differences made (-v) and
with dry-run (-n) feature.
Format the output of stty -a into a single
column, allowing diff to show just the single
feature(s) which may change.
Edit the output of ldd for the given
parameters, yielding only the absolute pathnames.
I use this script with less to remove
temporary files.
I use this script with less to open
special files.
Displays information about the user's
LD_LIBRARY_PATH environment variable (which
controls linking and loading of dynamic libraries.
If no arguments given, this script displays the names of the
directories in the library-path. If argument is given, it is
assumed to be a library name (e.g., "c" for -lc). All matching
names are displayed; the first is the one that will be
loaded.
Options (arguments beginning with "-") are passed to
'ls'.
Construct a manifest from the RCS description lines for the
selected (symbolic) version. The script assumes that the
following files are automatically generated from scripts:
MANIFEST – from this script
configure – from autoconf (configure.in,
aclocal.m4)
config.h – from configure
Other files are all managed via RCS, and are present in a
release only if they have had symbolic revisions added for that
release.
Compares the elements of an archive file, showing which are
different or missing from one archive.
Run command without /usr/local in
environment.
Strip leading "." from PATH to execute a command.
Strip environment, e.g., as if running in cron,
and run a command.
Strip locale environment-variables, i.e., leaving only
POSIX, and run a command.
Strip all terminfo/termcap variables from environment, run
command.
Remove makefile target (e.g., an object file) and run
make to recreate it.
For each "psf" font listed in /usr/share/consolefonts, make
a report showing which ISO-8859-x variants are covered, as well
as the line-drawing used by ncurses.
Analyze one or more terminfo descriptions to get some useful
statistics. This assumes that you have tic, toe and
infocmp.
For a given package, ask the question: what packages satisfy
its dependencies, and is there a
“provides” which can be used as an alias
for those.
Strips ANSI escape sequences and reduces
backspace/overstrikes.
The script uses POSIX
sed, which means that only
BREs (Basic Regular Expressions) are permitted.
I have tested this with FreeBSD 10, OpenBSD 5.3, Solaris 10 as
well as Debian 6 and later:
Here are two examples, using "cat -v" to make the
nonprinting characters visible:
-
manpage, using both SGR sequences for bold text, as well
as two variations on backspace/overstruck text
input
versus
output.
-
typescript using yum, which includes overstruck lines
using carriage returns (^M)
input
versus
output.
Read the database printed by “dircolors -p” and
show the different patterns with the indicated color by (as
done by GNU ls) using hardcoded SGR sequences.
The output can be viewed with “less -R”.
See
my answer using this script.
See
these alternatives which use the LS_COLORS
variable.
Successively load and display console fonts using either
showconsolefonts or a grid for the different ISO-8859-x locale
settings. Like the ncurses test-programs, provide for reading a
key for single-stepping or proceeding with a time-delay. It
does not use the perl Curses module, since that would interfere
with sending raw bytes to the screen.
Sort the list of parameters as if they are filenames
containing a version. I use this to sort the names of tar-balls
for my programs in builds as well as website maintenance.
By the way, I noticed a comment in
remove the low version number of file indicating
that GNU ls has a similar feature. However, the
rules for parsing the version are a little different, and in my
script the version number follows the program name.
Recursively
touch (change timestamps) on directories to reflect the
newest file inside the directory. I use this for tidying up
source-trees, e.g., as in the “-t”
option of archive (which uses
cpd).
the script...
Here are a few links which I found when looking for
comparable scripts or programs:
Oddly, most applications in this area simply update the
directories (and files) to the current date.
Run a command with the given locale. The first parameter is
the locale to use. The remaining parameters are the command to
run.
I wrote these scripts to provide a way to export information
from my RCS archives to Git, starting in July 2016 (see Git exports):
Upload a Git bundle to github
in my "testing" repository. That can be renamed, to
provide a "readonly" snapshot.
For a given project, use release2git -R to find its
labeled releases, and generate a manifest-file for each of these. Discount
the first line, and compare successive files to identify
changes from one release to the next.
For the given directories, fetch current Git. The script
also calls git-utimes to set file
timestamps to correspond to their last commit-date.
Although Git is “distributed” that ignores
performance. For useful results, you need a copy of each
repository on a local machine (just like RCS!). I use this
script with about 200 repositories that are related to my
work.
Make a summary report for files stored in Git, showing the
number of commits, and amount of change for each committer.
This script uses the Perl module Text::CharWidth,
because author names in Git are in UTF-8 (not US-ASCII).
Interestingly enough, Perl does not appear to have an
equivalent to wcwidth in its main distribution.
Here is part of the script's output (20 of 513 lines) for
the gnome-settings-daemon:
Author Files Commits Inserts Deletes
A S Alam 1 22 10784 7787
Aaron Plattner 2 2 205 10
Abderrahim Kitouni 1 1 2 1
Abduxukur Abdurixit 1 2 3074 2333
Adam Dingle 1 1 0 1
Adam Jackson 1 1 18 0
Adam Matoušek 1 1 1341 1657
Adel Gadllah 2 1 22 7
...
vasudeven 1 2 225 141
Žygimantas Beručka 1 2 2268 1837
Γιάννης Κουτσούκος 1 1 652 741
Γιώργος Στεφανάνης 1 1 620 188
Марко Костић 1 4 769 859
Милош Поповић 2 4 1814 1489
Мирослав Николић 2 22 27259 23656
Роман Донченко 2 3 16 15
િિશાલ ભલાણી 1 1 623 788
甘露(Gan Lu) 1 1 918 1035
TOTAL 990 6507 1550730 1162857
Your browser probably does not render this properly (expect
problems with fixed-pitch Unicode), but it displays correctly
in xterm:
Update file timestamps to their last commit date/time. This
is modified from the version written by Tom
Christiansen: the initial import of some Git's doesn't fit
pattern.
Initial script, using my modifications to rcs-fast-export
to export snapshots of mawk.
Rather than attempt to export an RCS archive to Git using
rcs-fast-export.rb (and have to do this
repeatedly), analyze an RCS archive and apply release snapshots
to a Git-ball incrementally. That loses most metadata (such as
the individual file timestamps, comments, etc), but should work
even with my larger archives.
I use this for merging changes from a separate
upstream repository, into my repository's
master branch.
Use release2git
to obtain a list of versions and files for a given cutoff date,
and apply a label to that based on the cutoff date.
I wrote these scripts to support generating map- and sym-files
for ncurses6 symbol versioning:
Count symbol usage for the given executables, and summarize
their usage with categories for these types:
- ncurses (if using ncurses extension)
- curses (if not using ncurses extension)
- wide(curses/ncurses)
- termcap
- terminfo
- slang
Compare two “.map” files, generate a report
showing the number of symbols added, deleted or renamed. A
“renamed” symbol is one whose scope differs or is
in a different version (the numeric plus “.” part
on the end of the label).
Optionally show only the number of symbols changed.
Extract symbols from a given patch-date for dialog.
Inspect the given (build-)tree for libraries and programs,
determining which symbols exported by the libraries are needed
by other libraries or the found programs. Print a list showing
the symbols, where they are declared, and their needed status
(global or local).
Using needed-syms, construct
“.map” and “.sym” files for the current
build-tree's configuration of ncurses.
The “.sym” files are trivial – a comment
header telling what options were used to generate the
configuration, and a sorted list of symbols.
The “.map” files require special effort, because
they are constructed as a series of version-nodes. So... to
generate a new one requires us to read the existing
“.map” file to relate the symbols' version and
scope to the previous version.
To allow for symbols being in termlib/ticlib/ncurses, always
generate the names as if they were in the separate library.
Doing this allows splitting the map-file into parts and
(possibly) making it work with Solaris, which does not accept a
map-file declaring a symbol not in the library.
Successively build ncurses for release-dates with several
combinations of configure options and merge the result to
generate usable map- and sym-files.
Read one or more “.map” files, generate a report
showing
- scope (global or local)
- symbol
- node-label
Optionally show only the number of symbols in each
scope.
Here are some scripts written by others which I have
modified.
In particular, those using a restrictive license (e.g., GPL) are
not bundled with the others:
This is a slightly-modified copy of altchars by
John H. Dubois III (from here). The original script
carries forward a documentation error from AT&T, for the
lantern pseudo-graphics character which was fixed in
ncurses in 1997 as a
result of work on lynx.
Starting in 2000
(for ncurses Ada95 binding), I have modified this script to
make the generated HTML validate, as well as to use
CSS. (The script is originally from
gnat, but unmaintained).
I use linklint, but (see this) it has been unmaintained
since around 2001. For example, see the Debian
bug-reporting system. As of early 2016, I have fixed most
of the problems which affect me:
- flush stderr/stdout, so I can pipe result inside vile
- change
%20's back to spaces to check for
actual files with spaces
- <a>-tags are not the only things that can have
id's or name's for
href
This is an improved version of Earl Hood's
man2html Perl script (see my expanded page on the
topic).
I modified this script in 2003 (it was originally from
emacs, but unmaintained):
- to allow for symbolic links to RCS directories, e.g., for
link2rcs.
- to avoid using bogus hostname for authors
This is a wrapper for
rpm2cpio which makes it easier to use. There
actually is more than one script named
“unrpm” — this is the
best-known, and was seen in a posting to the
fedora-legacy mailing list in
2004, stating that it came from the
fedora-rpmdevtools package. According to the
package's
changelog, it was removed in 2005.
This is useful for xterm
testing, found in Debian #588785
(from http://lindi.iki.fi/lindi/xcutselprint).
The scripts on this page are a small subset of my collection.
I use (predictably) another script to put together the tarballs
as well as creating the changelogs on this page (with the help of
rcs2log).
The "RCS_BASE" comment denotes the initial check-in using my
RCS wrappers. Those preserve the
modification time of the file, which in some cases predates the
system where I keep these files.
In a few cases (such as path), I had
created the script several years before. However, when I left the
Software Productivity Consortium, I did not make a copy of the
RCS archive for my general-purpose scripts (unlike ded and its friends). When moving to
different projects, path is something
that I need immediately; the source-controlled version tends to
lag.
Older versions (archives)
