Archive for the 'linux' Category

Project Unicorn!

Friday, May 27th, 2011

Hello everybody! Sorry it’s been so quiet over here … I’ve been preoccupied with unicorns.

The last three months have been rainy, indoorish months for me and Portland. Writing books is still my day job, but I’ve also be surprisingly, enjoyably, annoyingly busy with these smart lights. It’s been an educational odyssey: so far it’s required circuit-board fabrication, AVR programming in C and assembler, iPhone hacking, waterjet cutting, lamp design, bicycle maintainence, and lots and lots (lots!) of soldering. I’ve enjoyed it all, but what a time sink! (If only I could flip one of these AVR pins to source time instead of sinking it …)

Something about blinking, glowing light is still compelling to me, but I really do hope that the sun will come out soon and I’ll lose interest in this weak substitute. The goal I’ve set for myself is to have these lights mounted on my tallbike in time for Pedalpalooza, Portland’s three-week bicycling festival. I’ve always wanted ambulance lights on my tallbike, and these should provide the required traffic-shifting pep.
Read the rest of this entry »

The Shining $PATH

Friday, March 21st, 2008

The search-path algorithm for UNIX is broken, and has been broken for a long time.

$PATH is fine for specifying bin directories; in fact, there ought to be just one $PATH per host, which all processes can inherit, rather than users having to set such things in their dotfiles. $PATH isn’t obsolete, but it needs help.

The problem is with interpreting the order of the PATH variable. The historic algorithm, implemented by the first exec() and its descendants, is to search each of the dirs in $PATH, in the order in which they appear, until a file matching the provided command name is found, and then exec that file. It’s a simple algorithm: fast, easy to implement … but it’s also the cause of a whole genre of UNIX aggravations.
Read the rest of this entry »

named pipelines in unix

Tuesday, August 28th, 2007

while using a collection of filter and mungers and sorters to extract statistics from the apache logfiles in the traditional manner, it occurred to me that we really need a shell that’s able to create a thing called a ‘named pipeline’.

this is similar to, but not the same as, a named pipe. to wit: suppose my webserver’s filesystem is 80% full due to immense logfiles, and i am sifting those files for data. first, i have to combine them:

cat file1 fil2 file3 | sort | grep | munge > output

.. except actually file2 has got a bunch of crud from a previous egrep in it, and needs special treatment, so:

cut -d: -f2- file2 > file2.fixed
cat file1 file2.fixed file3 | sort | grep | munge > output

… only there’s not enough space on my filesystem for a second copy of most of the contents of file2, and anyway i’m only looking for a tiny percentage of that file.

what to do? well, named pipes are a handy unix-ism for this situation:

mknod file2.fixed p
cut -d: -f2- file2 > file2.fixed
cat file1 file2.fixed file3 | sort | grep | munge > output

that solves my space problem. only, i’m still re-running my sort over and over — kind of figuring it out as i go — and whenever i need to run it again, i have to re-start that stupid cut command. feh. what i need is the power of bash:

mknod file2.fixed p
(while true; do; cut -d: -f2- file2 > file2.fixed ; done) &
cat file1 file2.fixed file3 | sort | grep | munge > output

… which will re-start the command every time the pipe is opened for reading. that’s about what i want. only, instead of typing all this mess:

mknod foo p
(while true; do; cmd | cmd | cmd | cmd > p; done)&

i’d much rather just type this:

cmd | cmd | cmd | cmd |% p

when i type that special |% notation, the shell should first create the named pipe, then spawn a process to execute the pipeline into the named pipe, restarting it as necessary for as long as the shell is running and the named pipe exists.

in addition, it ought to have a mechanism for quietly halting when the named pipe is deleted, and maybe for automatically restarting such pipelines when they are found abandoned by a previous shell. and someone ought to be able to figure out what’s coming down that pipe without having to suck on it — i.e. there needs be a mechanism to inspect which commands are hooked to what pipes.

in short, the resulting named pipe ‘p’ is more than just a pipe, it’s a predefined pipeline that can be started and read from any time, but that consumes minimal resources until it is called upon. hence ‘named pipeline’.

(of course we’re consuming some memory here, with all those commands standing around blocking for output. but perhaps our clever implementation will avoid executing the commands until such time as their output is requested.)

if i was smarter or less lazy i might hack that %| syntax into the source of bash. but instead, i’m going to write a utility called ‘plumb’, maybe like so:

plumb p ‘cmd|cmd|sort|grep|etc’ # to create and start a pipeline
plumb p # to inspect
replumb p # to restart

sadly, the commands must be single-quoted, or the pipes escaped somehow, in order for this to work in the shell. but it’ll do for now.

making named-pipeline creation trivial allows me to construct what would otherwise be an unwiedly-long unix pipeline as a series of small, easily joined & inspected fittings. i can set up a large pile of sub-processes, each ready to filter the data another step, none of them consuming file space in the process of doing so, and with these craft my ultimate super-grep one step at a time.

plumb file2.fixed ‘cut -f2- -d: file2’
plumb sorted ‘cat file1 file2.fixed file3 | sort -u’
plumb sifted_a ‘egrep a sorted’
plumb sifted_b ‘egrep b sorted’
plumb formated ‘prettyfy sifted_a sifted_b’
plumb beer ‘mail -s “today’s traffic report” myboss@myjob < formatted’

what’s brilliant is, if i inspect ‘formatted’ and find an error introduced by ‘file2’, i can edit ‘file2’ to fix it, and ‘formatted’ will reflect the change without additional work or resource-consumption — an implementation of functional programming in plumbing, basically — thereby enabling:

00 17 * * * cat beer > /dev/mykle

daemontools suck!

Monday, June 4th, 2007

 (update 9/19/2008:

please allow me to clarify that the following snide, grumpy article discusses “daemontools,” DJB‘s free-software utilities for managing unattended processes, aka “daemons”, under UNIX environments.

It’s not at all about “DAEMON Tools Lite” which is some kind of DVD-ROM emulator for Windows sufferers.  I, having never used “DAEMON Tools Lite”, am unfit to pass judgment, except maybe upon the name “DAEMON Tools Lite,” which does actually suck a lot.  In three words the authors have managed to combine a tacky spelling, a capitalization crime and a copyright infringement, while neglecting to describe their product even slightly.  I suggest a re-name to “SNUGGLER Ultra Supreme Cock Matrix DVDMAX!!!” for at least some slight improvement.

anyway, on with the snide grumpiness ….)

msl runs on free software. one of our main sources of free software has been Daniel J. Bernstein, the obsessively secure author of our favorite MTA, qmail, and our favorite domain name server, djbdns.

if you follow djb’s instructions for installing djbdns, you’ll be led almost automatically to install the third major piece of djb technology, daemontools. daemontools is djb’s answer to various bugs and complaints with time-tested Unix components such as init and inetd. it includes a system for running daemons and other always-available services under Unix.

you can read what’s good about daemontools here. to read what sucks about daemontools, you’ve come to the right place.

things that suck about daemontools:

1) needing to be in a certain directory to run the command? user-interface error of the first degree.

2) no command-line help? no man pages? no coherence to any well-known online documentation system. clearly, djb hates them all and plans to write something better, but he hasn’t yet, but he’s suggesting people use his software blind, without docs. not a problem if you have a web browser. i didn’t. my whole network was down due to dns failures, and i had console-only access to a server with no x-window interface, just a single screen. this made bugs #1 and #2 extremely severe bugs; like putting a bag over my head and shackling my leg to a tree.

apparently DJB endorses HTML docs over man pages; he says web browsers are more evolved than ROFF readers, although one of those standards is twenty years older than the other.

if the interface to the software is command-line, then the interface to the documentation should be too. if ‘svc –help’ just listed the cryptic argument options of svc, i’d be an hour younger.

3) assumes a lot of low-level knowledge of unix file management. it requires shell commands for administration, instead of its own administration interface. djb says this is easier to automate;
editing config files is hard to automate. perhaps so. but if it’s so easy to automate, why isn’t it automatic?

4) envdir falls over with a confusing, useless error message when a directory is accidentally created in an env directory. since daemontools invites, requires even, users to be creating both files and directories within its labarinthine directory structures, it should tolerate exactly this kind of user error, and explain its own errors clearly and humbly.

5) then there’s the difficult to search nature of his documentation itself, but clearly that’s not his strong point. i mean, i enjoy the terseness and the smugness, but it’s just not welcoming to the reader. his headings in the djbdns section are a random list of thoughts.

6 ) his departure from unix logging is pointless. put your logs in a file, in a log directory. rotate that log however you like. give it a human readable name, not some hex code. likewise, put the date in a human-readable form. i have no problem with being y10k compliant, but do it in a human-readable way. log files are read by humans. nobody wants to go hunt for a special log file decoder ring just to read djb’s log files, esp given bugs like #4 (unclear error messages.)

tho i must admit i was stoned at the time. but what i’m looking for is an error message like:

“envdir is trying to read all the files in DIR, but one of those files is a directory, which is not allowed, and, though perhaps i should just try to ignore this problem as best i can, i refuse to do so on principle. ergo, i must die. farewell.”

9) in general: djb desires that sysadmins twiddle around in his maze, but his maze is full of twisty, turny passages, all different. the advantages of a single configuration file become painfully obvious; humans navigate documents more easily than filesystems.

he demands that they learn a new command for every single way they used to do something before. the ui compatability with initscripts or any other comparable system is nil.

really the ideal interface to manage a daemontools/djbdns/qmail installation is a windows file explorer. and that just ain’t right. bind has horrible config files, but just a few of them.

(now, having said all that, i really don’t mean to be complaining about daniel j. bernstein. he’s a great guy. my complaint is with this sofware. alas, his personality and his software are so deeply related in my mind that i began with technical complaints and now i have all but accused djb of plotting to kill my cat. sorry. sometimes i just get upset.)

just about all of these complaints come down to a central user interface problem: daemontools hasn’t got one. this is a classic example of what unix-haters hate about unix. the integration of the technology with the user is left as an exercise for the user. what’s odd to me is that qmail and djbdns are both, along with their other advantages, much easier for a sysadmin to grok, configure and live with than the respective dinosaurs they set out to slay. (sendmail and bind, respectively.) their UIs are chief among my reasons for using them.

however, the technologies that daemontools hope to usurp (initscripts. inittab, inetd, etc.) aren’t dinosaurs at all — they’ve been evolving as UNIX has evolved, and these days can answer most of djb’s complaints about them.