302 Found

Recently I was asked to write an “OS device driver” a.k.a. Linux kernel driver, or as I came to find out, really just a kernel module since there was no hardware involved. After locking up my system completely twice (glad it runs a journaling filesystem), I held down the hardware “reset” button for a few seconds, then crafted a sufficiently sharp albeit dangerous, stab into the world of kernel programming. I had suspected it, but a segfault in kernel-land is a major faux-pas, as is killing a user-space helper process the kernel was supposed to kill. Of course it is an understatement to say that a few guides have been written on topics here. Here are some requirements that were explored with my project:

1. intercept calls to kernel routines (mainly C system calls) from userspace in near real-time
2. time how fast said calling processes would then exit after the kernel zapped them with a friendly signal
3. do the aforementioned without creating feedback loops or locking up the kernel
4. manage the nuances of intercepting kernel routines (can this be done for every process? every call? should it?)

Initially before the requirements were ready, I wrote a quick shell script to discover the target process, send it a SIGTERM, and time the exit. As it turned-out, it was probably too slow to catch syscalls that completed quickly. There may have been other ways to use strace to fulfill the requirements but I ended-up taking a different approach.

#!/usr/bin/env bash
which strace >/dev/null || { echo 'need `strace` to run. Quitting.'; exit; }
p=`pidof java | tail -n1`
[ -n "$p" ] || { echo 'please start the java process before running this script'; exit; }
echo 'Sending SIGTERM to Java process '$p' on 1st "write()" syscall'
strace -e write -fp $p 2>&1 | read 
kill -SIGTERM $p
t=`( time {
    while kill -0 $p 2>/dev/null; do 
        sleep 0.000001
    done; 
    } ) 2>&1 \
| awk '/real/ { print $NF }'`
echo "Time to terminate: ${t}"

I ended-up deciding to write an application that would be part kernel-space, part user-space. Minimal time is spent in kernel-space, just enough to intercept the syscall and send a message of-sorts (real-time signal) to the user-space application which handles the friendly zapping of lucky processes being monitored (non-real-time SIGTERM or SIGKILL signal). One benefit of this approach is that the real-time kernel-to-userspace signals can be queued and do not get interrupted. This allows the userspace to branch off worker threads or fork processes to handle the requests in a less sensitive context.

The results of the subsequent “time-to-exit” timings, as well as other output is logged using syslog. Don’t ask me what the ultimate use-case is for this application, I also thought it was a bit strange when the client asked me to work on it. In any case, it was a great learning experience for me, and I enjoyed the opportunity to learn about an aspect of Linux programming that had until then, intimidated me. That said I came to the conclusion that for a beginner, writing a module is one of the best ways to cut your teeth with kernel programming. Here is a sample session intercepting a toy Java program:

• Top-Left: the userspace process receives a real-time signal from the kernel module containing the target PID, then fires a SIGTERM to that PID while timing how long it takes the target to exit
• Top-Right: the Java toy app opens a dummy text file and writes to it. Since this does not take very long, it simulates a delay to exit. A real “uninterruptible delay” to exit would involve a clean-up by the target in the case of SIGTERM, or in the case of SIGKILL a condition of “uninterruptible sleep” would be required such as on blocking i/o
• Bottom-Left: dmesg displays kernel messages, our module’s output begins at +47.. seconds. As one can see, the sys_write() call, or C-level write() call is being intercepted using Linux Kprobes
• Bottom-Right: my kernel module is inserted and removed from the kernel

Here is the module (disclaimer: I know my use of CamelCase/camelCase/lowercase_separated_by_underscores/UPPERCASE_SEPARATED_BY_UNDERSCORES and general coding style is probably way-off, and that some of the things my code is doing may be wrong/dangerous, and also that bla bla bla…. so yes this is “pre-alpha” code):

#include <asm/siginfo.h>    //siginfo
#include <linux/debugfs.h>
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/kprobes.h>
#include <linux/module.h>
#include <linux/moduleparam.h>
#include <linux/rcupdate.h> //rcu_read_lock
#include <linux/sched.h>    //find_task_by_pid_type
#include <linux/uaccess.h>  //copy_from_user
#include <linux/version.h>

#include "my_syscall_display.h"

#define USERMODE_EXEC_PATH      "/home/pablo/Desktop/probe_io_timer/probe_io_timer_u"
#define MY_DEBUGFS_MAX_COUNT    10
#define SIG_TEST                44  // we choose 44 as our signal number
                                    // (real-time signals are in the range of 33 to 64)
static char *target_taskname = "timer_test_target";
static char *target_syscall = "sys_write";
static int  myoffset = 0;
static int  myskipcount = 0;

/* 
 * module_param(foo, int, 0000)
 * The first param is the parameters name
 * The second param is it's data type
 * The final argument is the permissions bits, 
 * for exposing parameters in sysfs (if non-zero) at a later stage.
 */

module_param(target_taskname, charp, 0);
MODULE_PARM_DESC(target_taskname
        , "Name of program to monitor, Use this to get pid. Default is \"test\"");
module_param(target_syscall, charp, 0);
MODULE_PARM_DESC(target_syscall
        , "Name of system call or another OS symbol to monitor. Default is \"sys_write\"");
module_param(myoffset, int, 0);
MODULE_PARM_DESC(myoffset
        , "hexadecimal offset from the OS symbol name where monitoring will be done. Default is 0");
module_param(myskipcount, int, 0);
MODULE_PARM_DESC(myskipcount, "How many symbols to skip before taking action. Default is 0");

struct siginfo info;
struct task_struct *userspace_task;
static unsigned int userspace_task_pid;
struct dentry *file = NULL;

static unsigned int counter = 0;

int Pre_Handler(struct kprobe *p, struct pt_regs *regs)
{
    if( strcmp( current->comm, target_taskname)== 0){
        printk( "probe pid %d `%s`, count: %d/%d\n",
                current->pid, current->comm, ++counter, myskipcount);
        //printk("%s( %lu, %s, %lu)\n", target_syscall, regs->di, (char *)(regs->si), regs->dx);
        mySyscallPrint( target_syscall, regs);

        if ( counter >= myskipcount){
            if(userspace_task == NULL)
                printk("\tuserspace_task is still NULL. No signal sent.\n");
            else{
                info.si_int = current->pid;
                send_sig_info(SIG_TEST, &info, userspace_task);
    }   }   }
    return 0;
}

void Post_Handler(struct kprobe *p, struct pt_regs *regs, unsigned long flags)
{
    ;
}

static ssize_t write_pid(struct file *file, const char __user *buf, size_t count, loff_t *ppos)
{
    char mybuf[10];
    if(count > MY_DEBUGFS_MAX_COUNT)
        return -EINVAL;
    if( copy_from_user(mybuf, buf, count) != 0)
        return -EFAULT;
    sscanf(mybuf, "%d", &userspace_task_pid);
    printk("   helper pid = %d\n", userspace_task_pid);

    rcu_read_lock();
    if( userspace_task_pid != 0)
        userspace_task = pid_task( find_pid_ns( userspace_task_pid, &init_pid_ns), PIDTYPE_PID);
    if( userspace_task == NULL){
        printk( "    no such pid!\n");
        //rcu_read_unlock();
        //return -ENODEV;
    }
    rcu_read_unlock();

    return count;
}

static const struct file_operations my_fops = {
    .write = write_pid,
};

static struct kprobe kp;

static int __init myinit(void)
{
    int ret;
    char *argv[] = {USERMODE_EXEC_PATH, NULL };
    char *envp[] = {"PATH=/sbin:/usr/sbin:/bin:/usr/bin", NULL };

    printk("__Kprobekill module__\n");
    printk("target_taskname:            %s\n", target_taskname);
    printk("target_syscall:             %s\n", target_syscall);
    printk("myoffset (inside call):     %d\n", myoffset);
    printk("myskipcount (before signal):%d\n", myskipcount);
    printk("Signal sent to Target Progr:%s\n", "SIGTERM");

    kp.pre_handler = Pre_Handler;
    kp.post_handler = Post_Handler;
    kp.addr = (kprobe_opcode_t *) kallsyms_lookup_name( target_syscall);
    //kprobe_lookup_name( target_syscall, kp.addr);
    kp.addr += myoffset;
    //kp.addr = (kprobe_opcode_t *)0xffffffff815950a0; 

    memset(&info, 0, sizeof(struct siginfo));
    info.si_signo = SIG_TEST;
    info.si_code = SI_QUEUE;
    
    /*  we need to know the pid of the user space process
     *  -> we use debugfs for this. So 1st create the debugfs file,
     *  then set the userspace_task_pid to 0,
     *  exec the userspace, wait for successful exec,
     *  and block on it to write its PID to debugfs (w/in timeout?),
     *  Finally, get the task struct for that PID in order to deliver signals to it.
     *  If any of these steps in this module init chain fail, module must
     *  exit because it depends on that userspace process.
     *  Of course, since that userspace can be killed or crash, we must continue
     *  to not assume it is still there on subsequent transactions, and
     *  restart it, or module-exit if userspace is gone.
    */
    file = debugfs_create_file("signalconfpid", 0200, NULL, NULL, &my_fops);

    userspace_task_pid = 0;
    userspace_task = NULL;   

    printk("usermodehelper: init -");
    ret = call_usermodehelper(USERMODE_EXEC_PATH, argv, envp, UMH_WAIT_EXEC);
    if (ret != 0)
        printk(" error: %i\n", ret);
    else
        printk(" success\n");

    register_kprobe(&kp);
    printk("`%s` probe inserted on `%s` tasks\n", target_syscall, target_taskname);

    return 0;
}

void myexit(void)
{
    // same PID -- different signal
    if(userspace_task != NULL){
        info.si_signo = SIGTERM;
        info.si_int = 1234; // no msg to deliver except "bye-bye"
        send_sig_info( SIGTERM, &info, userspace_task);    //send the signal
    }
    // could've done this earlier?
    if( file != NULL)
        debugfs_remove(file);

    unregister_kprobe(&kp);
    printk("`%s` probe removed for `%s` tasks\n", target_syscall, target_taskname);
}

module_init(myinit);
module_exit(myexit);
MODULE_AUTHOR("Pablo");
//MODULE_AUTHOR("Manoj");
MODULE_DESCRIPTION("KPROBE MODULE");
MODULE_LICENSE("GPL");

and userspace:

/*
 * TODO:
 * 1. SIGTERM hndlr for clean shutdown
 * 2. process forks up to MAX_PROCS2TIME for timing workers
 * 3. circular FIFO pid2term of size MAX_PROCS2TIME
 * 4. does SIG_TEST really have to be hard-coded arbitrarily?
 */
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <limits.h>
#include <stdbool.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <syslog.h>
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>

#define MY_SYSCONF_PATH     "/sys/kernel/debug/signalconfpid" 
#define MY_SYSCONF_MAX_MSG  10
#define MAX_PROCS2TIME      10
#define SIG_TEST            44 /* hard-coded since SIGRTMIN is different in user and in kernel space */ 

#define MY_ERR(s)           syslog(LOG_NOTICE,s": %s",strerror( errno))

unsigned long getMicrotime();

int pid2term=0;

void receiveData( int n, siginfo_t *info, void *unused)
{
    pid2term = info->si_int;
}

int main( int argc, char **argv )
{
	int configfd;
	char buf[ MY_SYSCONF_MAX_MSG];

	/* kernel needs to know our pid to be able to send us a signal ->
 	 * we use debugfs for this -> do not forget to mount the debugfs!
 	 */
	configfd = open( MY_SYSCONF_PATH, O_WRONLY); 
	if( configfd < 0) {
		MY_ERR( "open");
		return -1;
	}
	sprintf( buf, "%i", getpid());
	if ( write( configfd, buf, strlen(buf) + 1) < 0) {
		MY_ERR( "fwrite"); 
		return -1;
	}

	/* now setup the signal handler for SIG_TEST 
 	 * SA_SIGINFO -> we want the signal handler function with 3 arguments
 	 */
	struct sigaction sig;
	sig.sa_sigaction = receiveData;
	sig.sa_flags = SA_SIGINFO;
	sigaction(SIG_TEST, &sig, NULL);

    /* for now copying one int to another, pid2term => pid2term_cpy 
     * as an "atomic" way to "safely" handle subsequent interrupts
     * part-way through a timing op
     */
    int pid2term_cpy;
    unsigned long b4, diff, sec;

    syslog(LOG_NOTICE, "entering wait loop");

    while( true){

        pause();

        if( pid2term != 0){            
            pid2term_cpy = pid2term;
            pid2term = 0;
            syslog(LOG_NOTICE, "Sending SIGTERM to PID %d\n", pid2term_cpy);
            b4 = getMicrotime();            
            kill( pid2term_cpy, SIGTERM);            
            while( kill( pid2term_cpy, 0) != -1)
                usleep( 1);
            if( errno != ESRCH) continue;
            diff = getMicrotime() - b4;
            sec = diff / 1e6;
            syslog(LOG_NOTICE, "PID %d: time2term was %lus,%luus\n"
                    , pid2term_cpy, sec, (unsigned long)(diff - ( sec * 1e6)));
        }
    }
	return 0;
}

unsigned long getMicrotime()
{
    struct timeval currentTime;
    gettimeofday(&currentTime, NULL);
    return currentTime.tv_sec * (int)1e6 + currentTime.tv_usec;
}

What did I learn?

• My hybrid K&R/Stroustrup/Allman-8/Whitesmiths/Horstmann/Ratliff/Lisp indentation style is WAY-wrong and could result in permanent, irreparable damage to the universe
• pass module params to module
• share params between kernel and userspace using debugfs
• real-time signals kernel to userspace
• Linux KProbes!
• some things strace can do/not-do
• tell a userspace process to fork-off from the kernel
• write log output from the kernel (printk) and from userspace (syslog)
• receive a parameter through a signal

External Links:
https://opensourceforu.com/2011/04/kernel-debugging-using-kprobe-and-jprobe/
https://www.ibm.com/developerworks/library/l-kprobes/index.html
https://blog.tanelpoder.com/2013/02/21/peeking-into-linux-kernel-land-using-proc-filesystem-for-quickndirty-troubleshooting/
http://www.vantagepoint.sg/blog/82-hooking-android-system-calls-for-pleasure-and-benefit
http://manpages.ubuntu.com/manpages/artful/man8/kprobe-perf.8.html
https://www.scalingphpbook.com/blog/2016/04/03/my-favorite-simple-php-debugging-tool.html
https://jlmedina123.wordpress.com/2013/08/13/current-variable-and-task_struct/
http://www.xml.com/ldd/chapter/book/ch02.html#t6
https://lwn.net/Articles/288056/
https://linux-kernel-labs.github.io/master/labs/kernel_modules.html#
https://syscalls.kernelgrok.com/
https://www.gnu.org/software/libc/manual/html_node/Atomic-Types.html
https://www.wikitechy.com/tutorials/linux/linux-api-to-list-running-processes
https://idea.popcount.org/2012-12-11-linux-process-states/
https://eklitzke.org/uninterruptible-sleep
https://stackoverflow.com/questions/223644/what-is-an-uninterruptable-process
http://stupefydeveloper.blogspot.com/2009/06/linux-creation-of-new-process.html
http://jkukunas.blogspot.com/2010/05/x86-linux-networking-system-calls.html
https://jvns.ca/blog/2016/01/18/guessing-linux-kernel-registers/
https://lwn.net/Articles/604515/
https://stackoverflow.com/questions/9782660/using-ptrace-to-find-out-what-exactly-does-the-arguments-signify-for-a-system-ca?rq=1
https://stackoverflow.com/questions/44612136/how-to-simulate-hung-task-in-linux/44612553#44612553
https://tuxthink.blogspot.com/2012/05/module-to-print-open-files-of-process.html

This BASh script was written during the process of solving the Functions and Fractals – Recursive Trees – Bash! problem on Hacker-Rank. Yes it is slight overkill for that problem, but adds neat interesting features that no sane developer should implement in a shell script – such as drawing arbitrary lines, “Y” shapes, and recursive trees to any number of iterations (“screen” size and computational/memory/time-constraints withstanding) on an ASCII-art “screen”. Why not a more-limited, simpler approach to the problem? During the dev process, I found that I benefitted from incrementally breaking down the problem to its simpler components: reliably drawing any arbitrary pixel, then a straight line, then a “Y”, then any number of those Y’s on the screen to form a tree. Probably the trickiest part was deciphering exactly *what* idiosyncratic method the author of the Hacker-Rank problem had used to draw his full tree, then replicating that in a manner that was automatic and repeatable across iterations. Although high-school level, the math was the fun part.

Can also be downloaded here

declare -a _scrn
_rows=63
_cols=100
max_iters=5
trunk_row=0

start_bl=$(( ($_rows + 1) / ( $max_iters - 1) ))
trunk_col=`expr $_cols / 2`

declare debug=''

function blankScreen()
{
    blank=`yes 2>/dev/null | head -n $(( $_cols + 1 )) | tr -d y | paste -sd_ -`
    for ((i=0; i<_rows; i++)); do
        _scrn[$i]=$blank
    done
}

declare new_row_val
function flipRow()
{
    new_row_val=$(( $_rows - $1 - 1 ))
}

function drawWood()
{
    row=$1; col=$2
    flipRow $row
    old_row=${_scrn[$new_row_val]}
    _scrn[$new_row_val]=${old_row:0:$col-1}1${old_row:$col}
}

function drawLine()
{
    x1=$1; y1=$2; x2=$3; y2=$4 
    if [ $x1 -eq $x2 ]; then
        x=$x1
    else
        m=`echo "scale=4; (${y2} - ${y1})/(${x2} - ${x1})" | bc`
        b=`echo "scale=4; ${y1} - ${m} * ${x1}" | bc`
    fi
    for y in `seq $y1 $y2`; do
        if [ $x1 -ne $x2 ]; then
            x=`printf '%.0f\n' $(echo "scale=4; (${y} - ${b})/${m}" | bc)`
        fi
        if [ -n "$debug" ]; then 
            echo drawWood $y $x 1>&2
        else
            drawWood $y $x
        fi
    done
}

function drawY()
{
    start_x=$1; start_y=$2; stem_len=$3
    drawLine $start_x   $start_y    $start_x    $(( $start_y + $stem_len ))
    start_y=$(( $start_y + $stem_len + 1 ))
    drawLine $(( $start_x - 1 ))    $start_y    \
            $(( $start_x - $stem_len - 1 ))     $(( $start_y + $stem_len ))
    drawLine $(( $start_x + 1 ))    $start_y    \
            $(( $start_x + $stem_len + 1 ))     $(( $start_y + $stem_len ))
}

function drawTree()
{
    iters=$1
    declare -a curr_ys
    bla=$(( $start_bl - 1 ))
    curr_ys[0]="${trunk_col}_${trunk_row}_${bla}"

    for curr_iter in `seq 1 $iters`; do
        declare -a next_ys
        next_ys_ind=0
        for y in ${curr_ys[*]}; do 
           declare -a unpakt=( `echo $y | tr _ ' '` )
           drawY ${unpakt[*]}
           prev_x=${unpakt[0]}
           prev_y=${unpakt[1]}
           prev_bla=${unpakt[2]}
           prev_bl=$(( $prev_bla + 1 ))
           next_y=$(( $prev_y + $prev_bl * 2 ))
           next_bla=$(( $prev_bl / 2 - 1 ))
           next_ys[$next_ys_ind]=$(( $prev_x - $prev_bl ))'_'$next_y'_'$next_bla
           next_ys_ind=$(( $next_ys_ind + 1 ))
           next_ys[$next_ys_ind]=$(( $prev_x + $prev_bl ))'_'$next_y'_'$next_bla
           next_ys_ind=$(( $next_ys_ind + 1 ))
        done 
        curr_ys=( ${next_ys[*]} )
        unset next_ys
    done
    unset curr_ys
}

blankScreen
echo 'Hello. You can enter "h" or "help" to view the documentation at any time, or enter commands to continue. "q" to quit.' 

while true ; do
    draw_screen=y
    read -u 0 -p '$ ' line
    declare -a cmd=( $line ) 
    c=${cmd[0]}
    cmd[0]=''

    case $c in
        c)
            unset _scrn
            blankScreen
            ;;
        d)
            if [ -n "$debug" ]; then debug=''; else debug='y'; fi
            draw_screen=''
            ;;
        h|help)
            echo '
This simple shell draws lines, single pixels, "Y"s, or a "fractal tree" on an ASCII-art "screen". Please enlarge your terminal to at least 100 columns wide & 64 rows high so that the "screen" can be displayed. Available to you our fearless user, are the following one-letter commands (note there is no error-checking):

c   Clear the screen
d   toggle Debug mode. When in debug mode, diagnostic information may be printed, but no screen will be displayed
h   read this Help again
l   draw a Line, format:
    l [x1] [y1] [x2] [y2]
    example:
    l 1 22 3 45
    x1,x2,y1,y2 should all be integers within the screen (0<=x<=100, 0<=y<=63)
p   draw a "Pixel" on the screen, format:
    p [row(y)] [col(x)]
    example:
    p 54 3
q   feeling Queasy, say buhbaiee
t   draw a "fractal Tree" with a branch iteration between 1 and 5:
    t [iter-level]
    example:
    t 5
y   draw a single "Y" or branching structure, format:
    y [root-x] [root-y] [stem-length]
    example:
    y 23 34 5

**Other commands, and parameters that fall outside valid ranges will result in undefined behavior such as:
    a) this script crashing
    b) inter-galactic thermonuclear war resulting in death & destruction at an apocalyptic scale
    c) your dog eating your child"s food and quietly peeing on the carpet yet again
'    
            draw_screen=''
            ;;
        l)
            drawLine ${cmd[@]}
            ;;
        p)
            drawWood ${cmd[@]}
            ;;
        q)
            break    
            ;;
        t)
            drawTree ${cmd[@]}
            ;;
        y)
            drawY ${cmd[@]}
            ;;
        *)
            echo 'I seem to be running with an nonexistent amount of disk space... ;-D'
            draw_screen=''
            ;;
    esac
    [ -n "$draw_screen" -a -z "$debug" ] && echo ${_scrn[*]} | tr ' ' $'\n'
    unset cmd
done;

Recently my Hobbit PA50ii blew a crankcase bearing.. Fun Stuff! So I was forced to either ditch my heavy, powerful motorcycle, or fix it. I chose the latter. After some fun times on treatland.tv (closest shipper to my town and a sweet outfit regardless), I received crankcase bearings, seals, full gasket kit and other goodies in the mail and it was time to get down & oily.

Being the ‘ped expert that I am, I put everything back together only to discover that old Betsy refused to idle and ran like [insert-expletive-here]. After trolling the forums on mopedarmy and spending more time in front of a blue phosphorescent LED array on youtube and other virtual venues I was convinced that my ‘ped’s woes probably boiled-down to some type of gross vaccuum leak. Since the bike would not even idle and surged dangerously when given even little blips on the throttle I had to go whole-hog and run a goood ol’ fashioned leakdown test. $200 and an air compressor later (just kidding…) $20 and a bicycle pump later I came up with this:

The goal is to seal up all the holes in the engine that should be there, then put a little air pressure, in order to detect the holes that should not be there. Soapy water served up from a common spray bottle is used to detect air leaks. Many guides have been written on this subject so I will not go in to them here. Here are the individual moving parts that, together, make up this glorious contraption:

The expansion plug was fit into the intake. Along with the spark plug, which is left tightly in place, this only leaves one hole in the system that should be there – the exhaust. I used the exhaust to introduce air pressure. At first, I thought I could simply plug up the exhaust port on the cylinder, but found that this was not possible because the port was not round! That is why my solution was to reuse a section of my old Hobbit Stock Exhaust pipe and plug the end of that. Since we are not running the engine for the test, parts will not get hot, and it is possible to use a rubber grommet at the exhaust port to make a seal with the lip of the pipe. I inserted the solid rubber stopper in the end of the pipe section after drilling a 3/64″ hole in the stopper and squeezing an ordinary basketball (or volleyball, etc,..) inflation needle through the hole. This allows a fairly air-tight seal, while allowing an ordinary bicycle pump to be attached to the end of the needle to put pressure into the system. This is the expansion plug in the intake:

Apparently we only want to put about 10 p.s.i. of pressure. More, and you risk doing damage.

Using this tester I figured out that my crankcase seals were on backwards (17mm ID was swapped with 15mm ID), then after fixing the seals, that my engine did not seem to have any leaks.

BTW, this exhaust-side tester only works when the piston is at Bottom-Dead-Center:

2-stroke snaps courtesy of http://www.animatedengines.com/twostroke.html

Thanks to these people for ideas and initial guidance:

Homemade Moped Leak Down Tester – Beyond the Carb Cleaner Check

Cheap DIY Two Stroke Leakdown Tester

<script type="text/javascript">
var rege = /Amstrad|Android|AvantGo|BlackBerry|Blazer|Brew|Cricket|Danger|DoCoMo|Elaine|Gamma|HD7|HP iPAQ|HTC|IEMobile|iPhone|iTunes|j2me|LG[/\-]|Linux arm|Mobile|MOT-|MSIE|Nintendo|Nitro|Nokia|nook browser|Opera Mini|Opera Mobi|Palm|PDA|phone|PLAYSTATION|PS2|Samsung|Sanyo|SEC-|SonyEricsson|Sprint|SPV|Symbian|SymbOS|Tablet|Teleca|Trident|Tungsten|Vodafone|WebPro/i;
var is_mobile = rege.test(navigator.userAgent);
</script>

REST stands for REpresentational State Transfer. A REST-ful web service is one that emphasizes the ideals of the REST architecture – flexibility (scalability & generality), low coupling (client and server can manage their resources however they want as long as they agree on the API), and lack of constraints with message formats (when compared to equivalent architectures like CORBA). For a more complete discussion of REST straight from the horse’s mouth see Roy Fielding’s PhD dissertation.

The idea of REpresentational State Transfer can be derived from thinking of computers on a network as having resources (data such as files, database records, etc,..) that they exchange in order to implement distributed applications. The components of a distributed application do not reside in one place (as is the case with a stand-alone program on a single computer such as a text editor), but across multiple machines (or hosts). The concept of “state transfer” refers to the ability of a client (the machine that initiates the transfer) to request CRUD (Create,Read,Update,Delete) transactions from a server. Each transaction will cause a change in state either on the client (requester) or the server (responder). “Read” transactions change the state of the client, while “Create”,”Update”, and “Delete” transactions are meant to change the state of resources on the server.

A Web Browser talking with a Web Server in order to implement the distributed application we know of as the browsable internet is a basic example of RESTful Web Services at work. Of course there are many other software & hardware layers involved in supporting the internet (per the OSI model), but let’s ignore those for now. We all know that your basic webpage address (also called a URL, or Uniform Resource Locator), starts with “http://” or “https://”. This is because the browser and web server are using HTTP (HyperText Transfer Protocol) or HTTPS (HTTP-Secure) to negotiate transactions such as loading a certain webpage or receiving file uploads. HTTP is a request/response protocol in that a client (such as the browser) requests a resource (such as a webpage “/exclusives/superpark-16-day-4-photos-recap/index.html” from host “www.snowboardermag.com”), and the server responds with an HTTP code and response body (the webpage if the request was successful).

Here is a sample request/response for the aforementioned webpage (slightly simplified)..

Browser says:

GET http://www.snowboardermag.com/exclusives/superpark-16-day-4-photos-recap/ HTTP/1.1
Host: www.snowboardermag.com
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:12.0) Gecko/20100101 Firefox/12.0

Server responds:

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Date: Tue, 29 May 2012 07:31:44 GMT
Server: Apache/2.2.3 (CentOS)

<!DOCTYPE html>
<!--[if IE 6]><html id="IE6" lang="en-US"><![endif]-->
<!--[if IE 7]><html id="IE7" lang="en-US"><![endif]-->
<!--[if IE 8]><html id="IE8" lang="en-US"><![endif]-->
<!--[if (gt IE 8)|!(IE)]><!-->
<html lang="en-US">
<!--<![endif]-->

    <head prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#">

        <meta charset="UTF-8" />
        <title>Superpark 16: Day 4 Photos & Recap | Snowboarder Magazine</title>
        <meta name="description" content="&lt;strong&gt;Recap: T. Bird&lt;/strong&gt;&lt;br clear=left&gt;
...

Let’s look at some key components of the request and response. The browser issues a “GET” method request for a resource named “http://www.snowboardermag.com/exclusives/superpark-16-day-4-photos-recap/” from a host (server) named “www.snowboardermag.com”. It also sends a few other bits of potentially useful information. The response from the server contains an HTTP code, 200 (which means “OK”), and the requested resource (the webpage). There are other HTTP request methods, but browsers mostly use only GET or POST. There are also many other HTTP response codes web servers will typically use:

• 301 “Moved Permanently”
• 302 “Found”
• 400 “Bad Request”
• 403 “Forbidden”
• 404 “Not Found”
• 500 “Internal Server Error”

To see another of these return codes in action, let’s check out a goofy request for a non-existent resource:

Browser says:

GET http://www.google.com/easter-bunny-and-his-furry-friends-on-mars/ HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:12.0) Gecko/20100101 Firefox/12.0

Server responds:

HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Date: Tue, 29 May 2012 07:40:34 GMT
  
  <title>Error 404 (Not Found)!!1</title>
...
  <p>The requested URL <pre>/easter-bunny-and-his-furry-friends-on-mars/</pre> was not found on this server.  <ins>That's all we know.</ins>
</p>

A wise browser will realize the folly of the request and let the user know to go fish. The server has also provided webpage content to alert the user in a more human-readable way that a request has been made for a resource that does not exist. The browser will display that content.

So there it is, a simple trimmed-down example of a REST Web Service. If we desire more functionality than just uploading/downloading content between a browser and web server, we will need to use other HTTP request methods besides GET and POST. Here is a more complete list of the methods we could use:

• GET
• POST
• DELETE
• PUT
• TRACE
• OPTIONS

HTTP request methods function like verbs. With the browser example a request can “GET” a resource (such as a webpage or image file), or “POST” data to the server for it to save as a file or in a database (this is the case when submitting an online form or uploading a document). The other methods are similar but rarely used with browsers. They are however available for other applications that use a REST API (Application Programming Interface). Each request will still contain 1) a method, 2) a resource identifier, and 3) optionally a request body (such as content to upload). Each response will contain 1) a response code, 2) optionally a response body (such as a webpage).

Request/Response Body Content is classified by what format is used to transmit information. As can be seen in the example responses above, “text/html” is a common response body “Content-Type” (also called a MIME type). There are a myriad of other content types available for request/response bodies. Here are some commonly used types:

• image/jpeg
• text/plain
• audio/mpeg
• text/xml
• application/json

MIME, or content-types are very similar to file extensions in that they identify what format the file (or content) is in and consequently, how best to interpret it. In fact, each of the above content-types could be given an extension if it were a file:

The last 2 types are particularly important for RESTful Web Services and are commonly used. JSON (JavaScript Object Notation) is a format that can be interpreted as Javascript (among other languages) and is a flexible, powerful, yet human-readable way to encode and transmit software objects and array data. XML is a generic markup language that is similar to HTML. In fact, HTML is a subset of XML.

This quick introduction will come to a REST with an example REST request that an application could use to GET a list of buckets (data storage containers) held by a certain user’s account, and the server’s XML response. The specific example is not so important, I’m just trying to illustrate how a REST API can be used outside of the browser/web-server examples given above to complete a client/server transaction. Note that the requested resource “/” refers to the “root” or top-level index of resources the server provides. The “Authorization” field in the request identifies to the server which account the application wants to query and provides authentication credentials to prove it is authorized to make the request. So the resulting request is asking the server for a top-level index of “S3 bucket” resources that are held by the account identified and authenticated in the “Authorization” field. In this particular example, the API specifies that the response body content-type will be XML. It could’ve also been in another format such as plain text or JSON.

(adapted from GET Service – Amazon Simple Storage Service 5/28/2012)

Application says:

GET / HTTP/1.1
Host: s3.amazonaws.com
Date: Wed, 01 Mar  2009 12:00:00 GMT
Authorization: AWS AKIAIOSFODNN7EXAMPLE:xQE0diMbLRepdf3YB+FIEXAMPLE=

Server responds:

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Tue, 29 May 2012 08:36:39 GMT
Server: AmazonS3

<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://doc.s3.amazonaws.com/2006-03-01">
  <Owner>
    <ID>bcaf1ffd86f461ca5fb16fd081034f</ID>
    <DisplayName>webfile</DisplayName>
  </Owner>
  <Buckets>
    <Bucket>
      <Name>quotes</Name>
      <CreationDate>2006-02-03T16:45:09.000Z</CreationDate>
    </Bucket>
    <Bucket>
      <Name>samples</Name>
      <CreationDate>2006-02-03T16:41:58.000Z</CreationDate>
    </Bucket>
  </Buckets>
</ListAllMyBucketsResult>

Lately I’ve been working a lot with modern JS frameworks like JQuery, MooTools, and Prototype. Coming from a procedural programming background the emphasis on anonymous functions, chaining, and some other things took some getting used-to. Although these frameworks are powerful and convenient, it can be nice to work with just pure Javascript sometimes for the sake of speed and to minimize dependencies on libraries outside of the browser.

Since being a kid, I occasionally have been prone to taking dozens of photographs as I travel or even during day-to-day strolls about the neighborhood. Around 2004 I looked into a few solutions for hosting and displaying my photos over the web. Overall I was disappointed with how heavy-weight and non-portable many of these solutions seemed to be. For example, there was no way most of these would render on my flip-phone and still look halfway decent on a full browser. So, I decided to scrounge around the web for Javascript code to do the job, and after adding a preloader, and more event handling came up with a super-simple, light-weight back-end/front-end solution in PHP & JS.

var num_of_slides = 35;
var slide_num = 0; // start at the beginning
var desc = new Array(num_of_slides);    // optional per image
var pics = new Array(num_of_slides);    // file paths
var imgs = new Array(num_of_slides);    // the actual image data
var auto = 0;   // auto-advance? this could be boolean
var interval;   // ms between slide in auto-mode
var tID = null; // timeout between slides in auto-mode
var load_cnt = 0;   // number of images that have been pre-loaded
var lID = null; // timeout for preloading all images

// most of the following are self-explanatory
function setInterval(i) {
    interval = i;
}
function closeWindow() {
    window.close();
}
// following functions ignore requests if in auto-mode
function firstSlide() {
    if (!auto) {
        slide_num = 0;
        changeSlide();
    }
}
function prevSlide() {
    if (!auto) { 
        slide_num = slide_num - 1;
        if (slide_num < 0) {
            slide_num = 0;
        }
        changeSlide();
    }
}
// only function that needs to differentiate between user/auto request
function nextSlide(called_by) {
    if (slide_num == num_of_slides - 1) { // if on last slide turn off auto
        auto = 0;
        document.getElementById('autoOn').checked = false;
    } else {
        if ((auto) && (called_by == "usr")) {
            return;
        }
        slide_num = slide_num + 1;
        changeSlide();
        if (auto) {
            tID = setTimeout('nextSlide(auto)', interval);
        }
    }
}
function lastSlide() {
    if (!auto) {
        slide_num = num_of_slides - 1;
        changeSlide();
    }
}
function changeSlide() {
    // following may not need to be evals
    eval('document.picbox.src = pics[slide_num]');
    eval('document.descform.descbox.value = desc[slide_num]');
}
function toggleAuto() {
    if (document.getElementById('autoOn').checked == true) {
        auto = 1;
        tID = setTimeout('nextSlide(auto)', interval);
    } else {
        clearTimeout(tID);
        auto = 0;
    }
}
// called when either all images are preloaded, or the preload timeout has fired
function slidesReady() {
    clearTimeout(lID);
    lID = null;
    // swap "still-loading" content for slideshow content
    document.getElementById('content').style.display = 'block';
    document.getElementById('loading').style.display = 'none';
}
function isLoaded() {
    // keep track of how many images preloaded and stop when done
    if (++load_cnt >= (num_of_slides-1)) {
        slidesReady();
    }
}
function preload() {
    // set a timeout for preloading. Currently set to 300 ms per image,
    // could be more dynamic given info about image size and connection speed
    lID = setTimeout('slidesReady()', 300*num_of_slides);
    for (i = 0; i < num_of_slides && lID != null; i++) {
        imgs[i] = new Image();
        imgs[i].src = pics[i];
        imgs[i].onLoad = isLoaded(); // example of functional-style/callback
    }
}

window.focus();
// following block parses the URL query string
// equivalent of $_GET['start'] only on the client side
// no error-checking is done here.
var loc_str = window.location + "";
URL_array = loc_str.split(/\?/);
var query_str = URL_array[URL_array.length - 1];
params = query_str.split(/&/);
for (i = 0; i < params.length; i++) {
	if (params[i].match(/start=/)) {
	    slide_num = parseInt(params[i].replace(/start=/, ""));
	}
}
// ---------------------------------
desc[0] = "An invisible 747";
pics[0] = "/photos/101.jpg";
desc[1] = "A Green Giraffe on Rollerblades";
pics[1] = "/photos/a_photo.jpg";
// ...
// slide data is statically defined here. of course it could come from anywhere, such as an AJAX request or crunching a hidden content on the page

Looks kinda-like Java but with more built-in front-end web functionality. This is how it renders on my 2008 Blackberry with its tiny, sharp display:



This is the gallery that points to the slides:

Like most command-line habitués, I’ve been using SSH and it’s buddies (scp, rsync, svn+ssh, sftp, vpn, tunneling), since around the time I wrote my first #include “stdio.h” main(){printf(“hello world\n”);} , and just got used to typing in username/password pairs every time I had to log in. Of course as with email, e-commerce sites, forums, and everything else on the wonderful intrawebs, I was schooled from an early stage to prioritize security (i.e.; not using my first name as the user and my dog’s name as the pass on all my accounts). This makes frequent log-ins on a dozen or so accounts considerably more enjoyable..

With a few recent projects the need to frequently login on many machines, and create automated tasks requiring user-less logins (backups,etc,..) got me looking into public/private keys and other methods. I had heard that there were all kinds of slick tricks that could be done with SSH, including baking cakes and putting satellites into orbit, but my intention was to achieve some level of convenience/security without reading a treatise the length of Crime and Punishment.

Public/Private keys

Without dissecting the full mechanism of encrypted communication let it be said that public/private keys are similar to physical locks and keys; once a lock is made, it is possible for an unauthorized entity to pick it, but it is relatively very difficult (depending on encryption type and key length). The “public” key is like a physical lock, and the “private” key like the physical key that is used to open it, through a challenge/response protocol. So it is relatively ok for many others (public) to know about the lock but we probably want to keep the key to ourselves. In order to use the lock/key pair we install a public key on the server we want to log in to, and keep the private key on our client. If we have more than one of these for logging into many machines it becomes a “keychain” in effect. Nuts & bolts for creating/storing/installing the key pairs:

# type can be dsa or rsa, generated on the client account. for
# automated tasks requiring user-less logins it may be necessary
# to use a blank password although this is less secure. There are
# ways around that reqt and one of them will be discussed further down
pancho@client$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/pancho/.ssh/id_dsa): /home/pancho/.ssh/id_dsa_key1        
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/pancho/.ssh/id_dsa_key1.
Your public key has been saved in /home/pancho/.ssh/id_dsa_key1.pub.
The key fingerprint is:
93:80:fe:8a:f9:c2:2a:f5:78:cf:2b:57:b3:8b:cb:f6 pancho@client
The key's randomart image is:
+--[ DSA 1024]----+
|                 |
|    *     **     |
|   **      *     |
|                 |
| *           *   |
|  *         *    |
|   * *******     |
|                 |
|                 |
+-----------------+

Now the private key will be stored as /home/pancho/.ssh/id_dsa_key1 while the public key is stored as /home/pancho/.ssh/id_dsa_key1.pub

There are clever and useful ways to automagically send the public key over to the server account (such as ssh-copy-id) but you can also just log in to it, open up its $HOME/.ssh/authorized_keys file and carefully append the shiny new public key (which is really the lock). It is a good idea to make a backup copy of the file and/or be careful when editing it because it may already contain other keys, and changing them will render their respective key-authenticated logins broken. Here is a sample authorized_keys file:

willamina@server$ cat ~/.ssh/authorized_keys
ssh-dss AAAAB3NzaC1kc3MAAACBAMjOAVT6RdfFjGM2Wk3L9sDe/u7qgFUGHF9kL+jFVLbX9qrPQqBh5wlicrGdL/Fish5u0VfyJD+R/8j71KVXFXLlDOVlxrXCXGBRpRuJwMj2jh0mrA2BaC3DZcCTF5sfOh5Fda+W3o/bvZ9GR+3/54Eov4NvamF/A1EISEuqw8SnAAAAFQCAbdkgeW2+BGJiKLZLmb+F2M2gAwAAAIARQE5/cr1eYCr++gPlGwa93nXu81a9EZm3msPTbfF5UTF3MTt27/nHKnZ2UXJtZo/cn3Eh4T8QS8q3wqeatdq2/oJ2ppbQBbY5dkY3M01he48ffYmVO+PggCAmzD7OuDwF2lpUW6Fg3Agz+xZEYDTuhoe+0jw7aOVw5FhHdKgVzAAAAIEAq8bekAARMEERzUw1sUUX6IiVqvnMWeWojq+bG8OvK43zD8dSTgLXjp3D9sFG4akceUfxrRUViN3oGLljg8F/NyZU8dteWnW6nMBNEuMGMRHcDjOiL4X3wTsZJuSeVHqKFw/d0NgPsPNEbYh03LbdrHvuK5ROvlVcIL50YmBM40Q= pedro@client1
ssh-dss AAAAB3NzaC1kc3MAAACBAJZfNfZ1ibOiWQu5nPghZA+eRLSWm+4QnjtmbaFVb+cb3p9HyNnlvcFCjVwJGYjlW6BbT+bqkoT8A1LVNFwtswd+9FEqm+G/BYJKLRYLHEvZvGiqFx81Y2XSorhAInvMSjFzW4nylb6RtZH0nuGUcLb0dzt/OG//Zr8yJBbXcRG1AAAAFQD63fOcG+OQ6hLoj3Uvez58HgsA7wAAAIEAjs0mwxbilSCXANtQpgLLWgQLXKlpZNpmg5WwLlWhmrPy0llq/9kijFBTaSJ4W6/T363wT6M4xwajDfDyoclzgW16RtmCGtAScGDk3RAQm/R7zKV5h0LKZKBN0b/RdhNUaYSTsJ9JXG+NV98lp+TkJ2bHaC3ffvplVMoFtoSl3TAAAACAZV3vv9lVPoFSHG5LKmh3TI2kIgJssIJCUbPWlMponkqLy5Cx9+xICY8r3zv/MeJK8FMwSuVObRZ1qygoc5OHaIiXHKLm5gW8HvdNzEltemDn//TiaT5HRrVaGY7Kxv1nEzvf+H7H23IQFPvm2NKn/WKGxvORtfj5FGWDpZQfyTA= pancho@client2

It is a good idea to set restrictive permissions on at least the private key and authorized_keys files, such as 0600. It may also be a good idea to change keypairs every now and then. ssh-keygen itself has many other options for managing the keys.

And Voila.. Log out of the server account and next time you log in from the client the server should prompt you for the password you used when the key-pair was generated (or not if none was used). The password-less part is especially useful for automated tasks but bad if someone gets a hold of the private key. So in general we want to keep using passwords, and all we’ve done with this whole key business is add another component to authentication making it a bit more secure. What a waste of time. But wait,.. ..Ladies & Gentlemen. In this corner, weighing in at 256 lbs, with 1024 victories, NaN defeats, hailing from inode 0xBADC0FFEE0DDF00D, block 0xdeadbeef, it’s,… S.S.H. Aagent!!… (cheers and applause) <ding>

ssh-agent

# calling ssh-add registers an identity with an ssh-agent for use in
# locally "un-locking" the private key for use in future log-ins. On
# my system, ssh-agent was already running as a daemon process and the
# necessary environment variables were set. On other systems it may be
# necessary to start/stop the agent through other means.
pancho@client$ env | grep SSH
SSH_AGENT_PID=1234
SSH_AUTH_SOCK=/tmp/ssh-nzpaAalp5678/agent.5678
pancho@client$ ssh-add -l
The agent has no identities.
pancho@client$ ssh-add ~/.ssh/id_dsa_key1
Enter passphrase for /home/pancho/.ssh/id_dsa_key1: 

Now as long as the agent is running and the registered identity is not deleted (which can be done), I can log in to the server with no password.. 😀

I have yet to investigate whether this will fail, but see no reason why identities for automated processes couldn’t also be registered once and remain usable as long as the same agent is running. Passwords would need to be re-entered if the machine is rebooted / ssh-agent killed / etc,..

There are many additional tools for managing ssh-agent and the “keychain” for logging into multiple accounts. They vary in degree of complexity and sophistication and include GUI apps, systems for managing large numbers of users across multiple networks and such, but I won’t go into them here.

As you can imagine, even with keys and ssh-agent, it can be a chore to keep track of all the keys, usernames, ip addresses/hostnames across all the client and server systems. Luckily SSH can use both a system-wide configuration file, and a per-user config to address this.

ssh_config

SSH_CONFIG(5)             Gentoo File Formats Manual             SSH_CONFIG(5)

NAME
     ssh_config -- OpenSSH SSH client configuration files
...
ssh obtains configuration data from the following sources in the following order:

1. command-line options

    2. user's configuration file (~/.ssh/config)
    3. system-wide configuration file (/etc/ssh/ssh_config)
...
The configuration file has the following format: 
...
Host' Restricts the following declarations (up to the next Host keyword) to be only for those hosts that match one of the patterns given after the keyword. '*' and '?' can be used as wildcards in the patterns. A single '*' as a pattern can be used to provide global defaults for all hosts. The host is the hostname argument given on the command line (i.e., the name is not converted to a canonicalized host name before matching). 
...
HostName
Specifies the real host name to log into. This can be used to specify nicknames or abbreviations for hosts. Default is the name given on the command line. Numeric IP addresses are also permitted (both on the command line and in HostName specifications). 
...
IdentityFile
Specifies a file from which the user's RSA or DSA authentication identity is read. The default is ~/.ssh/identity for protocol version 1, and ~/.ssh/id_rsa and ~/.ssh/id_dsa for protocol version 2. Additionally, any identities represented by the authentication agent will be used for authentication. The file name may use the tilde syntax to refer to a user's home directory. It is possible to have multiple identity files specified in configuration files; all these identities will be tried in sequence. 
...
User' Specifies the user to log in as. This can be useful when a different user name is used on different machines. This saves the trouble of having to remember to give the user name on the command line. 
...
SEE ALSO
     ssh(1)

AUTHORS
     OpenSSH is a derivative of the original and free ssh 1.2.12 release by Tatu Ylonen.
     Aaron Campbell, Bob Beck, Markus Friedl, Niels Provos, Theo de Raadt and Dug Song removed
     many bugs, re-added newer features and created OpenSSH.  Markus Friedl contributed the
     support for SSH protocol versions 1.5 and 2.0.

Gentoo/linux                     July 27, 2011                    Gentoo/linux

Yes, there are a ton of options. Many of them quite useful.. The ones shown in my copy & paste from the man page above are what I focused on for my purposes for now. Given a .ssh/config file like:

Host server1
   HostName example.com
   IdentityFile /home/pancho/.ssh/id_dsa_key1
   User willamina

Host server2
   HostName 203.0.113.123
   IdentityFile /home/pancho/.ssh/id_rsa_key2
   User tawhaki

Host daedalus
   HostName foo.org
   IdentityFile /home/pancho/.ssh/id_rsa_key3
   User curly

..I don’t have to keep track of all the hostnames/usernames/keys for all the accounts. Instead I can log-in with a simple “ssh daedalus”. Of course it is possible to use the same key for all the accounts. ssh will still prompt for the key passphrase every time if one was given, but that can be remedied with ssh-agent and ssh-add.

Finally a sample bit of scriptactic sugar to tie it all together (there are other apps/tools to address this such as this and this):

#!/bin/bash
# "host" here corresponds with host lines in local ssh_config file.
# if a key exists for the host, the user will be prompted for its
# passphrase on the 1st login for the running ssh-agent. Afterwards
# since the identity has been added no pass will be necessary

host="$1"

if [ -z "$host" ]; then
	echo 'usage: '`basename $0`' <host>'
	exit 1
fi

IdentityFile=`
	awk '	($1=="Host"){
			last_host=$2
		}
		($1=="IdentityFile" && last_host=="'$host'"){
			print $2
			exit 0
		}' $HOME/.ssh/config`


if [ -n "$IdentityFile" ]; then
	ssh-add -l \
	| grep -q "$IdentityFile" \
	||  ssh-add "$IdentityFile"
fi

/usr/bin/ssh "$host"

Recently I needed to compare a specific set of tables in two MySQL databases on a remote system where I did not have Admin privileges (but DB user credentials). Both databases had the same structure, just some different data. I didn’t want to see all the differences in all the tables because there were probably hundreds of them. Not having access to the server logs, I was considering doing some brute-force regex-driven diff of dump files or perhaps a complex SQL query. I knew there had to be a better way.

After fishing around I found a relatively stand-alone tool (Perl dependency, not much else) that could be set up and used quickly on a non-Admin *nix account: mysql_coldiff. It’s easy-to-use, well-documented, and most-importantly, scriptable..

Since I had about a dozen tables that I wanted to compare I wrote a wrapper script to automate the process and produce a nice report of the differences.

#!/bin/bash
DIFF_SCRIPT="$HOME/mysql_coldiff-1_0/mysql_coldiff"

db4credentials="$1"
query_list="$2"
out_file="$3"

if [ ! -e "$query_list" -o -z "$out_file" ]; then
	echo 'usage: '`basename $0`' <db4credentials> <query_list_file> <out_file>'
	echo '   db: seems only 1 set of credentials can be used for both DBs'
	echo '   query_list_file: list of tables to diff. 1st 2 lines are DB names'
	echo '      remaining lines have 3 fields per record:'
	echo '      <table_name> <id_column_name>  [IGNORE]'
	exit 1
else
	db1=`sed -ne '1 p' "$query_list"`
	db2=`sed -ne '2 p' "$query_list"`
fi

read -p "username for $db4credentials: " us
read -p "password for $db4credentials: " ps

awk '
	($0!="" && $NF!="IGNORE" && NR>2){
		gsub(/[\047"]/,"",$0) # strip devious chars for now
		print "------------------------------\n" $1 "\n------------------------------"
		cmd="'$DIFF_SCRIPT' -h localhost -u '$us' -p '$ps' -i " $2 " -n '$db1'." $1 " '$db2'." $1
		system(cmd);
		close(cmd)
}' "$query_list" > "$out_file"

This strategy seems to work best in situations where differences are minimal (but important). Case-in-point (or bird in hand?), I quickly found the modified field values in a table I had been looking for and changed them with phpMyAdmin. Here is the report I used to find those diffs (output condensed where “…” appears):

...
------------------------------
eav_attribute
------------------------------
We're comparing the the_database_number_1.eav_attribute using the attribute_id column and the database_number2.eav_attribute using the attribute_id column.
...
------------------------------
eav_attribute_group
------------------------------
...
------------------------------
eav_attribute_option
------------------------------
...
------------------------------
eav_entity_text
------------------------------
We're comparing the the_database_number_1.eav_entity_text using the value_id column and the database_number2.eav_entity_text using the value_id column.
...
------------------------------
eav_entity_type
------------------------------
We're comparing the the_database_number_1.eav_entity_type using the entity_type_id column and the database_number2.eav_entity_type using the entity_type_id column.
...
+---------------------------------------+----------------+------------------------------+
|                                       | entity_type_id |              increment_model |
| the_database_number_1.eav_entity_type +----------------+------------------------------+
|_______________________________________|             11 | eav/entity_increment_numeric |
|                                       |             11 | eav/entity_increment_alphabetic
|      database_number2.eav_entity_type +----------------+------------------------------+
|                                       | entity_type_id |              increment_model |
+---------------------------------------+----------------+------------------------------+
...

Gotchas?

• Had to use the same login credentials for both DBs. May be possible to use 2 sets?
• Not sure if mysql_coldiff handles tables where the primary key is not a single identity column, i.e.; tables that use a composite primary key. From glancing over the mysql_coldiff documentation I think it may not matter too much what you choose as the “index column”, but I figured it probably did, that’s why I added the “IGNORE” param to the input file for my wrapper script when I found that one of the tables I was hoping to compare had a composite primary key.
• There’s always ways to further investigate certain problems and solve them better. But if it ain’t broke neither are you.

Here are some rough directions for adding a new driver to the CVS source-tree for Player/Stage. Although these are specific to the “roboteq” driver they should work for other drivers. “…” in a code block means lines were omitted for clarity.

o.k., I have created a patch file for the changes made to the Player source tree in order to add my new Roboteq driver. Only glitch is that the patch does not include the two new files or the new directory:

i.e.;

position/
	roboteq/
		roboteq.cc
		Makefile.am

here is the process:

1. cvs checkout of Player source (a cvs checkout and build is its own process; check Player FAQs for more info)

2. drop the directory for the new driver (“roboteq” — position2d) in “player/server/drivers/position/” with its appropriately edited roboteq.cc (removed the extern “C” Extra stuff for building a shared object, otherwise same as the plugin driver).

3. add a new entry in “player/configure.ac”:

...
dnl Create the following Makefiles (from the Makefile.ams)
AC_OUTPUT(Makefile
...
server/drivers/position/roboteq/Makefile
...

4. add a new entry in “player/server/drivers/position/Makefile.am”:

...
SUBDIRS = isense microstrain vfh ascension bumpersafe lasersafe nav200 nd roboteq
...

5. add new entries in “player/server/libplayerdrivers/driverregistry.cc”:

...
#ifdef INCLUDE_ROBOTEQ
void roboteq_Register (DriverTable* table);
#endif
...
#ifdef INCLUDE_ROBOTEQ
  roboteq_Register(driverTable);
#endif
...

6. add new entry in “player/acinclude.m4”:

...
PLAYER_ADD_DRIVER([roboteq],[yes],[],[],[])
...

7. create “player/server/drivers/position/roboteq/Makefile.am”:

AM_CPPFLAGS = -Wall -I$(top_srcdir)
noinst_LTLIBRARIES =
if INCLUDE_ROBOTEQ
noinst_LTLIBRARIES += libroboteq.la
endif
libroboteq_la_SOURCES = roboteq.cc

8. run the usual

./bootstrap
./configure
./make && make install

if you want to make sure this worked

9. from the top-level source directory (player/)

cvs diff -u > registernewdriver.patch

to make a patch file of any existing files that have changed

10. cvs did not allow me to add any files to the repository without having write-privileges:

$ cvs add roboteq
cvs [server aborted]: "add" requires write access to the repository

so I just uploaded a tar.gz of the new directory with the patch file to patch tracker – don’t know if there is a better way.

This device reads and writes RFID tags. It is made by DLP Design and uses a serial-to-usb converter (Future Technology Devices International, Ltd). Kernel 2.6.20 already includes a driver that will work with this (usbserial and ftdi_sio), but my DLP-RFID1 device has its PID set to a value that is not defined in the kernel header file, so it is not recognized. In my case, “lsusb” listed the device as

ID 0403:fbfc Future Technology Devices International, Ltd

(with PID “fbfc”). As DLP Tech Support explained, the kernel driver is expecting a different PID (0x6001 or 0x6006). This howto is not based off of any suggestions made by them, but the PID hint sent me on my merry way.

ftdi_sio.h

Try plugging the device in and seeing if it is automatically detected.

dmesg | tail

if all you get is something like

usb 4-1: new full speed USB device using uhci_hcd and address 4
usb 4-1: configuration #1 chosen from 1 choice

Then the kernel knows something was plugged in but doesn’t know how to use it. If you also get something like

ftdi_sio 4-1:1.0: FTDI USB Serial Device converter detected
drivers/usb/serial/ftdi_sio.c: Detected FT232BM
usb 4-1: FTDI USB Serial Device converter now attached to ttyUSB0

then you’re set – the device is accessible thorough /dev/ttyUSB0. If not, make sure the kernel is configured to include the driver:

...
CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_FTDI_SIO=m
...

If not change it and recompile the kernel. Unplug and replug the device, if it is still not detected, edit the kernel source.

Fire up your favorite editor and point it to /usr/src/linux/drivers/usb/serial/ftdi_sio.h (kernel 2.6.20) Find the lines that say:

#define FTDI_8U232AM_PID 0x6001 /* Similar device to SIO above */
#define FTDI_8U232AM_ALT_PID 0x6006

and change the “alt_pid” line to:

#define FTDI_8U232AM_ALT_PID 0xfbfc

Save the file and recompile the kernel. Unplugging and replugging the device (and/or loading the modules,) should cause it to be detected now (confirm with dmesg).

This is probably not the best way to do this since now the device listed as pid 6006 won’t be detected; it is a quick hack and it works.   :-p

Also, if I upgrade to the next kernel source my edit will disappear and I will need to edit the new file. I could make a patch, hmm.. Hopefully this device will be included in the next kernel. One of these days I’ll learn how to write kernel patches..

One of the alternatives that was suggested by DLP was to reprogram the device to use a default PID (such as 6001). For now I’ll work on learning the serial settings and protocol to use the DLP-RFID1.