HackerRank “BASh Recursive Trees” Solution

This BASh script was written during the process of solving the Functions and Fractals – Recursive Trees – Bash! problem on Hacker-Rank. Yes it is slight overkill for that problem, but adds neat interesting features that no sane developer should implement in a shell script – such as drawing arbitrary lines, “Y” shapes, and recursive trees to any number of iterations (“screen” size and computational/memory/time-constraints withstanding) on an ASCII-art “screen”. Why not a more-limited, simpler approach to the problem? During the dev process, I found that I benefitted from incrementally breaking down the problem to its simpler components: reliably drawing any arbitrary pixel, then a straight line, then a “Y”, then any number of those Y’s on the screen to form a tree. Probably the trickiest part was deciphering exactly *what* idiosyncratic method the author of the Hacker-Rank problem had used to draw his full tree, then replicating that in a manner that was automatic and repeatable across iterations. Although high-school level, the math was the fun part.

Can also be downloaded here

declare -a _scrn

start_bl=$(( ($_rows + 1) / ( $max_iters - 1) ))
trunk_col=`expr $_cols / 2`

declare debug=''

function blankScreen()
    blank=`yes 2>/dev/null | head -n $(( $_cols + 1 )) | tr -d y | paste -sd_ -`
    for ((i=0; i<_rows; i++)); do

declare new_row_val
function flipRow()
    new_row_val=$(( $_rows - $1 - 1 ))

function drawWood()
    row=$1; col=$2
    flipRow $row

function drawLine()
    x1=$1; y1=$2; x2=$3; y2=$4
    if [ $x1 -eq $x2 ]; then
        m=`echo "scale=4; (${y2} - ${y1})/(${x2} - ${x1})" | bc`
        b=`echo "scale=4; ${y1} - ${m} * ${x1}" | bc`
    for y in `seq $y1 $y2`; do
        if [ $x1 -ne $x2 ]; then
            x=`printf '%.0f\n' $(echo "scale=4; (${y} - ${b})/${m}" | bc)`
        if [ -n "$debug" ]; then
            echo drawWood $y $x 1>&2
            drawWood $y $x

function drawY()
    start_x=$1; start_y=$2; stem_len=$3
    drawLine $start_x   $start_y    $start_x    $(( $start_y + $stem_len ))
    start_y=$(( $start_y + $stem_len + 1 ))
    drawLine $(( $start_x - 1 ))    $start_y    \
            $(( $start_x - $stem_len - 1 ))     $(( $start_y + $stem_len ))
    drawLine $(( $start_x + 1 ))    $start_y    \
            $(( $start_x + $stem_len + 1 ))     $(( $start_y + $stem_len ))

function drawTree()
    declare -a curr_ys
    bla=$(( $start_bl - 1 ))

    for curr_iter in `seq 1 $iters`; do
        declare -a next_ys
        for y in ${curr_ys[*]}; do
           declare -a unpakt=( `echo $y | tr _ ' '` )
           drawY ${unpakt[*]}
           prev_bl=$(( $prev_bla + 1 ))
           next_y=$(( $prev_y + $prev_bl * 2 ))
           next_bla=$(( $prev_bl / 2 - 1 ))
           next_ys[$next_ys_ind]=$(( $prev_x - $prev_bl ))'_'$next_y'_'$next_bla
           next_ys_ind=$(( $next_ys_ind + 1 ))
           next_ys[$next_ys_ind]=$(( $prev_x + $prev_bl ))'_'$next_y'_'$next_bla
           next_ys_ind=$(( $next_ys_ind + 1 ))
        curr_ys=( ${next_ys[*]} )
        unset next_ys
    unset curr_ys

echo 'Hello. You can enter "h" or "help" to view the documentation at any time, or enter commands to continue. "q" to quit.' 

while true ; do
    read -u 0 -p '$ ' line
    declare -a cmd=( $line )

    case $c in
            unset _scrn
            if [ -n "$debug" ]; then debug=''; else debug='y'; fi
            echo '
This simple shell draws lines, single pixels, "Y"s, or a "fractal tree" on an ASCII-art "screen". Please enlarge your terminal to at least 100 columns wide & 64 rows high so that the "screen" can be displayed. Available to you our fearless user, are the following one-letter commands (note there is no error-checking):

c   Clear the screen
d   toggle Debug mode. When in debug mode, diagnostic information may be printed, but no screen will be displayed
h   read this Help again
l   draw a Line, format:
    l [x1] [y1] [x2] [y2]
    l 1 22 3 45
    x1,x2,y1,y2 should all be integers within the screen (0<=x<=100, 0<=y<=63)
p   draw a "Pixel" on the screen, format:
    p [row(y)] [col(x)]
    p 54 3
q   feeling Queasy, say buhbaiee
t   draw a "fractal Tree" with a branch iteration between 1 and 5:
    t [iter-level]
    t 5
y   draw a single "Y" or branching structure, format:
    y [root-x] [root-y] [stem-length]
    y 23 34 5

**Other commands, and parameters that fall outside valid ranges will result in undefined behavior such as:
    a) this script crashing
    b) inter-galactic thermonuclear war resulting in death & destruction at an apocalyptic scale
    c) your dog eating your child"s food and quietly peeing on the carpet yet again
            drawLine ${cmd[@]}
            drawWood ${cmd[@]}
            drawTree ${cmd[@]}
            drawY ${cmd[@]}
            echo 'I seem to be running with an nonexistent amount of disk space... ;-D'
    [ -n "$draw_screen" -a -z "$debug" ] && echo ${_scrn[*]} | tr ' ' $'\n'
    unset cmd
Posted in Uncategorized | Leave a comment

Home-Made Hobbit Leakdown Tester

Recently my Hobbit PA50ii blew a crankcase bearing.. Fun Stuff! So I was forced to either ditch my heavy, powerful motorcycle, or fix it. I chose the latter. After some fun times on treatland.tv (closest shipper to my town and a sweet outfit regardless), I received crankcase bearings, seals, full gasket kit and other goodies in the mail and it was time to get down & oily.

Being the ‘ped expert that I am, I put everything back together only to discover that old Betsy refused to idle and ran like [insert-expletive-here]. After trolling the forums on mopedarmy and spending more time in front of a blue phosphorescent LED array on youtube and other virtual venues I was convinced that my ‘ped’s woes probably boiled-down to some type of gross vaccuum leak. Since the bike would not even idle and surged dangerously when given even little blips on the throttle I had to go whole-hog and run a goood ol’ fashioned leakdown test. $200 and an air compressor later (just kidding…) $20 and a bicycle pump later I came up with this:

Home-made Moped Leakdown Tester

The goal is to seal up all the holes in the engine that should be there, then put a little air pressure, in order to detect the holes that should not be there. Soapy water served up from a common spray bottle is used to detect air leaks. Many guides have been written on this subject so I will not go in to them here. Here are the individual moving parts that, together, make up this glorious contraption:

Moving Parts

The expansion plug was fit into the intake. Along with the spark plug, which is left tightly in place, this only leaves one hole in the system that should be there – the exhaust. I used the exhaust to introduce air pressure. At first, I thought I could simply plug up the exhaust port on the cylinder, but found that this was not possible because the port was not round! That is why my solution was to reuse a section of my old Hobbit Stock Exhaust pipe and plug the end of that. Since we are not running the engine for the test, parts will not get hot, and it is possible to use a rubber grommet at the exhaust port to make a seal with the lip of the pipe. I inserted the solid rubber stopper in the end of the pipe section after drilling a 3/64″ hole in the stopper and squeezing an ordinary basketball (or volleyball, etc,..) inflation needle through the hole. This allows a fairly air-tight seal, while allowing an ordinary bicycle pump to be attached to the end of the needle to put pressure into the system. This is the expansion plug in the intake:

Apparently we only want to put about 10 p.s.i. of pressure. More, and you risk doing damage.

Using this tester I figured out that my crankcase seals were on backwards (17mm ID was swapped with 15mm ID), then after fixing the seals, that my engine did not seem to have any leaks.

BTW, this exhaust-side tester only works when the piston is at Bottom-Dead-Center:


2-stroke snaps courtesy of http://www.animatedengines.com/twostroke.html

Thanks to these people for ideas and initial guidance:

Homemade Moped Leak Down Tester – Beyond the Carb Cleaner Check

Cheap DIY Two Stroke Leakdown Tester

Posted in Uncategorized | Leave a comment

Quick-n-Dirty Mobile Browser Regex Test

<script type="text/javascript">
var rege = /Amstrad|Android|AvantGo|BlackBerry|Blazer|Brew|Cricket|Danger|DoCoMo|Elaine|Gamma|HD7|HP iPAQ|HTC|IEMobile|iPhone|iTunes|j2me|LG[/\-]|Linux arm|Mobile|MOT-|MSIE|Nintendo|Nitro|Nokia|nook browser|Opera Mini|Opera Mobi|Palm|PDA|phone|PLAYSTATION|PS2|Samsung|Sanyo|SEC-|SonyEricsson|Sprint|SPV|Symbian|SymbOS|Tablet|Teleca|Trident|Tungsten|Vodafone|WebPro/i;
var is_mobile = rege.test(navigator.userAgent);
Posted in Uncategorized | Leave a comment

Give Your Web Services a REST

REST stands for REpresentational State Transfer. A REST-ful web service is one that emphasizes the ideals of the REST architecture – flexibility (scalability & generality), low coupling (client and server can manage their resources however they want as long as they agree on the API), and lack of constraints with message formats (when compared to equivalent architectures like CORBA). For a more complete discussion of REST straight from the horse’s mouth see Roy Fielding’s PhD dissertation.

The idea of REpresentational State Transfer can be derived from thinking of computers on a network as having resources (data such as files, database records, etc,..) that they exchange in order to implement distributed applications. The components of a distributed application do not reside in one place (as is the case with a stand-alone program on a single computer such as a text editor), but across multiple machines (or hosts). The concept of “state transfer” refers to the ability of a client (the machine that initiates the transfer) to request CRUD (Create,Read,Update,Delete) transactions from a server. Each transaction will cause a change in state either on the client (requester) or the server (responder). “Read” transactions change the state of the client, while “Create”,”Update”, and “Delete” transactions are meant to change the state of resources on the server.

A Web Browser talking with a Web Server in order to implement the distributed application we know of as the browsable internet is a basic example of RESTful Web Services at work. Of course there are many other software & hardware layers involved in supporting the internet (per the OSI model), but let’s ignore those for now. We all know that your basic webpage address (also called a URL, or Uniform Resource Locator), starts with “http://” or “https://”. This is because the browser and web server are using HTTP (HyperText Transfer Protocol) or HTTPS (HTTP-Secure) to negotiate transactions such as loading a certain webpage or receiving file uploads. HTTP is a request/response protocol in that a client (such as the browser) requests a resource (such as a webpage “/exclusives/superpark-16-day-4-photos-recap/index.html” from host “www.snowboardermag.com”), and the server responds with an HTTP code and response body (the webpage if the request was successful).

Here is a sample request/response for the aforementioned webpage (slightly simplified)..

Browser says:

GET http://www.snowboardermag.com/exclusives/superpark-16-day-4-photos-recap/ HTTP/1.1
Host: www.snowboardermag.com
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:12.0) Gecko/20100101 Firefox/12.0

Server responds:

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Date: Tue, 29 May 2012 07:31:44 GMT
Server: Apache/2.2.3 (CentOS)

<!DOCTYPE html>
<!--[if IE 6]><html id="IE6" lang="en-US"><![endif]-->
<!--[if IE 7]><html id="IE7" lang="en-US"><![endif]-->
<!--[if IE 8]><html id="IE8" lang="en-US"><![endif]-->
<!--[if (gt IE 8)|!(IE)]><!-->
<html lang="en-US">

    <head prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#">

        <meta charset="UTF-8" />
        <title>Superpark 16: Day 4 Photos & Recap | Snowboarder Magazine</title>
        <meta name="description" content="&lt;strong&gt;Recap: T. Bird&lt;/strong&gt;&lt;br clear=left&gt;

Let’s look at some key components of the request and response. The browser issues a “GET” method request for a resource named “http://www.snowboardermag.com/exclusives/superpark-16-day-4-photos-recap/” from a host (server) named “www.snowboardermag.com”. It also sends a few other bits of potentially useful information. The response from the server contains an HTTP code, 200 (which means “OK”), and the requested resource (the webpage). There are other HTTP request methods, but browsers mostly use only GET or POST. There are also many other HTTP response codes web servers will typically use:

  • 301 “Moved Permanently”
  • 302 “Found”
  • 400 “Bad Request”
  • 403 “Forbidden”
  • 404 “Not Found”
  • 500 “Internal Server Error”

To see another of these return codes in action, let’s check out a goofy request for a non-existent resource:

Browser says:

GET http://www.google.com/easter-bunny-and-his-furry-friends-on-mars/ HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:12.0) Gecko/20100101 Firefox/12.0

Server responds:

HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Date: Tue, 29 May 2012 07:40:34 GMT

  <title>Error 404 (Not Found)!!1</title>
  <p>The requested URL <pre>/easter-bunny-and-his-furry-friends-on-mars/</pre> was not found on this server.  <ins>That's all we know.</ins>

A wise browser will realize the folly of the request and let the user know to go fish. The server has also provided webpage content to alert the user in a more human-readable way that a request has been made for a resource that does not exist. The browser will display that content.

So there it is, a simple trimmed-down example of a REST Web Service. If we desire more functionality than just uploading/downloading content between a browser and web server, we will need to use other HTTP request methods besides GET and POST. Here is a more complete list of the methods we could use:

  • GET
  • POST
  • PUT

HTTP request methods function like verbs. With the browser example a request can “GET” a resource (such as a webpage or image file), or “POST” data to the server for it to save as a file or in a database (this is the case when submitting an online form or uploading a document). The other methods are similar but rarely used with browsers. They are however available for other applications that use a REST API (Application Programming Interface). Each request will still contain 1) a method, 2) a resource identifier, and 3) optionally a request body (such as content to upload). Each response will contain 1) a response code, 2) optionally a response body (such as a webpage).

Request/Response Body Content is classified by what format is used to transmit information. As can be seen in the example responses above, “text/html” is a common response body “Content-Type” (also called a MIME type). There are a myriad of other content types available for request/response bodies. Here are some commonly used types:

  • image/jpeg
  • text/plain
  • audio/mpeg
  • text/xml
  • application/json

MIME, or content-types are very similar to file extensions in that they identify what format the file (or content) is in and consequently, how best to interpret it. In fact, each of the above content-types could be given an extension if it were a file:

image/jpeg: .jpg
text/plain: .txt
audio/mpeg: .mp3
text/xml: .xml
application/json .json

The last 2 types are particularly important for RESTful Web Services and are commonly used. JSON (JavaScript Object Notation) is a format that can be interpreted as Javascript (among other languages) and is a flexible, powerful, yet human-readable way to encode and transmit software objects and array data. XML is a generic markup language that is similar to HTML. In fact, HTML is a subset of XML.

This quick introduction will come to a REST with an example REST request that an application could use to GET a list of buckets (data storage containers) held by a certain user’s account, and the server’s XML response. The specific example is not so important, I’m just trying to illustrate how a REST API can be used outside of the browser/web-server examples given above to complete a client/server transaction. Note that the requested resource “/” refers to the “root” or top-level index of resources the server provides. The “Authorization” field in the request identifies to the server which account the application wants to query and provides authentication credentials to prove it is authorized to make the request. So the resulting request is asking the server for a top-level index of “S3 bucket” resources that are held by the account identified and authenticated in the “Authorization” field. In this particular example, the API specifies that the response body content-type will be XML. It could’ve also been in another format such as plain text or JSON.

(adapted from GET Service – Amazon Simple Storage Service 5/28/2012)

Application says:

GET / HTTP/1.1
Host: s3.amazonaws.com
Date: Wed, 01 Mar  2009 12:00:00 GMT

Server responds:

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Tue, 29 May 2012 08:36:39 GMT
Server: AmazonS3

<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://doc.s3.amazonaws.com/2006-03-01">
Posted in Uncategorized | Leave a comment

Javascript Sans JPath

Lately I’ve been delving a lot more into modern JS frameworks like JQuery, MooTools, and Prototype. Coming from a procedural programming background the emphasis on anonymous functions, chaining, and some other things takes some getting used-to. Reminds me of Lisp, Python, or even Prolog (no not really). For kicks I decided to dig up an old-school pure Javascript piece of code that began as some samples I adapted from other pages around 2004 or so, and improve upon it – add a preloader, more event handling, etc,.. Sans XPath, here is what I came up with:

// can be set by back-end
var num_of_slides = 35;
// horrifying C-style globals
var slide_num = 0; // start at the beginning
var desc = new Array(num_of_slides);    // optional per image
var pics = new Array(num_of_slides);    // file paths
var imgs = new Array(num_of_slides);    // the actual image data
var auto = 0;   // auto-advance? this could be boolean
var interval;   // ms between slide in auto-mode
var tID = null; // timeout between slides in auto-mode
var load_cnt = 0;   // number of images that have been pre-loaded
var lID = null; // timeout for preloading all images

// most of the following are self-explanatory
function setInterval(i) {
    interval = i;
function closeWindow() {
// following functions ignore requests if in auto-mode
function firstSlide() {
    if (!auto) {
        slide_num = 0;
function prevSlide() {
    if (!auto) {
        slide_num = slide_num - 1;
        if (slide_num < 0) {
            slide_num = 0;
// only function that needs to differentiate between user/auto request
function nextSlide(called_by) {
    if (slide_num == num_of_slides - 1) { // if on last slide turn off auto
        auto = 0;
        document.getElementById('autoOn').checked = false;
    } else {
        if ((auto) && (called_by == "usr")) {
        slide_num = slide_num + 1;
        if (auto) {
            tID = setTimeout('nextSlide(auto)', interval);
function lastSlide() {
    if (!auto) {
        slide_num = num_of_slides - 1;
function changeSlide() {
    // following may not need to be evals
    eval('document.picbox.src = pics[slide_num]');
    eval('document.descform.descbox.value = desc[slide_num]');
function toggleAuto() {
    if (document.getElementById('autoOn').checked == true) {
        auto = 1;
        tID = setTimeout('nextSlide(auto)', interval);
    } else {
        auto = 0;
// called when either all images are preloaded, or the preload timeout has fired
function slidesReady() {
    lID = null;
    // swap "still-loading" content for slideshow content
    document.getElementById('content').style.display = 'block';
    document.getElementById('loading').style.display = 'none';
function isLoaded() {
    // keep track of how many images preloaded and stop when done
    if (++load_cnt >= (num_of_slides-1)) {
function preload() {
    // set a timeout for preloading. Currently set to 300 ms per image,
    // could be more dynamic given info about image size and connection speed
    lID = setTimeout('slidesReady()', 300*num_of_slides);
    for (i = 0; i < num_of_slides && lID != null; i++) {
        imgs[i] = new Image();
        imgs[i].src = pics[i];
        imgs[i].onLoad = isLoaded(); // example of functional-style/callback

// following block parses the URL query string
// equivalent of $_GET['start'] only on the client side
// no error-checking is done here.
var loc_str = window.location + "";
URL_array = loc_str.split(/\?/);
var query_str = URL_array[URL_array.length - 1];
params = query_str.split(/&/);
for (i = 0; i < params.length; i++) {
	if (params[i].match(/start=/)) {
	    slide_num = parseInt(params[i].replace(/start=/, ""));
// ---------------------------------
desc[0] = "An invisible 747";
pics[0] = "/photos/101.jpg";
desc[1] = "A Green Giraffe on Rollerblades";
pics[1] = "/photos/a_photo.jpg";
// ...
// slide data is statically defined here. of course it could come from anywhere, such as an AJAX request or crunching a hidden content on the page

Looks a bit like C, only without pointers, and with more built-in regular expression support. Not as OO as Java or Python. A quick read on the history of ECMAScript is useful. Admitted, this is not the most sophisticated example of JavaScript code, but the Big JS frameworks (around since ~2006) seem like an evolutionary step (paradigm shift?) from this. I am still wondering when/if browsers will just support the new coding conventions/functionality natively.

Posted in Uncategorized | Leave a comment

ssh-keygen, ssh-agent, ssh_config, and their password-less buddies

Like most command-line habitu├ęs, I’ve been using SSH and it’s buddies (scp, rsync, svn+ssh, sftp, vpn, tunneling), since around the time I wrote my first #include “stdio.h” main(){printf(“hello world\n”);} , and just got used to typing in username/password pairs every time I had to log in. Of course as with email, e-commerce sites, forums, and everything else on the wonderful intrawebs, I was schooled from an early stage to prioritize security (i.e.; not using my first name as the user and my dog’s name as the pass on all my accounts). This makes frequent log-ins on a dozen or so accounts considerably more enjoyable..

With a few recent projects the need to frequently login on many machines, and create automated tasks requiring user-less logins (backups,etc,..) got me looking into public/private keys and other methods. I had heard that there were all kinds of slick tricks that could be done with SSH, including baking cakes and putting satellites into orbit, but my intention was to achieve some level of convenience/security without reading a treatise the length of Crime and Punishment.

Public/Private keys

Without dissecting the full mechanism of encrypted communication let it be said that public/private keys are similar to physical locks and keys; once a lock is made, it is possible for an unauthorized entity to pick it, but it is relatively very difficult (depending on encryption type and key length). The “public” key is like a physical lock, and the “private” key like the physical key that is used to open it, through a challenge/response protocol. So it is relatively ok for many others (public) to know about the lock but we probably want to keep the key to ourselves. In order to use the lock/key pair we install a public key on the server we want to log in to, and keep the private key on our client. If we have more than one of these for logging into many machines it becomes a “keychain” in effect. Nuts & bolts for creating/storing/installing the key pairs:

# type can be dsa or rsa, generated on the client account. for
# automated tasks requiring user-less logins it may be necessary
# to use a blank password although this is less secure. There are
# ways around that reqt and one of them will be discussed further down
pancho@client$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/pancho/.ssh/id_dsa): /home/pancho/.ssh/id_dsa_key1
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/pancho/.ssh/id_dsa_key1.
Your public key has been saved in /home/pancho/.ssh/id_dsa_key1.pub.
The key fingerprint is:
93:80:fe:8a:f9:c2:2a:f5:78:cf:2b:57:b3:8b:cb:f6 pancho@client
The key's randomart image is:
+--[ DSA 1024]----+
|                 |
|    *     **     |
|   **      *     |
|                 |
| *           *   |
|  *         *    |
|   * *******     |
|                 |
|                 |

Now the private key will be stored as /home/pancho/.ssh/id_dsa_key1 while the public key is stored as /home/pancho/.ssh/id_dsa_key1.pub

There are clever and useful ways to automagically send the public key over to the server account (such as ssh-copy-id) but you can also just log in to it, open up its $HOME/.ssh/authorized_keys file and carefully append the shiny new public key (which is really the lock). It is a good idea to make a backup copy of the file and/or be careful when editing it because it may already contain other keys, and changing them will render their respective key-authenticated logins broken. Here is a sample authorized_keys file:

willamina@server$ cat ~/.ssh/authorized_keys
ssh-dss AAAAB3NzaC1kc3MAAACBAMjOAVT6RdfFjGM2Wk3L9sDe/u7qgFUGHF9kL+jFVLbX9qrPQqBh5wlicrGdL/Fish5u0VfyJD+R/8j71KVXFXLlDOVlxrXCXGBRpRuJwMj2jh0mrA2BaC3DZcCTF5sfOh5Fda+W3o/bvZ9GR+3/54Eov4NvamF/A1EISEuqw8SnAAAAFQCAbdkgeW2+BGJiKLZLmb+F2M2gAwAAAIARQE5/cr1eYCr++gPlGwa93nXu81a9EZm3msPTbfF5UTF3MTt27/nHKnZ2UXJtZo/cn3Eh4T8QS8q3wqeatdq2/oJ2ppbQBbY5dkY3M01he48ffYmVO+PggCAmzD7OuDwF2lpUW6Fg3Agz+xZEYDTuhoe+0jw7aOVw5FhHdKgVzAAAAIEAq8bekAARMEERzUw1sUUX6IiVqvnMWeWojq+bG8OvK43zD8dSTgLXjp3D9sFG4akceUfxrRUViN3oGLljg8F/NyZU8dteWnW6nMBNEuMGMRHcDjOiL4X3wTsZJuSeVHqKFw/d0NgPsPNEbYh03LbdrHvuK5ROvlVcIL50YmBM40Q= pedro@client1
ssh-dss AAAAB3NzaC1kc3MAAACBAJZfNfZ1ibOiWQu5nPghZA+eRLSWm+4QnjtmbaFVb+cb3p9HyNnlvcFCjVwJGYjlW6BbT+bqkoT8A1LVNFwtswd+9FEqm+G/BYJKLRYLHEvZvGiqFx81Y2XSorhAInvMSjFzW4nylb6RtZH0nuGUcLb0dzt/OG//Zr8yJBbXcRG1AAAAFQD63fOcG+OQ6hLoj3Uvez58HgsA7wAAAIEAjs0mwxbilSCXANtQpgLLWgQLXKlpZNpmg5WwLlWhmrPy0llq/9kijFBTaSJ4W6/T363wT6M4xwajDfDyoclzgW16RtmCGtAScGDk3RAQm/R7zKV5h0LKZKBN0b/RdhNUaYSTsJ9JXG+NV98lp+TkJ2bHaC3ffvplVMoFtoSl3TAAAACAZV3vv9lVPoFSHG5LKmh3TI2kIgJssIJCUbPWlMponkqLy5Cx9+xICY8r3zv/MeJK8FMwSuVObRZ1qygoc5OHaIiXHKLm5gW8HvdNzEltemDn//TiaT5HRrVaGY7Kxv1nEzvf+H7H23IQFPvm2NKn/WKGxvORtfj5FGWDpZQfyTA= pancho@client2

It is a good idea to set restrictive permissions on at least the private key and authorized_keys files, such as 0600. It may also be a good idea to change keypairs every now and then. ssh-keygen itself has many other options for managing the keys.

And Voila.. Log out of the server account and next time you log in from the client the server should prompt you for the password you used when the key-pair was generated (or not if none was used). The password-less part is especially useful for automated tasks but bad if someone gets a hold of the private key. So in general we want to keep using passwords, and all we’ve done with this whole key business is add another component to authentication making it a bit more secure. What a waste of time. But wait,.. ..Ladies & Gentlemen. In this corner, weighing in at 256 lbs, with 1024 victories, NaN defeats, hailing from inode 0xBADC0FFEE0DDF00D, block 0xdeadbeef, it’s,… S.S.H. Aagent!!… (cheers and applause) <ding>


# calling ssh-add registers an identity with an ssh-agent for use in
# locally "un-locking" the private key for use in future log-ins. On
# my system, ssh-agent was already running as a daemon process and the
# necessary environment variables were set. On other systems it may be
# necessary to start/stop the agent through other means.
pancho@client$ env | grep SSH
pancho@client$ ssh-add -l
The agent has no identities.
pancho@client$ ssh-add ~/.ssh/id_dsa_key1
Enter passphrase for /home/pancho/.ssh/id_dsa_key1:

Now as long as the agent is running and the registered identity is not deleted (which can be done), I can log in to the server with no password.. :-D

I have yet to investigate whether this will fail, but see no reason why identities for automated processes couldn’t also be registered once and remain usable as long as the same agent is running. Passwords would need to be re-entered if the machine is rebooted / ssh-agent killed / etc,..
wonderful key hanger by Janice Warren
There are many additional tools for managing ssh-agent and the “keychain” for logging into multiple accounts. They vary in degree of complexity and sophistication and include GUI apps, systems for managing large numbers of users across multiple networks and such, but I won’t go into them here.

As you can imagine, even with keys and ssh-agent, it can be a chore to keep track of all the keys, usernames, ip addresses/hostnames across all the client and server systems. Luckily SSH can use both a system-wide configuration file, and a per-user config to address this.


SSH_CONFIG(5)             Gentoo File Formats Manual             SSH_CONFIG(5)

     ssh_config -- OpenSSH SSH client configuration files
ssh obtains configuration data from the following sources in the following order:

1. command-line options

    2. user's configuration file (~/.ssh/config)
    3. system-wide configuration file (/etc/ssh/ssh_config)
The configuration file has the following format:
Host' Restricts the following declarations (up to the next Host keyword) to be only for those hosts that match one of the patterns given after the keyword. '*' and '?' can be used as wildcards in the patterns. A single '*' as a pattern can be used to provide global defaults for all hosts. The host is the hostname argument given on the command line (i.e., the name is not converted to a canonicalized host name before matching).
Specifies the real host name to log into. This can be used to specify nicknames or abbreviations for hosts. Default is the name given on the command line. Numeric IP addresses are also permitted (both on the command line and in HostName specifications).
Specifies a file from which the user's RSA or DSA authentication identity is read. The default is ~/.ssh/identity for protocol version 1, and ~/.ssh/id_rsa and ~/.ssh/id_dsa for protocol version 2. Additionally, any identities represented by the authentication agent will be used for authentication. The file name may use the tilde syntax to refer to a user's home directory. It is possible to have multiple identity files specified in configuration files; all these identities will be tried in sequence.
User' Specifies the user to log in as. This can be useful when a different user name is used on different machines. This saves the trouble of having to remember to give the user name on the command line.

     OpenSSH is a derivative of the original and free ssh 1.2.12 release by Tatu Ylonen.
     Aaron Campbell, Bob Beck, Markus Friedl, Niels Provos, Theo de Raadt and Dug Song removed
     many bugs, re-added newer features and created OpenSSH.  Markus Friedl contributed the
     support for SSH protocol versions 1.5 and 2.0.

Gentoo/linux                     July 27, 2011                    Gentoo/linux

Yes, there are a ton of options. Many of them quite useful.. The ones shown in my copy & paste from the man page above are what I focused on for my purposes for now. Given a .ssh/config file like:

Host server1
   HostName example.com
   IdentityFile /home/pancho/.ssh/id_dsa_key1
   User willamina

Host server2
   IdentityFile /home/pancho/.ssh/id_rsa_key2
   User tawhaki

Host daedalus
   HostName foo.org
   IdentityFile /home/pancho/.ssh/id_rsa_key3
   User curly

..I don’t have to keep track of all the hostnames/usernames/keys for all the accounts. Instead I can log-in with a simple “ssh daedalus”. Of course it is possible to use the same key for all the accounts. ssh will still prompt for the key passphrase every time if one was given, but that can be remedied with ssh-agent and ssh-add.

Finally a sample bit of scriptactic sugar to tie it all together (there are other apps/tools to address this such as this and this):

# "host" here corresponds with host lines in local ssh_config file.
# if a key exists for the host, the user will be prompted for its
# passphrase on the 1st login for the running ssh-agent. Afterwards
# since the identity has been added no pass will be necessary


if [ -z "$host" ]; then
	echo 'usage: '`basename $0`' <host>'
	exit 1

	awk '	($1=="Host"){
		($1=="IdentityFile" && last_host=="'$host'"){
			print $2
			exit 0
		}' $HOME/.ssh/config`

if [ -n "$IdentityFile" ]; then
	ssh-add -l \
	| grep -q "$IdentityFile" \
	||  ssh-add "$IdentityFile"

/usr/bin/ssh "$host"
Posted in Uncategorized | 2 Comments

Diff’ing Data Needles Across 2 MySQL Haystacks

Recently I needed to compare a specific set of tables in two MySQL databases on a remote system where I did not have Admin privileges (but DB user credentials). Both databases had the same structure, just some different data. I didn’t want to see all the differences in all the tables because there were probably hundreds of them. Not having access to the server logs, I was considering doing some brute-force regex-driven diff of dump files or perhaps a complex SQL query. I knew there had to be a better way.

After fishing around I found a relatively stand-alone tool (Perl dependency, not much else) that could be set up and used quickly on a non-Admin *nix account: mysql_coldiff. It’s easy-to-use, well-documented, and most-importantly, scriptable..

Since I had about a dozen tables that I wanted to compare I wrote a wrapper script to automate the process and produce a nice report of the differences.



if [ ! -e "$query_list" -o -z "$out_file" ]; then
	echo 'usage: '`basename $0`' <db4credentials> <query_list_file> <out_file>'
	echo '   db: seems only 1 set of credentials can be used for both DBs'
	echo '   query_list_file: list of tables to diff. 1st 2 lines are DB names'
	echo '      remaining lines have 3 fields per record:'
	echo '      <table_name> <id_column_name>  [IGNORE]'
	exit 1
	db1=`sed -ne '1 p' "$query_list"`
	db2=`sed -ne '2 p' "$query_list"`

read -p "username for $db4credentials: " us
read -p "password for $db4credentials: " ps

awk '
	($0!="" && $NF!="IGNORE" && NR>2){
		gsub(/[\047"]/,"",$0) # strip devious chars for now
		print "------------------------------\n" $1 "\n------------------------------"
		cmd="'$DIFF_SCRIPT' -h localhost -u '$us' -p '$ps' -i " $2 " -n '$db1'." $1 " '$db2'." $1
}' "$query_list" > "$out_file"

This strategy seems to work best in situations where differences are minimal (but important). Case-in-point (or bird in hand?), I quickly found the modified field values in a table I had been looking for and changed them with phpMyAdmin. Here is the report I used to find those diffs (output condensed where “…” appears):

We're comparing the the_database_number_1.eav_attribute using the attribute_id column and the database_number2.eav_attribute using the attribute_id column.
We're comparing the the_database_number_1.eav_entity_text using the value_id column and the database_number2.eav_entity_text using the value_id column.
We're comparing the the_database_number_1.eav_entity_type using the entity_type_id column and the database_number2.eav_entity_type using the entity_type_id column.
|                                       | entity_type_id |              increment_model |
| the_database_number_1.eav_entity_type +----------------+------------------------------+
|_______________________________________|             11 | eav/entity_increment_numeric |
|                                       |             11 | eav/entity_increment_alphabetic
|      database_number2.eav_entity_type +----------------+------------------------------+
|                                       | entity_type_id |              increment_model |


  • Had to use the same login credentials for both DBs. May be possible to use 2 sets?
  • Not sure if mysql_coldiff handles tables where the primary key is not a single identity column, i.e.; tables that use a composite primary key. From glancing over the mysql_coldiff documentation I think it may not matter too much what you choose as the “index column”, but I figured it probably did, that’s why I added the “IGNORE” param to the input file for my wrapper script when I found that one of the tables I was hoping to compare had a composite primary key.
  • There’s always ways to further investigate certain problems and solve them better. But if it ain’t broke neither are you.
Posted in Uncategorized | Leave a comment

Player Driver Howto

Here are some rough directions for adding a new driver to the CVS source-tree for Player/Stage. Although these are specific to the “roboteq” driver they should work for other drivers. “…” in a code block means lines were omitted for clarity.

o.k., I have created a patch file for the changes made to the Player source tree in order to add my new Roboteq driver. Only glitch is that the patch does not include the two new files or the new directory:



here is the process:

1. cvs checkout of Player source (a cvs checkout and build is its own process; check Player FAQs for more info)

2. drop the directory for the new driver (“roboteq” — position2d) in “player/server/drivers/position/” with its appropriately edited roboteq.cc (removed the extern “C” Extra stuff for building a shared object, otherwise same as the plugin driver).

3. add a new entry in “player/configure.ac”:

dnl Create the following Makefiles (from the Makefile.ams)

4. add a new entry in “player/server/drivers/position/Makefile.am”:

SUBDIRS = isense microstrain vfh ascension bumpersafe lasersafe nav200 nd roboteq

5. add new entries in “player/server/libplayerdrivers/driverregistry.cc”:

void roboteq_Register (DriverTable* table);

6. add new entry in “player/acinclude.m4″:


7. create “player/server/drivers/position/roboteq/Makefile.am”:

AM_CPPFLAGS = -Wall -I$(top_srcdir)
noinst_LTLIBRARIES += libroboteq.la
libroboteq_la_SOURCES = roboteq.cc

8. run the usual

./make && make install

if you want to make sure this worked

9. from the top-level source directory (player/)

cvs diff -u > registernewdriver.patch

to make a patch file of any existing files that have changed

10. cvs did not allow me to add any files to the repository without having write-privileges:

$ cvs add roboteq
cvs [server aborted]: "add" requires write access to the repository

so I just uploaded a tar.gz of the new directory with the patch file to patch tracker – don’t know if there is a better way.

Posted in Uncategorized | Leave a comment

DLP-RFID1 usb id 0403:fbfc in Linux

This device reads and writes RFID tags. It is made by DLP Design and uses a serial-to-usb converter (Future Technology Devices International, Ltd). Kernel 2.6.20 already includes a driver that will work with this (usbserial and ftdi_sio), but my DLP-RFID1 device has its PID set to a value that is not defined in the kernel header file, so it is not recognized. In my case, “lsusb” listed the device as

ID 0403:fbfc Future Technology Devices International, Ltd

(with PID “fbfc”). As DLP Tech Support explained, the kernel driver is expecting a different PID (0×6001 or 0×6006). This howto is not based off of any suggestions made by them, but the PID hint sent me on my merry way.


Try plugging the device in and seeing if it is automatically detected.

dmesg | tail

if all you get is something like

usb 4-1: new full speed USB device using uhci_hcd and address 4
usb 4-1: configuration #1 chosen from 1 choice

Then the kernel knows something was plugged in but doesn’t know how to use it. If you also get something like

ftdi_sio 4-1:1.0: FTDI USB Serial Device converter detected
drivers/usb/serial/ftdi_sio.c: Detected FT232BM
usb 4-1: FTDI USB Serial Device converter now attached to ttyUSB0

then you’re set – the device is accessible thorough /dev/ttyUSB0. If not, make sure the kernel is configured to include the driver:


If not change it and recompile the kernel. Unplug and replug the device, if it is still not detected, edit the kernel source.

Fire up your favorite editor and point it to /usr/src/linux/drivers/usb/serial/ftdi_sio.h (kernel 2.6.20) Find the lines that say:

#define FTDI_8U232AM_PID 0x6001 /* Similar device to SIO above */
#define FTDI_8U232AM_ALT_PID 0x6006

and change the “alt_pid” line to:

#define FTDI_8U232AM_ALT_PID 0xfbfc

Save the file and recompile the kernel. Unplugging and replugging the device (and/or loading the modules,) should cause it to be detected now (confirm with dmesg).

This is probably not the best way to do this since now the device listed as pid 6006 won’t be detected; it is a quick hack and it works.   :-p

Also, if I upgrade to the next kernel source my edit will disappear and I will need to edit the new file. I could make a patch, hmm.. Hopefully this device will be included in the next kernel. One of these days I’ll learn how to write kernel patches..

One of the alternatives that was suggested by DLP was to reprogram the device to use a default PID (such as 6001). For now I’ll work on learning the serial settings and protocol to use the DLP-RFID1.

Posted in Uncategorized | Leave a comment

Motorola v195 with T-Mobile on Gentoo Linux

The Motorola_v195 can be used as a modem to connect a laptop to the internet wherever cell phone reception is good enough given the right provider plan. I have the internet plan as an add-on to my T-Mobile account..


A mini-usb cable will connect the phone to the laptop. Before connecting, build it a driver. Set


in /usr/src/linux/.config (I am using kernel 2.6.20)

then recompile the kernel:

mount /boot
cd /usr/src/linux
make && make modules_install && make install
umount /boot

Make sure the phone is on and connect the cable. Wait a second or two..

dmesg | tail

should mumble something to the effect of

usb 2-2: new full speed USB device using uhci_hcd and address 3
usb 2-2: configuration #1 chosen from 2 choices
cdc_acm 2-2:1.0: ttyACM0: USB ACM device
usbcore: registered new driver cdc_acm
drivers/usb/class/cdc-acm.c: v0.25:USB Abstract Control Model driver for USB modems and ISDN adapters

You can also check if the new device exists

ls /dev/ttyACM*

Dial-Up using PPP

The gentoo init script method worked for getting the connection to the GPRS network. I plucked the configuration settings from T-Mobile’s Mac_OS_X_guide. You might want to keep an eye on their settings in case they change.

If you don’t already have PPP:

emerge -av ppp

Create a link for the init script:

ln -s /etc/init.d/net.lo /etc/init.d/net.ppp0

Here is my /etc/conf.d/net:

 # /etc/conf.d/net:
 # $Header: /home/cvsroot/gentoo-src/rc-scripts/etc/conf.d/net,v 1.7 2002/11/18 19:39:22 azarah Exp $
 config_eth0=( "null" )
 config_eth1=( "dhcp" )
 dhcpcd_eth1="-t 10"
 dhcp_eth1="release nontp nonis nosendhost"
 # PPP
 config_ppp0=( "ppp" )
 # Each PPP interface requires an interface to use as a "Link"
 link_ppp0="/dev/ttyACM0"   # Most PPP links will use a serial port
 # PPP requires at least a username. You can optionally set a password here too
 # If you don't, then it will use the password specified in /etc/ppp/*-secrets
 # against the specified username
 # NOTE: You can set a blank password like so
 # The PPP daemon has many options you can specify - although there are many
 # and may seem daunting, it is recommended that you read the pppd man page
 # before enabling any of them
 # "maxfail 0" # WARNING: It's not recommended you use this
 #   # if you don't specify maxfail then we assume 0
  "updetach"  # If not set, "/etc/init.d/net.ppp0 start" will return
 #   # immediately,  without waiting the link to come up
 #   # for the first time.
 #   # Do not use it for dial-on-demand links!
  "debug"  # Enables syslog debugging
  "noauth" # Do not require the peer to authenticate itself
  "defaultroute" # Make this PPP interface the default route
  "usepeerdns" # Use the DNS settings provided by PPP
 # On demand options
 # "demand"  # Enable dial on demand
 # "idle 30"  # Link goes down after 30 seconds of inactivity
 # "" # Phony IP addresses
  "ipcp-accept-remote" # Accept the peers idea of remote address
  "ipcp-accept-local" # Accept the peers idea of local address
  "holdoff 3"  # Wait 3 seconds after link dies before re-starting
 # Dead peer detection
 # "lcp-echo-interval 15" # Send a LCP echo every 15 seconds
 # "lcp-echo-failure 3" # Make peer dead after 3 consective
 #    # echo-requests
 # Compression options - use these to completely disable compression
 # novj
  noaccomp noccp nobsdcomp nodeflate nopcomp novjccomp
 # Dial-up settings
 # "lock"    # Lock serial port
  "115200"   # Set the serial port baud rate
 # "modem crtscts"   # Enable hardware flow control
 # "" # Local and remote IP addresses
 # Dial-up PPP users need to specify at least one telephone number
 phone_number_ppp0=( "*99#" ) # Maximum 2 phone numbers are supported
 # They will also need a chat script - here's a good one
 # 'ABORT' 'Invalid Login'
 # 'ABORT' 'Login incorrect'
  'TIMEOUT' '5'
  '' 'AT+IPR=115200'
  'OK' 'ATZ'
  'OK' 'AT+cgdcont=1,"IP","internet3.voicestream.com"' # Put your modem initialization string here
  'OK' 'ATDT\T'
  'TIMEOUT' '60'
  'CONNECT' ''
  'TIMEOUT' '5'
  '~--' ''

A few things to note:

  • The eth0 and eth1 settings (top of file) are irrelevant, but the gateway is set only for eth1, so as long as ppp0 is the only device that is started (up,) the gateway will come from the PPP connection. This is because routing problems will likely be related to interference from other interfaces.
  • Username and password are not specified
  • The only compression option not disabled is “novj” because the T-Mobile instructions ask to use TCP Header Compression
  • The serial port baud rate is set to “115200″
  • The chat script has most of the ABORT lines commented out — e.g.; we’re not looking for a dial tone
  • The
        '' 'AT+IPR=115200'

    chat line will ask the v195 to set it’s baud rate to match PPP (115,200)

Start and stop the connection with the usual suspects:

/etc/init.d/net.ppp0 start
/etc/init.d/net.ppp0 stop

The start script should reply with

 * Starting ppp0
 *   Bringing up ppp0
 *     ppp
 *       Running pppd ...
 *       ppp0 received address xxx.xxx.xxx.xxx/yy

ifconfig ppp0 should confirm this. Go ahead and check your spam,err.. email while you’re out in Coldspot, Saskatchewan.

Posted in Uncategorized | Leave a comment