Stefan Talpalaru, CTO

It's official: prolonged sitting is a risk factor for all-cause mortality, independent of physical activity. Time to switch to a stand-up desk I you haven't done so already.

Category: 42

My Samsung SpinPoint F3 (HD103SJ) hard disk started exposing bad blocks to the operating system after 5210 hours of use (10 months from installation). For some reason it killed the kernel and compromised some filesystem structures preventing booting. At this point, the recommended procedure is to initiate the recovery immediately, copying whatever blocks you can copy to a new disk, but I was reluctant to buy a new hard disk now that the prices are doubled so I went ahead and ran fsck from SystemRescueCD. It worked, with the loss of a few replaceable files, but new bad blocks kept appearing so I caved in and bought a Seagate Barracuda 7200.12 (ST31000524AS) with the same 1TB capacity.

Using SystemRescueCD again I decided to clone individual partitions with GParted. It’s easier to clone the whole disk, but since the original GPT partition table was created on OS X, it wasted about 1.5GB on empty space between partitions (for future super-secret use or whatever reason the appletards have for doing it) and I wanted that space back. So I did the GPT creation and partition copy/paste dance with GParted, only to see it crash when told to retry the bad blocks. Sure enough, that wasn’t the tool I was looking for. Some more digging and I found ddrescue (not to be mistaken with dd_rescue). With some hints from the Forensics Wiki I was able to come up with the following strategy (sda is the bad disk, sdb is the good one):

ddrescue -f -n /dev/sda1 /dev/sdb1 logfile1
ddrescue -f -d -r3 /dev/sda1 /dev/sdb1 logfile1

The first command skips the bad blocks (-n) for a quick copy while the second one tries hard to retrieve them with direct access (-d) retrying for a maximum of 3 times (-r3). I ran the first command in a shell loop for all the partition pairs, and the second one for the 3 that reported bad blocks. One of them was recovered completely (after more than 3 retries, though), one still had 512 bytes left after a few runs and the last one got stuck on the ‘splitting’ phase for more than 15 minutes (weird behavior for approximately 16KB of unreadable data, but it seems that letting ddrescue run for hours/days is not uncommon).

After the recovery, I repaired the filesystems on sdb, eliminating the bad block lists copied from sda:

fsck.ext4 -f -y -L /dev/null /dev/sdb1

Category: Linux

Zeromq sockets are the cat's meow but they do not provide connection status information. Of course, this is by design, so messages can be sent before the endpoint starts listening. There's some transparent buffering going on and in most cases this behavior is actually desired. Sometimes, though, we need to take different actions based on the availability of a server. A background task server, for example. If the server is not alive we might want to run the task in foreground instead of ditching it or letting it wait in the send buffer (think of a short lived client process).

The solution lies in what the zeromq guide calls the Lazy Pirate Pattern - make a pair of request-reply sockets, have the client make a request and wait for a reply within a given timeout. The example there is a bit too complicated, so here's a simplified version:

#!/usr/bin/env python

import zmq
from zmq.eventloop import ioloop
import argparse

context = zmq.Context()
ALIVE_URL = 'tcp://127.0.0.1:5556'
ALIVE_TIMEOUT = 1000 # in ms

def alive_handler(alive_socket, *args, **kwargs):
    request = alive_socket.recv()
    reply = 'pong'
    alive_socket.send(reply)

def server():
    print 'starting the server'
    alive_socket = context.socket(zmq.REP)
    alive_socket.bind(ALIVE_URL)
   
    io_loop = ioloop.IOLoop.instance()
    io_loop.add_handler(alive_socket, alive_handler, io_loop.READ)
    io_loop.start()

def client():
    print 'starting the client'
    alive_socket = context.socket(zmq.REQ)
    alive_socket.connect(ALIVE_URL)
    poll = zmq.Poller()
    poll.register(alive_socket, zmq.POLLIN)
    request = 'ping'
    alive_socket.send(request)
    socks = dict(poll.poll(ALIVE_TIMEOUT))
    if socks.get(alive_socket) == zmq.POLLIN:
        reply = alive_socket.recv()
        print 'the server is alive'
    else:
        print 'consider the server dead'
        # we can't use this socket any more
        alive_socket.setsockopt(zmq.LINGER, 0)
        alive_socket.close()
        poll.unregister(alive_socket)


def main():
    parser = argparse.ArgumentParser(description='zeromq connection status checking demo')
    parser.add_argument('--server', action='store_true')
    parser.add_argument('--client', action='store_true')
    args = parser.parse_args()
    if args.server:
        server()
    elif args.client:
        client()
    else:
        parser.print_help()

if __name__ == '__main__':
    main()

Category: Python

Working with GeoTIFF leads sooner or later to files that have an uncompressed size bigger than the available disk space. This means that it's not possible to resize such a TIFF file with imagemagick (or graphicsmagick) because convert insists on using a temporary file for the raw image data. Sure enough, "converting" from TIFF to TIFF while resizing can be done on the fly, with a small memory buffer and that's exactly what gdal_translate (from the GDAL project) does. Here it is downsizing a huge file in a matter of seconds:

gdal_translate -outsize 10% 10% -co COMPRESS=DEFLATE big.tif small.tif

Category: Linux

If you have a Canon IXUS 220HS (ELPH 300HS) camera with the 1.01G firmware version, this is your lucky day: now you can run CHDK on it.

Category: CHDK

We all know that Skype is a terrible piece of software, with the Linux version buggy and left behind the main code base, but lately it's been crashing like a Top Gear funny men in an overpowered car. Each time it starts and there are chat windows with new messages to be shown, the damn thing segfaults (at least on 64 bit, statically linked against Qt). While trying to find more details about the problem, I stumbled upon an unlikely fix: running skype under gdb seems to stabilize it. Is it the delay introduced by the debugger interfering with some concurrency bugs or some other mechanism at work? No idea. Just that this is how I run the rubbish right now:

LD_LIBRARY_PATH="/opt/skype" gdb /opt/skype/skype

and then type 'r' in the debugger. It hasn't crashed once under gdb so there's no stack trace available.

Category: Linux

If you enable the SandyBridge New Acceleration (SNA) in the xf86-video-intel driver on a GM965 chipset with a dual-monitor setup your GPU will hang with the kernel DRM driver crying like a little girl and both screens going blank. While the rest of xorg is still working and your current session is accessible through x11vnc, you will need to reboot to restore the video output.

Category: Linux

One of the first things you learn about optimizing web applications is how to cache processed data in memory. Just install memcached - the easy to use key-value store - and you're set. Until you reach its limits, that is...

Memcached does not allow keys larger than 250 bytes or values bigger than 1MB. What happens when you really need bigger keys/values? You move on to redis.

If you are using a proper framework (and you should) there's probably already a redis plugin for you (here's the Django one). Now all you have to do is install redis and configure it to suit your caching needs. As a rule of thumb, speed and low resource usage are more important than data persistence. Here's my configuration (only the relevant differences from the default redis.conf):

daemonize yes
bind 127.0.0.1
loglevel notice
logfile /var/log/redis/redis.log
dir /var/lib/redis/
maxmemory 67108864

Category: caching

Starting with django-1.3.1 you'll get redirect failures for reverse proxy setups (cherokee and cherrypy in my case). The reason is that relative paths that you feed to HttpResponseRedirect are converted into absolute URLs using META['HTTP_HOST'] which is actually the IP of the reverse proxy. To fix this add the following line to settings.py:

USE_X_FORWARDED_HOST = True

More details in the release notes.

Category: Django

I don't know how high ranking python is for automating system administration tasks but when I had to script remote ssh commands, I thought I'd give python a try. The main SSH library is paramiko and, while lacking in the documentation department, it's rather rich in features.

Besides the usual root login, I had a more difficult use case: using a regular user with full sudo rights instead of the superuser. These are the functions I came up with:

import paramiko
import getpass

def ssh_connection(user, host, port=22, password=None, key_filename=None):
    """
    with password='' you will be prompted for a password when the script runs
    """
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    if password == '':
        # ask for one on stdin
        password = getpass.getpass('Password for %s@%s: ' % (user, host))
    ssh.connect(host, port=port, username=user, password=password, key_filename=key_filename)
    # custom attributes
    ssh.user = user
    if user == 'root':
        ssh.homedir = '/root'
    else:
        ssh.homedir = '/home/%s' % user
    ssh.password = password
    ssh.use_sudo = False
    return ssh

def run_remote(ssh, cmd, check_exit_status=True, verbose=True):
    chan = ssh.get_transport().open_session()
    stdin = chan.makefile('wb')
    stdout = chan.makefile('rb')
    stderr = chan.makefile_stderr('rb')
    processed_cmd = cmd
    if ssh.use_sudo:
        processed_cmd = 'sudo -S bash -c "%s"' % cmd.replace('"', '\\"')
    chan.exec_command(processed_cmd)
    if stdout.channel.closed is False: # If stdout is still open then sudo is asking us for a password
        stdin.write('%s\n' % ssh.password)
        stdin.flush()
    result = {
        'stdout': [],
        'stderr': [],
    }
    exit_status = chan.recv_exit_status()
    result['exit_status'] = exit_status
    def print_output():
        for line in stdout:
            result['stdout'].append(line)
            print line,
        for line in stderr:
            result['stderr'].append(line)
            print line,
    if check_exit_status and exit_status != 0:
        print_output()
        print 'non-zero exit status (%d) when running "%s"' % (exit_status, cmd)
        exit(exit_status)
    if verbose:
        print processed_cmd
        print_output()
    return result

# set up a ssh connection for root and ask for the password interactively
myconn = ssh_connection('root', 'example.com', password='')
# run some commands with the default settings
run_remote(myconn,
          """
           pwd
           cd /var/log
           pwd
           ls -la
          """)

# set up a connection for a regular user with full sudo rights and use it for running commands with root privileges
sudoconn = ssh_connection('jim', 'example.com', password='imanewbieandileavepasswordsintextfiles')
sudoconn.use_sudo = True
run_remote(sudoconn,
          """
           whoami
           pwd
           cd /var/log
           pwd
           ls -la
          """)

Category: Python