Testing a new SD card under linux

SD cards are a common target for product forgery. Many cards on the market (be it eBay or retailers) are rebranded, have fake capacity or are simply bad ones initially removed from production that are resold afterwards by third parties. Bunnie has an excellent article on fake SD cards and how to test them. For more info on SD cards, check out this flash card survey.

The card I bought, a Sandisk 16GB class 4 Micro SDHC memory card, including adapter

I recently bought an SD card from Sandisk, one of the biggest target for forgeries. It was labeled “Sandisk 16GB class 4 Micro SDHC memory card”. I bought it on eBay, it was coming from China, and it was cheap (16GB for 8€ including shipping). Of course, I wanted to make sure that I got a good one, so I ran a couple of tests. (BTW, I guess the best up-front-protection against fakes is not to buy the cheapest offer on the market.)

Here’s what I did under Ubuntu Linux for testing my card. I ran two tests: one for the performance using flashbench, and another one to do a full-write test in order to reveal capacity problems. My laptop only has a USB card reader, so that I cannot read the manufacturer and other info of the card. Not the best for testing, but well.

Attention: the example below writes to /dev/sdb, which is the device for the card in my case. In your case, it might well be a disk. Make sure that you use the proper device (often something like /dev/mmcblk1 if you have a real card reader).

Running flashbench:

$ ./test_sd_card.sh /dev/sdb
4MiB    10.9M/s
2MiB    4.06M/s
1MiB    4.05M/s
512KiB  2.73M/s
256KiB  1.99M/s
128KiB  924K/s  
64KiB   508K/s  
32KiB   282K/s  
16KiB   194K/s  
8KiB    761K/s  
4KiB    959K/s

So, looking at the flashbench wiki this seems to be fairly reasonable for a class 4 card.

Next, I did the full-write test, to see if the card was actually had the promised capacity. Obviously, you need a bit of space on your disk for this. We’re creating a random data file, write it to the SD card, read it back, and check for differences.

Create a file with the required capacity, filled with random data. We’re writing blocks of 1024 bytes, so adjust the count to the capacity required (16GB in my case).

dd if=/dev/urandom of=rnd_init_data bs=1024 count=16000000

Now, we write the data to the SD card.

$ dd if=rnd_init_data of=/dev/sdb bs=1024 count=16000000
15558145+0 records in
15558144+0 records out
15931539456 bytes (16 GB) copied, 536.286 s, 29.7 MB/s

You can see, we didn’t manage to write all the bytes to the card. This is expected, as we don’t know the capacity of the card down to the last byte. As long as the capacity reported by dd (16 GB) is close to the expected result, we’re happy.

Now, we copy the bytes written to a new file, which we use for later comparison. Notice here the count, which we adapted to the bytes written to the card (15931539456 / 1024 = 15558144). I am sure this can be done easier using tools like truncate, but I didn’t want to start messing with programs that I don’t know by heart.

$ dd if=rnd_init_data of=rnd_written_data bs=1024 count=15558144
$ rm -f rnd_init_data

Read back the data from the card:

$ dd if=/dev/sdb of=rnd_read_data bs=1024 count=15558144

Finally, we’re comparing the two data files:

$ md5sum rnd*
dfed784abed6662926eb01f7fb5359ca  rnd_written_data
dfed784abed6662926eb01f7fb5359ca  rnd_read_data

Et voilà, my card is genuine (or at least does neither have abysmal performance nor fake capacity). Yay, I’ve been lucky!

Don’t forget to clean up (remove files, reformat card):

$ rm -f rnd_written_data rnd_read_data
$ fdisk /dev/sdb    # keys: o, n, p, 1, <accept defaults 2 times>, t, c, w
$ mkfs.vfat -F 32 /dev/sdb1

(The fdisk line creates a single primary FAT32 partition on the card.)

How to get Ubuntu Live running

This article described how to boot Ubuntu from a bootable CD without installing anything on the harddrive. It is meant for the many users of my script that described how to adjust the fan thresholds of a Dell PowerEdge.

Step 1
Download Ubuntu and burn it on a CD. The latest version with 32-bit will do fine.

Step 2
Boot from the CD and select “Try Ubuntu” (don’t install for obvious reasons). This allows you to try Ubuntu without installing anything. You can find more detailed instructions on the download page.

Ubuntu Search Menu

Step 3
After Ubuntu booted, click on the Ubuntu button on the top left. A search menu as seen on the right should open.

Step 4
Type “terminal” into the search box. Ubuntu should give you an icon for the corresponding application. Click on it to start the terminal.

Screenshot of the terminal


Step 5
You should now have a console prompt open. Type the following into the prompt to install the dependencies required for the script:

sudo apt-get install freeipmi freeipmi-tools openipmi ipmitool python wget

This should install the software necessary to query your server using IPMI. IPMI is a protocol for server management. See this link for more information regarding IPMI on Ubuntu.

That’s it! You can continue here: How to adjust the fan thresholds of a Dell PowerEdge.

Python/psycopg2/PostgeSQL: script for bulk inserts using COPY with progress indicator

Woah, what a title. :) I needed a script for inserting bulk data into a PostgreSQL database. Actually, I had a script already, written in Perl, and it was so slow that I needed a better and faster replacement. As I am slowly replacing all my Bash/Perl scripts with Python-pedants I aimed at doing the same here.

I decided to use psycopg2 for a Python-PostgreSQL binding. The copy_from method proved to be very fast; exactly what I needed. BUT I also needed a progress indicator. And while I’ve found a some people out there looking for exactly the same thing, I couldn’t find a solution. So here’s my script for doing this:

#!/usr/bin/python
import psycopg2
import sys
import os

class ReadFileProgress:

  def __init__(self, filename):
    self.datafile = open(filename)
    self.totalRecords = 0
    self.totalBytes = os.stat(filename).st_size
    self.readBytes = 0

    # skip header line
    self.datafile.readline()
    # count records
    for i, l in enumerate(self.datafile):
      pass
    self.totalRecords = i + 1
    sys.stderr.write("Number of records: %d\n" % (self.totalRecords))
    # rewind
    self.datafile.seek(0)
    # skip header line
    self.datafile.readline()
    # start progress
    self.perc5 = self.totalBytes / 20.0
    self.perc5count = 0
    self.lastPerc5 = 0
    sys.stderr.write("Writing records: 0%")

  # count bytes and display progress while doing so
  def countBytes(self, size=0):
    self.readBytes += size
    if (self.readBytes - self.lastPerc5 >= self.perc5):
      self.lastPerc5 = self.readBytes

      if (int(self.readBytes / self.perc5) == 5):
        sys.stderr.write("25%")
      elif (int(self.readBytes / self.perc5) == 10):
        sys.stderr.write("50%")
      elif (int(self.readBytes / self.perc5) == 15):
        sys.stderr.write("75%")
      else:
        sys.stderr.write(".")

      sys.stderr.flush()

  def readline(self, size=None):
    countBytes(size)
    return self.datafile.readline(size)
 
  def read(self, size=None):
    self.countBytes(size)
    return self.datafile.read(size)

  def close(self):
    sys.stderr.write("100%\n")
    self.datafile.close()

def main():
  config = dict()
  config['tablename']="tablename"
  config['filename']="filename"
  config['connstring'] = "host='?' dbname='?' user='?' password='?'"
  config['droptable'] = False
  config['createtable'] = False
  config['rowsdef'] = "id serial PRIMARY KEY, number integer NOT NULL"
  config['filecolumns'] = ['id','number']

  try:
    # get a connection, if a connect cannot be made an exception will be raised here
    conn = psycopg2.connect(config['connstring'])
    # conn.cursor will return a cursor object, you can use this cursor to perform queries
    cursor = conn.cursor()

    # drop table if requested (and it exists)
    cursor.execute("SELECT * FROM information_schema.tables WHERE table_name=%s", (config['tablename'],))
    if (config['droptable'] and bool(cursor.rowcount)):
      cursor.execute("DROP TABLE "+config['tablename']+";")

    # create the table if requested
    if (config['createtable'] and not bool(cursor.rowcount)):
      cursor.execute("CREATE TABLE "+config['tablename']+" ("+config['rowsdef']+");")

    # create a fileprogress object and copy the data to the database
    datafile=ReadFileProgress(config['filename'])
    cursor.copy_from(file=datafile, table=config['tablename'], sep='\t', null='\N', size=8192, columns=config['filecolumns'])
    datafile.close()

    # commit and clsoe
    cursor.close()
    conn.commit()

    sys.stdout.write("Transaction finished successfully.\n")

  except:
    exceptionType, exceptionValue, exceptionTraceback = sys.exc_info()
    sys.exit("Database connection failed!\n ->%s" % (exceptionValue))


if __name__ == "__main__":
  sys.exit(main())

Basically, the progress counter is a wrapper around the file object. It simply outputs the percentage of bytes read. The main program is very straight forward. The script was written with an experiment-specific config file parsing which I do not include here. Thus, you have to find your own way to set the config variables in the beginning of the main program.

Latex+Beamer+PDF+embedded movies

Finally I found a way to embed videos into Latex-generated PDFs in a platform-independent way!

Download and install flashmovie.sty:

wget http://mirror.ctan.org/macros/latex/contrib/flashmovie.zip
unzip flashmovie.zip
cp flashmovie/flashmovie.sty $YOUR_TEX_DOCUMENT_DIR

Additionally, you need a flash-based video player. Free option:

cp flashmovie/flashmovie/player_flv_maxi.swf $YOUR_TEX_DOCUMENT_DIR

I preferred the Longtail Video Player which is free for personal use:

wget http://www.longtailvideo.com/jw/upload/mediaplayer.zip
unzip mediaplayer.zip
cp mediaplayer-5.7/player.swf $YOUR_TEX_DOCUMENT_DIR

This is the mencoder command I use for converting my videos:

mencoder -nosound -forceidx -of lavf -ovc lavc -lavcopts vcodec=flv:vbitrate=2500:mbd=2:mv0:trell:v4mv:cbp:last_pred=3 -o video.flv video.avi

And finally, a bare example showing how to create an embedded, full-screen movie in a presentation using the beamer package:

\RequirePackage{flashmovie}
\documentclass[utf8x]{beamer}
\usepackage[absolute,overlay]{textpos}
\setlength{\TPHorizModule}{1mm}
\setlength{\TPVertModule}{1mm}

\begin{document}
  \begin{frame}[plain]
    \begin{textblock}{12.8}(0,0)
      \flashmovie[auto=1,loop=1,controlbar=0,engine=jw-player,width=12.8cm,height=9.6cm]{taros_talk.flv}
    \end{textblock}
  \end{frame}
\end{document
}

The options are pretty much self-explanatory. Ah yes, the whole thing needs an Adobe Reader >=9.0.

Update: The latest version of Adobe Reader for Linux (9.4-2) gives a “3D parsing error” upon opening a page with an embedded flash video. Downgrade to 9.4-1 (Ubuntu package name is acroread_9.4-1, you can get it here) and everything’s fine

Minecraft on Linux

Minecraft Logo
I struggled to get Minecraft up and running on my laptop (Ubuntu 10.04). I copied an older user folder from a windows machine, and nothing worked – I only got a black screen after starting the game. This seems to be quite common, and seems to be cause by library mismatches.

There are a million complicated howtos out there how to fix this, but for me it was sufficient to let Minecraft re-download all its libraries:

$ rm -rf ~/.minecraft/bin

Then, start the game and wait for the download to finish. Et voila!

The folder on windows is %APPDATA%\Roaming\.minecraft\bin\. An alternative route is to select “force update” in the Minecraft option screen.

Dell DRAC/Remote Console discoveries

1. Logout else lockout

The system has usually a limited number of users, and if you don’t logout properly multiple times you find yourself in the position that all slots have been taken. In this case, you have to wait for a timeout…

2. Firefox & Java

On Linux, the only supported browser is Firefox. On standard Ubunut, this does not work in combination with the DRAC as Ubuntu installs the Icetea Java plugin by default. Switch to the Sun Java plugin:

sudo update-alternatives --set mozilla-javaplugin.so /usr/lib/jvm/java-6-sun/jre/lib/i386/libnpjp2.so

3. Firewalls

On many systems, you will have problems to access the DRAC (or at least the remote console) because of firewalls in your way. You can forward the whole thing by using a host inside the network as an proxy. For this, you have to tunnel the connections through SSH as follows:

sudo ssh -L 443:<DRAC_IP>:443 -L 5900:<DRAC_IP>:5900 -L 5901:<DRAC_IP>:5901 -l <LOGIN> -N <SSH_PROXY_SERVER> -o ExitOnForwardFailure=yes

Update: I created a project page for my server.

Mute sound on screen lock

My coworkers got annoyed that my music continued playing when I was away from my desk. As I am anyways locking the screen automatically after 5 minutes (if I forget to do it by keyboard shortcut), I thought I might use this event for muting as well.

Below is the script I came up with. It needs the inotify-tools (sudo apt-get install inotify-tools) to work. Thanks to Vermind on the ubuntuforums for figuring out the mute stuff!

The advantage of this script is that it works with the auto-lock of the screensaver as well as the pre-configured shortcuts for screen locking.

#!/bin/bash
################################################################################
# mute-on-lock.sh
# Mutes sound and pauses amarok when the screen gets locked
# Version 1.0
#
# written by Arnuschky, mute stuff by Vermind (ubuntu forums)
#
# http://ubuntuforums.org/showthread.php?t=1364597
################################################################################

########################################
# Settings
########################################

mute=1            # Mute the sound while locked. 0=do not mute.
amarok=1          # Pause/unpause amarok on lock
card=0            # Sound card. 0 == default, first card.
channel="Master"  # Which volume to control. Examples: Master, PCM, Headphone
step=3            # Fade step: The volume increment/decrement amount
interval=0.2      # Seconds between increments/decrements when fading


########################################
# Functions
########################################

# get current volume
function getvol {
    vol=$( ${m} sget ${channel} | awk '
$0 ~ "%" {
    if ($4 ~ "%") {
        print $3;
    } else if ($5 ~ "%") {
    print $4;
    exit 0;
    }
}'
)
}

# fades volume up or down
function fade {
    # get current volume
    getvol
    if [ "$1" == "down" ]; then
        # fade down:
        while [ ${vol} -gt 0 ]; do
            ${m} set ${channel} ${step}- >/dev/null
            let vol-=${step}
        sleep ${interval}
        done
        ${m} set ${channel} mute >/dev/null
   else
       # first unmute, then fade up
        ${m} set ${channel} unmute >/dev/null
        while [ ${orig} -gt ${vol} ]; do
            let vol+=${step}
            if [ ${vol} -gt ${orig} ]; then vol=${orig}; fi
            ${m} set ${channel} ${step}+ >/dev/null
            sleep ${interval}
        done
        ${m} set ${channel} ${orig} >/dev/null
   fi
}

########################################
# Startup
########################################

if [ "${mute}" == "1" ]; then
    # short mixer command with card
    m="amixer -c ${card}"
    # record original volume value
    getvol
    orig=$vol
fi

########################################
# Loop                                 #
########################################

# start dbus monitor
monitorfile=`mktemp`
dbus-monitor --session "type='signal',interface='org.gnome.ScreenSaver',member='ActiveChanged'" &> ${monitorfile} &
pid=$!

# loop on changes on the monitorfile
while inotifywait -e modify ${monitorfile} &> /dev/null; do
  # on file change, grab the last line and filter it
  lastline=`tail -n1 ${monitorfile}`
  echo ${lastline} | grep -e "boolean true" -e "boolean false" &> /dev/null
  if [ $? -ne 0 ]; then
    continue
  fi

  # grab the lock state and act on it
  lockstate=`echo ${lastline} | sed 's/^.*boolean \(.*\)/\1/'`
  if [ ${lockstate} == "true" ]; then
    if [ "${mute}" == "1" ]; then
      fade down
    fi
    if [ "${amarok}" == "1" ]; then
      amarok --pause
    fi
  elif [ ${lockstate} == "false" ]; then
    if [ "${amarok}" == "1" ]; then
      amarok --play
    fi
    if [ "${mute}" == "1" ]; then
      fade up
    fi
  fi
done

# kill monitor & cleanup
kill $pid
rm -f ${monitorfile}

exit 0

How to install this? Put it into e.g. /home/youruser/bin/mute-on-lock.sh. Then, go to System -> Preferences -> Startup Applications and add your script. Log in& out and you’re done!

Installing Dell OpenManage on Ubuntu 11.04 64bit

Installing Dell OpenManage 6.50 on Ubuntu 11.04 64bit (Meerkat? Olifant? damn names) was a pain. Mostly because Debian-based systems are only supported since recently and thus there is too much outdated info out there (mostly grisly hacks how to force the rpm-based install to work on your system).

Thus, here a very quick summary:

  1. Follow instructions on this page:http://linux.dell.com/repo/community/deb/latest/
  2. Afterwards, in order to be able to authenticate, edit /etc/pam.d/omauth as follows:
    -auth required /lib32/security/pam_unix.so nullok
    -auth required /lib32/security/pam_nologin.so
    -account required /lib32/security/pam_unix.so nullok
    +auth required /lib/x86_64-linux-gnu/security/pam_unix.so nullok
    +auth required /lib/x86_64-linux-gnu/security/pam_nologin.so
    +account required /lib/x86_64-linux-gnu/security/pam_unix.so nullok

Surprisingly painless (if I would have had this information 5 hours earlier). Ah yes, don’t forget to open your firewall for port tcp 1311… :)

Update: I created a project page for my server.

TraMoS – Traffic Management over SNMP

General Info

This software allows to query, display and administrate Linux traffic shaping rules over SNMP. TraMoS was a project I did at university, and I used it for a while at home.

The server-side is based on SNMP (d’oh), tc and htb. More infos on the underlying Linux software can be found at the Linux Advanced Routing howto. I wrote the server side, which consists of a SNMP daemon interfacing with NET-SNMP, in C.

The client is a Java Swing client that allows to graphically administrate hierarchical rulesets. It has been written by a collegue of mine.

Download TraMoS

Unfortunately, I’ve lost the final version of the documentation, but here’s the final, working software:

Required SNMP Server

The project requires Net-SNMP as a base daemon.
Net-SNMP includes everything required (libraries and the like). The server side part of TraMos is an extension to Net-SNMP.

MIB Module Informations

Our own OID prefix is: 1.3.6.1.4.1.19399.8.42

As the hierarchical structure of the HTB traffic shaping classes could not be directly modeled in a MIB tree, all classes are mapped to flat tables in the tree.

Download MIBs

Here’s the MIB tree:
+--traMIB(1)
   |
   +--traIfTable(1)
   |  |
   |  +--traIfEntry(1)
   |     |  Index: traIfID
   |     |
   |     +-- -R-- INTEGER   traIfID(1)
   |     |        Range: 1..2147483647
   |     +-- -R-- String    traIfName(2)
   |     |        Size: 0..255
   |     +-- -R-- String    traIfComment(3)
   |     |        Size: 0..1024
   |     +-- -R-- INTEGER   traIfTraffic(4)
   |              Range: 0..2147483647
   |
   +--traClassTable(2)
   |  |
   |  +--traClassEntry(1)
   |     |  Index: traClassIfID, traClassID
   |     |
   |     +-- -R-- INTEGER   traClassIfID(1)
   |     |        Range: 1..2147483647
   |     +-- -R-- INTEGER   traClassID(2)
   |     |        Range: 1..2147483647
   |     +-- -R-- INTEGER   traClassParent(3)
   |     |        Range: -1..2147483647
   |     +-- -R-- String    traClassComment(4)
   |     |        Size: 0..1024
   |     +-- -R-- INTEGER   traClassRate(5)
   |     |        Range: 1..2147483647
   |     +-- -R-- INTEGER   traClassCeil(6)
   |     |        Range: 1..2147483647
   |     +-- -R-- INTEGER   traClassBurst(7)
   |     |        Range: 1..2147483647
   |     +-- -R-- INTEGER   traClassPriority(8)
   |     |        Range: 1..2147483647
   |     +-- -R-- INTEGER   traClassQuantum(9)
   |              Range: 1..2147483647
   |
   +--traRuleTable(3)
   |  |
   |  +--traRuleEntry(1)
   |     |  Index: traRuleIfID, traRuleClassID, traRuleID
   |     |
   |     +-- -R-- INTEGER   traRuleIfID(1)
   |     |        Range: 1..2147483647
   |     +-- -R-- INTEGER   traRuleClassID(2)
   |     |        Range: 1..2147483647
   |     +-- -R-- INTEGER   traRuleID(3)
   |     |        Range: 1..2147483647
   |     +-- -R-- String    traRuleProtocol(4)
   |     |        Size: 0..15
   |     +-- -R-- String    traRuleDest(5)
   |     |        Size: 0..19
   |     +-- -R-- INTEGER   traRuleDestPort(6)
   |     |        Range: 0..65536
   |     +-- -R-- String    traRuleSource(7)
   |     |        Size: 0..19
   |     +-- -R-- INTEGER   traRuleSourcePort(8)
   |     |        Range: 0..65536
   |     +-- -R-- String    traRuleTos(9)
   |     |        Size: 0..15
   |     +-- -R-- String    traRuleLength(10)
   |              Size: 0..15
   |
   +--traStatTable(4)
   |  |
   |  +--traStatEntry(1)
   |     |  Index: traStatIfID, traStatID
   |     |
   |     +-- -R-- INTEGER   traStatIfID(1)
   |     |        Range: 1..2147483647
   |     +-- -R-- INTEGER   traStatID(2)
   |     |        Range: 1..2147483647
   |     +-- -R-- String    traStatProtocol(3)
   |     |        Size: 0..15
   |     +-- -R-- INTEGER   traStatPort(4)
   |     |        Range: 1..65536
   |     +-- -R-- INTEGER   traStatTraffic(5)
   |              Range: 1..2147483647
   |
   +--traMIBConformance(6)
      |
      +--traMIBCompliances(1)
      |  |
      |  +--traMIBCompliance(1)
      |
      +--traMIBGroups(2)
         |
         +--traGroup(1)