Detecting endianness when making a FreeBSD port makefile

The title really says it all, my porting foo and Google foo was insufficent when I was working on a port for the IMS Corpus Workbench this weekend. I had to turn to the good people at #bsdports@EFnet and to aid me was @stass. He supplied me with what I at time suspected to be the only way, to use a regexp to extract the information from endian.h (yuck!). Here are the two options he provided.

LITTLE_ENDIAN!= ${EGREP} \
'^#define[[:space:]]+_BYTE_ORDER[[:space:]]+[^[:space:]]+$' \
/usr/include/machine/endian.h | ${AWK} '{if ($3 == \
"_LITTLE_ENDIAN") print 1;}'

LITTLE_ENDIAN!= ${AWK} \
'/^#define[[:space:]]+_BYTE_ORDER[[:space:]]+[^[:space:]]+$/
{if ($3 == "_LITTLE_ENDIAN") print 1;}' \
/usr/include/machine/endian.h

These work just fine for an ordinary shell but the same thing in a makefile needs some small adjustments. I prefered the purely AWK one, so here it is.

LITTLE_ENDIAN= ${AWK} \
'/^\#define[[:space:]]+_BYTE_ORDER[[:space:]]+[^[:space:]]+$$/ ' + \
'{if ($$3 == "_LITTLE_ENDIAN") print 1;}' /usr/include/machine/endian.h

Now hopefully the Google Bot will crawl over this and allow the next porter to find what I could not. Once again, thank you @stass. =)

Njorak isn't dead (yet (now lives on a lone E6400))

Yes, I admit it! I am still a Dvorak typer. I tried going back recently when learning Vim using my new Vim book, but I just can't. Something about Qwerty makes me squirm when trying it. Perhaps it's just that I have become accustomed to the strangeness of Dvorak but in general I don't see any reason to switch back. Sure, I am not that quick when I get in front of someone's Qwerty laptop, but I'll manage.

Now with my new Dell E6400 it's time for yet another njorak layout. Since Swedish characters are too precious to miss out on, but not precious enough to get rid of all the good things about pure and clean Dvorak (hmm, I does sound like a fanatic don't I?). Here it is, the all magical njorak for Dell Latitude E6400 using the Windows key to access Swedish characters.

! njorak, not-quite-dvorak. Version 1.2
! By Pontus Stenetorp
! Derived from us dvorak, accesses Swedish characters through
! altgr or similar combos. Also swaps caps and esc.
! Apply it to an ordinary dvorak keymap using xmodmap.
! Dell Latitude E6400 version
! still dedicated to spatrik

! Set up Swedish special characters
keycode 43 = d D aring
keycode 44 = h H adiaeresis
keycode 45 = t T odiaeresis

! Windows key on a Dell Latitude E6400
keycode 115 = Mode_switch

! Swap escape and caps lock
remove Lock = Caps_Lock
remove Control = Escape
keycode 66 = Escape
keycode 9 = Caps_Lock
add Lock = Caps_Lock
add Control = Escape

I have also added a Caps-lock Escape switch to this version. The reason, there is never a good reason not to remove caps-lock. The only I reason I can come up with is that it's not expected behaviour for anyone who is new to your layout. But being a Vim user (and a Dvorak user, which makes most people perplexed when trying out my keyboard) and also having taken a couple of seconds to ponder on which key I use the most, the switch really is a sane thing to do.

Portssearch alias and portmaster

In order to manage FreeBSD ports I have used portupgrade for some time. It provides neat facilites such as the command portinstall and portupgrade, I am sure that the use of these commands is pretty straight forward. The standard way to install a port is the following.

> cd /usr/ports/category/portname
> make install clean

Instead portupgrade allows you to do.

> portinstall portname

Or

> portinstall category/portname

One problem with portupgrade is that it's written in Ruby. One might argue that this is not really a problem, but when you have a fresh FreeBSD install you don't want to drag in Ruby as your first package. This takes time and you would rather focus on something else.

Luckily there is an alternative that I quite recently stumbled upon. portmaster supports pretty much the same set of operations but is written in shell script. It's quick to install and you are ready to go in a jiffy. It also handles fetching and port configuration first and then starts building and installing.

> portmaster category/portname

The only problem is that it enforces the use of category/portname as arguments. This is most likely to avoid conflicts since the portname might conflict with other portnames. To resolve this portupgrade presents all "likely" matches and asks you to choose from them, portmaster just dismisses your arguments as invalid. I have considered adding this functionality in some way. Perhaps printing "likely" matches and/or conflicts and then terminating would be a good option?

So, now we have a tool to handle port installation with ease. Then we need a way to search for ports when we don't know the actual name or category. There are of course several such tools but out of the box ports supplies the following way.

> cd /usr/ports
> make search name=searchstring

Or if you want to search in dependencies etc. you can always use key instead of name. But this usually give you far too many hits to browse through. But the output even from name is usually a bit too much, listing dependencies, web addresses and maintainers. For example.

> cd /usr/ports
> make search name=hugs98
Port: hugs98-200609_2
Path: /usr/ports/lang/hugs
Info: An interpreter for the functional programming language Haskell 98
Maint: haskell@FreeBSD.org
B-deps: bash-3.2.39_1 bison-2.3_4,1 damageproto-1.1.0_2 fixesproto-4.0 gettext-0.17_1 gmake-3.81_3 inputproto-1.4.2.1 kbproto-1.0.3 libGL-7.0.3 libGLU-7.0.3 libICE-1.0.4_1,1 libSM-1.0.3_1,1 libX11-1.1.3_1,1 libXau-1.0.3_2 libXdamage-1.1.1 libXdmcp-1.0.2_1 libXext-1.0.3,1 libXfixes-4.0.3_1 libXi-1.1.3,1 libXmu-1.0.3,1 libXt-1.0.5_1 libXxf86vm-1.0.1 libdrm-2.3.1 libglut-7.0.3 libiconv-1.11_1 m4-1.4.11,1 pkg-config-0.23_1 xextproto-7.0.2 xf86vidmodeproto-2.2.2 xproto-7.0.10_1
R-deps: damageproto-1.1.0_2 fixesproto-4.0 inputproto-1.4.2.1 kbproto-1.0.3 libGL-7.0.3 libGLU-7.0.3 libICE-1.0.4_1,1 libSM-1.0.3_1,1 libX11-1.1.3_1,1 libXau-1.0.3_2 libXdamage-1.1.1 libXdmcp-1.0.2_1 libXext-1.0.3,1 libXfixes-4.0.3_1 libXi-1.1.3,1 libXmu-1.0.3,1 libXt-1.0.5_1 libXxf86vm-1.0.1 libdrm-2.3.1 libglut-7.0.3 pkg-config-0.23_1 xextproto-7.0.2 xf86vidmodeproto-2.2.2 xproto-7.0.10_1
WWW: http://www.haskell.org/hugs/

Usually I remedy this using grep, but perhaps I could do better with some scripting? I quickly hacked together an alias to add to my cshrc that would allow to simply type.

> portssearch searchstring

For example.

> portssearch hugs
Port: hugs98-200609_2
Path: /usr/ports/lang/hugs
Info: An interpreter for the functional programming language Haskell 98

Port: ohugs-0.5_5
Path: /usr/ports/lang/ohugs
Info: Interpreter for Haskell with object-oriented features

All this with just two lines, an extra line to work around some difficulties with escaping strings.

set AWKSTR='{ if ( $0 ~ /^Info:/ ) { print $0; print ""; } else { print $0; } }'
alias portssearch "make -C /usr/ports search name=\!:1 | grep -E '^(Port|Path|In
fo):' | awk '$AWKSTR'"

The only problem I have encountered so far is when a port lacks a comment in it's Makefile and Info: is left out you end up with some ugly output. Like this.

Port: japanese/diclookup-emacs20
Port: japanese/navi2ch-emacs20
Port: textproc/dictum-emacs22

But it's usually a non-issue. For now, I am happy with portmaster and my little portssearch alias.

For more information on portmaster see http://dougbarton.us/portmaster.html

Philosophy of change

Recently I involuntary got involved in a religious debate (yes, again). My observation of how my counterpart argued became apparent pretty quickly. Change, is bad. Now, we should further define change in this particular context. Change, as in changes made to laid down principles or things considered to be facts.

If you become a part of a doctrine, a book that rigidly defines good and bad, your habits and your afterlife. You get the feeling of certainness, at least when it comes to these things. Certainness feels good, doesn't it? You can stop worrying about these specific issues, because they have been taken care of. This observation upon religion is not new, but what are the consequences?

Any change made to these principles or any element that might cause your principles to change is now causing you to feel less certain. Can god exist if we have evolved from microbes? Can my morale be justified if we are not created? Asking those questions hurt, I know, I have been there. You are faced with what seems to be two options, to abandon your principles and become a "devil", someone with no morale or certainty or to shut your eyes, perhaps even work against these "forces". For these "forces" must truly be "evil" if they tempt you to doubt? Even though they make no such claims.

But I see it in another light. I see change as good. My opponent brings up things such as Einstein proving that the ether is not a necessity. That it is indeed possible that science is wrong and that it _will_ change. I agree, I wholeheartedly agree that science will change and I think that most likely science is wrong. Wrong of course means not in perfect harmony with the physical world. Theories will change in accordance to future observations.

But my opponent and I don't agree upon change. I think that change is good, I don't think that the claims of science are rigid and forever true. Instead, they will change, but most likely for the better. You proved me wrong, thank you, thank you for allowing me to discard what was faulty and move on. To me, change is good, change is the very essence of living. I strive to pass something on, not what I learnt from those before me, but what I learnt, verified and tried to take further. This goes both regarding morale and knowledge. For how else can my children be wiser than me? I am the rope, humanity is the rope dancer.

Sorting all the mail

I am subscribed to numerous mailing lists, ranging from FreeBSD to Python. As a result, I get a couple of hundreds of mail every day. Most of my mail boxes, such as the one I got from my school can't withstand such a bombardment for very long. Within matter of days the inbox will be full.

My old solution was to use Thunderbird on my workstation to filter out all the mails into a directory structure on my disk. That way the mails were pulled from the server and stored at a place where storage space was plenty. It worked, but it was a bit ugly. As soon as the machine was turned off or my connection went down I would have my inbox filled until a got home and started the workstation again. Now, not being at home and not having a workstation things needed to change. But how?

GMail is kind of neat, and free. You get a couple of gigabytes of storage and Google gets to index your mail and give you relevant ads. The only thing that really bothers me is that GMail sometimes uses a different terminology which clashes with ordinary email terminology. You use "labels", which really are folders on the IMAP server (took me a couple of seconds to understand what it meant to "tag" an email and to "archive it"). Sure, GMail is to date _the_ best and most intuitive way to display mails, but it is a bit confusing at times. Would I make a similar user-interface definitely, there's a lot of thought put into it.

Anyway, on to the point. I unsubscribed my old email, signed up for a GMail account and subscribed my new email to about 75 lists. Then used the GMail UI to "label" all my incoming mails from the mailing lists, mails from Python/Dev are now labeled as python/dev and are thus stored in the //INBOX/python/dev directory. It's tedious, I got there. I then added my GMail account to my mail client of choice (Thunderbird) and I could read them just like in the old days, but being accessible from any computer and not having to worry about filling my inbox.

Now, everything could have been just fine. But Thunderbird was not playing along. As a default Thunderbird won't check other boxes than the inbox. This is most likely due to the fact that it takes quite some time to check all the directories. But I just won't find it comfortable to click every single folder every time I want to check for mail. You can force Thunderbird to check every single one of them, but doing so requires you to edit the properties for all your directories (no, you can't change more than one at the time). This, is tedious, _very_ tedious.

To make matters worse, for my ordinary inbox I sort mails by date. This is really good for simple conversations but for a mailing list like FreeBSD/Current, it just becomes a huge mess. For these cases you want to sort by threads, as in, a conversation becomes a tree, the first mail is the root, all mails are leaves and an edge is established to the mail which the mail replied to. This also requires you to change the properties on a per-directory basis. I have three operating systems and a workstation at work. I don't want to reboot to check my mail, so setting it up would force me to make about 200 clicks per client. This is not acceptable, so I turned to Google.

I found the following. There is a configuration editor and there is a way to achieve this without grinding your teeth for all too long.

Check all IMAP folders for new mail
http://www.mozilla.org/support/thunderbird/tips#beh_downloadstartup

Sort order for mail/news not configurable by default
https://bugzilla.mozilla.org/show_bug.cgi?id=86845#c74

Just remember that you need to delete the old directories information files in order to rebuild them with the new defaults (in my case, threaded sorting). Why not use a one-liner? Remember to shut down Thunderbird before applying it and then start Thunderbirds afterwards and you are all done.

find ${PATH_TO_GMAIL_INBOX} -name '*.msf' | xargs rm

I hope you enjoy the "easier way to do it" (tm) as much as I did.

SSSB and automation

Recently I began reading up a bit on this whole web thing (yes, I lost my interest in coding HTML and web pages back in grade school). Things have changed and I better learn a bit since it will be beneficial for my work on my thesis. I wanted to try out authentication and HTML forms. Here's the story.

SSSB is a company that rents out flats to students in Stockholm. Back in the good old days (see 2005) you registered and for every day you got a day in line. These days could be traded in for a flat when you needed one, the more days, the better the flat. But, those things changed, the rules of the game are basically the same but you need to log in now and then to show that you are active enough. This of course, is tedious, most other non-student flat renting companies in Stockholm either don't bother or charge a fee. Already back in 2005 this annoyed me a bit but now being in the US and all it's a gun to my head. If I forget I will not have anywhere to live when I get back, or worse, I may loose three years of days. This seemed like an excellent task for a small hack.

The script I created has been tested a bit, but as always, use it at your own risk and take into consideration any moral implications. I will most likely not deploy it, but at least it might prove useful to someone.

#!/usr/bin/env python

'''
Author: Pontus Stenetorp
Version: $Id: sssb.py,v 1.2 2008/12/21 08:07:04 ninjin Exp $

Simple script to keep logging on the SSSB website to keep your days in line
for a flat in Stockholm. Meant to be executed by cron now and then.

This does not violate the agreement on sssb.se but the morality of automating
this is somewhat questionable. *sigh* I remember the good old days before this
log in every 'n' days silliness.
'''

import os
import random
import sys
import time
import urllib
import urllib2

### Constants
## User settings
USERNAME = '1234567890'
PASSWORD = '1234'

# The maximum amount of days to wait between logins, this value should be
# lower than the actual required amount of days (90)
WAIT_MAX = 90/2

# The minimum amount of days to wait between logins, this is to not log in
# all too often
WAIT_MIN = 14

# Faking is _not_ nice, but it is sometimes the only option
FAKE_USERAGENT = True

# If you want to fake, you may choose from the list below and give the index
# here. If set to negative the script will choose one at random and note it
# in the data file for future logins.
FAKE_USERAGENT_INDEX = -1

## Web page specifics
SSSB_URL = 'http://www.sssb.se/index.php?page=login'
VALUES = {'m6input_username': USERNAME, 'm6input_password': PASSWORD,
      'mact': 'FrontEndUsers,m6,do_login,1', 'm6returnid': '794',
      'page': '794'}

## Script internals
DATA_FILE_PATH = os.path.expanduser('~/.sssb_data')

# Look for this to confirm a successful login
LOGGED_IN_MATCH = 'Du %sr inloggad som:' % ''.join(['&', 'auml;'])

SECONDS_PER_DAY = 60 * 60 * 24

# If you are honest you will show that you are actually running a script
REAL_USERAGENT = 'SSSB login Python script 1.0'

# Thank you http://www.zytrax.com/tech/web/browser_ids.htm for the user-agents
# A mix of user agents to choose from, ranging from MSIE to Opera on several
# operating systems.
FAKE_USERAGENTS = [
      'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; SLCC1; ' +
      '.NET CLR 2.0.50727; .NET CLR 3.0.04506; Media Center PC 5.0; .NET ' +
      'CLR 1.1.4322)',
      'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SU 3.1; SLCC1; ' +
      '.NET CLR 2.0.50727; .NET CLR 3.0.04506; .NET CLR 1.1.4322; Tablet ' +
      'PC 2.0; .NET CLR 3.5.21022)',
      'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.8pre) ' +
      'Gecko/20071019 Firefox/2.0.0.8 Navigator/9.0.0.1',
      'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.5) ' +
      'Gecko/20050519 Netscape/8.0.1',
      'Opera/9.60 (X11; Linux i686; U; en) Presto/2.1.1, Opera/9.02 ' +
      '(Windows NT 5.0; U; en)',
      'Mozilla/5.0 (Windows NT 5.1; U; en) Opera 8.00',
      'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 ' +
      '(KHTML, like Gecko) Version/3.1.2 Safari/525.21']
###

def main(argv=None):

  def read_data_file():
      data_file = open(DATA_FILE_PATH, 'r')
      last_login, next_login, user_agent = [line.strip() for line
              in data_file]
      data_file.close()
      return int(last_login), int(next_login), user_agent

  def write_data_file(last_login, next_login, user_agent):
      if os.path.isfile(DATA_FILE_PATH):
          os.remove(DATA_FILE_PATH)
      data_file = open(DATA_FILE_PATH, 'w')
      data_file.write(str(last_login) + '\n')
      data_file.write(str(next_login) + '\n')
      data_file.write(user_agent + '\n')
      data_file.close()
      return

  if not os.path.isfile(DATA_FILE_PATH):
      if not FAKE_USERAGENT:
          user_agent = REAL_USERAGENT
      elif FAKE_USERAGENT_INDEX < user_agent =" random.choice(FAKE_USERAGENTS)" user_agent =" FAKE_USERAGENTS[FAKE_USERAGENT_INDEX]" next_login =" 0," user_agent =" read_data_file()" current_time =" int(time.time())">= next_login:
      opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
      data = urllib.urlencode(VALUES)
      request = urllib2.Request(SSSB_URL, data)
      request.add_header('User-agent', user_agent)
      response = opener.open(request)
    
      logged_in_line = [line for line in response if
              LOGGED_IN_MATCH in line]

      if not logged_in_line:
          print 'FATAL: Log-in failed'
          return -1

      next_login = random.randint(
              current_time + WAIT_MIN * SECONDS_PER_DAY,
              current_time +  WAIT_MAX * SECONDS_PER_DAY)
      last_login = current_time
      write_data_file(last_login, next_login, user_agent)
  return 0

if __name__ == '__main__':

  sys.exit(main(argv=sys.argv))

At line #53 you may notice a strange ''.join(), Blogger finds it a good idea to translate my escaped Swedish characters. This is a remedy, so enjoy the special Blogger version which is slightly uglier than my own version.

EDIT:
Blogger is really _terrible_ at displaying code, one almost weeps when you take a look at it. All the 80 width formatting killed by static width. *sigh* Paste it into an editor and take it away from this awful place.

P990i USB issues

Recently I got a new cell phone, my old one being 9 years old it was quite a change. I wanted one that played music, had WiFi (since I spend most of my time at school and then I could check things up without using my laptop) etc. I found a used Sony Ericsson P990i for 1000 SEK which was quite reasonable.

It all worked fine until I transfered some music to it and the sound was all choppy and made my ears bleed. I phoned SE and they said that they never had heard of such an error. I was puzzled, what could cause this?

I tried analyzing the encoding of the files, nothing strange about it. But when I transfered a song that I considered to be more likely not to contain errors it played flawlessly. How come? It was encoded in the same way. I even checked using SE encoding and transfer tool.

Then I got it, when I played the song transfered from the phone to the computer it was still choppy. What must have happened is that SE tries to make the P990i act like a USB utility but somehow does not do it accurately (I find this likely, the phone needs a bundle of drivers), or that FreeBSD has some sort of bug in the USB handling (I find this unlikely). The first file was transfered using FreeBSD, ending up corrupted, and the second one using Windows, working just fine. This is not a pleasant surprise but now I can finally use my SE head phones.