HTPC project, part 3

Ahem, my mistake. I had thought that the one cable coming from the PSU must be the one to connect to the power fan contacts on the mobo. Actually that was the floppy disk power connector. Lucky I didn't burn the mobo.

After that little fix, the device started as expected. I quickly installed a minimal Ubuntu installation instructions provided by the XBMC Wiki. I installed OpenSSH and Samba straight from the installer because I knew I need them both. I am using the Intel GPU built into the i3 processor so I didn't need to install any drivers for that.

There were some minor issues because the instructions are for a bit older Ubuntus (including wrong apt-key values. The correct ones were easy enough to Google.

I also set a static address for the device for obvious reasons.

My hdd setup is like this:

On the SSD disk:

• 256MB /boot,
• The rest /.

The two 2TB disks have the following:

• 8GB /swap on disk 1,
• 8GB /var/log on disk 2,
• 10 GB /home on disk 1,
• 10 GB /tmp on disk 2,
• and the rest is a RAID1 partition in /data.

Afterwards I thought maybe the /home and /tmp disks should have been bigger (50GB) just in case. The idea is that static stuff is on the SSD disk and variable on the HDD's. I don't anticipate I will save anything important in /home so no need to use RAID on that.

There are still some issues:

• The power button doesn't turn the device off (and I did install upower).
• Samba speed is not very good ( ~5MiB/s).
• I have gotten the "*ERROR* EDID checksum is invalid, remainder is 143" two times. Looks like this has something to do with the Intel graphics controller. Lots of talk, no resolutions in the Linux world yet.
• The remote worked as a remote after installing lirc but only until I rebooted. Now it acts like a mouse again. I chose "Soundgraph iMON Antec Veris" as the IR receiver and "None" as transmitter, these should be right. There are some odd debug lines in syslog that I have to look into later.
• LCD is not working yet but it looks like there are solutions for this.
• The Nexus case fan is way too loud for this application. I'll go buy a Noctua instead. Then I should be able to drop the speed of the CPU fan to a minimum - I tried this already and it's practically silent then. But I need to run it a lot faster if I want to avoid using the case fan.

There are things I still need to do, including:

• Install the plugin that allows me to rip DVD's with this device.
• Install autostarter stuff so that the device starts automatically to XBMC on boot.

HTPC project, part 2 - no good

Damn, the thing wouldn't even start after I put it together. Pressing the power button does exactly zilch. I checked the cabling at least five times, read the chassis and motherboard documentation I don't know how many times and everything seems ok. I even checked the power button with my multimeter - it works.

I had to take the thing to the shop since I don't have spare parts to try and replace parts to see which one is faulty. I believe it must be one of PSU, motherboard or chassis. PSU does give some power since the LCD on the chassis lights up and a LED is turned on on the motherboard. But maybe it doesn't give the +12V correctly? Or for some reason the PSU is not signalled to start pumping out power when the power button is pressed.

My main suspect is the motherboard, though. Lets see what they say in the shop. This delay is just annoying since I cant' start installing Ubuntu and XBMC.

HTPC project, part 1

I decided that since I have some spare time before Christmas (I'm on vacation, wife is not) I'd build a HTPC. After a weekend of getting my bearings since the whole issue is pretty unknown to me I came up with this list of items (all of the items needed to be available with a two-day notice):

• Antec Fusion Remote Black HTPC chassis.
• Intel i3-540 processor.
• Asus P7H55-M Pro motherboard.
• Kingston HyperX Blu 4GB 1600MHz DDR3 (2*2GB) memory.
• Kingston SSDNow V-Series 30GB drive.
• 2 Western Digital Caviar Green 2TB SATA II hard drives.
• LG CH10LS20 Blu-ray Combo drive.
• Scythe BIG Shuriken SCBSK-1000 cooler.
• Nexus PWM Series Silent Fan 120mm.

Of course as I started I realised the chassis actually didn't include a power supply, so I later picked this up:

• Nexus RX-5300 530W.

A bit overkill but this was the best option on offer.

This should prove to be a powerful package with some nice features:

• Silent, this is something I aimed for. The chassis is planned like that (hard drives have special silenced cage, the power source is placed on rubber foots etc), the chassis fan is one of the quietest on the market and the cooler isn't bad either. I would have wanted the Ninja Mini cooler, but Rev B which supports the LGA1156 is unavailable. Ninja Mini is so efficient that I could run it without a dedicated fan because the chassis cooler on the Fusion chassis blows straight at the cooler (I plan to try this with Shuriken as well). The power source is also perhaps the quietest on the market.
• Lots of space to save the media files, I plan to use a software RAID 2.
• No need for a separate video card as the i3 has one built-in. Saves energy and is much cheaper since a suitable silent card would cost something.
• The motherboard has built-in HDMI.
• The SSD disk which I will make the system disk will make boot-up very quick.
• Good looking chassis that fits with my other equipment.
• A built-in remote which should work with lirc (Linux Infrared Remote Control).
• And some other stuff I have already forgotten.

So I collected all the stuff on the kitchen table and started the assembly (this is the point I realised I was missing something). I assembled several PC's before and it's not something I particularly enjoy, but since I had to purchase the parts from two different stores it would have been quite expensive to let one of them do it.

Some highlights/lessons learned:

• The chassis comes with only two brass standoffs! WTF? Three is the bare minimum, IMO.
• With the brass standoffs and the backplate from the motherboard's box, the motherboard was too high up. So I had to put the standoffs to the front of the motherboard. This is of course means the motherboard is not exactly even, but no can do.
• I hate those damn backplates!
• I also can't figure out why Scythe insists on making these kinds of tighteners for their products. They are always tricky to install and require way too much force.
• What happened to ZIP (Zero Insertion Pressure)? I had to use way too much pressure on the lever that secures the processor, as well.
• You need really, really modest amounts of silicone on the processor. I think I overdid it this time, thankfully the cooler hides the evidence.
• Always check the drive cages before screwing all 8 screws! Yes, I initially put the Blu-ray player the wrong way and spent a good ten minutes figuring why I can't get the cage back inside the chassis.
• The motherboard doesn't have a Firewire connector. Wish I had realised it earlier instead of reading through the manual twice.
• Neither does it have a built-in USB connector, so in order to connect the LCD I needed to use a normal USB socket. Because there are no suitable holes in the chassis, this meant some ugly solutions.
• There is no place for the SSD disk (the chassis has space for 2*3,5" internal and one 5,25" external drive). But as it happens, the drive cage that holds the optical disk has plenty of space and some suitable screw holes so I attached it there with two screws on the other side. Should be sufficient, it's not like the disk weights anything.

So I did what I could lacking the power supply. Quite fiddly as the instructions for the chassis aren't exactly the best I have read and there are plenty of connectors to figure out. And it took a goddam eternity to get the motherboard lined up with the standoffs and the damn backplate! But this is what I did:

• Removed the stock fans from the chassis and put a plate in front of the other fan hole.
• Inserted the Nexus chassis fan.
• Changed the backplate.
• Inserted the motherboard (it helped that I moved all of the cables away from the motherboard compartment).
• Inserted the CPU and cooler.
• Inserted the air flow vents that help the air flow from the chassis fan through the CPU.
• Inserted the WD drives to their positions.
• Inserted the Blu-ray player and SSD disk to the cage.
• Connected all the cables except for SATA and cables that come from the power supply (obviously).

And that was it for today. Tomorrow: put in the power source, finish cabling, create a Ubuntu installation CD and attempt to install the OS.

My plan is to run XBMC eventually so that it boots up straight to XBMC and I can control the system with the remote. Any administrative duties I'll do with ssh.

Subgrouping C code in Doxygen

The Doxygen documentation about grouping concerns itself about C++ code and using the groups with plain C wasn't quite as forward as I thought. I wanted to create a main group (shown as a top-level page) and sub-groups below it in a manner where the files belonging to a sub-group are not shown on the ancestor group pages. E.g. I wanted the structure to be something like

Main
  main.c
  Module
    module.c
    helper.c

The files wouldn't obviously been shown on the listing, but they would appear in the Main and Module pages' Files section.

After a bit of hacking, this is how I accomplished it:

main.c:
/** @file main.c
 * 
 * @brief
 * (other Doxygen tags)
 * 
 * @ingroup main_group
 */

/** @defgroup main_group
    This is the main group of the application.
 */

module.c:
/** @file module.c
 * (other tags)
 * @ingroup module
 */

/** @defgroup module
 * This is the sub group of the application. */
 * @ingroup main_group */

helper.c:
/** @file helper.c
 * (other tags)
 * @ingroup module
 */

So the trick is to define separately the new groups and what group (if any) they belong to. If I tried to have @defgroup inside the file header block, things just didn't seem to work. At the best main.c was not shown under main_group. I think this is because @ingroup only takes into account the previous suitable tag. So if there is a @defgroup before it, @ingroup adds this group to whatever group @ingroup defines, and the file level inclusion is not done.

soundKonverter failing to convert FLAC to AAC

Looks like something was broken in Kubuntu 10.10 because converting FLACs to AACs seems to have stopped working in soundKonverter. I let Dolphin convert my CDs to FLAC and then use soundKonverter to convert them to AAC for iPod. Last time I did this (not too long ago) everything went smoothly, now soundKonverter just told me the conversion had failed.

After some digging I found out that the log was actually in ~/.kde/share/apps/soundkonverter/log (obviously I first looked in /var/log). The logs told me "Unknown encoder 'libfaac'". The package was there, though.

Googling, I came up with this page. I first tried installing ffmpeg, x264 and libx264-dev by hand, but that didn't help. But following the instructions to the letter did (though I use aptitude instead of apt-get). It's just somewhat annoying having to install packages outside of the package manager.

Creating sequence charts with Doxygen

I wrote a small Python script called trace2dox that one can use to turn trace logs in message sequence charts into Doxygen. I'll write instructions later, but for now the script and an example log and msc output can be found from Gitorious. You just need to create the logs somehow and then run it through this script to create the charts you need. You can filter, colour etc. the output.

For the project I wrote this script I created a C macro that wraps the message sending function and adds logging capabilities. This way I didn't need to change anything in the actual source code.

I'll write better documentation once I have time.

A way to handle big merges

I had the task of merging two branches with lots of changes. The problem was I wasn't very familiar with the code but I took the task because I have lots of experience in this sort of stuff, although this merge was the biggest I have done. The branches were over a year old and unfortunately it looked like they were differentiated as best as possible. For instance doing refactoring in one branch but not bothering to do it in the other. An extra challenge was that the quality wasn't very good and documentation was somewhat lacking. A typical situation, in other words.

So I thought about this for a while before plunging head in. Quickly I figured out that the best way to figure out what was needed would be a three-way compare where I would see the two branch heads and their common ancestor. This way I would immediately see where each difference was made (and in many cases the same blocks were changed in both branches, but in different ways). I tried a few merge tools but only one really did what I needed: KDiff3. It's by no means the perfect tool (it has some usability issues) but with it I could easily have the ancestor on the left pane, the other branch in the middle and the other - where I wanted to do the merge - in the right pane and I could edit the final product easily in the merge window at the bottom.

The branches were originally in Subversion, but I moved them to Git because it's far quicker and has a much better support for branching (and I knew it better).

In theory I could have configured KDiff3 to work from a single clone, but I figured it was just as well to to have each commit on it's own clone, just to be on the safe side.

So I had three clones: initial, branchA and branchB. And off I went.

At first I tried to do this the thinking man's way, i.e. try to actually merge the code, but it became quickly obvious that this would take me months since most changes were non-trivial. "If I change this, where else does it affect?"

So I turned this into an idiot-proof process: I simply flagged each change something like this:

#if defined(BRANCHA)
...
#else
...
#endif

I had the following possibilities:
1. Something was changed in A, but not B (B is the same as ancestor),
2. Something was changed in B, but not A,
3. Something was changed in both A and B and in the same way,
4. Something was changed in both A and B but differently.

1 and two would be handled like the example above (unless a very trivial change, like an added logging command), third situation would mean no flagging and the fourth would reguire a if defined... elseif defined...else to keep the original code for later reference.

There were some 650 places I needed to flag but after that the code was in one branch and the cleanup was even possible. Now it is just a case of looking at each change and deciding whether it can be used on both branches or if this really is a differentiation. The latter can then possibly be made into run-time flags which is preferrable because this would mean the binary is the same for both branches. This in turn would mean automated testing would be a lot faster since most of the similar code could be tested just once rather than twice. And considering the changes I think the number of different code segments is well below 100 (rest is mostly refactoring that can be used in both branches).

This was definitely the best way because mistakes basically meant the code wouldn't compile rather than anything more complex. The mistakes I made usually involved putting the #endif directive on the wrong side of a brace or some similar thing and with the right technique even those are relatively easy to find.

After everything was done and the SW was built, tests passed. Like that, and for both branches. So what I spent figuring out the best way was saved many times over during testing and debugging.

Removing Linux kernels

I think this might be the best way to remove those old kernel versions that just clutter up the Grub menu: from tuxtweaks. This is just a friendly reminder for myself next time I need this.

On a related matter, damn I hate www proxies! Eclipse updates and plug-in installation doesn't work at all. I have an earlier blog post about this issue, but this solution didn't help me last week. CDT just won't install on top of the platform Eclipse available in the Ubuntu package repository because it lacks an endless number of dependencies if you try it from a local archive file. So I had to download the CDT version manually. Thankfully PyDev doesn't have any pesky dependencies, so I was able to install that on top of the CDT version no problem.

Passwordless SSH login

I was fiddling with NFS trying to get it to work (previous post) when after a reboot I run into an odd problem: suddenly I couldn't do a passwordless login any more. Took me a while to figure out but there was a very clear error in the syslog: I had changed the permissions in my server home directory to 777 (allow all) to make sure my problems weren't access rights related and the server refused to look at the ssh keys because of this.

A good reason not to work, mind as you never should set the rights like that.

And, btw, the best way to put your ssh key to the server:

ssh-copy-id -i [identity file] [user]@server.

NFS and bind - no good together

I had problems getting my NFS server to work. The whole client side seemed to have stuck to some state that wasn't anymore (plus I got a few stale NFS handles as a bonus).

In the end the problem was that what I was looking at was a bound directory and that just didn't work. I have a /share directory on the server and I had bound /home to /share/home and this wouldn't work. Once I removed the binding from /etc/fstab and just added the /home directory directly to /etc/exports, NFS started working properly.

I don't have the expertise to figure why this is, I'm just happy I got that fixed.

Transferring an SVN repository to Gitorious

Moving SVN repositories to Git is obviously inherently a good idea since Git is a far better vcs. I needed to transfer a very large SVN repository to Gitorious and it did take me a while to figure out how to do it. Turned out git-svn is not the best tool for this. I tried and it took almost two days to fetch the svn repository as a git repository.

Then I found the svn2git Ruby script and that did the trick. Here's how:

• Start by installing git-core and git-svn. Obviously you also need a Ruby environment and the gem package.
  
• Then install svn2git:

gem install svn2git --source http://gemcutter.org

• Then you need to create a committers list because svn has user id's whereas git uses email addresses to identify committers:
  

  svn log | grep -E "r[0-9]+ \| [a-z]+ \|" | awk '{print $3}' | sort | uniq > committers.txt

• Edit this file so that each user id has a unique email address in the form of

user_id = User Name <user@email.com>

• Then fetch the repository:

 svn2git svn://server/path/to/repo --authors committers.txt

• At this point, create the repository into Gitorious.
• Finally, push the contents to Gitorious:

git push --all git@gitorious.org:project/repo.git

And that's it. Just took me a few tries to get the svn repository converted correctly.

Gaming problems

Gah, it seems I always find the odd problems in SW.

Like when I tried to install Baldur's Gate and BG2 + EasyTutu. EasyTutu is a nice mod that allows you to play BG with the BG2 engine and "easy" refers to the fact it's supposedly much easier to install than the plain Tutu.

First problem I hit was that BG wouldn't install. I found out that I did manage it in safe mode, but that's obviously a bit painful. Later I discovered that setting the computer to Gaming mode with Game Booster also does the trick, which helped a lot.

Anyways, BG and BG2 were installed and patched. Next problem was that EasyTutu failed almost at the last hurdle when it was installing some core files (tutucore.exe is some such) claiming it didn't have access to ".\". Which is directory the installer supposedly created. I think the problem is/was that the directory was created read-only. And after the failure, the install app cleaned the directory.

I got around this by killing the setup with task manager and running the last app by hand. First time I thought I had done something wrong because the game wanted CD 2 in the drive and I didn't realise it was for BG2, not BG. Next day I figured this out, but obviously I needed to install everything one more time.

Plain Tutu didn't install either, instead it crashed.

Now I think I'll risk it and install the Widescreen mod that allows screen resolutions more fit for modern LCD screens.

Another problem I had was with my mouse, a Razer Copperhead. Apparently it's a common problem where the mouse button starts generating double clicks instead of single ones. Fortunately it seems I managed to fix it quite easily by opening the mouse. Here's how:
- The screw that holds the mouse together is under the back slider on the bottom. Just peel off the slider and you should see the screw. Make sure you have the right size screwdriver because it's pretty tight and you might damage it otherwise.
- The top slides off by pulling it backwards. Might be a bit tight, but don't overdo it or you might break something.
- The offending part is a small white button on top of a slightly larger black one. I used a flat-head screwdriver to pull it very slightly up and then I pushed it back down. That's it.
- While you are at it, you might remove the excess lint around the wheel as well.
- Putting everything back together is just a reverse of pulling it apart. It took me a while to get the top back on, but the trick was to push it forward, it doesn't just snap on.

There's an other way that might work if you don't want to open the mouse.

Gitorious installation gotchas

Again I spent couple of days just setting up Gitorious. It's much easier than it used to be thanks to good instructions but still a bit of a pain. The part where most people hit the biggest problems is not setting up the site but repository creation phase which never seems to work on the first go. Here's a few hints on that:

1. Double and triple check you have every package and gem installed and with correct version (config/environment.rb hardcodes versions for some gems). It's really easy to miss one package.
   
2. Check that you have created all the directories required and that they are owned by the git user. tmp/pids is one nasty directory that is easy to forget.
3. If you copy-paste stuff from the instructions, fix the quotation and other special characters! This one is extremely annoying, bit me again. Not only are the quotation marks anything but the ones required I spent a good hour figuring out why the poller wouldn't start. There was nothing in any log to suggest any problems, but it turned out one of the hyphens in the init script was in fact some odd character (an extended hyphen) that I just didn't pick up when looking at the script.
4. I figured out that there was a problem with initialisation by the fact that tmp/pids had nothing in it. So check if there is. You can also do ps -ef | grep poller.
5. You can also try to run the poller from the command line by giving "run" as a parameter. This will print any errors to the stdout.
6. Logs that are of interest that you might want to check: anything in /var/www/gitorious/log, /var/www/gitorious/tmp/pids and /var/log/apache2 as well as /var/log/boot.log, /var/log/boot and /var/log/syslog. All of these might sometimes include clues to what went wrong.
7. Check also that stompserver and git-daemon are running (ps -ef).
8. Make absolutely sure that you have edited the correct section in the config files, it's easy to accidentally edit the development section instead of production.
   
9. And finally, don't try to recreate a repo that failed because the poller wasn't running - doesn't work for me and I again spent an hour trying to find an error that wasn't. Test your setup by creating a completely new repository. I think this is a bug in Gitorious.
   

Useful tools for debugging a DNS server

Having been trying to set up a DNS server and a domain with bind9, I have found the following commands useful:

dig domain.name - queries the DNS server for the information regarding domain.name

dig -x xx.xx.xx.xx - queries the reverse DNS information for IP address xx.xx.xx.xx from the DNS server.

dig has many other uses, too. Very useful tool. One useful parameter is @xx.xx.xx.xx which tells dig to use the DNS server at address xx.xx.xx.xx for the query.

named-checkzone domain.name domain.name.db - Run in the directory where the domain.name.db file is located, this checks that the file is correct.

nslookup domain.name - Gets the IP address and some other info.

Links:
dig HOWTO
named-checkzone manual page
nslookup manual page

A simple developer process with Git

There seems to be quite a few questions about how to work with Git in a project because the whole concept of distributed version control is unclear. My proposal for the starting point for developers new to the concept looks like this (Gitorious is used here as the tool to host the central repository used to share the code):


The idea is to create a new branch for every feature the developer is working on. This helps the merge/rebase operations which in turn motivates the developers to do it often enough. An advanced concept would include quashing the commits with rebase --interactive just before releasing the code to the central repository. This blog post explains how to do it. This would help keep the commit log clean and simple. The feature branch is also a must in my opinion if this feature is used because it alters the history and you never, ever (*) do that to code that has already been released. If the developer works solely on the trunk of his repository, there is a greater risk that he does the rebase to already published code.

(*) See "Perils of Rebasing" in the excellent ProGit book.

Extreme debugging

This is a tale of the hardest bug I have ever debugged.

I was working on a baseband module in the DSP level as an integrator. The system run on a chip with three cores, a receiver, a transmitter and application core which dealt with user and control planes.

We had a problem that sometimes one of the cores seemed to crash because it went unresponsive. Halting the core in the debugger it looked like it was still running, though. The place was different so it looked like it should work, it just wasn't replying to messages from other cores etc. Depending on the source package it could take anything from 30 seconds to hours before this occured and even within a package this was pretty random (but still some packages crashed faster than others).

After quite a lot of debugging and writing test code it became apparent that one core alone would not crash so this had something to do with several cores doing something simultaneously. One thing I for instance did was a trace system for this particular problem. The SW already had one but that was more about the logical way the SW worked rather than at a function and interrupt level. This only provided some help, problem was we didn't have any tools available to debug several cores at the same time. We did have an emulator that we could use to trace the SW a core at a time. I tried my own trace first because the log that the emulator created was absolutely massive as it traced every single assembler level operation the core did. And because the amount was so large and the USB connection so slow I could only collect the program counter (PC). This of course could be merged with the SW image and C code so one can walk through the code in the order it was run.

I spent hours and hours looking at traces and this was not helped by the fact that often it was some other core crashing than the one I had connected the emulator to. Problem was, the tails of the traces didn't usually seem to have anything in common. In other words the call stacks were very different.

But having watched how the code behaved and looking at the traces I still started to get a pattern. There were always one of two things happening some time (thousands or tens of thousands of cycles) before the crash. There was either an inter-core message or access to the HW semaphore block. It was not easy to spot because it was so much earlier so that's why it took literally weeks before this pattern started to appear in my head.

So I started suspecting that maybe there was an issue when one core was sending a message and another was accessing the semaphore block. I didn't have a way to test this because it was impossible to synchronize the cores to such extent that I could write some logger for this as the way the cores were started required messages from upper layer going to each core etc. Any timestamps would have been in an order of tens of thousands of cycles off and even when one core crashed the others would still run for quite some time (speaking from DSP's point of view).

In the end I took the baseband system to our customer who happened to have logic analyzers with high enough frequenct support and our schematics. The schematics unfortunately showed that there weren't too much debugging support from HW level. The chip could have supported all sorts of signals but they were not connected anywhere. Basically all I had that I could use was seven GPIO pins.

So working on my hunch I wrote some simple macros that turned a pin on or off and inserted these to strategic places in the code. The logic analyzer was then connected to the board to show the state of the pins. After several crashes it started to look like my hunch was right: there was always two operations from different cores very close to each other. So it was time to check the chip block diagram and there was an interesting thing there: all the blocks in the chip are connected to a common DMA crossbar (from all three cores). As it happens, both the block involved with the message passing code and the HW semaphore were behind the same bridge (which in turn was connected to the crossbar).

A colleague then wrote code that basically didn't do much else than to use those two blocks from two cores and upon running it one of the cores crashed immediately. Proof positive that we had found the source of the problem: the chip's DMA engine had an issue.

We sent this code to the chip vendor and only took a day before they responded that there was indeed a bug in the HW. It took another week before full analysis was delivered:

There were several HW blocks connected to the same bridge. Therefore the bridge had a queue when ever accesses were made. Each access was given a number in the queue and the block was to send a message to the bridge when that queue item was free again (=access request was handled by the block). However, two of the blocks had a bug (HW semaphore and another we didn't use) where, if this freeing operation wasn't acknowledged back by the bridge they wouldn't try to send it again. This could happen if two blocks tried to send it at exactly the same time. In that case the bridge would only know of one of them.

What made the bug so damn hard to debug was that this wasn't the point when things failed. Instead the core continued to run and use the queue (which was used as a circular buffer) until it came back to the item that was not freed. The bridge was programmed to assume that all resources were used up and stall the core until the item was freed. Of course because the block had already sent the command and didn't send it again this caused an eternal stall for that core. So depending on what all of the cores were doing the time difference between the offending command and the stall varied greatly.

Moral of the story? Debugging is labour but sometimes also instinct. Work only gets you so far.

Eclipse, proxies and Kubuntu 10.04

It was not a nice realisation that it took a while to get Eclipse to work on Kubuntu 10.04 and a company proxy. The package that comes from the Ubuntu repository didn't want to install cdt (C++ Development Toolkit) because of some issues with components that should have been installed automatically. I couldn't resolve that, so I downloaded the C/C++ version of Eclipse from the eclipse.org website.

But this version didn't like the proxy. It just wouldn't connect until I found an explanation.

So, in effect you need to add the following line to the eclipse.ini file and after that things should work:

-Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclient

Massive improvement with code snippets in this blog

I have been pretty unhappy with the way code snippets were shown in this blog. But I just stumbled upon a blog post that shows how to add syntax highlightning to them. I did it like that and added bsh and bash shell support by adding this line to the template

<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushBash.js' type='text/javascript'/> 

and the results are very nice. It works for C, Perl, Java and Ruby, too (and others I won't list here)!

I also changed the template to one that stretches to the width of the window instead of using a very narrow column for the text - something that made code lines wrap all too easily.

.. and never mind that. I'll post the code snippets to Smipple, and add their embed script instead. It's a service that is somehow connected to Google as is Blogger so hopefully they will work well together.

OSS meets Agile methods

I had an interesting revelation yesterday: there are several things in common between open source software (philosophy) and agile development methods (at least Scrum). Of course OSS in itself is pretty agile, but more as a by-product of the field rather than something by design.

Transparency. In both the aim is to be open about things. In OSS the big thing is of course that the work artifact is there for all to see and usually developers are pretty open about what they plan. In agile software development the aim is to have everyone in the project to be able to know where the project is at at any given time. In Scrum there is a host of things with this purpose: the Daily Scrums, the Burndown charts, the reviews, the whole aim of having the team sit near each other and so on.

Trust. OSS creates this by default - anyone can check the product. No hidden agendas. In Scrum trust is both delivered and expected in many layers: for the teams to work well, the managers need to trust the teams enough to give them the power to make decisions, in return they should receive accurate information on where the project is. This creates mutual trust. Teams within themselves should also trust each member, otherwise performance will suffer.

Self-organization. In open-source development people can pick and choose what they want to do and in what order. In Scrum this sort of self-organizing is also a core idea, the management should not intefere with day-to-day work. Of course commercial development usually has more constraints in what can be done, but still, to get best possible performance it's better to let the team decide on the minor details regarding job organization. Micro-management isn't very effective.

These are all, as can be seen, inter-connected. It's harder to have one without the other. And they are all very beneficial.

On top of this, the open-source has one huge advantage for Scrum teams: since the source code is freely modifyable teams and companies can modify them to fit their needs. In Scrum the idea of modifying the process, methods and tools to fit the current project as well as possible is a core idea.

Testing and Scrum

There is a small problem with Scrum when it comes to testing when a full complement of tests takes a long time. In wireless products a full set of tests can take weeks and even what you would call release tests can take days. This of course doesn't lend well to the concept of Continuous Integration when you can't run your tests even nightly, never mind after every commit. On the other hand you don't want to allow untested code to your code repository trunk so what to do?

Here's my idea, pulled from the way many Linux components do it. It assumes a version control system with fast branching and merging, incidentally Git fits this very well. The idea is to have the tree trunk as your "unstable" version. Code going to the unstable undergoes a test set that can be run quickly. When ever a release is needed / wanted, a testing branch is created. This version undergoes heavy testing and, if needed, small patches can be applied directly to this version instead of going through the unstable. If many changes are required, it's probably better to scratch this version. This shouldn't happen often if unstable testing is decent enough. But when tests pass, a stable version is created and the patches are merged back to the trunk.

This way the developers don't need to worry about anything but the trunk (of course they should still use branches when developing new features) and new features can be integrated all the time full-filling the Continuous Integration idea and staying flexible while still having the extra layer of testing required before releases are made. This creates the minimum possible amount of overhead while keeping things safe. The following figure illustrates this idea:

Error handling in Scrum

I posted a question to Stack Overflow some time ago about how to fit error handling in Scrum. I re-read the answers and the first one got me thinking and here's how I could see doing it:

There is a "Defects" task in the Product Backlog. Size is estimated by past experience or just guesstimated.

During the Sprint Planning, a new task is created in the PB, which could be called "Defects for Sprint X" (X = sprint number). The Product Owner adds items to this, team could add some etc. After all the bugs that are planned to be fixed are added to the task, the team evaluates them for the task size estimation like they do for any other tasks. This task is set to the top of the PB and normal Sprint planning resumes.

The main "Defects" task can also be updated at this point to reflect the work that will be done during this Sprint, i.e. by decreasing the size of this task. It should also be updated for any other new developments, for instance once the bugs start rolling in, the size can be adjusted. This all has the nice effect that we have an estimation of one of the things that often gets overlooked.

This way we will handle the defects like other work that has not been completely planned yet. We have a rough idea of how much work should be left for this task but we can also follow our estimations over time. It should be worrying if the "Defects" task doesn't get any smaller during the consequent Sprints. A draw a crude picture of this process to illustrate the idea:

Wiki as a more serious documentation tool

Wikis are great for creating documents collaboratively. They are easy and quick to edit by anyone and there is no overhead like you usually have with normal documentation.

But they have one fatal flaw when it comes to using them to document something: you have no way of evaluating a certain version. Because anyone can write anything, the version could simply have false information, knowlingly or unknowlingly written. This is the reason why project documentation is reviewed, people make mistakes.

This got me thinking of how to combine the best of both worlds and I quickly iterated my initial thought to something much more useful.

Initially I thought that each document could have different, per wiki selectable phases like
- "Draft": the latest version)
- "Proposed": someone thinks this is a good version and
- "Accepted": someone with the authority to say that a version is correct has done so.

Users could then choose which version of the pages they want to see. When implementing something, for instance, they might only want to look at Accepted pages.

This is advantageous, but not collaboration in the sense wikis are meant to be. Why not let the users decide which are good versions?

So my second idea was that users could vote for the correctness of each version and you could set per wiki when a document is deemed "accepted", say five votes.

This would already be a lot better idea, but why make it so strict? How about not having states at all, rather just show how many (and possibly who) has thought this version is good and letting users select their own standards of what is acceptable?

Or, better yet, combine all these and let the Wiki admins (e.g. project members) decide how to use it. Some projects might still like that certain persons check the pages because they understand the system best, i.e. democracy doesn't necessarily work while some would prefer to decide how many votes are regarded as "Accepted" and lastly some might just want to know the number of up (or down) votes.

There is still the problem of how to tell something is not right. With a normal wiki, you change the errors, but in this case you would "lose" the votes. On the other hand, this would maybe be exactly what we want - obviously the page was incorrect so people voting for it were in a sense wrong. Having down votes would still necessitate fixing the errors, so instead of voting a page down the users should just fix them.

I don't know of any wikis that have this sort of system. It would be interesting to know if someone else has had the same idea.

Huawei e169 HSDPA and Windows 7

A lot of people seem to have problems getting the Huawei e169 modem to work on Windows 7. I have a fairly old version (I got it about a year ago) and did notice that the instructions given for Win XP and Vista didn't work. I got the stick to work without updating the firmware by doing the following in our Acer mini laptop:

1. Boot the laptop without the device attached.
2. Attach the device and wait a bit.
3. Remove the device, wait a few seconds and put it back in. Wait until the device driver has been installed (Win7 said something about this in the lower right hand corner).
   
4. Reboot.
   
5. Win7 again opens the dialog that asks if it should run the autorun program. This time do it. This will install the Mobile Partner.

After this I had to fix some settings in the Mobile Partner to get it to work with my ISP but after that the device worked.

Annoying NetBeans bug

Oh man, what an annoying bug. After I updated the project file from version control, NetBeans suddenly decided it can't find some of the packages that worked perfectly well until then. Main problem was org.jdesktop, but even the main class was missing! I tried running clean several times with no effect and restarting NB.

What helped was to remove the NB cache located in ~/.netbeans/x.x/var/cache (x.x = NB version number). I simply closed down NB and deleted the whole cache dir and after restart everything works!

Addendum: this seems to happen regularily if I work on the same project with versions 6.7.1 and 6.8. Whenever I first edit the files with the older version, the newer version requires me to do this fix. I have no problems doing it the other way around.

Wake up, Google!

I finally decided I need to do something with the fact that since I'm using Ubuntu at work but the email they use is Exchange and I can't be arsed to check my laptop all the time, I keep forgetting appointments because they are only visible in Outlook's calendar.

So I installed Google Calendar Sync and started using Google Calendar on my work box. It works, except for one very annoying thing: Live Meetings are not shown in GC. And it's not like Google doesn't know about it. And it shouldn't be too big an issue to fix, either. So wakey, wakey, Google! It's an annoying bug that affects a lot of people, me included.

Unhashing Gitorious paths

Gitorious hashes the repository paths which can be an issue if you need to access them from somewhere else (e.g. Redmine). I wrote a script that creates sane links to the repositories with an easily followable structure. Probably the best way to use this is to add this to crontab to be run every now and again (every 10 minutes?).

openSUSE Build Server

Damn, I wished I had known about this last summer. Novell has developed a build server with which you can build different Linux distributions to different architectures. Something I used a lot of time on, and couldn't do for others than Intel-based processors since with standard tools you can't produce packages to other processor architectures. Well, now I know if I ever need to create a build system again.

openSUSE Build Server

MySql with Java in Ubuntu

I had some problems connecting to a MySql database from a Java application in Ubuntu. This a short description on what is needed. This assumes you have MySql and some database set already.

First, you need to install the package:

sudo aptitude install libmysql-java

Then you need to make sure that the classpath is correct:

CLASSPATH=$CLASSPATH:/usr/share/java/
export CLASSPATH

This is a good point to logout/login...

The code to connect to the database looks something like

Nokia S60 Music Player bug

So I got this new N86 8MP since I wanted a phone with a good camera (and N86 has the best Nokia has to offer) that could also replace my iPod which is breaking down. I think the hard drive is giving up. I went for Nokia because I'm used to them and the T9 works ok with Finnish as well. Other makers' phones tend to have problems with that.

I bought a 16GB memory card and copied some stuff there. The OVI Suite, unsurprisingly, didn't work too well so I reverted to keeping the folders in order by hand. This worked well to start with but then I added a few new albums to the card on the weekend and blast it! The band nor the albums were nowhere to be seen in the library. The tracks were in the track listing, though, so the player could see them.

I spend a good time figuring out the problem, including checking the ID3 tags and trying to fix them. Fortunately I finally found this link and following those instructions the music library was once again updated. This rebuilds the entire library. Fortunately it's that easy so I can just about manage even if I have to do it every time as I don't add new stuff that often.

Small issue with Gitorious

I created a simple project to gitorious.org so that I can easily work on it from several locations. Of course I had issues, from one location I had problems getting through the firewall but I also had a minor issue from home.

Gitorious tells that I can clone the repository with

git clone http://git.gitorious.org/scheator/scheator.git

which worked but when I tried to push I got this error:

fatal: protocol error: expected sha/ref, got '
----------------------------------------------
The git:// protocol is read-only.

Please use the push url as listed on the repository page.
----------------------------------------------'

Reason is that the url for the repository is incorrect in .git/config. It looked like

url = git://gitorious.org/scheator/scheator.git

when in fact it should look like

url = git@gitorious.org:scheator/scheator.git

After I fixed that, it works. I can't figure out why this can't be correct in the first place without the need to fix it by hand.

NetBeans not finding the main class

Oh man, this was an annoying problem. I suddenly got the error "Main class not found" from NetBeans even though everything was ok (the class was there, so was main(), project properties were correct etc). After some digging I found the simplest of solutions:

touch the main class file, i.e. change the date so that NetBeans detects it has been changed and recompile.

Whew. I was getting worried there for a moment.

Things I should know

It's surprising how little math us programmers know, me included. You would think that working on something that is more or less completely based on mathematics (deep down, your computer is basically just adding, subtracting, comparing and moving numbers) people on the field would know it. Well, that's not true, and a lot of it has to do with the fact how it is handled in school and university. Steve Yegge has a great article about the whole issue. It's actually so great I got an urge to learn it myself! Well, I'll enroll to an algorithms course next Autumn in any case..

And sometimes you run into articles that make you say wow, because you thought you knew most of the stuff, at least in principle. This document by Brian Kernighan of C and Unix fame back from 1984 did that to me. Wow. Simply wow. What is intriguing is what this should mean for open source software. One of the big principles of OSS is that a lot of people can check it so the likelihood of it having Trojan horses is small. Is it, really?

Checking ISBN with Java

I couldn't find any Java implementation to check both ISBN-10 and ISBN-13 so I wrote my own from scratch for a small assignment I had.