Monday, 15 January 2018

Using btrfs to speed up Android development

Building Android from the scratch is slow. In my work machine building Oreo takes about three hours. This is ok, if you don't need to do it too often but sometimes you have to run a full build for one reason or other. Often because something got corrupted and partial build won't finish.

A nice way to improve this would be to create a golden image which has a full build and make a copy of that when you want to do some work. If this new area gets corrupted, just make a new copy. The problem with this approach is it takes a lot of disk space. Just one workspace is almost 200 gigabytes and since SSD is the preferred disk type since it's much quicker to build using one of them it gets quite expensive if you need to support several variants. I have a terabyte of space for this on my work machine but I have potentially four different products to support.

Btrfs is an almost perfect solution: by using snapshots space requirements are much smaller. Btrfs only makes a new copy if a file is changed and since most files are unaltered the savings are huge.

Following is for my work machine which has two 500 gigabyte SSDs for this. Data is striped while metadata is mirrored.

sudo apt install btrfs-tools
sudo mkfs.btrfs -d raid0 -m raid1 /dev/sdc /dev/sddsudo

mkfs.btrfs /dev/sdc -L Work
mkdir /work
sudo mount -t btrfs /dev/sdc /work

You might need to change ownership with chown.
To add the file system to /etc/fstab you can use something like this:

UUID= /work btrfs defaults 0 0
UUID can be the UUID of any disk in the btrfs file system.

After this it's time to create the base volume. Base volume only contains the source code and is the only volume that is synced from main repository.

mkdir /work/product-BASE_VOL
btrfs subvolume create /work/product-BASE_VOL

Android source is fetched to this directory.

Next we create a snapshot of the base volume for building:

btrfs subvolume snapshot /work/product-BASE_VOL /work/product-BUILD-variant

A full build is done in this snapshot.

Now when we want to code something, we create a snapshot of /work/product-BUILD-variant and work in that directory. No need to do a full build. If we somehow mess up, just delete this directory (you might want to save changes first, or just use mv to rename the directory) and create a new snapshot.

The actual magic happens when you use a script to update the base volume and build directories every night. It just syncs the base volume and replaces the build directory with a fresh build. Now you have a fresh build every morning with the latest changes! Again, you might want to keep a copy of the old build directory for a while just in case.

You add this to your crontab by running crontab -e and adding the following:

MAILTO=""
PATH=/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin
01 00 * * * /path/to/script/update-golden.sh btrfspath product variant

Note that you might need to alter the PATH variable. This will run the script every day one minute after midnight.

This script will create a backup of the previous build directory. Old build directories are best removed with 

btrfs subvolume delete

References:
https://btrfs.wiki.kernel.org/index.php/Getting_started
https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices

Wednesday, 20 January 2016

Issue with c.vim

I got a new PC at work and for some reason home directory is located in /opt/home while /home is just a link to that directory.

This caused an issue with c.vim. Namely it thought I had a system-wide installation so was looking for templates under /opt/share rather than $HOME. The core issue is an if clause that checks if the script file is under $HOME. Problem was that path to the script file is under /opt/home while $HOME is set to /home and thus the script thinks it's not in the home directory.

I fixed this issue rather heavy-handedly by forcing my installation to only include local installation. I did this by editing c.vim and under the part where paths are set for Unix/Linux, removing all the stuff not needed, i.e. the if clause and everything inside else branch.

Friday, 5 December 2014

Installing Erlang on Ubuntu & Emacs

This was quite simple.

From https://www.erlang-solutions.com/downloads/download-erlang-otp:
wget http://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb
sudo dpkg -i erlang-solutions_1.0_all.deb

sudo apt-get update
sudo apt-get install erlang erlang-doc erlang-manpages

The page says erlang-mode also needs to be installed separately but it seemed to be installed already (or it was included in the erlang package).

After this all I needed to do in Emacs was to add this to .emacs:
(setq load-path (cons "/usr/lib/erlang/lib/tools-2.7/emacs"      load-path)) 
(setq erlang-root-dir "/usr/lib/erlang") 
(setq exec-path (cons "/usr/lib/erlang/" exec-path)) 
(require 'erlang-start)
Now I need to figure out how to tell flycheck of all the directories it needs & set up TAGS.

Friday, 7 November 2014

Git prompt

A very cool and useful way to avoid issues with working on the wrong git branch is to use the git-prompt.sh script. I sometimes manage to do this mistake. I need to check something which means changing the branch, I'm distracted and when I get back to work I forget to change the branch back. This can get messy. But with git-prompt.sh current branch is always shown in the prompt! I read about this from the Git book at http://git-scm.com/book/en/v2/Git-in-Other-Environments-Git-in-Bash which also mentions git-completion.bash. I incorporated that into my .bashrc, too, but I feel that's a lot less useful than the prompt.

I changed the instructions a bit and made the prompt like this:
 export PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w$(__git_ps1 " (%s)")\$ '
This will add my username and machine name in the prompt as well (I always use to have them).

Saturday, 27 September 2014

Ubuntu stops at Grub menu

I had this annoying issue with my Ubuntu server. Every time it rebooted, quite often because of a power failure, it wouldn't boot properly. I needed to go the storage room where I have it and connect a laptop with a serial cable. Well, I got an UPS to at least stop this from happening inadvertently but I decided to fix the issue once and for all when I need to reboot it for other reasons. And it turns out that since 12.04 LTS there has been a very simple solution. Just adding the following to /etc/default/grub fixes this issue:
GRUB_RECORDFAIL_TIMEOUT=0

The problem is that if the system goes down without proper shutdown, default behaviour is to stop in the Grub menu. This is pretty ridiculous for servers.

As it happens I managed to buy the only UPS (from APC) without remote access, so I can't set the system to shutdown in case of power failure. But the UPS should be able to keep the system running for at least 30 minutes, easily, and our power failures are not that long. Think there has been one that long once and that was because our mains was cut by some workers.

Thursday, 18 September 2014

Emacs evil mode

I decided to have a look at Emacs evil mode (the Vi mode for Emacs). Vim is not really suitable as an IDE. It's an editor and anything else tends to be a bit of a hack. Debugger is one such issue, it's not very convenient to use gdb inside Vim. As an added bonus I can use Lisp instead of Vimscript to write scripts.

But I like Vim's editing better than Emacs's endless control-meta-this-and-that approach. It's faster and more convenient. I might try Emacs without Evil mode at some point, I'm a pragmatist after all but at least this way I don't have to learn the entire Emacs editing thingy just to get started. I'm familiar with M-x, C-x C-f, C-x C-f and the like which should suffice for now.

Installing was almost like a breeze. Had I figured out Ubuntu 12.04 uses emacs23 instead of emacs24 which all the help I can google points to I would have had an easier time. But here's how I got started:

sudo add-apt-repository ppa:cassou/emacssudo apt-get updatesudo apt-get install emacs24git clone http://www.dr-qubit.org/git/undo-tree.git ~/.emacs.d/undo-treegit clone git://gitorious.org/evil/evil.git ~/.emacs.d/evil


Then I added the following to .emacs:
(add-to-list 'load-path "~/.emacs.d/evil")  (require 'evil)  (evil-mode 1)
(load-theme 'wombat)

Last line loads the wombat colour scheme. I use wombat256 in Vim.

Now I just need to figure out how to get the same functionality as my favourite Vim plugins have. For instance, CtrlP. Someone suggested ido mode, but I can't seem to get it to do recursive search. In CtrlP you don't have to know where a file is to find it.

Thursday, 3 July 2014

Headphones in Ubuntu

I bought a set of Beyerdynamic DT660's (got them cheap) to use at work. But when I plugged them into the work machine (runs Ubuntu 12.02) sound quality was horrible. Bass was non-existent ans high frequencies intolerable. In other words it was impossible to listen to music. I first tried an equalizer (qpaq) and while it did improve the sound a bit it was still not good enough.

I just got a Behringer U-Control UCA202 USB-DAC with headphone amp and boy is this a big improvment! At 29€ a no-brainer. Installation was also totally painless. This would suggest the issue is with the Intel HDA driver. So if you hate the sound that you get from analog output from your Linux box, get one of these (or any other USB-DAC with headphone amp). You won't regret it.