I got the load down to 380MB, but that’s still not good enough.

With the optimisation that I’d done to reduce memory load, I was starting to find that static assets are taking a while to load, typically all of the items on a standard e-commerce product listing page was taking around 3s to display. This was giving a negative user experience, so today I set about seeing what I could do to speed up the static assets, and keep a low memory footprint.

I’d heard people talking about running Lighttpd and Apache together, using Lighttpd to serve static assets – this seemed like a good idea, so I went ahead and configured Apache to listen on 127.0.0.1 only, and on a port that wasn’t 80.
This meant that I could have Lighttpd running on port 80, and filtering what traffic went to Apache.

Already I was down to 0.5s page-load speed, and down to 300MB memory footprint. Good, but could I do better?

Nginx came to the rescue, very simple set up for all of my virtual hosts, which fed directly into Apache for all server-side scripts, with nginx serving static assets. Down to a 0.2s page load, 280MB RAM usage and barely any CPU load regardless of the amount the server gets hit.

[ ~ ]
[ alex@server ] pstree
init-+-crond
 |-dhclient
 |-httpd---14*[httpd]
 |-mingetty
 |-mysqld_safe---mysqld---9*[{mysqld}]
 |-ntpd
 |-perl
 |-rsyslogd---2*[{rsyslogd}]
 |-screen---bash---irssi---figlet
 |-3*[screen---bash---irssi]
 |-screen-+-bash---irssi
 |        `-bash
 |-2*[screen---bash]
 |-sshd-+-3*[sshd---sshd---bash---screen]
 |      `-sshd---sshd---bash---pstree
 |-svnserve
 |-udevd
 `-znc

pgrep - lookup processes based on name or other attributes

To list all process id’s owned by root, we could:

 # ps -ef | egrep '^root ' | awk '{print $2}'
1
2
96
187
2053
...

Or we could:

# pgrep -u root
1
2
96
187
2053
...

split - for splitting  a larger file into manageable chunks

[ bbc.closing.sites.archive ]
[ alex@server ] ls -al
drwxrwxr-x 3 alex alex      4096 Apr 19 19:58 .
drwxrwxr-x 3 alex alex      4096 Apr  5 19:55 ..
-rw-rw-r-- 1 alex alex 916635648 Feb 10 15:04 bbc.co.uk.tgz
[ bbc.closing.sites.archive ]
[ alex@server ] split -b 128m bbc.co.uk.tgz bbc.co.uk.tgz_
[ bbc.closing.sites.archive ]
[ alex@server ] ls -al
-rw-rw-r-- 1 alex alex 916635648 Feb 10 15:04 bbc.co.uk.tgz
-rw-rw-r-- 1 alex alex 134217728 Apr 19 19:58 bbc.co.uk.tgz_aa
-rw-rw-r-- 1 alex alex 134217728 Apr 19 19:58 bbc.co.uk.tgz_ab
-rw-rw-r-- 1 alex alex 134217728 Apr 19 19:58 bbc.co.uk.tgz_ac
-rw-rw-r-- 1 alex alex 134217728 Apr 19 19:58 bbc.co.uk.tgz_ad
-rw-rw-r-- 1 alex alex 134217728 Apr 19 19:58 bbc.co.uk.tgz_ae
-rw-rw-r-- 1 alex alex 134217728 Apr 19 19:58 bbc.co.uk.tgz_af
-rw-rw-r-- 1 alex alex 111329280 Apr 19 19:58 bbc.co.uk.tgz_ag

nl - number the lines and output to stdout

[ bbc.closing.sites.archive ]
[ alex@server ] nl bbc.closing.sites.archive.txt | head
1  On Monday 24th January 2011 the BBC announced [1] that it would be
2  restructuring its online department - with 360 job losses and the
3  deletion of 200 of its top level directories (including the websites
4  that live under them - eg http://www.bbc.co.uk/blast).  172 of of those
5  top level directories [2] are due to be deleted within the coming 12
6  months.

7  Most of these sites are already 'mothballed' [3], which means that the

xmlwf - check whether an XML file is well formed

[ ~ ]
[ alex@server ] curl -s 'http://millsie.net' > index.html
[ ~ ]
[ alex@server ] xmlwf index.html
index.html:387:154: not well-formed (invalid token)

Eclipse does have this feature, but it is often overlooked – simple open a file in Eclipse, in the Perspective that you are programming in, and hit CTRL + SPACE. You will be given a popup similar to the one below, and you can start typing the name of the method/function you want to call, and it will show you a brief description, and the parameters that it takes, without having to look them up online – how handy is that!

In order to sign up to, well, any website these days, you have to give them your email address, in order to prove you are a real person, but a simple trick lets you track who is giving who your email address to who.

Say you sign up to my blog, your email address is i_love_biscuits@biscuitlover.com – by simply adding +<websitename> to your email address before the @ symbol, you can, relatively reliably, check who is selling your email address!

For example, i_love_biscuits+millsie.net@biscuitlover.com passes 99% of validators due to it being in RFC 2822 – however, some validators are too strict and won’t let you use it.

Using the +<websitename> or any other +<randompieceoftext> is a step closer to finding out who is selling your email address (or, giving it out for free!) – and the message will still enter your inbox, at which point you can create filters etc based on the To: field of the email to filter it down even more.

Whilst I do try and give as much as I can to work, I really like sleeping, so I wasn’t going to get up and start messing around in their CMS when I’m half awake.

I think of the ‘at’ command as a one-off cronjob, and it’s syntax is really intuitive: here are some examples of usage:

To run the command svn up at 1am tomorrow:

# at 1am tomorrow
svn up

To run the script ~/start_backups.sh at 10:00am on 31st July:

#at 10am Jul 31
~/start_backups.sh

To view jobs in the list:

atq

To remove jobs from the list:

atrm [job number]

This is really annoying.

You can, however, tell the SDK to use an external path for the datastore, so it will survive a shutdown.

I do this on an app by app basis, so I know where everything is saved and I feel more comfortable with it

1. Create a folder somewhere memerable(!)
2. Edit the settings for each app in the Python SDK, and add the following to the ‘Extra Flags’ field: –datastore_path=/path/to/folder/you/just/created

All done, you can now reboot and keep your datastore intact!

The usage is really simple, and I find it really useful, especially when using scp to transfer archives between servers.

Usage is really simple: pv database.sql | mysql -uusername -ppassword database

There is also quite a nice comment in the description of pv’s man page:

A more complicated example using numeric output to feed into the dialog(1) program for a full-screen progress display:

(tar cf - . \
| pv -n -s 'du -sb . | awk '{print $1}'' \
| gzip -9 > out.tgz) 2>&1 \
| dialog --gauge 'Progress' 7 70

Frequent use of this third form is not recommended as it may cause the programmer to overheat.

From the Cyanogen(mod) website:

The ZTE Blade (aka Venus or Orange San Francisco) is a touchscreen Android smartphone manufactured by ZTE for the European market. The device has various versions, differing in aspects such as screen type (AMOLED/TFT), camera (3.2MP or 5.0MP), and RAM (256MB or 512MB), although it is now clear that all European models have 512MB, but the Greek and older Hungarian versions of the phone have been limited to 256MB by software. A custom update is available on the internet to resize the partitions in order to get back the 256MB that’s originally taken up by software.

1. Download the Stable Mod for ZTE Blade
2. Download & install the Java SE JDK
3. Download & install the latest Android SDK Starter Tools as documented on the Cyanogen(mod) wiki
4. I tried using Amon_Ra’s recovery method, but that didn’t work for me. So we will use the ClockworkMod Recovery tool.
5. Download the latest version of ClockworkMod Recovery
6. Download the ZTE Blade drivers (Windows only)
7. Assuming that Android SDK is set up correctly, open a terminal and enter the following:
   # adb reboot bootloader
8. The phone will reboot and a single green android should appear on the screen, and remain there
9. Open a terminal and enter the following:
   # fastboot flash recovery /path/to/recovery-clockwork-3.0.1.4-blade.img
   ClockworkMod will now be installed in the recovery sector of the device
10. Enter USB Mount mode in Clockword Mod by using the volume buttons to scroll through options, home to select, and back to go back.
11. Copy the CyanogenMod .zip file to the mounted SD card, and optionally, Google Apps .zip for the device (I recommend this, it includes Anroid Market etc) – Leave USB Mount mode
12. Select the option to Wipe data/factory reset.
13. Then select the option to Wipe cache partition.
14. Select Install zip from sdcard.
15. Select Choose zip from sdcard.
16. Select the CyanogenMod update.zip.
    Optionally, repeat steps 14 & 15 and then selecting the Google Apps .zip
17. Once the installation has finished, select +++++Go Back+++++ to get back to the main menu, and select the Reboot system now option. The ZTE Blade should now boot into CyanogenMod.