Archive for March, 2010

Facebook throttling the last viral channel

Wednesday, March 24th, 2010

Up until today, there was still one viral channel left on Facebook that allowed your applications to grow organically: the wall posts.

Today Facebook cracked down on it as explained in their developer blog.

The consequences are amazing as you can see the overnight 50% drop in DAUs for quiz applications that relied heavily on wall posts like Social Interview (graphs courtesy of developeranalytics.com):
Social Interview DAUs down 50 percent

So from a canvas application you can still publish stories to walls, but you now have to ask the user each and every time to confirm with the “Publish this story to your friend’s Facebook Wall?” popup like this one:

Before you could let the user choose not to have this popup every time they want to post something on a wall by using the Facebook.showPermissionDialog() javascript call…:

Facebook.showPermissionDialog('publish_stream', callback);

function callback (perms) {
  if (!perms) {
    message('You did not grant the special permission to post to friends wall without being prompted.');
  } else {
    message('You can now publish to walls without being prompted.');
  }
}

… that would prompt for your permission to, hem, not be prompted when publishing a story to a wall (“Allow {Application X} to publish posts or comments without prompting me.”):

Once you had this permission granted by your user, you could just call streamPublish() with the auto_publish parameter set to true to avoid the popup:

Facebook.streamPublish(null, attachment, actionLinks, target_id, "Add a personal message", callback, true);

In practice it didn’t quite work because even though the language in the permission dialog is clear that you would not see any popup anymore, you would still either:
- get the popup all the time (and that was a Facebook bug)
- or reach a throttling limit (streamPublish() would return “Feed action request limit reached”) and would not be able to post anymore.

Facebook is very obscure on what the limits are, contrary to the requests limits which you can see from the Statistics of your application in the Developer app (tab “Allocations”).

So you had to use a trick to circumvent this throttling by asking for a totally unrelated permission: the Offline permission.

Facebook.showPermissionDialog('offline_access', callback);

That would ask the user for yet another permission using the “Allow Constant Authorization.” popup:

This workaround worked well until today, as you now get a lot more “Feed action request limit reached” errors.
So much more that Facebook added an extra tab in the Statistics of your application called “API Errors”:

What’s remaining now for virality on Facebook?
- the notifications are gone
- the invites are throttled in an obscure way so you never see your allocation of invites per user per day go up
- the wallposts are throttled
- … but you now have access to users emails!

This is a clear move from Facebook to capture more revenues by forcing developers to buy ads from them.

The true organic virality is gone, unless developers become spammers and start hammering the email addresses of Facebook users.
This will happen unfortunately, and users will just opt-out, so it’s a short lived solution.

Is it time to move back to MySpace?

InnoDB engine disabled when specifying a buffer pool size too high

Friday, March 12th, 2010

The InnoDB engine will (silently) not start if innodb_buffer_pool_size too big.

You modified your my.cnf to increase the performance of your DB by tweaking the InnoDB variables.
Then you restarted MySQL for the changes to take effect, everything seems fine, no error message.

Except your InnoDB tables are no longer working.

mysql> SELECT VERSION();
+-----------------------+
| VERSION()             |
+-----------------------+
| 5.1.37-1ubuntu5.1-log |
+-----------------------+
1 row in set (0.00 sec)

mysql>
mysql> SHOW CREATE TABLE users;
ERROR 1286 (42000): Unknown table engine 'InnoDB'
mysql>

You can’t believe your eyes and start double checking for confirmation:

mysql> SHOW VARIABLES LIKE 'have_innodb';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| have_innodb   | NO    |
+---------------+-------+
1 row in set (0.00 sec)
mysql>
mysql> SHOW ENGINES \G
*************************** 1. row ***************************
      Engine: MyISAM
     Support: DEFAULT
     Comment: Default engine as of MySQL 3.23 with great performance
Transactions: NO
          XA: NO
  Savepoints: NO
*************************** 2. row ***************************
      Engine: MRG_MYISAM
     Support: YES
     Comment: Collection of identical MyISAM tables
Transactions: NO
          XA: NO
  Savepoints: NO
*************************** 3. row ***************************
      Engine: BLACKHOLE
     Support: YES
     Comment: /dev/null storage engine (anything you write to it disappears)
Transactions: NO
          XA: NO
  Savepoints: NO
*************************** 4. row ***************************
      Engine: CSV
     Support: YES
     Comment: CSV storage engine
Transactions: NO
          XA: NO
  Savepoints: NO
*************************** 5. row ***************************
      Engine: MEMORY
     Support: YES
     Comment: Hash based, stored in memory, useful for temporary tables
Transactions: NO
          XA: NO
  Savepoints: NO
*************************** 6. row ***************************
      Engine: FEDERATED
     Support: NO
     Comment: Federated MySQL storage engine
Transactions: NULL
          XA: NULL
  Savepoints: NULL
*************************** 7. row ***************************
      Engine: ARCHIVE
     Support: YES
     Comment: Archive storage engine
Transactions: NO
          XA: NO
  Savepoints: NO
7 rows in set (0.00 sec)

mysql>

Yep, InnoDB is definitely gone.

If you try to create an InnoDB table unfortunately the command will not fail:
MySQL will create a MyISAM table behind your back without telling you it couldn’t create an InnoDB table. Ouch.

Examining your /etc/mysql/my.cnf and you see you typed some extra zeros:

innodb_buffer_pool_size=10000M

Probably you were thinking assigning 1G to the buffer pool when in fact it was 10GB, and your server only has 2GB of memory.
Change that to:

innodb_buffer_pool_size=1000M

Restart MySQL and you get your InnoDB tables working again.

mysql> SHOW VARIABLES LIKE 'have_innodb';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| have_innodb   | YES   |
+---------------+-------+
1 row in set (0.00 sec)

mysql>

It’s amazing but MySQL will actually not complain about a value too high when you restart it.
It looked like this to me when the value was too high:

bash# /etc/init.d/mysql restart
 * Stopping MySQL database server mysqld    [ OK ]
 * Starting MySQL database server mysqld    [ OK ]
 * Checking for corrupt, not cleanly closed and upgrade needing tables.
bash#

On an Amazon EC2 Standard Default Small Instance running Ubuntu server, I was surprised to find the limit is very low.
It’s a 32-bit platform with 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), and 160 GB of local instance storage.

InnoDB could not start when innodb_buffer_pool_size was set to 512M.
However it worked with 256M.

Enabling Memcached on BlueHost

Thursday, March 11th, 2010

Bluehost is a pretty good web hosting company, cheap and leaving you in total control of your website.
That is if you know your way around.
I highly recommend them for dirt cheap hosting yet access to powerful features.

A common complaint against any cheap hosting is the lack of memcached installation by default.
It really doesn’t make sense even business-wise for them because memcached would allow to use less CPU/Memory per site hosted, meaning less total servers, meaning less power drawn, meaning big savings.
The hosting plans charging usually a fixed amount per month or year, reducing CPU/Memory/Servers would really make sense as the savings would be on directly on them.

Anyway, now we need to setup memcached ourselves on a machine we don’t have root access to.
No big deal, we will just have to compile everything ourselves.

Here’s how.

1/ Create an installation folder.

As we cannot write to /usr/, we cannot install any software system wide, so everything will remain local in our home folder.
Let’s call it ~/install/
It will be where you install all your compiled software, acting like the /usr/ folder we don’t have access to.

bash# mkdir ~/install

2/ Compile libevent

Why libevent? Because we will need it later to compile memcached.

bash# wget http://www.monkey.org/~provos/libevent-1.4.13-stable.tar.gz
bash# tar zxf libevent-1.4.13-stable.tar.gz
bash# cd libevent-1.4.13-stable
bash# ./configure -prefix=~/install/
bash# make
bash# make install

libevent is now neatly installed in our home directory at ~/install/

3/ Compile memcached

bash# wget http://memcached.googlecode.com/files/memcached-1.4.4.tar.gz
bash# tar xzf memcached-1.4.4.tar.gz
bash# cd memcached-1.4.4
bash# ./configure --with-libevent=~/install/ --prefix=~/install/
bash# make
bash# make install

Neat, now we have memcached installed in ~/install/bin/, so let’s try to run it.

bash# cd ~/install/bin
bash# ./memcached
failed to set rlimit for open files. Try running as root or requesting smaller maxconns value.
bash# ./memcached -c 100
^CSIGINT handled
bash#

So we hit a limitation: as we’re not root, our memcached server is limited to only 100 connections instead of the default 1024.
Maybe we can live with that.

Now the server part is done, we need to be able to use the PHP commands to communicate with memcached.
This is found in the PEAR packages.

4/ Install the PHP memcache client

(the memcache client doesn’t have the ‘d’ at the end)

bash# pecl install memcache
downloading memcache-2.2.5.tgz ...
Starting to download memcache-2.2.5.tgz (35,981 bytes)
..........done: 35,981 bytes
11 source files, building
running: phpize
Configuring for:
PHP Api Version:         20041225
Zend Module Api No:      20060613
Zend Extension Api No:   220060519
Enable memcache session handler support? [yes] :
Notice: Use of undefined constant STDIN - assumed 'STDIN' in CLI.php on line 304

Warning: fgets(): supplied argument is not a valid stream resource in CLI.php on line 304

Warning: fgets(): supplied argument is not a valid stream resource in /root/lib/php/PEAR/Frontend/CLI.php on line 304
building in /usr/local/src/pear/pear-build-bottoml8/memcache-2.2.5
ERROR: could not create build dir: /usr/local/src/pear/pear-build-bottoml8/memcache-2.2.5

Ouch, we’re not root so pecl fails.
No big deal: pecl already downloaded memcache-2.2.5.tgz for us, let’s compile it manually.

bash# tar xzf memcache-2.2.5.tgz
bash# cd memcache-2.2.5
bash# phpize
bash# ./configure --prefix=~/install/
bash# make
bash# make test
bash# make install
Installing shared extensions:     /usr/lib/php/extensions/no-debug-non-zts-20060613/
cp: cannot create regular file `/usr/lib/php/extensions/no-debug-non-zts-20060613/#INST@7000#': Read-only file system
make: *** [install-modules] Error 1
bash#
bash# ls modules/
./  ../  memcache.so*
bash# cp memcache.so ~/install/lib/

Ok, make install failed, but we have already what we need: memcache.so in the memcache-2.2.5/modules/ directory.
It’s the extension we need to load in PHP (in the php.ini file) so we can use the PHP commands to access memcached.
We just moved it to ~/install/lib/ so it’s comfy with the other libraries we compiled earlier (libevent and memcached).

5/ Load the memcache module for PHP

Now for the tricky part:
As we don’t have root access, we cannot just change the server’s php.ini file to include our freshly brewed memcache.so and restart Apache to make the changes taken into account.

But Bluehost provides us with a neat way to use our own php.ini file.
By default this option is disabled.
In their cPanel it’s listed under the category ‘Software / Services’ as ‘PHP Config’.

By default the setting is Standard PHP5, we need to change that to the third option: PHP5 (FastCGI)
It will directly load the php.ini it finds in your ~/public_html/ directory.

On this same page of the cPanel, there’s an interesting button “INSTALL PHP.INI MASTER FILE”.
Just click it and magically the server’s php.ini file is copied in ~/public_html/php.ini.default

It’s now our base php.ini we’ll use to load memcache. Let’s edit it.

bash# cd ~/public_html/
bash# mv php.ini.default php.ini
bash# vi php.ini

In php.ini, find the line that reads:
extension_dir = “/usr/local/lib/php/extensions/no-debug-non-zts-20060613″
and replace it with
extension_dir = “/home/your_bluehost_user_name/install/lib”
so that we have access to our memcache.so lib.
At the end if php.ini, before the last line that loads the Zend Optimizer, include this line to load memcache.so:
extension=memcache.so

There should be some other libraries being loaded here:
;extension=pdo.so
;extension=pdo_sqlite.so
;extension=sqlite.so
;extension=pdo_mysql.so

As we changed the extension_dir path these libraries are not accessible anymore so you can just comment them.
extension_dir does not support listing multiple directories so I wish we didn’t have to do that.
At some point there was a proposal to make it happen, don’t know if it’s implemented yet, I couldn’t find anything about it.
Worst case scenario if you need PDO (and you probably do), you just need to compile it and install it in ~/install/lib/

Now that php.ini is modified, the server will pick up the configuration automagically.
But you may have to wait a little bit as php.ini will not be read again right away, you have to wait for the next poll, a few minutes.

6/ Let’s use memcached!

Now everything’s installed, let’s finally use memcached.

Create a memcache.php test file that will connect to memcached and just return the server version:

<?php
$memcache = new Memcache;
$memcache->addServer('localhost', 11211);
$memcache->connect('localhost', 11211);

echo $memcache->getVersion();
?>

Start the memcached in verbose mode so we can see our connection arriving:

bash# cd ~/install/bin/
bash#  ./memcached -h
memcached 1.4.4
-p <num>      TCP port number to listen on (default: 11211)
-U <num>      UDP port number to listen on (default: 11211, 0 is off)
-s <file>     UNIX socket path to listen on (disables network support)
-a <mask>     access mask for UNIX socket, in octal (default: 0700)
-l <ip_addr>  interface to listen on (default: INADDR_ANY, all addresses)
-d            run as a daemon
-r            maximize core file limit
-u <username> assume identity of <username> (only when run as root)
-m <num>      max memory to use for items in megabytes (default: 64 MB)
-M            return error on memory exhausted (rather than removing items)
-c <num>      max simultaneous connections (default: 1024)
-k            lock down all paged memory.  Note that there is a
              limit on how much memory you may lock.  Trying to
              allocate more than that would fail, so be sure you
              set the limit correctly for the user you started
              the daemon with (not for -u <username> user;
              under sh this is done with 'ulimit -S -l NUM_KB').
-v            verbose (print errors/warnings while in event loop)
-vv           very verbose (also print client commands/reponses)
-vvv          extremely verbose (also print internal state transitions)
-h            print this help and exit
-i            print memcached and libevent license
-P <file>     save PID in <file>, only used with -d option
-f <factor>   chunk size growth factor (default: 1.25)
-n <bytes>    minimum space allocated for key+value+flags (default: 48)
-L            Try to use large memory pages (if available). Increasing
              the memory page size could reduce the number of TLB misses
              and improve the performance. In order to get large pages
              from the OS, memcached will allocate the total item-cache
              in one large chunk.
-D <char>     Use <char> as the delimiter between key prefixes and IDs.
              This is used for per-prefix stats reporting. The default is
              ":" (colon). If this option is specified, stats collection
              is turned on automatically; if not, then it may be turned on
              by sending the "stats detail on" command to the server.
-t <num>      number of threads to use (default: 4)
-R            Maximum number of requests per event, limits the number of
              requests process for a given connection to prevent
              starvation (default: 20)
-C            Disable use of CAS
-b            Set the backlog queue limit (default: 1024)
-B            Binding protocol - one of ascii, binary, or auto (default)
-I            Override the size of each slab page. Adjusts max item size
              (default: 1mb, min: 1k, max: 128m)
bash#
bash# ./memcached -c 100 -vvv -l 127.0.0.1
slab class   1: chunk size        96 perslab   10922
slab class   2: chunk size       120 perslab    8738
slab class   3: chunk size       152 perslab    6898
slab class   4: chunk size       192 perslab    5461
...
slab class  40: chunk size    616944 perslab       1
slab class  41: chunk size    771184 perslab       1
slab class  42: chunk size   1048576 perslab       1
<26 server listening (auto-negotiate)
<27 send buffer was 124928, now 268435456
<27 server listening (udp)

---- Access the memcache.php file here in your browser ----

<28 new auto-negotiating client connection
<29 new auto-negotiating client connection
29: going from conn_new_cmd to conn_waiting
29: going from conn_waiting to conn_read
29: going from conn_read to conn_parse_cmd
29: Client using the ascii protocol
<29 version
>29 VERSION 1.4.4
29: going from conn_parse_cmd to conn_write
29: going from conn_write to conn_new_cmd
29: going from conn_new_cmd to conn_waiting
29: going from conn_waiting to conn_read
28: going from conn_new_cmd to conn_waiting
28: going from conn_waiting to conn_read
28: going from conn_read to conn_closing
<28 connection closed.

Tada! And magically the browser is displaying a magnificent “1.4.4“.

You got yourself a memcached server running on BlueHost.

Did you find this post useful? Buy this man a beer:

Setting up MySQL replication

Wednesday, March 10th, 2010

In this article the Master is 200.126.187.013 and the Slave is 200.126.187.066

On the Master, edit /etc/mysql/my.cnf:

server-id        = 126187013  # choose a unique integer
log_bin          = /var/log/mysql/mysql-bin.log
bind-address   = 0.0.0.0  # listen on all interfaces (by default 127.0.0.1 is used)
binlog_do_db   = mydb    # replicate only the 'mydb' database

Test your new configuration for syntax error:

bash# mysqld --help

Make sure the skip-networking option is not enabled.
Make sure server-id is unique. You can for instance use the last 3 octets of the IP address of the server.

Create a replication user on the Master using the IP address of the Slave that will access it (200.126.187.66):

mysql> select version();
+-------------------+
| version()         |
+-------------------+
| 5.1.37-1ubuntu5.1 |
+-------------------+
1 row in set (0.00 sec)
mysql> CREATE USER 'replication'@'200.126.187.66' IDENTIFIED BY 'slavepass';
mysql> GRANT REPLICATION SLAVE ON mydb.users TO 'replication'@'200.126.187.66';
mysql> GRANT SELECT ON mydb.users TO replication;

Restart MySQL for the changes to take effect:

bash# /etc/init.d/mysql restart

On the Slave, edit /etc/mysql/my.cnf to set the server-id:

server-id  = 126187066    # choose a unique integer

Copy the current database on the Master with mysqldump and load it on the Slave.

Check the current status of the Master by retrieving its log file name and position:

mysql> SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000003 |    23577 | mydb         |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec)
mysql>

Start the replication on the Slave.

mysql> CHANGE MASTER TO MASTER_HOST='200.126.187.013', MASTER_USER='replication', MASTER_PASSWORD='slavepass', MASTER_LOG_FILE='mysql-bin.000003', MASTER_LOG_POS=0;
mysql> START SLAVE;

We did take some shortcuts here: the proper way would have been to:
1/ prevent the Master from committing anything
2/ backup the DB on the Master
3/ check the log file name and log position on the Master (‘SHOW MASTER STATUS’)
4/ load the backup on the Slave
5/ start replication using the log file name and position on the Master
6/ unlock the Master so the commits can happen again

A nightmare if the Master is already a live production server because it means from step 1/ to 6/ the Master is dead not committing anything.

As we did a ‘dirty’ backup, the replication will bump into errors that will stop the Slave (mainly because of entries already on the Slave, so we can safely skip them).
List them on the Slave with ’show slave status \G’ (look at the ‘Last_Error’) and if you can skip it, do it this way:

mysql> SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1;
mysql> START SLAVE;

Continue skipping the failed insert (if they can be skipped of course) until the slave completely catches up with the Master.

How do I know when the Slave is done catching up with the Master?

Issue a ‘SHOW SLAVE STATUS’ on the Slave:

mysql> SHOW SLAVE STATUS \G
*************************** 1. row ***************************
             Slave_IO_State: Waiting for master to send event
                Master_Host: 200.126.187.013
                Master_User: replication
                Master_Port: 3306
              Connect_Retry: 60
            Master_Log_File: mysql-bin.000003
        Read_Master_Log_Pos: 339528
             Relay_Log_File: prod1-relay-bin.000004
              Relay_Log_Pos: 339665
      Relay_Master_Log_File: mysql-bin.000003
           Slave_IO_Running: Yes
          Slave_SQL_Running: Yes
            Replicate_Do_DB:
        Replicate_Ignore_DB:
         Replicate_Do_Table:
     Replicate_Ignore_Table:
    Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
                 Last_Errno: 0
                 Last_Error:
               Skip_Counter: 0
        Exec_Master_Log_Pos: 339528
            Relay_Log_Space: 339665
            Until_Condition: None
             Until_Log_File:
              Until_Log_Pos: 0
         Master_SSL_Allowed: No
         Master_SSL_CA_File:
         Master_SSL_CA_Path:
            Master_SSL_Cert:
          Master_SSL_Cipher:
             Master_SSL_Key:
      Seconds_Behind_Master: 0
1 row in set (0.00 sec)

mysql>

You can see everything is fine because:
- ‘Seconds Behind Master’ reports 0, so the Slave is done catching up (it’s in sync with the Master)
- Both Slave IO and Slave SQL are running
- The Slave is just waiting for new data to come in: ‘Waiting for master to send event’

Source

Changing the timezone on a Ubuntu server

Wednesday, March 10th, 2010
bash# sudo dpkg-reconfigure tzdata

Then just follow the instructions on the screen to select your time zone.

While we’re at it, let’s sync the time with the atomic clock:

bash# sudo ntpdate pool.ntp.org
10 Mar 09:55:19 ntpdate[23121]: adjust time server 173.8.198.243 offset -0.009830 sec

Making the default Wordpress Kubrick theme wider

Tuesday, March 9th, 2010

Using this tutorial to make Kubrick wider in Wordpress 2.9.2 will not work right away as the tutorial was written for 2.7, but the basics remain the same:

1/ Change the 4 images to 984px

kubrickheader.jpg
kubrickfooter.jpg
kubrickbgwide.jpg
kubrickbg-ltr.jpg

2/ Change header-img.php as in the tutorial

$corners = array(
        0 => array ( 25, 734+224 ),
        1 => array ( 23, 736+224 ),
        2 => array ( 22, 737+224 ),
        3 => array ( 21, 738+224 ),
        4 => array ( 21, 738+224 ),
        177 => array ( 21, 738+224 ),
        178 => array ( 21, 738+224 ),
        179 => array ( 22, 737+224 ),
        180 => array ( 23, 736+224 ),
        181 => array ( 25, 734+224 ),
        );
...
$x2 = 740+224;
...
$x2 = 739+224;

4/ Modify style.css

#headerimg {
        width: 984px;
}
#page {
        width: 984px;
}
#header {
        width: 984px;
}
.narrowcolumn {
        width: 660px;
}
.widecolumn {
        width: 660px;
}
#footer {
        width: 984px;
}
#sidebar {
        margin-left: 745px;
}

VoilĂ 

Display APC cache statistics

Tuesday, March 9th, 2010

Use apc_cache_info()

echo "<pre>".print_r(apc_cache_info())."</pre>"

Array
(
    [num_slots] => 2000
    [ttl] => 0
    [num_hits] => 9
    [num_misses] => 3
    [start_time] => 1123958803
    [cache_list] => Array
        (
            [0] => Array
                (
                    [filename] => /path/to/apc_test.php
                    [device] => 29954
                    [inode] => 1130511
                    [type] => file
                    [num_hits] => 1
                    [mtime] => 1123960686
                    [creation_time] => 1123960696
                    [deletion_time] => 0
                    [access_time] => 1123962864
                    [ref_count] => 1
                    [mem_size] => 677
                )
            [1] => Array (...iterates for each cached file)
)

The Madoff Experiment

Tuesday, March 9th, 2010

The Madoff Experiment is currently in progress, check back later for the results…

Spring Blog, all fresh

Monday, March 8th, 2010

Migrating all the entries to WordPress for spring cleaning.

Enjoy.

CakePHP: Convert a complex find condition from 1.1 to 1.3

Sunday, March 7th, 2010

First, findAll() no longer exists in 1.3, you need to use find(‘all’).
Then the conditions array structure is different, moving from a key/value pair array to an array with values only.

This is the query in 1.1:

 $conditions = array("OR" => array(
      'Item.expiration'       => 0,
      'Item.expiration >'     => "NOW()"
    )
 );
 $items = $this->Item->findAll($conditions);

Re-written for 1.3:

$conditions = array("OR" => array(
      'Item.expiration = 0',
      'Item.expiration > NOW()' // single string
    )
 );
 $params = array(
   'conditions'    => $conditions
);
 $items = $this->Item->find('all', $params);

Source