Cloning a B-Translator Server

Why do I need to clone a B-Translator server? An obvious reason is to have a backup server. In case that something goes wrong with the main server I can quickly switch to the backup server, until I find and fix the problem.

A clone can also be used for testing. Before applying something on the main server I can test it first on a clone/backup server.

Another reason for cloning can be load-balancing. It is possible to keep several clone servers synchronised with each-other, and then the work can be shared between them with a load balancer. I haven't tried this yet, but I think that it should work.

It can be possible and even simple to make identical clones just by copying the physical disk or the filesystem. However I prefer to build a new server from scratch and copy only the relevant data from the old server.

Table of Contents

Installation

First, I build a new server from scratch:

mkdir /var/chroot
cd /var/chroot/
git clone --branch v1.0 https://github.com/dashohoxha/B-Translator.git
nohup nice B-Translator/install/install.sh btr-1 &
tail -f nohup.out

Then configure it:

chroot btr-1/ /tmp/install/config.sh

And then reboot the host:

reboot  ## it is advisable to reboot the host after this installation

The last step of configuration is about setting the primary language of translation (sq in my case), and a list of auxiliary languages. This list should include at least fr, since it is used instead of POT (PO template) files, when they are missing. But the features related to auxiliary languages are not implemented yet, so it is not much useful to include other languages on that list, and I leave just fr.

Setting the domain name

If I can assign a domain name to this copy of B-Translator (for example test.l10n.fs.al), but during the configuration it is not given properly, then I can correct/change it like this:

chroot /var/chroot/btr-1
cd /var/www/btr/profiles/btranslator
install/config/domain.sh

If this is a local copy (installed on my local machine) and its domain name is for example l10n.org.xy, then I also add this domain name on the file /etc/hosts of my machine:

127.0.0.1       l10n.org.xy  localhost

This way I can access it by typing https://l10n.org.xy on the browser location (not https://127.0.0.1 or https://localhost).

Replacing NGINX with Apache2

I have used NGINX as the default web server because it is supposed to be faster and more responsive under heavy load. Unfortunately it has been working very slow, for some reasons that I have not been able to find and fix. On the log file /var/log/nginx/btr.error.log I see messages like this:

2013/07/15 13:31:57 [info] 10294#0: *47 client closed prematurely connection while SSL handshaking, client: 127.0.0.1, server: l10n.org.xx
2013/07/15 13:31:57 [info] 10294#0: *48 client closed prematurely connection while SSL handshaking, client: 127.0.0.1, server: l10n.org.xx

Note: If find out what is wrong with NGINX and manage to fix it, please let me know.

So, I replace NGINX with Apache2 like this:

chroot /var/chroot/btr-1
cd /var/www/btr/profiles/btranslator
dev/apache2.sh start

It will stop services nginx, php5-fpm and memcached, will start apache2, will disable drupal module memcache, and will modify properly settings.php.

However there are also some changes outside chroot that should be done. The script dev/apache2.sh start cannot do them automatically (because it runs inside the chroot), so I have to do them manually. On the init script /etc/init.d/chroot-btr-1 I make this change:

#CHROOT_SERVICES="cron php5-fpm memcached mysql nginx"
CHROOT_SERVICES="cron mysql apache2"

Enable other drupal features

After installation I also enable some features that are optional and are not enabled by default during installation:

drush features-list
drush --yes pm-enable btr_simplenews
drush --yes pm-enable btr_mass_contact
## drush --yes pm-enable btr_googleanalytics
## drush --yes pm-enable btr_drupalchat

Enable the feature btr_fb

Note: The feature btr_fb is not yet finished and tested properly.

Enable it like this:

## drush --yes pm-enable btr_fb

After installing btr_fb, the configuration part related to FB should be un-commented, at the end of the file /var/www/btr/sites/default/settings.php:

// /* fb config
$conf['fb_api_file'] = 'profiles/btranslator/libraries/facebook-php-sdk/src/facebook.php';
include "profiles/btranslator/modules/contrib/fb/fb_url_rewrite.inc";
include "profiles/btranslator/modules/contrib/fb/fb_settings.inc";
if (!headers_sent()) {
  header('P3P: CP="We do not have a P3P policy."');
}
// fb config */

If you forget to do it, you will notice performance degrade with the site.

Start ssh

If this copy of B-Translator is remote, then I install ssh as well for accessing it easily and for using remote drush commands:

chroot /var/chroot/btr-1
cd /var/www/btr/profiles/btranslator
dev/install-sshd.sh

This script will also take care to change the ssh port to 2201, in order to avoid any conflicts with any existing daemon on the host environment, and also for increased security.

For drush remote access to work correctly, the public/private key ssh access should be set up and configured as well. For more detailed instructions on how to do it see: http://dashohoxha.blogspot.com/2012/08/how-to-secure-ubuntu-server.html

Reset the password of drupal admin

I almost always forget the password of admin (the primary user of Drupal) that I assign during installation. So, I have to reset it:

drush user-password admin --password="new-password"

By the way, on the file /etc/drush/drushrc.php you can see this drush setting:

<?php
// by default use the B-Translator root directory
$options['r'] = '/var/www/btr';

This means that the root directory of drush is always /var/www/btr, no matter where we call it from.

Transfer content

On the master (main/live) server, I export all the content as a feature, with the help of the module node_export. But first I have to disable the existing btr_content feature, otherwise the feature export will fail.

drush --yes pm-disable btr_content
drush --yes features-export --destination=/tmp btr_content_1 node_export_features
tar --create --gzip --file=btr_content_1.tgz --directory=/tmp btr_content_1

Now I transfer btr_content_1.tgz to the clone server and replace the existing content with it:

cd /var/www/btr/profiles/btranslator
cd modules/features/
tar --extract --gunzip --file=btr_content_1.tgz
drush --yes pm-disable btr_content
drush delete-all all  ## delete all existing nodes
drush --yes pm-enable btr_content_1

Fix path aliases and menus

Path aliases and some menus have to be fixed (recreated) manually. I couldn't find any modules, drush commands or scripts that can transfer them automatically. If you know any tricks to export/import them automatically, please let me know.

I transfer manually the configuration of the Homebox as well. I open admin/structure/homebox on both sites (the main and the clone), export the configuration of 'dashboard' from the main, then copy/paste and import it on the clone.

Transfer drupal private settings

Private settings are those variables that are site specific and cannot be included in features, for example: disqus_domain, disqus_userapikey, disqus_publickey, disqus_secretkey, etc.

We can transfer them like this:

  • Save them on the main site:
    cd /var/www/btr/profiles/btranslator
    modules/features/save-private-vars.sh
    

    It will generate the file restore-private-vars.php.

  • Transfer restore-private-vars.php to the clone site and then apply it like this:
    drush php-script restore-private-vars.php
    

Note: If the clone site will be used for testing, you may consider to edit manually the file restore-private-vars.php, before applying it, and change some values. For example, I usually change email addresses from info@l10n.fs.al to info+test@l10n.fs.al. I also enable email rerouting by changing these variables:

$variables['reroute_email_enable'] = 1;
$variables['reroute_email_enable_message'] = 1;

Sync users and contributions

Now the cloned site is almost identical with the primary site in terms of Drupal settings and configuration and in terms of translation data (projects that are imported, strings and their translations, etc.). What is still missing is the list of users that are registered on the primary site, as well as their contributions (votes and suggested translations).

We can transfer them like this:

  • Export them on the primary site:
    cd /var/www/data
    db/export-users.sh
    db/export-contributions.sh
    
  • Transfer *.sql.gz files to the clone site and import them:
    cd /var/www/data
    db/import-users.sh  users-20130717.sql.gz
    db/import-contributions.sh contributions-00000000-20130717.sql.gz
    

Note: Once you have transferred all the users, the clone site will send them daily a string for review, in addition to the one that is sent by the primary site. We can prevent this by disabling the cron. Edit the file /etc/cron.d/drupal7 and comment the line that starts the cron.

Sync vocabulary data

Vocabulary is a pseudo-project, its PO file does not really belong to the translation of any program. I use it to collect interesting terms and translations that I encounter while translating the other projects. It can help as a reminder (in case that I forget the translation of a term). It is also useful for discussing translations of difficult terms with other people, and indirectly it helps to ensure consistency among the translations. Terms are added to vocabulary by the translators. In order to transfer them from one instance of B-Translator to another, it can be exported as a PO file on one system and imported to the other.

  • Export vocabulary-sq.po:
    cd /var/www/data
    export/export.sh misc vocabulary sq $(pwd)/tmp
    mv tmp/vocabulary/vocabulary-sq.po .
    rm -rf tmp
    
  • Transfer vocabulary-sq.po to the other system and them import it:
    cd /var/www/data
    mv vocabulary-sq.po vocabulary/
    import/vocabulary.sh
    

Get and import PO files

The database of translations is almost empty (it has only the PO files that were imported for testing during installation). Downloading and importing all the PO files is easy (but it takes a long time).

Download (get) PO files

cd /var/www/data

nohup nice get/all.sh &
tail -f nohup.out

or

cd get/
./gnome.sh
./kde.sh
./firefox-os.sh
./drupal.sh
./office.sh
./mozilla.sh
./wordpress.sh
./ubuntu.sh

Note: These scripts get the data from some URL. They should be checked first, to make sure that the URL still works or that we are getting the latest version.

Note: Make sure that hostname is listed on /etc/hosts otherwise the command svn checkout will not work (strange, but that's how it is). For example if the output of the command hostname is dashamir, then /etc/hosts should look like this:

127.0.0.1 l10n.org.xx localhost dashamir

Import PO files

cd /var/www/data
nohup nice import/all.sh &
tail -f nohup.out

Switching to the new server

Suppose that I want to make the cloned server primary. In this case there are some steps that should be done:

  • Transfer GoogleApps verification files. I use GoogleApps for email accounts etc. (it offers 10 email accounts for free). To verify that I own this domain, GoogleApps requests me to put a certain file on the root of my webserver. This file looks like google9350a51ac2d503bf.html and I place it on /var/www/btr.
  • Transfer SSL certificates. I have obtained a free SSL certificate for my site (see: http://arstechnica.com/security/2009/12/how-to-get-set-with-a-secure-sertificate-for-free/). The configuration on /etc/nginx/sites-available/default looks like this:
    ssl_certificate         /etc/ssl/certs/ssl-cert-l10n_fs_al.pem;
    ssl_certificate_key     /etc/ssl/private/ssl-cert-l10n_fs_al.key;
    

    The corresponding configuration on /etc/apache2/sites-available/default-ssl looks like this:

    SSLCertificateFile    /etc/ssl/certs/ssl-cert-l10n_fs_al.pem
    SSLCertificateKeyFile /etc/ssl/private/ssl-cert-l10n_fs_al.key
    

    The files .pem and .key need to be transferred to the new server and the configuration files of nginx and apache2 should be modified properly.

  • Enable cron. Since I have disabled cron (on a test site), I have to enable it again by un-commenting it on /etc/cron.d/drupal7.
  • Replace test settings with live settings. Export drupal setting on the main site with modules/features/save-private-vars.sh and then import them on the new site with drush php-script restore-private-vars.php.
  • On the DNS server I change the record of l10n.fs.al to point to the new IP. But the DNS change may take about 2 days to be propagated worldwide. So, after 2-3 days I do again a transfer of users and contributions from the old server to the new one. These transfer operations are designed to be idempotent, which means that the result will be the same even if they are applied many times.

Author: Dashamir Hoxha <dashohoxha@gmail.com>

Date: 2014-06-23 19:32:35 CEST

HTML generated by org-mode 6.33x in emacs 23