Blog

Migrate/Upgrade Joomla 1.0 to Joomla 2.5

Oct 18 2015

Joomla Migrate CMS

I am always looking for a new challenge and not too long ago one presented its self to me that I couldn't turn down. I was browsing the local news sites and one of them was down. I wasn't given a maintenance page or an error message so I moved on.

Twenty-four hours later the site was still down so I looked a little further and found that I couldn't even ping the server it was hosted on. Keep in mind this is a small town (population 4500) news station. So got in contact with the people who ran it and offered to help them if they needed it.

What I didn't know was in the end I would be migrating their website from Joomla 1.0 to Joomla 2.5 and moving all of their content into the K2 Component. The site is updated 5 days a week and I didn't want to do this on the weekend so I had to script this so the total migration lasted less than an hour and resulted in ZERO down time.

If you find any errors or have anything to add please let me know in the comments below. If you have a migration script for any of the areas that I am missing send it over and I will add it in.

Background

Let me give a little background on the migration.

  • Database: 190MB
  • Document Root: 500G
  • First Post: June 12, 2004
  • 19000 articles
  • 14000 images

Joomla clearly hadn't been updated in years and neither had the extensions. I eventually learned the site had already been hacked and restored once a few years ago but there was little interest in keeping it up to date over the years.

Challenges

The list of challenges with the migration goes on and on. The first and most obvious are the Joomla! system requirements and then there are the less obvious items like image names.

  • Joomla! 1.0 will not run on PHP 5.3
  • Joomla! 1.0 had no image name control in com_content
  • K2 can't handle this large of a migration without timing out.
  • K2 can't import the images from Joomla! 1.0
  • K2 stores the image in 7 different sizes and the names as an md5 hash
  • Database schema has changed.

I am sure this list would have been a little longer if I had to manage a large users table or migrate other extensions but there really wasn't much else that needed to be moved.

Basic Setup

The first time you run this it may take some time especially if you have a large number of images that need to convert.

I probably ran this script 100 times to make minor tweaks and to make sure everything was correct so you will see a few backups and a truncate that you may not have expected. This just means you can run this script as many times as you would like and make as many changes as possible without having to manually remove any data from previous import attempts. Some lines are commented out so they don't run. Read the comments to understand why.

Originally this was a one off script so I entered the username, password, directory path, and database every time they were needed. This clearly would have been a good time to just assign those items to a variable so the script could be more portable. I don't have another migration to do so in order to make this more portable I am willing to work with someone to get that done.

We are going to be using a combination of php and bash scripts to get this done. We are actually going to import the content with the k2 importer but you may need to make a small change in one of the import scripts to avoid crashing your server.

That means you are going to need shell access to the server you are trying to make these changes on. If you don't have shell access you can look at converting the bash scripts into php scripts and try using the php exec() function to run the needed bash commands.

You are also going to need image-magick installed on the server to re-size the images for k2. "yum -y install image-magick"

We are going to have a total of three databases to help ensure our data is safe. The three databases will be called j_live (live site on old server), j_migrate (used to store data on new server), j_new (the new 2.5 install on the new server).

Install and setup the latest version of Joomla! 2.5 and K2 on the j_new database. Be sure to write down the database username and password. We are going to need them a few times later.

Since I only needed to provide access to the site for a couple of users I added the users manually.

Getting Started - Old Server to New

Create two folders one in the J2.5 root called "migrate" we will place all of our scripts and modified files in here and another called "migrate" in /tmp/.

In the J2.5/migrate folder create a file called "migrate.sh". This file will do all of the heavy lifting for us. It will migrate our images, and databases, create our backups, and call the other scripts we need.

 echo Dump j_old from old server;
ssh 
 root@192.168.0.2 "cd /tmp/migrate; mysqldump -u username -p'password' j_old > j_old.sql";

cd /tmp/migrate;
echo Get j_old from old server;
scp 
 root@192.168.0.2:/tmp/migrate/j_old.sql .;

echo Make backup of current j_migrate;
mysqldump -u username -p'password' j_migrate > j_migrate.sql.bak;

echo dumping j_migrate to timestamped file;
backupname=j_migrate`date`.sql; 
mysqldump -u username -p'password' j_migrate > /tmp/"$backupname";

echo Insert j_old into j_migrate
mysql -u username -p'password' j_migrate < j_old.sql;

# Every time you attempt a new import we will dump and re-import 
# the categories from K2 in order to keep all of the changes you have made to them
echo Dump j_new k2_categories to reload later so we keep all of our layout configs;
mysqldump -u username -p'password' j_new jos_k2_categories > k2_categories.sql;

echo Truncate the j_new jos_content table to receive the content from j_migrate;
mysql -u username -p'password' j_new -e "truncate jos_content";

# I used PHP to  modify the data for the migration
# from 1.0 to 2.5 there are settings in here that need to be changes also.
echo move data from j_migrate to j_new; php /home/public_html/migrate/content_update.php; 

echo sync the images from old server to joomla 2.5 install;
rsync -av 192.168.0.2:/home/jasonbrennan/public_html/images/* /home/public_html/images/.

#echo Removing all k2 cached images don't need unless you want to start the image migration over
#rm -rf /home/public_html/migrate/media/k2/items/cache/*

echo Copying images for the k2 system feel free to change the image widths below.
array=(`php /home/public_html/migrate/get_images.php`);

for i in "${array[@]}"
do
        id=`echo $i | tr -d '\r' | awk -F',,,' '{print $2}'`;
        idmd5=`echo -n "Image$id" | openssl md5`;
        imagename=`echo $i | sed s/SSPP/'\\ '/g | awk -F',,,' '{print $1}'`; 
        imageext=`echo $imagename | awk -F'.' '{print $2}'`;
        imagename=`echo $imagename | sed s/PERPER/'\.'/g | sed 's/ /\\ /g'`;
        oldfile=`echo /home/public_html/images/stories/$imagename`;

        newg=`echo /home/public_html/media/k2/items/cache/$idmd5'_General'.$imageext`;
        newxs=`echo /home/public_html/media/k2/items/cache/$idmd5'_XS'.$imageext`;
        news=`echo /home/public_html/media/k2/items/cache/$idmd5'_S'.$imageext`;
        newm=`echo /home/public_html/media/k2/items/cache/$idmd5'_M'.$imageext`;
        newl=`echo /home/public_html/media/k2/items/cache/$idmd5'_L'.$imageext`;
        newxl=`echo /home/public_html/media/k2/items/cache/$idmd5'_XL'.$imageext`;
        newsrc=`echo /home/public_html/media/k2/items/src/$idmd5.$imageext`;

    width=`identify -ping -format %w "$oldfile"`;

        #echo 'Create SRC Image from original';
        echo $id;
    if [ ! -f $newsrc ];
    then
        cp "$oldfile" $newsrc;
    fi

        #echo 'Create General Image from original';
    if [ ! -f $newg ];
    then
        cp "$oldfile" $newg;
      newwidth=300;
    (( $width > $newwidth )) && mogrify -resize $newwidth "$newg" && echo $newg;
  fi

        #echo 'Create XS Image from original';
  if [ ! -f $newxs ];
  then
    cp "$oldfile" $newxs;
    newwidth=100;
    (( $width > $newwidth )) && mogrify -resize $newwidth "$newxs" && echo $newxs;
  fi

        #echo 'Create S Image from original';
  if [ ! -f $news ];
  then
    cp "$oldfile" $news;
    newwidth=200;
    (( $width > $newwidth )) && mogrify -resize $newwidth "$news" && echo $news;
  fi

        #echo 'Create M Image from original';
  if [ ! -f $newm ];
  then
    cp "$oldfile" $newm;
    newwidth=400;
    (( $width > $newwidth )) && mogrify -resize $newwidth "$newm" && echo $newm;
  fi

        #echo 'Create L Image from original';
  if [ ! -f $newl ];
  then
    cp "$oldfile" $newl;
    newwidth=600;
    (( $width > $newwidth )) && mogrify -resize $newwidth "$newl" && echo $newl;
  fi

        #echo 'Create XL Image from original';
  if [ ! -f $newxl ];
  then
    cp "$oldfile" $newxl;
    newwidth=900;
    (( $width > $newwidth )) && mogrify -resize $newwidth "$newxl" && echo $newxl;
  fi

done

echo Trucating j_new k2 categories table;
mysql -u username -p'password' j_new -e "truncate jos_k2_categories";

echo Trucating j_new k2 content table;
mysql -u username -p'password' j_new -e "truncate jos_k2_items";

echo Copy modified items.php to reduce import time.;
cp /home/public_html/migrate/items.php.mod  /home/public_html/administrator/components/com_k2/models/items.php;

echo Import data from j_new jos_content to k2 using k2 import;

read -p 'After you run the k2 import press enter to restart this script'; 

echo restore category table;
mysql -u username -p'password' j_new < k2_categories.sql;

echo restore k2 items.php;
cp /home/public_html/migrate/items.php.ori  /home/public_html/administrator/components/com_k2/models/items.php;

echo All data has been migrated;

The first thing "migrate.sh" does is to move the database from the old server to the new one and begins to convert the jos_content database tables with a script called "content_update.php".

Migrating Content

Create a file called "content_update.php" in the J2.5/migrate directory.

<?php
#The root username and password since we are moving data between databases.
$db = mysql_connect("localhost","root","password");
mysql_select_db('j_migrate', $db);



$query = 'insert into j_new.jos_content
  (id,title,title_alias,introtext,`fulltext`,state,sectionid,mask,catid,created,created_by,created_by_alias,modified,modified_by,checked_out,checked_out_time,publish_up,publish_down,images,urls,attribs,version,parentid,ordering,metakey,metadesc,access,hits) 
select 
    id,title,title_alias,introtext,`fulltext`,state,sectionid,mask,catid,created,created_by,created_by_alias,modified,modified_by,checked_out,checked_out_time,publish_up,publish_down,images,urls,attribs,version,parentid,ordering,metakey,metadesc,access,hits from jos_content;';

mysql_query($query);

mysql_select_db('j_new', $db);
/*
 * All of the categories need to be created in the J2.5 content manager by hand. 
 * I needed about 50 of then. The array key is the old category id and
 * the value is the new category id.
 *
 * The ordering of these is extremely important. You cannot have a value
 * that has a matching key further down the array. This will cause data
 * to get re-categorized multiple times. See below....
 *
 *  $change_list = array(
 *    '93'  => '10',  // Value 10
 *    '108' => '11',  // News
 *    '97'  => '12',  // ML
 *    '10'  => '13',  // Value 10 is changed to 13
 *    '13'  => '39',  // Changed a third time to 39
 *
 * If you compare category counts beetween J1.5 and K2 you can find any 
 * errors you may have made in this.
 *
 */
$change_list = array(
                '93'  => '10',  // Category Name
                '108' => '11',  // Category Name
                '97'  => '12',  // Category Name
                '98'  => '13',  // Category Name
               ); 
foreach ($change_list as $old => $new) {
  $query = 'update jos_content set catid = '.$new.' where catid = '.$old.';'; 
  echo $query;
  mysql_query($query);
  echo mysql_error() . "\n";
}

// Need to change the state -1 to 2 due to archive position change.
$query = 'update jos_content set state = 2 where state = \'-1\';';
mysql_query($query);

// Need to make some small adjustments for changes made in joomla 2.5'
$query = 'update jos_content set sectionid = 0;';
mysql_query($query);
$query = 'update jos_content set asset_id = 95;';
mysql_query($query);
$query = 'update jos_content set access = 1;';
mysql_query($query);
?>  

At this point all of our data has been migrated into J2.5 and our migrate.sh script is moving on to rsync the images folder from the old server to the new one.

Migrating Images

I had imported the images a couple of times and then decided I wanted to change the sizes of them all so on line 35 of the "migrate.sh" I added a line that will remove all of the previously converted images. I have this commented out by default so if you need just remove the hash and run the script.

We need to get a list of images that need to be imported from the j_migrate database. We will do this using a script called "get_images.php".

Create "get_images.php" in your J2.5/migrate folder and include what I have below.

<?php
/* 
 * This is where you are going to make any file name changes
 * you need to get the data into the bash script and running
 * as clean as possible. Of the thousands of images I had to 
 * convert there was only one that I had to move manually
 * after all of the scripts ran.
 *
 */

// You can use the username and password for the j_migrate db
$db = mysql_connect("localhost","root","password");
mysql_select_db('j_migrate', $db);

// Select all of the image names from the articles in the j_migrate database
$query = 'select id, images from jos_content where id != \'\' order by id desc;';
$results = mysql_query($query);

While($assoc = mysql_fetch_assoc($results)){
  $image = trim($assoc['images']);
  if(!empty($image)) {
    $image = explode('|',$image);
    $image = $image['0'];
    $image = trim($image);  
    // there were images with spaces so I used "SSPP" to mark the spaces.
    $image = trim(preg_replace('/\s/', 'SSPP', $image));
    if(substr_count($image, '.pngSSPP') || substr_count($image, '.jpgSSPP') || substr_count($image, '\.PNGSSPP ') || substr_count($image, '\.JPGSSPP ')){
      $image = explode('SSPP',$image);
      $image = $image['0'];
    }

    $count = substr_count($image, '.');
    if($count > 1) {
      $new = $count - 1;  
      $image = preg_replace('/\./', 'PERPER', $image, $new);
    }
    // return the result string to the bash script to import to k2
    echo trim($image).',,,'.$assoc['id']." ";
  }
}
?>

Next we will start to move the images from the J2.5 images folder into the K2 cache and convert the file names into an md5 sum as required by K2. If you look at this part of the script you can set the image width by changing the "newwidth" variable for each of the respective sizes.

Finishing it up For the last couple of steps we will be migrating our data from the J_new com_content into k2. The "migrate.sh" will truncate the k2_categories table and the k2_items table to make sure they are clean and ready to import new data from a new migration attempt.

The script will prompt you to log into J2.5 and run the K2 import tool. I ran into a small problem here though. I found that this was running for hours and eventually failing in a number of ways. Either apache would timeout or the server would run out of memory and swap space.

The k2 import tool also generates tags for each post and it is extremely resource and time intensive. Since J1.0 didn't have this this feature I figured it was best to just turn that part of the import off.

I have two files in the J2.5/migrate folder one is called "items.php.ori" and the other is called "items.php.mod". Line 125 places the modified version of this file into K2 and line 135 places the original back in it's place after the migration is complete.

Make two copies of the "administrator/components/com_k2/models/items.php" and name them like I did above. By commenting out line 1095 "if($preserveItemIDs){" and it's entire else statement you will disable the tag generation.

Be sure to keep these two lines or you won't actually save any data.

$K2Item->id = $item->id;
$db->insertObject('#__k2_items', $K2Item);

The last thing to happen is for us to reinsert the k2_categories table that we exported earlier. Remember this is because we an preserve our preferences for the layouts of the category pages.




Menu

Contact Us

Contact form submitted!
We will be in touch soon.