Home / Blog

Getting FileConveyor working with Cloud Files

Ever since I started working with Drupal and Cloud Files (Rackspace's CDN), I've had the experts telling me that I should use FileConveyor to transfer all of my files from the local disk up to Cloud Files. And I really wanted to.. but EVERY time I've tried to the supposedly super simple fileconveyor, it is broken. Not a single piece of functionality in that damn thing seems to work out of the box. This matter is complicated by the fact that Rackspace just upgraded the Cloud Files apis to a new system, so the only libs that are even possibly working are using the old deprecated Cloud Files apis. *sigh* Well, I finally got one single test to work with FileConveyor and Cloud Files, so I'm going to document what I had to do to get it working because I'm sure I'll need this later on.

Install fileconveyor and other things

The following set of commands all use different methods of installing software, because, well... that's the only way I was able to get it to work. # pip install -e git+https://github.com/wimleers/fileconveyor@master#egg=fileconveyor # yum install python-paramiko # easy_install "django-cumulus==1.0.10" The first one installs fileconveyor into the current directory (for me, /root) python-paramiko is the SSH client (not sure if it's needed for Cloud Files, but I was trying to test with SFTP at one point and it was needed for that) The django-cumulus is necessary to get a working version of the Cloud Files storage API.

Enabling ServiceNet

Once I created my config.xml (below), I was able to get this working for DFW Cloud Files containers, but none of my ORD containers would work. I'm running this from a Rackspace Cloud Server so that means I can use the internal private network for my API calls and I thought (rightly) that it might fix my problem. So, I edited:  

/usr/lib/python2.6/site-packages/django_cumulus-1.0.10-py2.6.egg/cumulus/settings.py and set all references to SERVICENET to be True instead of False.

Configuration

I have no idea why they designed the system this way, and hopefully I'm just missing something because this is stupid, but it order for the system to work you have to store your config.xml file in the fileconveyor directories and it MUST be named config.xml. You can't even just pass the name of your config file on the command line.. so I saved my file as: /root/src/fileconveyor/fileconveyor/config.xml

<?xml version="1.0" encoding="UTF-8"?>
<config>
	<sources ignoredDirs="CVS:.svn">
	    <source name="drupal" scanPath="/root/fctest" />
	</sources>
	
	<servers>
		<server name="Rackspace Cloud Files" transporter="cloudfiles">
			<username>myusername</username>
			<api_key>xxxxxxxxxxxxxxxx</api_key>
			<container>ContainerName</container>
		</server>
	</servers>
	
	
	<rules>
	    <rule for="drupal" label="Generated Files">
	    	<destinations>
	    		<destination server="Rackspace Cloud Files" path="test" /> 
	    	</destinations>
	    </rule>
	</rules>
	
</config>

 

I created a directory in /root/fctest which would be the root of my synchronized directories. (That's the source entry) I created a server entry with my Cloud Files info. The rule entry tells fileconveyor what to do. It says for "drupal" (which is the name of my source entry), it should convey all files to the destination server (which is named "Rackspace Cloud Files") and everything in /root/fctest should be transferred to the ContainerName/test directory.

Starting fileconveyor

 

# python /root/src/fileconveyor/fileconveyor/arbitrator.py

Good luck. You'll need it.