sfMogileFSPlugin
sfMogileFS enables symfony applications to interact with [MogileFS]. MogileFS is an anagram for "OMG Files" and was created by [http://www.livejournal.com LiveJournal] to handle the storage, replication and retrieval of the large amount of file uploads they were, and continue to, experience. Many of the web's most popular sites use MogileFS as their file store. This includes [http://www.digg.com Digg], [http://www.last.fm Last.fm], http://www.guba.com Guba, and others.
Prerequisites
The following must be installed and setup before using this plugin.
How It Works
Perlbal is a load balancer that runs on port 80 with a pool of apache servers running behind it. In addition to dispatching requests to the internal servers, Perlbal also sends the critical X-REPROXY-URL header to MogileFS. MogileFS sends the file back based on the key given.
MogileFS Terminology
There are four main components to a MogileFS setup:
- tracker(s): talk to the MogileFS database. By default run on port 6001.
- database (a MySQL Cluster is recommended but simply one db server will work): maintains a list of storage nodes, files, etc.
- storage node(s). Physical disks (hard drives) where files are stored. Mogstored defaults to port 7500.
- client library (aka sfMogileFSPlugin) - talks to trackers (mogilefsd).
Storing files:
- domain: Top level separation of files. File keys are unique within domains. The Socialhouse.com domain (socialhouse) is defined within the socialhouse application in app.yml. A domain consists of a set of classes that define the files within the domain. class
- key: A unique textual string that identifies a file. Keys are unique within domains. Examples of keys: userpicture:34:39, phonepost:93:3834, userbackup:15.
- class: Part of exactly one domain. A class, in effect, only specifies the minimum replica count of a file. Examples of Socialhouse classes: image, thumbnail, video.
- minimum replica count: Property of a class. This defines how many times the files in that class need to be replicated onto different devices in order to ensure redundancy among the data and prevent loss.
- file: Every file is part of exactly one class. A file is a defined collection of bits uploaded to MogileFS to store. Files are replicated according to their minimum replica count. Each file has a key, is a part of one class, and is located in one domain.
Plugin Installation
$ php symfony plugin-install http://plugins.symfony-project.com/sfMogileFSPlugin
In your app.yml file add:
dev:
sfMogileFSPlugin:
domain: mogilefs_domain
timeout: 5
trackers: [222.222.222.2:6001](111.111.111.1:6001,)
You will also need to define settings for your production environment.
Modify your routing.yml (feel free to change "i", module, and action to whatever you'd like).
@mfs
url: /i/:mkey
param: { module: file, action: showMogileFSFile }
This plugin was designed to primarily add and retrieve files from MogileFS. You will need to create your own media handling model(s), module(s) and action(s), though an example setup is given below.
Installation complete. Clear your cache.
Plugin Files
The plugin contains 4 classes and one helper. They are phpdocumented and unit tests are included in the tests/ folder.
- sfMogileFSBaseFile.class.php - contains methods shared by both sfMogileFSFile and sfMogileFSRemote file.
- sfMogileFSFile.class.php - represents a file before it's saved to MogileFS.
- sfMogileFSRemoteFile.class.php - represents a file retrieved from MogileFS.
- sfMogileFS.class.php - contains static methods for interacting with the MogileFS trackers.
- MogileFSAssetsHelper.php - wrapper for existing symfony helper's suchc as url_for and image_tag.
To output an image file from your template:
To add a file to MogileFS:
$mfile = new sfMogileFSFile();
$mfile->setKey($mfs_key);
$mfile->setClass($mfs_class);
$mfile->setFile($local_file);
$mfile->save(); // saves to MogileFS
Alternatively:
$mfile = new sfMogileFSFile($key, $class, $file);
$mfile->save();
To retrieve a file directly from MogileFS as a string:
$mfile = sfMogileFS::loadFile($mfs_key);
To get a valid tracker path:
$mfile = sfMogileFS::loadFile($mfs_key);
$mfile->getPath();
Example Setup
This is a complete example. However, You should only use this as a guide and should make changes based on your own application and needs.
The example assumes you already have a file in MogileFS with the key "cab9f3c712d04de874dafb0af0a0bf03e303e6e0".
schema.yml:
files:
_attributes: { phpName: File }
id:
user_id: { type: integer, foreignTable: users, foreignReference: id, onDelete: cascade }
file_type_id: { type: integer, foreignTable: file_types, foreignReference: id, onDelete: cascade }
mime_type: varchar(128)
extension: varchar(8)
title: varchar(128)
description: varchar(255)
created_at:
updated_at:
photos:
_attributes: { phpName: Photo }
id:
file_id: { type: integer, foreignTable: files, foreignReference: id, onDelete: cascade }
photo_type_id: { type: integer, foreignTable: photo_types, foreignReference: id, onDelete: cascade }
photo_status_type_id: { type: integer, foreignTable: photo_status_types, foreignReference: id, onDelete: cascade }
mogilefs_key: { type: varchar(40), phpName: MogileFSKey, required: true }
width: integer
height: integer
filesize: integer
created_at:
routing.yml:
mfs:
url: /i/:mkey
param: { module: file, action: showMogileFSFile }
/apps//modules/file/actions/uploadAction.class.php:
class uploadAction extends sfAction
{
public function execute()
{
if ($this->getRequest()->getMethod() == sfRequest::POST && $this->getRequest()->hasFiles())
{
if ($this->getRequest()->hasFileErrors())
{
$this->setFlash('notice', 'An error occurred while uploading ('.$this->getRequest->getFileError('file').')');
$this->redirect('@upload');
}
// save() method adds file to File and Photo tables as well as MogileFS
$file = new File();
$file->setUserId($this->getUser()->getId());
$file->setFileTypeId(1); // image
$file->setMimeType($this->getRequest()->getFileType('file'));
$file->setExtension($this->getRequest()->getFileExtension('file'));
$file->setTitle($this->getRequestParameter('title'));
$file->setDescription($this->getRequestParameter('description'));
$file->setTmpFile($this->getRequest()->getFilePath('file')); // not actually part of model but used by it
$file->setFilesize($this->getRequest()->getFileSize('file')); // not actually part of the model but used by it
$file->save();
// delete upload
unlink($file->getTmpFile());
// redirect to upload form
$this->setFlash('notice', 'Your file has been uploaded');
$this->redirect('@upload');
}
else
{
// display upload form
return sfView::SUCCESS;
}
}
//lib/model/Photo.php:
class Photo extends BasePhoto
{
public function save($con = null)
{
if ($this->isNew())
{
$this->setMogileFSKey(hash('sha1', $this->getFileId().':'.$this->getId()));
}
parent::save($con);
}
}
//lib/model/File.php:
class File extends BaseFile
{
protected $filesize;
protected $tmp_file;
public function setFilesize($fileSize)
{
$this->filesize = $fileSize;
}
public function getFilesize()
{
return $this->filesize;
}
public function setTmpFile($filePath)
{
$this->tmp_file = $filePath;
}
public function getTmpFile()
{
return $this->tmp_file;
}
/**
* save
* adds original file to file table, sub-file (photo, etc) table and MogileFS
* All steps are wrapped in a transaction.
*
* @param mixed $con
* @access public
* @return void
*/
public function save($con = null)
{
if ($this->isNew())
{
list($width, $height) = getimagesize($this->getTmpFile());
// store it
$con = Propel::getConnection();
try
{
$con->begin();
// save to File table
parent::save($con);
// add to photo table
if ($this->getFileTypeId() == 1)
{
$photo = new Photo();
$photo->setFileId($this->getId()); // parent file
$photo->setPhotoTypeId(1); // original photo
$photo->setPhotoStatusTypeId(1); // uploaded but not converted
$photo->setWidth($width);
$photo->setHeight($height);
$photo->setFilesize($this->getFilesize());
$photo->save();
}
// save to MogileFS
$sfMogileFSFile = new sfMogileFSFile();
$sfMogileFSFile->setKey($photo->getMogileFSKey());
$sfMogileFSFile->setClass('orig_photo');
$sfMogileFSFile->setFile($this->getTmpFile());
$sfMogileFSFile->setFilesize($photo->getFilesize());
$sfMogileFSFile->save();
$con->commit();
}
catch (Exception $e)
{
$con->rollback();
throw $e;
}
}
}
}
/apps//modules/file/actions/showMogileFSFileAction.class.php:
class showMogileFSFileAction extends sfAction
{
public function execute()
{
// remove file's extension
$mogileFSKey = preg_replace('@\.[\w\d]+$@', '', $this->getRequestParameter('mkey'));
// get mogilefs key from photo table
$c = new Criteria();
$c->add(PhotoPeer::MOGILEFS_KEY, $mogileFSKey);
$photo = PhotoPeer::doSelectOne($c);
// load path from mogilefs
$sfMogileFSRemoteFile = sfMogileFS::loadFile($mogileFSKey);
// call perlbal's x-reproxy-url and set mime type
$this->getResponse()->setHttpHeader('Content-type', $photo->getFile()->getMimeType(), true);
$this->getResponse()->setHttpHeader('X-REPROXY-URL', $sfMogileFSRemoteFile->getPath(), true);
return sfView::HEADER_ONLY;
}
}
layout.php (or any template):
This results in the following HTML rendering:
<img src="/index.php/i/cab9f3c712d04de874dafb0af0a0bf03e303e6e0.jpg" />
Todo
- More detailed How It Works explanation
- Example of calling MogileFS using a separate php script and mod_rewrite (instead of having to launch symfony for every image call)
- Backend module with MogileFS stats, etc.
Changelog
trunk
2007-07-19 | 0.6.0 Alpha
- bmeynell: Initial release
Benjamin Meynell 