ThetaJoin Blog

Non-Drupal URIs and Lighttpd drupal.lua Script

mark — Sun, 27 Jun 2010 22:56:54 +0000

I have a few other URLs that don't belong to Drupal. If these URLs are received on port 80, they get redirected to port 443 using mod_rewrite. The problem is drupal.lua script will cause non-Drupal URLs to simply get into a redirection loop. So, for example:

http://blog.example.org/nondrupalurl will send constant 301s back to the browser until it times out. What I had to do at the top of drupal.lua was this:

if lighty.env['uri.path']:match('nondrupalurl') then
--  print('URI Path Match ' .. lighty.env['uri.path'])
  return
end

So far, this solution works but it's not very elegant, especially if there are a bunch of non-Drupal URLs. My site only has two, so it's not that big of a deal to hard code this into drupal.lua.

The other thing I'd like to know is how do you do your Lua dev?

Review: MySQL Admin Cookbook

mark — Tue, 11 May 2010 04:40:17 +0000

MySQL Admin Cookbook from Packt Publishing, authored by Daniel Schneller & Udo Schwedt, is a new addition to the MySQL literary genre with 99 great recipes for mastering MySQL configuration and administration. Announcing quick answers to common problems, these 99 recipes cover a broad spectrum of managing MySQL, and are appropriate for new MySQL administrators as well as more experienced administrators. MySQL Admin Cookbook is available in dead tree and PDF formats. The book's website is: https://www.packtpub.com/mysql-admin-cookbook/book with errata found at http://www/packtpub.com/support.

Subjects Covered

Replication
Indexing Tools
Backing Up & Restoring MySQL Data
Managing Data
Monitoring & Analyzing MySQL Installation
Configuring MySQL
MySQL User Management
Managing Schemas
Good to Know

All in that order.

Recipe Format

All recipes follow a common format:

Introduction of what this recipe will accomplish and why you'd want to use it.
Getting Ready – This section outlines the requirements for the recipe, user privileges, programming language familiarity, additional software required, and where to find additional information.
How to do it – Step-by-step instructions and expected output from these steps. All the reader has to do is follow the steps.
How it works – The details on how the recipe works. If you're unsure what happens during the How to do it section, be sure to spend all the time needed to understand how the recipe works. Don't implement any recipe unless you fully understand what it's doing and how it works.
There's more – Following on from the How it works section, this part will give the reader additional information to expand the functionality of the recipe, or provide other tools that provide the same, or nearly the same functionality. If there are additional concerns to look out for, it'll be included here as well.

MySQL Admin Cookbook Review

MySQL Admin Cookbook really is not designed to be read cover-to-cover as I did for this review, but it left me with a clunkiness about chapter order. There were a few of the recipes I thought were too basic – creating a basic user, creating indexes and tables. Those DBA functions should be automatic before diving into a book with this level of technical content. After those little bits of personal opinion, overall, I really liked the book. I put a few of the recipes into action with the servers I'm responsible for – Using a custom prompt, I would rely on the tab within terminal to see which server I was on, but that tab is missing when I SSH into a server from my phone. I quickly used Display query results page by page and Default pager as well since it makes it much easier to use the MySQL command-line client on smaller screens. Sometimes, it's the small things that make a large impact.

I work mostly from the command-line, but I like the way MySQL Admin Cookbook used both command-line tools and MySQL GUI tools to demonstrate the recipes. The authors acknowledge that the MySQL GUI tools are end of life, but they are still useful tools, so don't discount them. One of the things I thought was left out was using phpMyAdmin as one of the “GUI” tools. If you've never heard of phpMyAdmin, it is a very popular web-based MySQL interface used by many web hosting companies. One of the reviewers is heavily involved with phpMyAdmin development as well, so I would think at least one recipe in Chapter 3: Tools should have used phpMyAdmin.

One of the most popular command-line tools outside of MySQL command-line is Maatkit, which is mentioned several times throughout the book. On this subject, Maatkit could really use its own book ( hint, hint Baron! ), but I digress – hopefully in a positive direction.

At the time of this review, the was no errata on the Pakt website, and I couldn't add anything to the errata anyway. I've not tried all the recipes for accuracy, but I couldn't find any tpyos or mispelings.

If you'd like to see another DBA's perspective on MySQL server management or gain additional MySQL technical expertise, pick up a copy of MySQL Admin Cookbook.

If you'd like your MySQL, Drupal, or other F/OSS book reviewed on this site, please contact reviews@thetajoin.com.

Configuring JW FLV Player 5.x in Drupal to Stream Videos Using Amazon S3 & Cloudfront (Part III)

mark — Mon, 15 Mar 2010 22:33:16 +0000

Part III will show how to configure Drupal content with an embedded JW FLV Player 5.x media player. Get started by downloading the player and unzipping. We'll use Amazon Cloudfront to distribute the player instead of using our own webserver. After all, that's what a Content Delivery Network is for!

The media player is comprised of a few javascript files, so they need to be stored in an S3 bucket that's configured for downloading instead of streaming:

ThetaJoin uses an Amazon Cloudfront distribution based on an Amazon S3 bucket named: cf.thetajoin.com. Using the DNS masking trick, DNS was configured as a CNAME so cf.thetajoin.com pointed to the Amazon Cloudfront of http://d3hezglo7u107t.cloudfront.net. We now have one Amazon Cloudfront distribution based on an Amazon S3 bucket that contains the video files to stream and have a download distribution based on another Amazon S3 bucket that contains the media player. The delivery method for an Amazon Cloudfront distribution can only be one of streaming or download, it can't be both at the same time. That's why we have two Amazon Cloudfront distributions.

Upload the folder that contains the JW FLV Player, by default that's mediaplayer-viral, to your Amazon S3 bucket that's the source for your download Amazon Cloudfront distribution. When that's finished, change the ACL to that folder so everyone has read only access (make sure the apply to subfolders is checked):

The media player is now ready for distribution using Amazon Cloudfront. The next step is to embed the HTML required to display the player, configure it, and finally play the video from the Amazon Cloudfront distribution that's configured for streaming. There are two ways to add this content to a node, the first way is via HTML object, the second way is using a WYSIWYG editor that's able to embed the video for you.

Manual Way

The manual way to include your media player within Drupal content is to add the <object></object> tags directly as HTML, especially if you're comfortable with editing HTML. This approach doesn't require any Drupal modules to be installed and actually should work on just about any other content management system as well. The full <object></object> tags for the iPod repair video:

<object id="iPod" style="width: 640px; height: 480px; display: block; margin-left: auto; margin-right: auto;" width="640" height="480" data="http://cf.thetajoin.com/mediaplayer-viral/player-viral.swf" type="application/x-shockwave-flash">
<param name="data" value="http://cf.thetajoin.com/mediaplayer-viral/player-viral.swf" />
<param name="quality" value="autohigh" />
<param name="wmode" value="transparent" />
<param name="flashvars" value="file=iPodVideo5GRepair.flv&amp;streamer=rtmp://streams.thetajoin.com/cfx/st" />
<param name="name" value="iPod" />
<param name="src" value="http://cf.thetajoin.com/mediaplayer-viral/player-viral.swf" />
<param name="bgcolor" value="undefined" /></object>

The values where you see the cf.thetajoin.com domain need to be replaced with your own Amazon Cloud Front distribution URL that's configured for downloading. Width/height values can be set to the size of player you'd like displayed. Look for the flashvars param, replace file=iPodVideo5GRepair.flv, with the name of your FLV file, and adjust the streamer= to point to your Amazon Cloud Front distribution that's configured for streaming. Don't forget to add the /cfx/st without the trailing slash. The second to last param "src" also needs to be changed to point to your Amazon Cloud Front distribution that's configured for downloading.

WYSIWYG Editor Method

Most popular WYSWYG editors have an toolbar button to assist with the embedding of streaming content. This example will use TinyMCE to embed the JW FLV Flash player into Drupal content so you don't have to edit HTML.

Clicking on that toolbar button presents the following screen:

Type dropdown is set to Flash, File/URL textbox contains the Amazon Cloudfront URL to the JW FLV Player. Dimensions of your player can be set as well. Click on the Advanced tab next:

Give an Id & Name to your embedded object. The key textbox in the Advanced tab is the Flashvars textbox. Take a look at the flashvars param in the manual way, copy and paste after value= into this textbox. Be sure to click the Update button to save your configuration. Once this is done, your video will be displayed within your newly embedded video player like below.

The quality of the video isn't the best after the conversion from OGV to FLV format, so it's not any problem with Amazon Cloudfront streaming this video, it's the videographer! This is a 44 minute video demonstrating how to replace the screen and headphone jack of an iPod Video 5th Generation. If you try to repair your iPod using this video, you do so at your own risk!

I'm working on Part IV, stay tuned!!

-- Mark

Preparing Amazon S3 & CloudFront for Streaming Flash Videos from Your Drupal Site (Part II)

mark — Tue, 09 Mar 2010 14:00:00 +0000

This is Part II of an unknown number of posts about using Amazon S3 & CloudFront to host your Drupal site's streaming media. Part I covered a very basic approach using an Ogg Vorbis Video formatted file delivered over HTTP. Although this will work over HTTP with limited support from embedded media players, this isn't the internet standard for delivering streaming videos. One standard protocol to use is the real time messaging protocol or rtmp, developed by Adobe Systems. From the Amazon CloudFront manual:

"CloudFront uses Adobe Flash Media Server 3.5 to stream on-demand content with Adobe's
Real-Time Messaging Protocol (RTMP). CloudFront accepts RTMP requests over port 1935 and
port 80."

This means flash videos can be streamed from Amazon CloudFront. This post will show how to convert that Ogg Vorbis Video file to flash video format on Ubunta 9.10 and then configure Amazon CloudFront to support streaming the flash video.

1. Converting Ogg Vorbis to Flash Video

In order to convert the ogv file to flv format, ffmpeg needs to be installed. On Ubuntu 9.10:

sudo apt-get install ffmpeg

2. Convert the ogv file to flv format:

ffmpeg -i iPodVideo5GRepair.ogv -s 800x600 iPodVideo5GRepair.flv

The -s ffmpeg option resizes the video to 800x600. The original video file was recorded at 1600x1000. This reduced the file size from 440MB to 124MB. Still 44 minutes in length though.

3. Create Amazon S3 Bucket

Amazon CloudFront needs an Amazon S3 bucket to store the flv file. CloudFront uses Amazon S3 as the source for the flv file to stream. Amazon CloudFront only supports the distribution of content, not the storage. Using your favorite Amazon S3 client, create a bucket that's named like your internet domain name. In the case of ThetaJoin, a bucket was created named streams.thetajoin.com.

4. Configure Bucket for Streaming Content

CloudFront supports content distribution using two different methods - download and streaming. The download method is used for static content such as PDFs, JPG, or PNG files. The other method is the streaming method that's used to stream either audio or video files. A CloudFront distribution can only support one of these methods at a time, hence the creation of the streams.thetajoin.com bucket to support the CloudFront distribution that will stream the flash video file. Log in to the Amazon Web Services (AWS) Manager to access CloudFront: https://console.aws.amazon.com/cloudfront/home . Once there, the streams.thetajoin.com bucket can be configured to support CloudFront streaming. Click on the Create Distribution button:

Choose Streaming for the Delivery Method, select the Amazon S3 bucket, give it a CNAME the same as the Amazon S3 bucket name, and make sure it's enabled. Creating the CloudFront streaming distribution can take some time. AWS Manager will show that the distribution was created:

The streams.thetajoin.com bucket has the following attributes:

Delivery Method: Streaming
Domain Name: s9vf9f1gnogon.cloudfront.net
Origin Bucket: streams.thetajoin.com.s3.amazonaws.com
State: Deployed
Status: Enabled

The streams.thetajoin.com bucket is ready to support streaming of any streamable content stored there.

5. Configure DNS

The URL to the CloudFront distribution is s9vf9f1gnogon.cloudfront.net, which is on the ugly side and doesn't maintain your site domain name. It would be great for your site branding to use a portion of the origin bucket URL, streams.thetajoin.com only. To do this, you'll need to configure your DNS server with a CNAME value that's the name of your Amazon S3 bucket, that points to the Domain Name of your CloudFront distribution. In ThetaJoin's case streams.thetajoin.com points to s9vf9f1gnogon.cloudfront.net. Now, when a URL with the domain streams.thetajoin.com is requested from the media player, the DNS server will "redirect" the URL to s9vf9f1gnogon.cloudfront.net. From the end user, it'll look like your streaming content is being delivered from your own private content delivery network.

Amazon S3 and CloudFront are configured to support streaming the flash video that was converted in step 1. Part III of this series will show how to add streaming content to your site.

Drupal Integration with Amazon Cloud Front Streaming Video Demo

mark — Sun, 07 Mar 2010 05:45:25 +0000

During SandCAMP 2010, I held a session on Drupal Integration with Amazon S3 & Cloudfront, you can download the session slides too. As I was preparing for this session, Amazon had announced that Cloud Front was able to support streaming video and audio, but I didn't have enough time to put a demo together until now.

I created a Cloud Front demo video repairing my son's 5th Generation iPod Video, replacing the screen and headphone jack. This video was created using a Logitech® Webcam Pro 9000 into a Dell Inspiron 1525 laptop running Ubuntu 9.10 Karmic Koala. Cheese recording software was used to record this 44 minute, 441MB video. The video is 1600x1000 OGG video at 5 frames per second using the Theora codec. The audio is in mono with the Vorbis codec sampled at 44100hz with a bitrate of 80kbps.

The demo video was tested under Firefox 3.5.8 on the same laptop that was used for the recording, and it worked just fine. Same with Opera 10. IE wanted to download the video, so there maybe some other codecs that need to be installed in order to view the OGG video format streamed. Should work fine under Mac OSX too, but I have no way to test. Doesn't work on iTouch using Safari though.

What makes integrating Drupal with Cloud Front is that you can have your own private, unlimited video and audio content delivery network. This allows ultimate flexibility, no limits on the size or length of your video, choose your own video format, zero alteration of your video like what happens with the major video websites. With pay as you go, Cloud Front can really make your Drupal site shine.

The video was created simply as a demo of Cloud Front. If you decide to try to repair your own iPod - you do so entirely at your own risk! Please don't email me if you blow things up!!

I'm very interested in Cloud Front performance, please leave a comment on your experience with Cloud Front streaming.

-- Mark

Fun With Information Schema - How Big is My Drupal Database??

mark — Sat, 13 Feb 2010 22:45:07 +0000

Lately, I've had a few people ask me "How much data is in our database?". Sure, you can look at the file sizes of MyISAM tables and indexes and get a ballpark figure, but what if you need exact results, or are running InnoDB storage engine? That proves to be more challenging! In playing around with the information_schema, I've put together some queries to help:

Calculate Index Sizes

mysql> SELECT CONCAT(ROUND(SUM(index_length)/(1024*1024*1024), 2), ' GB') AS 'Total Index Size'
FROM information_schema.TABLES 
WHERE table_schema LIKE 'database';

+------------------+
|Total Index Size  |
+------------------+
|1.70 GB           |
+------------------+

1 row in set (1.60 sec)

Calculate the Total Size of Stored Data

mysql> SELECT CONCAT(ROUND(SUM(data_length)/(1024*1024*1024), 2), ' GB') AS 'Total Data Size' 
FROM information_schema.TABLES 
WHERE table_schema LIKE 'database';

+-----------------+
| Total Data Size |
+-----------------+
|3.01 GB |
+-----------------+

1 row in set (1.35 sec)

Per Table Sizes

SELECT CONCAT(table_schema,'.',table_name) AS 'Table Name', CONCAT(ROUND(table_rows/1000000,2),'M') AS 'Number of Rows', 
CONCAT(ROUND(data_length/(1024*1024*1024),2),'G') AS 'Data Size', 
CONCAT(ROUND(index_length/(1024*1024*1024),2),'G') AS 'Index Size' ,
CONCAT(ROUND((data_length+index_length)/(1024*1024*1024),2),'G') AS'Total'
FROM information_schema.TABLES 
WHERE table_schema LIKE 'database';

Just replace database with the partial name of your database you need to analyze. Yes, I know, those wonderful Maatkit tools contains mk-find which can do the same thing, but then you won't learn about the information_schema database!! It's important to also know that running the above commands on a server that's very busy can result in slowing it down even more. The information schema tables are 'virtual tables' they don't exist like regular database tables. Just be aware, and if you need to collect this information on a busy database, make a backup copy of your Drupal database and move it to a development server.

Drush for Multisite Cron

mark — Wed, 03 Feb 2010 05:59:59 +0000

When running multiple Drupal sites on the same server, it can be challenging to run cron.php for each installed site. Drush can be used to solve this problem nicely.

First thing is to download drush:

http://drupal.org/project/drush

untar the tarball

tar xvzf drush-All-Versions-2.1.tar.gz

As root, copy all of the files and subdirectories in the drush directory:

cp -rp * /usr/local/bin

By placing all the drush related files and directories into /usr/local/bin, this will allow cron to execute drush. Test drush out from the commandline first like this:

drush --root=/path/to/drupal --uri=http://www.thetajoin.com cron

It should return OK ofter running. Verify in your Drupal logs that cron ran sucessfully.

After testing, start edit cron to run the above command:

crontab -e

*/17 * * * * /usr/local/bin/drush --root=/path/to/drupal --uri=http://blog.thetajoin.com cron
*/19 * * * * /usr/local/bin/drush --root=/path/to/drupal --uri=http://www.thetajoin.com cron

To make sure each site on the server wasn't updated at the exact same time, pick a time that's a prime number. I would like my Drupal sites updated roughly every 15 to 20 mins, so 19 minutes and 17 minutes will work just fine.

There you have it! Multisite cron using drush!

-- Mark