Russell Brooks

How To Get Any Video Into iTunes and Apple TV Fast With No Quality Loss While Maintaining the 5.1 Soundtrack

2012-11-10T20:09:00+00:00

TL; DR

Apple TV - Set "Dolby Digital" to On rather than Auto.
MP4 Tools to convert.
- Set "Device" to "Apple TV".
- Choose "Pass Thru" for Video.
- Choose "Pass Thru" for Audio.
  - If "Pass Thru" is grayed, the audio isn't compatible with ATV. Select "AAC 5.1".
- Choose "Mux (soft)" for Subtitles.
Subler to add complete metadata and cover art into the video file.

That will remux the video in mere minutes, which causes zero quality loss. It causes no lengthy, 2-hour-long, quality-reducing transcode (like HandBrake does).

Now that you have weeks of work summarized into three steps, what follows is the research leading up to the above steps, as well as some tips on other tools.

Myths

Myths I am busting with this post:

"I can't get some videos into iTunes & Apple TV."
"Conversion takes forever."
"Quality is lost with converting."
"Apple TV doesn't do 5.1 soundtracks."
"To play 5.1, Apple TV requires 2 audio streams per container: AC-3 5.1 and a 2-channel AAC."

Glossary

Some vocabulary first:

Container - Sometimes "Format" or "Container Format". The file itself. Examples of container formats: AVI, MPG, MP4, MKV, WebM
Streams - A container file can have essentially 3 types of streams inside it: Video, Audio, and Subtitles.
Channel - Audio streams can be further broken up into channels. Stereo audio uses 2 channels. 5.1 audio uses 6 channels.
Codec - Sometimes "Compression Format". The method by which the video or audio streams are compressed.
- Video Codecs: mp2v, h264, vp8
- Audio Codecs: DTS, AAC, AC3

It's very important to understand the difference between the two types of conversion methods, Transcoding and Remuxing:

Transcoding - Sometimes called "re-encoding". Converting a stream to a different codec, bitrate (compression), or resolution. This means taking a fresh sample of every single frame. Transcoding is time-consuming and, with lossy codecs, causes generation-loss.
Demuxing - Breaking a container into its component streams. Example: A video called somevideo.mp4 might contain an h264 video and an AAC audio stream. Demuxing it would create two files, one containing only video, one only audio. Both of these files would be exact copies of the stream in the container (no generation loss):
- somevideo.h264 - Only video
- somevideo.aac - Only audio
Muxing - Combining individual stream files into a single container file. In the example above, somevideo.h264 and somevideo.aac would be combined together into a file called somevideo.mp4. That file would contain full video and audio tracks. Again, because there is no transcoding happening, there is no loss in quality when doing this.
Remuxing - Sometimes referred to as "converting container format". This means passing an exact copy of the streams through to a new container. Because nothing was transcoded, there is no loss in quality, and remuxing a 2-hour movie takes minutes not hours.

So remuxing is always preferred to transcoding. It's an order of magnitude faster, you leave the streams untouched, and avoid the generation-loss associated with transcoding.

What Apple TV Supports

There are some others, but the core ones we care about are:

Containers - MP4
Video Codecs - h264
Audio Codecs - AAC, AC-3 (via passthrough)

Container

Most downloaded videos come in an MP4 container or perhaps AVI, MOV, or MKV (Metroska). MP4's can sometimes have an M4V file extension. If the file is MP4, M4V, or MOV, it can be dropped directly into iTunes, provided its audio stream is AAC or AC3. More on that very important point in a sec.

Video Stream

Nearly every video you download from the Internet, not matter what the container, has an h264 video stream in it. h264 is fully compatible with Apple TV. 99% of the time, you can simply remux to MP4 very quickly without incurring a massively long transcode of the video stream. If it came as an MP4, you can leave it MP4, again provided the audio stream is compatible.

Audio Stream

Audio is the area from which the majority of problems arise. People often report problems importing h264 video into iTunes for Apple TV playback and, in most cases, it is because of an incompatible audio stream or an incorrect Dolby Digital setting in Apple TV. You'll see reports such as, "I imported an h264 MOV into iTunes and it's not working on Apple TV! I don't get any sound! I thought Apple TV supported h264. And MOV is Apple's own format!"

Correct, MOV is Apple's own format, and h264 is very much supported on Apple TV. It's not the MOV that is incompatible, nor is it the h264 video stream. It is the audio stream inside the MOV that is incompatible. Only AC-3 and AAC audio codecs are supported by Apple TV, but there is a "gotcha" with AAC and 5.1 sound.

The DTS Problem

I see many videos, especially MKVs (Metroska video), coming with a DTS 5.1 soundtrack. DTS is not compatible with Apple TV because the MP4 container does not support DTS. Most conversion (remuxing) software will display an error when you try to remux a video with DTS to MP4. To get video with DTS into iTunes and Apple TV, its audio stream must be transcoded to AC-3 or AAC.

The AAC Problem

If you are like me, you connected Apple TV to your reciever using solely an HDMI cable. True, there is a TOSLINK (optical) output on the back of the Apple TV, but my logical thought was, "OK, TOSLINK doesn't apply to my setup. HDMI is capable of carrying all kinds of multichannel digital audio, right? One world, one cable." I went ahead using only the HDMI cable.

My second logical thought was, "I'll see if I can center my movie library around 5.1 AAC audio, it being the newer codec than AC-3."

With 5.1 AAC videos over HDMI, I quicky discovered that only 2 audio channels were coming out of my 7.1 surround system. [gasp!]

Through trial and error, I found that I could get 5.1 sound to play if I did either of these:

Transcoded to AC-3 5.1 audio (Dolby Digital).
Used a TOSLINK cable between my Apple TV and my receiver.

TOSLINK wasn't an option for me because my receiver does not allow you to select a different, non-HDMI audio source while watching HDMI.

To summarize, Apple TV comes configured like this:

AAC 5.1 - Decoded in the Apple TV:
- HDMI - Only 2 channels sent to reciever (by default).
- TOSLINK - Full 5.1 sent to receiver.
AC-3 5.1 - NOT decoded in the Apple TV:
- HDMI - Full 5.1 sent to receiver via "Pass Thru".
- TOSLINK - Full 5.1 sent to receiver via "Pass Thru".

"Pass Thru", under the AC-3 bullet above, means that the audio stream is not decoded within Apple TV. Your reciver must be able to decode AC-3. AC-3 is a backcronym for "Dolby Digital", which was invented in the 90's. Any reciever manufactured in the last 10 or 15 years has support for Dolby Digital.

The Apple TV, by default, will only send 2 of the 6 channels of a video's AAC audio over HDMI. So I began standardizing on AC-3 5.1 soundtracks until I discovered the magic Apple TV setting:

The Solution: The Magic Setting

Turn "Dolby Digital" to On. The setting mentioned in this Apple TV support article.

Now the 5.1 soundtracks in my AAC videos are finally audible!

Right now you're thinking, "Why would a setting called 'Dolby Digital' have anything to do with the non-Dolby codec, AAC?"

My guess: The verbose way of saying what this setting's actual function is, would be, "It's OK to send decoded 5.1 over HDMI." To us audio geeks, the option is probably better called, "Bitstream 5.1 over HDMI", but that is far more cryptic to the average user. The average user simply knows the "Dolby Digital" brand to mean "good sound" or "surround sound". Apple, following the rules of good User-Interface Design, usually avoid as best they can, acronyms and abbreviations. They mean nothing to most users, so in using them, you communicate nothing to most of your audience.

Why would '2 channels' be the default over HDMI? Two possible answers I can see:

Perhaps during Apple's testing, there could have been some fringe issues with 5.1 AAC over some older HDMI setups or versions, like static or no sound as the above support article indicates. Apple probably opted for the safer 2-channel default because "audio not coming out my rear speakers is better than static coming out of all of them".
I saw "HDMI negotiation" as another possible cause. I haven't dug deeper into this, but I saw one person speculating that there is a HDMI-negotiation process that occurs between your receiver and your Apple TV to determine the # of simultaneous bitstream channels that your receiver can handle. If that type of negotiation is not a feature of your receiver, or this process fails in some way, perhaps it defaults back to 2 channels.

The Apple TV Setup Guide says something like, "If you have a Dolby Digital receiver, set this to On."

Tools

Conversion

MP4 Tools

I've had the best success with MP4 Tools. It focuses on Apple TV and remuxing, although it will transcode too. It's free if you are OK with nag screens, or $5 to make those go away (which I did). A very small price for a great, easy-to-use tool that works.

Open your video with MP4 Tools. Go into the MP4 tab.
Choose the "Apple TV" device in the lower right.
- It will gray out settings that aren't compatible with Apple TV.
Check off the streams you want to include: Audio, Video, Subtitles.
Choose "Pass Thru" for Video.
- This copies the h264 stream to the new container. No quality loss.
Choose "Pass Thru" for Audio if it's already 5.1 AC-3 or AAC.
- If "Pass Thru" is grayed out, choose AAC 5.1.
Choose "Mux (soft)" for subtitles.
- "Hard" will burn them in to the frames and cause a 2-hour-long transcode.
Leave everything else as is. Click "Convert".

Because we're remuxing, a 2-hour movie only take maybe 2 or 3 minutes. If the audio had to transcode, then only a few minutes more.

MP4 Tools leaves the individual video and audio stream files on your drive after it's done, and log files, in addition to the remuxed video. I turn this feature off in Preferences so the software cleans up after itself, leaving only the remuxed video.

Soft subtitles sometimes don't work in ATV. Sometimes you have to burn them in. When you burn them in, "Pass Thru" video grays out. Of course, you have to transcode video to burn in subtitles (a several-hour process). Tip: To see the Subtitles menu in Apple TV, hold down the Select button on the remote (the big round one in the middle). A subtitle-selection menu appears.

If "Header Compression" was used in an MKV, MP4 Tools will prompt you remux it away. Just say "Yes". It will create a new MKV file with Header Compression disabled. Simply run that one through MP4 Tools in the same way, again remuxing it so there is no quality loss. This is another area where this tool succeeds where others do not (FFmpeg).

You can basically stop here, and be on your way with high-quality 5.1 videos in iTunes and Apple TV. But from here, you'll likely want to write full metadata into the file, including hi-res cover art, that will make your personally converted videos look just as fancy as the ones you downloaded from Apple. They'll appear in iTunes and ATV with all the nice metadata: Title, Synopsis, MPAA Rating, Cover Art, Director, Actors, Genre, etc. What follows is a tool that does just that, as well as some other tools you should keep on hand.

Metadata

How to make all your videos have that "nice-looking" metadata that shows up in iTunes and Apple TV:

Subler - Subler has remuxing features, but it's primarily a metadata editor. It downloads the video's complete metadata from the TMDb and TVDB databases online, then writes that into your video file.
- Tip: If Subler crashes upon Save, just relaunch it and Save the metadata again. It does very occasinally crash, but I haven't noticed that adversely affect the video files at all.

"Seeing" the Streams

It's a good idea to keep some free software on-hand that allows you to view the stream data inside video containers.

MediaInfo - Peer into all of the metadata inside a video file. A keep the free command-line version of this app installed. Install Homebrew, then brew install mediainfo. There is also a GUI one in the App Store for a couple bucks.
FFprobe - Also a simply install via Homebrew: brew install ffmpeg.
VLC - Press Command-I for "Video Information". Go into the "Codec Details" tab and you'll see the Video, Audio, and Subtitle streams. Expand the arrow next to each of the streams to see more info, including the codec of each one.

Other Conversion Tools

These tools can do both remuxing and transcoding, but I didn't get into them in this article for reasons listed:

FFmpeg - FFmpeg is an enormously powerful command-line tool, and is one of my favorites. It is absolutely capable of remuxing to Apple TV-compatible formats, but my goal with this article was to show some simple, GUI-based conversion tools, geared toward remuxing and Apple TV, that I could recommend to non-nerds. And FFmpeg sometimes has trouble remuxing MKV's when "Header Compression" was used in the MKV.
iFFmpeg - A nice GUI wrapper for FFmpeg with some good, Apple-centric presets. Of course, FFmpeg's overwhelming array of settings and options bubbles up to this app. Great for learning FFmpeg, though, and discovering its secrets.
VLC - VLC does have some admirable transcoding and remuxing features, but being a media player, that's not its focus. It doesn't always succeed, or its resultant files don't import into iTunes. Although, I keep VLC installed at all times. It's an excellent media player that plays every format in the world.

Transcoding Only: Last Resort

HandBrake - Handbrake is a great tool, but is strictly a transcoding app, i.e., lossy conversion. Unfortunately, it doesn't do straight remuxing. Exhaust your efforts first with the above tools before resorting to HandBrake. I often see folks recommending HandBrake to get things into iTunes. That is unfortunate because they are imposing unnecessary, 2-hour-long, quality-reducing transcodes on all of their videos.

Install PostgreSQL 9 on Ubuntu Linux

2010-12-04T20:10:00+00:00

Tested on: Ubuntu 10.04

Add Repo

:::bash
sudo nano /etc/apt/sources.list

Append lines:

:::bash
## Postgres 9 hasn't been backported yet.  Manually added repos.
deb http://ppa.launchpad.net/pitti/postgresql/ubuntu lucid main 
deb-src http://ppa.launchpad.net/pitti/postgresql/ubuntu lucid main

Go to this site and expand the "Technical details about this PPA" panel. Under Signing Key, copy everything to the right of the slash [/], and paste into the command below. In this example, it is "8683D8A2".

:::bash
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 8683D8A2
apt-get update

Remove Old Versions

Careful! These commands completely remove your old PostgreSQL version. Back up data first.

:::bash
sudo service postgresql-8.4 stop
sudo apt-get purge postgresql*

Install

:::bash
sudo apt-get install postgresql-9.0

Verify you can log in:

:::bash
sudo -u postgres psql

You probably want to create a user for your web app while you're in there:

:::sql
CREATE USER www_db WITH PASSWORD 'jw8s0F49eQ';

Open External Access

If you plan to access Postgres externally, modify conf file to make Postgres listen on * instead of localhost. Modify Host-Based Authentication file to make Postgres respond to more IP ranges than solely 127.0.0.1. See my other article on this.

Instrumentation

So pgAdmin doesn't complain:

:::bash
apt-get install postgresql-contrib-9.0
sudo -u postgres psql -U postgres -d postgres < /usr/share/postgresql/9.0/contrib/adminpack.sql

Server-Side Email on OS X

2010-11-27T22:32:00+00:00

This will make it so that you can send email from Terminal sessions, server-side scripts, and web pages. We are also going to tell it to route email through your ISP's email servers: Gmail, in this case.

Works on: OS 10.5 and 10.6

Postfix

Conveniently, the powerful and popular SMTP server, Postfix, is already installed on OS X. In this example, we're going to tell it to route outgoing email through your Gmail account.

Edit Postfix's config file:

:::bash
sudo nano /etc/postfix/main.cf

Press Esc-/ to go to the bottom of the file. Add the block of text below to the bottom, changing relayhost to your outgoing SMTP-server address. In this example, we are using port 587, which TLS-encrypted mail travels through. Verify your ISP supports TLS and port 587. Nearly all do.

:::bash
relayhost = smtp.gmail.com:587
smtp_sasl_auth_enable = yes
smtp_use_tls = yes
smtp_enforce_tls = yes
smtp_sasl_security_options =
smtp_sasl_tls_security_options =
smtp_sasl_tls_verified_security_options =
smtp_tls_loglevel = 2
# Log is here /var/log/mail.log  
smtp_sasl_password_maps = hash:/etc/postfix/smtp_sasl_passwords
smtp_tls_per_site = hash:/etc/postfix/smtp_tls_sites
tls_random_source = dev:/dev/urandom

Create the file to hold login credentials:

:::bash
sudo nano /etc/postfix/smtp_sasl_passwords

Add this line to it, changing it to match your SMTP server address and credentials:

:::bash
smtp.gmail.com username:password

Edit this file:

:::bash
sudo nano /etc/postfix/smtp_tls_sites

Add this line to it, changing mail server accordingly:

:::bash
smtp.gmail.com MUST_NOPEERMATCH

Run these commands:

:::bash
cd /etc/postfix
sudo chmod go-rx smtp_sasl_passwords
sudo postmap smtp_sasl_passwords
sudo postmap smtp_tls_sites

Test

Change email addresses to yours:

:::bash
echo "Hello" | mail -s "Test" me@domain.com

The above may not work if your ISP requires a valid From address, so also try:

:::bash
printf "Subject: TestnHello" | sendmail -f me@domain.com me@domain.com

Both of these worked with Gmail, in my case.

If things fail, tail out the log:

:::bash
tail -f /var/log/mail.log

Send more email in a new Terminal tab while watching the logs.

Install Rails 3 on OS X with Devise, RSpec, HAML, and jQuery Support

2010-11-26T18:20:00+00:00

Goals

PostgreSQL - Instead of MySQL
Devise - Instead of AuthLogic or Restful Authentication
HAML - Instead of ERB
jQuery - Instead of Prototype
RSpec - Instead of Test-Unit

Prerequisites

Ruby 1.9.2 - Installed via Homebrew. These instructions not tested with RVM.
Rails 3.0.3 - gem install rails
PostgreSQL - See my notes on installing PostgreSQL 9 on OS X via Homebrew.

Create App

:::bash
rails new myapp
cd myapp

Create Database

Change config/database.yml accordingly to point to your dev DB, then:

:::bash
rake db:create

Gems

Add to Gemfile in the root of your app:

:::ruby
gem 'autotest'
gem 'devise'
gem 'haml-rails' # Not haml
gem 'hpricot'
gem 'pg'
group :development, :test do
  gem 'rspec-rails', '~> 2.1'
end
gem 'ruby_parser'

Run sudo bundle install to install all gems. Note haml-rails instead of haml. haml-rails is newer and includes Rails 3 generators. Also, hpricot and ruby_parser are necessary or Devise's generators fail.

Devise

Follow the Devise instructions at the end. There are a couple of manual steps. If you missed it, those steps are:

Add to config/environments/development.rb:

:::ruby
config.action_mailer.default_url_options = { :host => 'localhost:3000' }

Add to config/environments/production.rb:

:::ruby
config.action_mailer.default_url_options = { :host => 'russbrooks.com:3000' }

Add to config/routes.rb:

:::ruby
root :to => "home#index"

Since we are going to use HAML templates, rename app/views/layouts/application.html.erb to application.html.haml, then paste:

:::ruby
!!!
%html{ :lang => 'en', 'xml:lang' => 'en', :xmlns => 'http://www.w3.org/1999/xhtml' }
  %head
    %title
      My New Site
    = stylesheet_link_tag :all
    = javascript_include_tag :defaults
    = csrf_meta_tag
  %body{ :class => @body_class }
    #wrap
      #header
      #body
        %p{ :class => :notice }
          = notice
        %p{ :class => :alert }
          = alert
        = yield
      #footer

Remove Rails' welcome page or it will render instead of home#index:

:::bash
rm public/index.html

Use HAML Instead of ERB

Add this to config/application.rb anywhere inside the Application class:

:::ruby
# Default template engine for Rails Generators. Use HAML instead of ERB.
config.generators do |g|
  g.template_engine :haml
end

The generators are now built in to the HAML-Rails gem. There is no need to add the rails3-generators gem.

Use RSpec Instead of Test-Unit

We already added the necessary gems to Gemfile and did a bundle install above, so just run the RSpec installation generator:

:::bash
rails g rspec:install

This makes Rails generators produce RSpec tests instead of Test-Unit tests.

Use jQuery Instead of Prototype

:::bash
rake rails:template LOCATION=http://github.com/lleger/Rails-3-jQuery/raw/master/jquery.rb

Generate User Model and DB Table

:::bash
rails g devise User
rake db:migrate

Use rake routes to see the routes Devise created for sign_up, sign_in, sign_out, etc. You should be able to access those in a browser to test them at this point. See the Devise docs for more info.

From there, you might want to start with some scaffolding to ensure everything worked right, for example, that it generates HAML, RSpec tests, and is using jQuery. For example, maybe create a "Post" resource if you are creating a blog:

:::bash
rails generate scaffold Post title:string teaser:string body:text

Install PostgreSQL 9 on OS X

2010-11-25T18:27:00+00:00

Homebrew

You're not still using MacPorts, are you? Please go get Homebrew immediately, the new-hotness package manager for OS X written in Ruby. It amazing how simple Homebrew makes this process in comparison to my old instructions on this topic. It's also nice to be able to peer into friendly-Ruby Homebrew Formulae to see the exact build commands.

Uninstall Old Versions

Initially, I tried a brew install postgresql over the top of the existing version that was installed via non-Homebrew means. When it was complete, psql --version was still showing 8.4.x. I found it is best to uninstall any old PostgreSQL versions first. Of course, back up your data with pgAdmin or pg_dump.

First, uninstall any Homebrew versions. The --force option makes it uninstall all versions.

:::bash
brew rm postgresql --force

Via Uninstaller

Second, uninstall any versions installed by other means.

If you installed using EnterpriseDB, you likely have an uninstall-postgresql.app file in your install directory. Double-click that to uninstall their Postgres install.

Manual

I went the manual route. Change paths accordingly for your version.

:::bash
sudo /sbin/SystemStarter stop postgresql-8.4
sudo rm -rf /Applications/PostgreSQL\ 8.4
sudo rm -rf /etc/postgres-reg.ini
sudo rm -rf /Library/StartupItems/postgresql-8.4
sudo rm -rf /Library/PostgreSQL/8.4
sudo dscl . delete /users/postgres

Edit /etc/profile and delete any lines that reference "postgres". For me, there was just one line that had to do with loading Postgres' Environment file.

:::bash
nano /etc/profile

I rebooted here, to make sure the Postgres startup items were truly gone. Open "Activity Monitor" and ensure there are no "postgres" processes.

Install PostgreSQL

:::bash
brew update
brew install postgresql

Follow the instructions at the end of the install to initialize the DB, add startup items, and start Postgres. For me, it was:

:::bash
initdb /usr/local/var/postgres
cp /usr/local/Cellar/postgresql/9.1.4/homebrew.mxcl.postgresql.plist ~/Library/LaunchAgents/
launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start

Important Note About Authentication

The Homebrew formula does not create a "postgres" user, as was common up to this point. Out of the box, you authenticate against the DB using the same user account under which you ran brew install. It's perfectly acceptable to leave it that way on your Dev machine. Don't forget to change that if this is a Production environment.

Instrumentation

Install Instrumentation so pgAdmin doesn't yell at you. Run this as the same user under which you installed Postgres.

:::bash
psql postgres -c 'CREATE EXTENSION "adminpack";'

Test

Launch pgAdmin, add a DB for "localhost", and use your username and password.

Install the PG Gem

This installs the "pg" gem that Matz himself wrote. It is recommended over the old "postgres" gem. I do not believe you need to give it Architecture Flags anymore (env ARCHFLAGS="-arch x86_64" gem install pg), so just do:

:::bash
gem install pg

LFTP Cheatsheet

2010-11-19T20:06:00+00:00

I don't use FTP personally. But, on a professional level, LFTP is becoming a super-reliable option for me when automating FTP transfers with script. LFTP is a more robust FTP client than just plain FTP or cURL. Unlike those clients, it retries a few times when transmission fails, has mirroring features, and supports simultaneous multi-file transfers, recursion, and Regular-Expression matches. Speed, from what I've experienced, is identical.

Upload a File

:::bash
lftp -e 'put /local/path/yourfile.mp4; bye' -u user,password ftp.foo.com

Like all FTP clients, LFTP correctly uses Passive mode by default. Passive is newer than Active. It was designed to solve the problems with Active, and will go through the greatest array of firewalls without incident. [Never use Active.]

Connection Timeout

LFTP will keep trying forever I believe, so it's wise to specify a timeout, especially if you are scripting this. In this example, timeout is 10 seconds:

:::bash
lftp -e 'set net:timeout 10; put /local/path/yourfile.mp4; bye' -u user,password ftp.foo.com

Active Mode

For what it's worth, Active mode -- older, stinkier, and won't go through your firewall:

:::bash
lftp -e 'set ftp:passive-mode false; set net:timeout 10; put /local/path/yourfile.mp4; bye' -u user,password ftp.foo.com

Download

Uses Binary mode by default:

:::bash
lftp -e 'set net:timeout 10; get yourfile.mp4; bye' -u user,password ftp.foo.com

If you need to put it in a specific local directory:

:::bash
lftp -e 'set net:timeout 10; get yourfile.mp4 -o /local/path/yourfile.mp4; bye' -u user,password ftp.foo.com

Mirroring

Recursive Upload

When using the mirror command, LFTP is recursive by default.

Mirror everything from the /local/path to the root of the remote FTP site, including all subdirectories and their files. The -R switch means "reverse mirror" which means "put" [upload]. Remote path is simply /, the root.

:::bash
lftp -e 'mirror -R /local/path/ /' -u user,password ftp.foo.com

Recursive Download

To mirror remote to local, just omit the -R param and swap remote path with local. Be careful with this. Don't overwrite your local changes.

:::bash
lftp -e 'mirror / /local/path/' -u user,password ftp.foo.com

Regular-Expression–Match Upload

Only include certain file types. Mirror everything in the /local/path recursively, but only transfer files that end with .htm, .html, .css, and .js.

:::bash
lftp -e 'mirror -R -i "\.(html?|css|js)$" /local/path/ /' -u user,password ftp.foo.com

Non-Recursive

Deploy only root-level files, not the entire directory tree. The -r switch disables recursion. In this example, we are also only deploying HTML, CSS, and JavaScript:

:::bash
lftp -e 'mirror -R -r -i "\.(html?|css|js)$" /local/path/ /' -u user,password ftp.foo.com

Special Characters in Username or Password

Special characters in your username or password must be escaped with backslashes:

:::bash
lftp -e 'put /local/path/yourfile.mp4; bye' -u user,password\!\! ftp.foo.com

IMO, FTP is not the right protocol for Internet file transfer of any kind. But, unfortunately, it is in place everywhere so you do find yourself stuck with it as the only option from time to time. [SFTP is the right protocol.]

Rails Cache and MemCache on Passenger with Smart Spawning

2010-10-20T22:15:00+00:00

This article applies to Passenger 2.2.x. Passenger 3 was released today, so I'm not certain if this issue applies to that yet.

Smart Spawning

Passenger's "Smart Spawning" is enabled by default, but does not play nice with MemCacheD.

If you recall from the Mongrel days, Mongrel_Cluster loads an entire copy of of your app and the Rails framework into each Mongrel. Mongrel is analogous to an "app process" in Passenger. For efficiency reasons, when Smart Spawning is active, Passenger shares your app code and Rails framework between app processes. Unfortunately, this can seriously confuse MemCacheD. Different app processes can contend for the same item in MemCache, unbeknownst to MemCache, causing one of those two MemCache connections to drop mid-stride. If you have Rails.cache configured to use MemCache as its store, this can cause periodic incomplete cache reads. Since everything in Rails.cache is marshaled on its way in, if a MemCache connection is interrupted, a cache read can be truncated, which causes demarshaling to fail, yielding a cryptic "Marshal data too small" error. You may encounter other errors such as MemCache IO errors and timeouts.

Lazy Solution

To solve this, some developers back Passenger off to "Conservative Spawning". But that, like Mongrel_Cluster, stores a copy of your app and the Rails framework in each app process. This wastes memory and app-process [re]spawn time. I didn't want to concede to that.

Correct Solution

I found a clue in the Phusion Passenger User's Guide under Spawning Methods Explained. If you scroll a little further, there's a section called "11.3.1. Example 1: Memcached connection sharing (harmful)". It provides a small code snippet for your environment.rb file. This code resets the connection to MemCache when an app process is [re]started. But, unfortunately their code snippet contains a some psuedo-code, reestablish_connection_to_memcached, instead of actual code for the solution. I see some solutions online that use a CACHE constant which would mean a global replace of "Rails.cache" throughout your app. We were already using the standard Rails.cache application-wide, so I wanted to stick with that. So, I call the reset() method on Rails.cache instead [but only if Rails.cache's store is set to MemCache].

:::ruby
if defined?(PhusionPassenger)
  PhusionPassenger.on_event(:starting_worker_process) do |forked|
    if forked
      Rails.cache.instance_variable_get(:@data).reset if Rails.cache.class == ActiveSupport::Cache::MemCacheStore
    end
  end
end

Also update MemCacheD and your memcache-client gem to latest. They're faster and better. And add a require 'memcache' to environment.rb, so Rails use the new gem and not its bundled old one.

Restart MemCacheD:

:::bash
service memcached restart

...and Passenger [from the root of your app]:

:::bash
touch tmp/restart.txt

Rails.cache.write and Rails.cache.fetch should now be shuttling data in and out of MemCacheD unscathed.

BMW S-1000RR Test Ride

2010-03-20T17:45:00+00:00

Everything you’ve read and heard is true. Germany really has crafted a groundbreaking masterwork in superbike engineering. Sport-riding enthusiasts in the audience will know I am talking about the brand-new BMW S-1000RR, BMW’s first-ever foray into the superbike arena. May I say that Bavarian Motorwerks has not merely dropped a disposable, cookie-cutter, half-hearted hero into that arena, but a full-on fire-breathing dragon of epic, Tolkienian proportions.

The salesman gave me the test-ride rules as we stood next to the running motorcycle, in Beemer green, onlookers gazing in admiration, while we conversed in that slightly-raised, talking-over-a-running-bike tone, “You track-ride a 1098 and a CBR-1000, so you know what you’re doing, but I have to go over the rules. The bike is electronically limited to 8,000 RPMs during the break-in period, so just be aware of that. BMW requires that we only demo in ‘Rain Mode’.”

“150 horsepower,” I interrupted.

“Exactly. You can switch to other modes, but it doesn’t matter. The demo bike is configured to remain in Rain Mode.”

“No problem,” I affirmed.

“About the Quick-Shifter, if you’re gonna try it, don’t let off the throttle between gears. Just stay one the throttle while up-shifting clutchless,” as he made the universally understood wrist motion with his right hand.

“Check.”

He noodled various switches and poked his head closer to the display, “Traction Control on. ABS active. Trip timer reset. That’s about it. Enjoy! Don’t crash it!” [chuckles]

“How many have you guys sold already?”, I inquired pulling my helmet onto my head.

“5 pre-orders right now. Every single person we have let ride it, has bought it.”

“Well, let’s see if I can maintain that trend for ya,” as I slung my leg over.

Accelerating out of the parking lot, I was surprised how much power the bike actually had in Rain Mode. Regardless of what the dealer had told me, I still had to at least try putting it in Race mode. I frantically thumbed the tiny gray “Mode” button with my right thumb and squinted at the dash. I watched as the little arrow cycled up to SPORT mode, then RACE.

RAIN
SPORT
RACE <

I attempted a hard acceleration, but the bike was clearly still in RAIN mode. Keep in mind that even in RAIN mode, the horsepower is equivalent to the most powerful 1,000’s of 6 years ago. There is a forth, even more powerful, mode beyond RACE, called “SLICK”, but you have to flip a switch under the seat to activate it.

I noticed one particular difference immediately, and this was a small difference but it was an indicator of the thoughtful engineering to come, found throughout this machinery. The turn-signal button moved all of a micron in each direction. No huge 1/4-inch push with your thumb was necessary. All it took was a tiny, effortless nudge with your left thumb and *snikt* the blinker was active. The same applied for turning it off.

I stopped at a red light with some fast, smooth road ahead and thought it the perfect time to try some full-throttle, clutchless upshifts. The light greened and I was off. I snapped through several gears, in the tuck, hitting 8,000 each time. I even felt the temporary “break-in” limiter kick in on a gear or two. I could not believe the smoothness and speed at which this motorcycle changed gears. I realized I was actually being too hard on the shifter with my left foot, being used to my Ducati’s fine but noticeably less effortless gearbox. I made a mental note to shift more easily.

I arrived at another light, made a right onto a buttery county road, and really gave it the goose. This time, clucthless upshifts were executed by merely thinking. I identified the bike’s gorgeous and distinct engine sound as Mozart’s Symphony No. 25 in G Minor. Strudel to my ears, as I tucked it cleanly around a beautiful left-hand sweeper at 80 MPH, leaning off, knee out. I rolled on smoothly but assertively and absolutely launched myself out of the corner. 90, 100, 110, 120… as fast as you can say it. Various expletives of wonderment came to mind as my spine shivered and goosebumps appeared on my skin. Rapidly approaching a line of cars doing 50, I had a chance to feel the race-class brakes. I one-fingered the lever and zzzzzzzzzzZZZZZZ the sintered brake pads made their sound, the nose gently dove, and the bike decelerated with all the stability and rapidity of a world-class superbike. As I railed it down through the gears, habitually blipping the throttle each time, I noticed a certain smoothing of my downshifting. Ah, I remembered… the slipper clutch (or “back-torque-limiting” to be precise). It was time to put that to the test. I found a clean spot to pass and snapped it effortlessly down two gears and whacked it to the wood. The front-wheel skimmed the Tarmac as the bike had me past a line of 5 cars in seconds. Rounding a right-hand sweeper, a red light rapidly approached. Slowing from 125 again, I intentionally downshifted sloppily, not blipping the throttle at all. The Beemer absorbed my ham-fisted shifts with all the efficiency of Deutschland.

I stood at the red light and thought, “5 miles on the trip odometer. I have to turn around at some point. This is it. Let’s make it count.” I had read about the Wheelie-Control System so, being a recreational wheelier myself, I really wanted to feel that. The light went green and I gave it everything an 8,000 RPM limit would afford me. I pinned it. The front wheel eagerly attempted to lift into the stratosphere, with the Rain-mode-limited 150 horsepower behind it, but Germany’s finest electronics prevented that from happening. I felt the wheelie control engage as the bike’s front wheel just skimmed along at perhaps an inch off the ground. But, even in Rain mode, MAN did the motorcycle accelerate! I could feel the precession in the engineering beneath me, everything working in perfect harmony. The engine note is nothing short of beautiful. I cannot wait to feel it in SLICK mode on the track. I also cannot wait to feel this motorcycle when the shackles of an 8000-RPM limit are removed and the full power of its 14,200 redline can be felt. I can’t even imagine. Let’s remember that it was only six years ago that 600cc motorcycles had 14,000-RPM redlines. BMW has now exceeded this with a 1,000. The rest of the liter bikes in this class all have sub-14,000 redlines. According to BMW, the engine can spin significantly faster, but they limited it to 14.2 for production durability reasons.

I’m seeing mixed reports online about the Wheelie-Control System. I saw one report that indicated it is completely disabled in RACE and SLICK modes. I saw a report that only in SLICK mode is it disabled. I saw other reports that indicated that it is active in all modes, only allowing you to get the front wheel off the ground 12” to 16”, and only for 5 seconds (which, as a recreational wheelier, would be disappointing).

The bike is absolutely effortless to ride. It is the lightest, most nimble 1,000 I have ever ridden. Every lever, switch, and control on that motorcycle are manipulated by making the most infinitesimal movement with your hands, fingers, and feet. It has highly user-friendly ergos with everything in the right places. It felt very slim for an inline-4, definitely slimmer than my CBR-1000. The power curve felt pin-straight. It is also remarkable that BMW has managed to pack every possible gadget onto this bike at this insanely reasonable price point: DTC (with lean-angle sensor), ABS, wheelie-control, track-ready suspension, track-ready brakes, slipper clutch, quick shifter, and a free-breathing exhaust. I did the math, and to do the same to a new CBR, R1, or GSX-R with aftermarket kits, would bring you up to around $21,000, and you still wouldn’t have power modes or true DTC with wheel sensors. BMW has just, plain nailed it.

On the way back, I was waiting at another light, and I glanced at the SUV driver to my right. He was looking down at the bike, hitting his passenger in the shoulder to get him to also look, and he had that “HOLY SHNIKIES! WTF IS THAT?” look on his face with a smile. He saw me looking over, so I slowly reached to the throttle and give it a few twists for him, rotating the handgrip with only my thumb and index finger to demonstrate to him the precision of this wondrous machine. “GROOOOOWL!! GROOOOOOWL!! GROOOOOOOOOOOOOOOOOWWLLL!!”, the S-1000 sang its beautiful song. His eyes widened to disproportionate levels, a huge grin came across his face, as he gave the thumbs-up of approval. I had my mirrored shield on, but I’m sure he saw my upper torso and head vibrating up and down as I giggled in absolute astonishment, looking down at the marvel I was straddling, shaking my head, arms out, palms up as if to say, “Isn’t it just…?”

Disable .DS_Store Files on Mac OS X

2009-12-30T21:22:00+00:00

Have you ever noticed that your Mac leaves small .DS_Store files on network shares and wished you could shut that off?

To disabled them, enter or paste these commands into a Terminal window:

:::bash
defaults write com.apple.desktopservices DSDontWriteNetworkStores true
su -
defaults write com.apple.desktopservices DSDontWriteNetworkStores true

Log out of your Mac from the Apple Menu, then log back in again.

The second command [su -] logs us in as Root, then we perform the command again as Root. It is unlikely you will be browsing network shares as Root, but we do it just in case.

The above command does not prevent these .DS_Store files from being created locally, which is a good thing. The OS needs them locally.

The only thing stored in these files is some unnecessary things like scrollbar positions, icon size, and window dimensions. Trust me, you can safely live without them. I prefer that my Mac be a good citizen and not leave a trail of liter strewn throughout the network. [I should mention that Windows does the same thing with its thumbnail cache files.]

Embed a Static HTML Page in a Rails View

2009-11-18T22:00:00+00:00

During a recent Rails project, I had the need to embed some static HTML pages within a View that were wrapped in the application’s default Layout. I found it interesting that there was no apparent simple way to do this, for example SSI.

Things that didn’t work:

render :file in the Controller. It worked, but you cannot wrap it in a Layout.
render :file in the View
render :partial
render :inline
render :text
render :action

I discovered a loosely documented method called render_to_string(), but it did not work directly in the View because render_to_string() has a visibility of “Protected” in ActionController::Base. In other words, it is only accessible in ActionController::Base and subclasses thereof. The Controller is a subclass of ActionController but the View is not.

Questionable

One egregious way around this, that I’ve seen online, is to simply make render_to_string() public in your Controller:

:::ruby
class MyController < ApplicationController
  public :render_to_string
end

Then access it in the View using the controller. reference:

:::ruby
controller.render_to_string :file => '/aboslute/path/to/file.html'

Note that what is being shown above, in the MyController class, you cannot do in other OOP languages like C#, Java, and PHP. We are effortlessly overriding a method’s Protected visibility and making it Public.

This raises the question, “Why even have visibility modifiers in Ruby?”

Ruby is very open and trusting of the client-code programmer. In Ruby, marking a member as Private or Protected essentially means, “If you try to access this member in a way that violates its visibility, we will certainly let you know [by throwing an error], but if you really want to, you can still access it [by adding a line of code].”

The object masters will cry, “But this breaks Encapsulation, an OOP fundamental! What happens if the implementation changes?”

True, but by the same token, it allows you to do some powerful and flexible things in Ruby that you can’t do in other languages. [Plenty of examples online.]

Correct Solution

I opted to store the contents of render_to_string in an attribute in the Controller:

:::ruby
@file = render_to_string :file => '/aboslute/path/to/file.html'

Then output the @file attribute in the View.