flimflam

Compiling a Realtime (PREEMPT RT) Kernel in Debian 5.0/Lenny

OK, let’s just get right to it. We both know why you’re here and you’re probably raring to get JACK and some client apps running in RT mode with low latencies as quickly as possible.

Make sure you have the requisite packages:

$ apt-get install kernel-package libncurses5-dev fakeroot wget bzip2 -->
    build-essential zlib1g-dev

By the way, there’s no need to build your kernel in /usr/src or to do anything as root until the actual installation. I like to create a ~/KERNEL dir to work in:

$ mkdir ~/KERNEL
$ cd ~/KERNEL

The next thing you’ll want to do is grab the source for desired matching versions of the kernel and the RT patch. For this post, I’ll use: 2.6.26.8.

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.26.8.tar.bz2
$ wget http://www.kernel.org/pub/linux/kernel/projects/rt/patch-2.6.26.8-rt16.bz2

Unpack, do some finagling, jerry-rigging, and ceremonial black magick:

$ tar xvjf linux-2.6.26.8.tar.bz2
$ mv linux-2.6.26.8 linux-2.6.26.8-rt16
$ ln -s linux-2.6.31.6-rt16 linux
$ bzcat ../patch-2.6.26.8-rt16.bz2 | patch -p1
$ cd linux

At this point, you have a few options for handling your kernel config. You could start with your current kernel’s config:

$ make oldconfig

If you want to start fresh (well, sort of …) skip that step and go right to:

$ make menuconfig

A few important settings should be pre-configured for you at this point, but to be sure, double-check the following:

Processor type and features ---> Preemption Mode (Complete Preemption (Real-Time))
Processor type and features ---> Timer frequency (1000 HZ)
Kernel hacking ---> [ ] Kernel debugging

IMPORTANT: note that Kernel debugging is DISABLED.

Now, just build your kernel in that oh-so-special Debian way … but first, a time saver: If you have a dual core processor, use all of it!

$ export CONCURRENCY_LEVEL=2

Of course, if you have a quad core proc, I imagine you can do the math. (If you have a tri core proc, I have some real-estate you might be interested in.) Carry on:

$ make-kpkg clean
$ time fakeroot make-kpkg --initrd kernel_image kernel_headers

NOTE: you could specify a -rev switch on the make command. For example: time fakeroot make-kpkg --initrd -rev my_custom_kernel_v1 kernel_image kernel_headers

Now is a good time to go pour yourself a pint … or three … this could take a while.

If all goes well, you can now install your kernel packages (found in your ~/KERNEL dir.) If you get an error, it’s usually because of a missing package, but is occasionally a known issue that could require you making modifications to your .config, or even so far as tweaking some source. No matter, the Google machine is your best bet.

Let’s assume all went well:

$ cd ..
$ su
$ dpkg -i linux-headers-*.deb linux-image-*.deb

NOTE: of course, if you have multiple versions of the .deb files in your ~/KERNEL dir, be more specific.

Reboot. And you should see your new kernel in the GRUB menu. Now go pour another pint.

Tags: audio, DAW, debian, kernel, low latency, preempt, realtime, rt

Why Everyone Hates Sidney Crosby

So, if you’re not a hockey fan, this is probably as interesting to you as a stranger’s colonoscopy. But I am a hockey fan. An avid, no, rabid, hockey fan.

Being a hockey fan while the Stanley Cup playoffs are going on means you check ESPN.com/nhl 3 times a day and you count down the days until John Buccigross’ next column. That countdown was over yesterday and the latest installment is here: http://sports.espn.go.com/nhl/playoffs/2009/news?columnist=buccigross_john&id=4193279.

Somewhere in the middle, Bucci asks, “Why does everyone hate Sidney Crosby?” And he goes on to say, “It always has baffled me. Why is someone who dedicates his life to hockey — in how he trains, eats, lives and plays so fast and in such breathtaking and entertaining fashion — so reviled?”

This got me thinking, and I sent this on to the resident ESPN Puckhead:

Bucci –

In response to your “What a Cup would mean” column: the fact that people hating Sidney Crosby baffles you, in turn, baffles me. From my perspective, it’s a simple matter of a few things that form something of a negative synergy for the typical NHL fan — and by “typical NHL fan”, I mean one that does not root for the team that Sidney plays for but is forced to watch him in virtually every other nationally broadcast game for two years while uncomfortably listening to the color commentator fall just short of exclaiming, “I want to make breakfast for Sidney Crosby in the morning.”:

1. The NHL over saturated its fanbase with almost desperate Sid the Kid marketing in his rookie year. Suddenly, the game that so many of us have loved and faithfully followed for so long had been re-branded: “HOCKEY, featuring Sidney Crosby.” Or, if I am in a really cantankerous mood, the white noise began to sound like, “Tune in Sunday to watch Sidney Crosby play hockey and to see at least 7 spots chronicling Sidney’s childhood and his love for the game’s roots on Canada’s ponds.” Pardon me while I puke on my shoelaces.

2. He whined for his first two seasons. Period. No one can argue this point with a straight face while passing a breathalizer test. Yes, perhaps the scrutiny was more intense for a kid who could skate on the ice whether it was frozen or not, but the fact is that he was a whiner.
Combine this with reason #1, and you get the recipe for the annoying brat that’s most likely to be suspended by a jockstrap from the football field flagpole on homecoming Sunday in his sophomore year.

3. He is as charismatic as a box of hair. In the media, he seems to possess little genuine likability and garners virtually no unspoken respect. Add reasons #1 and #2 together, and one could surmise that Sidney has a non-trivial debt to pay off in the Guy I’d Like to Drink Beer With category. Unfortunately, Crosby does not seem to possess the power of persuasion required to make people like him if he weren’t The Greatest Hockey Player In The World.

But let’s be candid: I don’t know Sidney Crosby, so I can’t tell you if I genuinely dislike him. The real tragedy is that I *think* I dislike him, I *want* to dislike him, and I secretly want to see the league’s poster child fall on his face on the big stage, all as the result of the NHL botching its best opportunity to sell its product in almost 15 years.

Cheers,
R. Sheaves
Boston, MA

Tags: hockey, john buccigross, nhl, stanley cup playoffs, sydney crosby

Kill Your iPod

Admittedly, I do not own an iPod, but consider it a symbol for my point. I got to thinking about this the other day when I became wholly frustrated at not being able to legally make backup copies of movies that I thought I had purchased. Naturally, I started cursing copy protection technolgies and pined for the days of recording the latest Van Halen LP — these were made of vinyl, kiddies — to cassette so I could listen to it in my Z28 stereo on the way to school. (Insert nostalgiac moment here.)

Was the music on that duped cassette “free”? Of course not, but I was free to do for my personal use with that music as I so chose because I had purchased that right upon paying cash monies to Warner Bros. for the vinyl with the grooves on it. As we all know, somewhere along the line since then, my rights got swapped for restrictions.

Don’t let the entertainment industries dupe you into believing that Digital Rights Management protects the artists. Pffft. It clearly protects corporate profits. Hell, the latest lobbying by the record companies in Washington argues that copy protection ultimately helps to prevent terrorist attacks. WTF? (On the flip side, one might credibly claim that Hannah Montana is, in fact, a terrorist attack on young teenagers.)

Anyhow, off to work for me. I leave you with this:

What is DRM?

Digital Restrictions Management – it restricts what you
and your family can do with the electronic devices and
media purchased. It is an attempt by technology and
media companies to take away your rights.
DRM software and hardware monitors and controls
your family’s behavior.

iPod users are restricted from transferring their music to
other non­Apple devices because the music downloaded
from iTunes is encrypted ­ locked with DRM. Apple
allows you to write an audio CD, but will leave you with
very lousy sound quality if you ever want to take your
music to a new portable device in a compressed format.

Sony Music has secretly planted DRM “rootkits” on
computers when users purchased music CDs from them.
DRM is more than a nuisance. The film and music
industry are setting the agenda to increase their control.
They have demanded that technology companies impose
DRM to deliver for them what their political lobbying to
change copyright law never has: they aim to turn every
interaction with a published work into a transaction,
abolishing fair use and the commons, and making
copyright last forever. By accepting DRM users
unwittingly surrender their rights and invite a deeper
surveillance. This will put your family’s viewing,
listening, reading, browsing records on file with them.

What gives them that right?

DRM Means: No fair use. No purchase and resell. No
private copies. No sharing. No backup. No swapping. No
mix tapes. No privacy. No commons. No control over
our computers. No control over our electronic devices.
The conversion of our homes into apparatus to monitor
our interaction with published works and web sites.

What you can do

Stay away from DRM­dependent products like Blu­ray
and HD­DVD, iTunes, Windows Media Player, Zune,
Amazon Unbox…  Stay away from retailers who insist on
making DRM part of the package.  Stop financing the
people who want to restrict you. Find out more at
www.DefectiveByDesign.org

Now, brush up on some ways to protect yourself, and convert all of your CDs to OGG format. Cheers!

Tags: drm

The Best Blog Post I’ve Read All Week

Stranger in a Strange Land

From a guy named “Linus.” Last name, “Torvalds.” Enough said.

Tags: Linux

REST, CRUD, and Resources with Ruby on Rails

Unless you’re not a Software Engineer, or you are but have spent the past 18 months on a different planet, you’ve probably heard all about how Rails 2.0 is RESTful. Now, before we get too far into what exactly “RESTful” means, let’s set a reference point: the only document I am aware of that nobody disputes — too much, anyway — is Roy Fielding’s dissertation on Representational State Transfer (REST). So, for the sake of brevity, let’s assume we don’t need to elaborate the finer points of the REST architectural style and just take Fielding’s work at face value.

Resources and Rails

REST basically states that messages between system components — like browsers and servers — are requests and responses that represent resources. That is to say, resources are “conceptual mappings”, but are not necessarily mapped to databases, models, or controllers. They can be mutable (like the page number that a book is currently opened to), and they can be singular or plural (book vs. books.) In a phrase, a resource is essentially a high level description of something at a particular point in time: the time at which your request is responded to. Some examples:

• Your tab at the local pub
• An investment portfolio
• The contents of a public library
• A book

As mentioned at the outset, Rails 2.0 supports REST. However, note that like most of Rails, the support is highly opinionated. The implementation is rather particular and expects that we have “drunk the Kool-Aid.” If we don’t approach it in this manner, we’ll probably miss out on the benefits REST on Rails has to offer.

Clean up your CRUD

Another concept of which I am certain you and I likely have a common understanding is Creating, Reading, Updating, and Destroying data, or CRUD for short. For anyone who has ever worked on a software system, these are the familiar building blocks of a pervasive and persistent data storage mechanism — it also happens to be what we get out of the Rails scaffold generator with a single command:

$ ./script/generate scaffold book title:string abstract:text isbn:string

And with that, we get a model, a controller with all the fixin’s (more on this in a moment), view templates, migration scripts, and — this is the wicked cool part — a RESTful CRUD implementation. Let’s create the database and spin up the server to check things out:

$ rake db:create
$ rake db:migrate
$ ./script/server

Now, point your browser to http://localhost:3000/books. Go ahead and play with the screens — create, read, update and delete books until your fingers hurt.

Keeping it simple

It’s so simple that it’s easy to forget how simple it is. With this (somewhat obtuse) statement, I am not referring to Rails or Ruby, but rather since we tend to approach operations involving databases in our applications through overly-complex abstractions and convoluted design patterns, it’s all too easy to forget just how uncomplicated CRUD really is. And this temporary amnesia often leads us to complicate code with misguided action names that obfuscate the actual database operation they perform. Consider an action named replace_book_title. See my point? Even though the controller is not obligated to map to the database the way the model is, things become far simpler — more conventional, one might say — when actions are named similarly or altogether after the CRUD operations they perform.

But what about routing? Even if we were so disciplined as to name all of our actions adhering to strict CRUD conventions, we could still screw the whole thing up by specifying stupid names for the routes — routes with names like add_book. (Remember when I said that Rails was opinionated?) And this is where Rails’ REST implementation saves us from doing stupid things…

map.resources :books

With this line in your routes.rb file, you get four named routes that allow you to call seven different controller actions — all with eerily CRUD-ish names like those I have been ranting about for the past 5 minutes. These actions are: index, show, new, create, edit, update, and destroy. To help get our heads wrapped around this concept, let’s look at all the routes:

$ rake routes
[output edited for brevity ...]
GET     /books               {:action=>"index", :controller=>"books"}
POST    /books               {:action=>"create", :controller=>"books"}
GET     /books/new           {:action=>"new", :controller=>"books"}
GET     /books/:id/edit      {:action=>"edit", :controller=>"books"}
GET     /books/:id           {:action=>"show", :controller=>"books"}
PUT     /books/:id           {:action=>"update", :controller=>"books"}
DELETE  /books/:id           {:action=>"destroy", :controller=>"books"}

Now, the first thing you’ll notice is that there is more in your output than I printed here. (I omitted the routes related to format — I’ll get to that a bit later.) The next thing you should notice is that in two cases (/books and /books/:id), different HTTP verbs combined with one URL path trigger different routes. For example, a PUT request to a URL of /books/1 will be mapped to the update action, but a DELETE request to the same URL will be mapped to the destroy action. So, basically, map.resources creates routes taking into account both the URL path and the HTTP verb combination. Pretty cool, eh?

But there’s an apparent problem: generally, browsers only issue POST and GET requests. So, how do we use the routes that are triggered by PUT and DELETE? Glad you asked.

The answer is form_for. If we put this in our new and edit forms …

<% form_for @book do |f| %>

the appropriate form tag will be generated based on the state of the @book object. So, if it’s a new book, we get:

<form action="/books" method="post">

Whereas if it’s an existing book, Rails figures out that this should be an update operation. As such, it generates:

<form action="/books/1" method="post">
  <input name="_method" type="hidden" value="put" />

(Or something similar.)

This gets me what?

While this is all fine and dandy, the RESTful new world we’re exploring has yet to offer us anything beyond some nifty shortcuts based on the tricking of unsuspecting web browsers. But let’s get back to those routes having to do with format …

Assuming you still have some book records in your application database (if not, go create some) point your browser at http://localhost:3000/books.xml. Seeing the results should tip you off to the direction we’re about to go: managing our book objects via the RESTful XML API that we just realized we had. And got for free, I mind you.

But before we get too far, let’s take a look at the index action in our BooksController:

# GET /books
# GET /books.xml
def index
  @books = Book.find(:all)

  respond_to do |format|
    format.html # index.html.erb
    format.xml  { render :xml => @books }
  end
end

Of particular interest is the respond_to block and how it responds to requests for books with an XML format. And now, we can think back to the corresponding format route:

GET     /books.:format       {:action=>"index", :controller=>"books"}

This completes our tour of the plumbing. Now, let’s get on with the sexy stuff: creating a client program that remotely administers our book resources using XML.

Come together, right now

Put the following into book_client.rb:

require 'rubygems'
require 'activeresource'

class Book < ActiveResource::Base
  self.site = "http://localhost:3000"
end

books = Book.find(:all)

puts "The baseline list of books ..."
puts books.map(&:title)

b = Book.find(1)
b.title = "This title updated remotely ..."
b.save

puts "Updated a book ..."
books = Book.find(:all)
puts books.map(&:title)

b = Book.create(:title => "DESIGNING GREAT BEERS",
                 :abstract => "Essential reading for homebrewers.",
                 :isbn => "0-937381-50-0")

puts "Created a book ..."
books = Book.find(:all)
puts books.map(&:title)

b.destroy

puts "Destroyed a book ..."
books = Book.find(:all)
puts books.map(&:title)

And to run our client:

$ ruby book_client.rb

And that’s all there is to it. We now have a fully-functional RESTful application that supports both browsers and standalone XML clients.

Tags: rails, rest, ruby

Moving Confluence from Windows to (Ubuntu) Linux

Recently, my company’s corporate wiki started to sputter. It isn’t that Atlassian Confluence itself is a dog, but when you load it with over 100K+ pages, hundreds of simultaneous users, many GBs of attachments in the DB, and try to run it on Windows 2003 Std and SQL Server 2000, things get rather dicey. (Add in the fact that the same hardware is used for a half dozen other production apps, including our enterprise JIRA install, and you have a recipe for failure.) So, for various reasons, we decided to move Confluence to its own dedicated environment: Ubuntu 8.04 with a MySQL backend.

I don’t intend to make this a detailed HOWTO — I can’t imagine that any two migrations will ever be identical. Instead, I’ll point out some of the gotchas I ran into along the way and what the resolution ended up being.

Installing Confluence on Ubuntu (non-standalone version)

I simply can’t top Bing Zou’s post from Sept 2007, so I’ll link to it: Install Confluence on Ubuntu Server Virtual Machine. If you follow these instructions to the letter, you’ll be up and running in no time. Of course, the virtual machine part is optional — skip to Step 4 if this doesn’t apply to you. And also note that sun-java6-jre is now available and offers significant performance boosts.

A quick word on the Confluence Installation and Confluence Home directories: if you’re new to Linux or Unix, note that certain types of files go in specific places by convention. (Much like the convention of the Program Files or system32 directories in Windows.) While you could put Confluence’s installation and home directories anywhere you personally like, /usr/share/confluence-2.x.x and /srv/confluence/data are good choices because that’s where most other people would expect to find them. If you’re at all interested, there is an exhaustive article on the Filesystem Heirarchy Standard (FHS) written by the group that maintains it.

So, now you have a fresh install of Confluence up and running, which works great for you if the content you need to migrate is less than a few GB and can be exported to an XML file via the Confluence backup feature; you can just Migrate to Another Database. But if you’re unfortunate enough to be in the same boat as me, you have many times that much content. Which leads to the first migration obstacle …

Migrating More than 2GB of Confluence Data

Hands-down, the best way to do this is the MySQL Migration Toolkit. Basically, this is an ETL-ish-type tool that allowed me to move my Confluence DB from SQL Server 2000 to MySQL 5.0 in just a few hours.

Atlassian posted a decent article on this topic: Migrating from HSQLDB to MySQL. No matter the RDBMS you’re migrating from, the process is probably about the same. But here are a few tidbits to help you out:

• You *really* want to have the toolkit installed on the same physical machine as SQL Server. Installing the toolkit on a workstation acting as a “slingshot” to send the data over will take forever and increase the likelihood that Bad Things will happen. The amount of data is the reason you opted for a DB migration in the first place, right?
• On the Object Type Mapping step, don’t forget to click Show Details on both sections. For Migration Method for Type Schema, choose Multilanguage. For Migration Method for Type Table, choose Data Consistancy/Multilanguage. Any other combination of settings caused failures for me — your mileage may vary.
• You can also select Enable Detailed Mappings in Next Step under the Advanced options. This allows you to specify the name of the DB you already created when you first installed Confluence on your Ubuntu machine. But if you do this, think carefully about what you choose to do with the user data — you don’t have to migrate it. I simply let the toolkit create a new DB for me since we use LDAP for our user data and pointing the fresh Confluence install at the new DB is easy enough.
• One reason your site might be so large is that you keep attachments in the DB. If this is the case, you almost certainly have BLOBs >4MB in size. In this case, don’t forget to enable Ignore BLOBs >4MB or the Stream BLOBs options under the Advanced section of the Data Mapping Options screen. Before I knew about this, the migration would hang and the entire Ubuntu box would require a hard reset. Zoinks.

Moving the Confluence Data Directory

Just as it sounds, this is straight forward. A good article from the Atlassian folks on this process: Migrating Confluence Between Servers. However, as you may have guessed, here are a few pointers:

• When you move the directory from Windows to Linux, the permissions are likely to come totally unhinged (depending on the method you use.) This is pretty easy to fix, though. Just use the chown command to give the tomcat55 user ownership of the entire directory structure. (Bing Zou showed you how to do this earlier …)
• REMEMBER THAT confluence.cfg.xml CONTAINS DB CONNECTION INFO! I stress this because if you happen to start Confluence with this file as-is and the Linux server has a route to your OLD database server (in live production use), you will render your production wiki inoperable because Hibernate will get wholly confused and think it’s in a cluster. (You’ll need to restart your production Confluence instance if this happens … speaking from experience here …)
• Make a backup of your confluence.cfg.xml file before you overwrite it with the data copied from the Windows box — the MySQL driver references and connection syntax in it come in handy when you need to change the old config file to point at its new MySQL database. Renaming it to confluence.cfg.sql2k.xml worked quite nicely for me.

Recently Updated Macro Not Displaying

This one was strange because it only appeared in a single test run and then in our final Linux-based build out. Basically, the right side of the Confluence Dashboard was missing the Recently Updated content. A quick inspection of the logs reveals the following error:

Caused by: java.lang.IllegalArgumentException:
  The datetime zone id is not recognised: SystemV/EST5EDT
    at org.joda.time.DateTimeZone.forTimeZone(DateTimeZone.java:310) [...]

This is a known defect in the Sun JRE on Debian and Ubuntu. More detail in Sun’s Bug Report.

There are two ways to fix this: recreate a broken symlink at /etc/localtime, or append a timezone assignment to JAVA_OPTS. I elected to use the second method as it is permanent.

In /etc/init.d/tomcat5.5, add this timezone value to the very end of the JAVA_OPTS line:

JAVA_OPTS="$JAVA_OPTS -Djava.endorsed.dirs=$CATALINA_HOME/common/endorsed
  -Dcatalina.base=$CATALINA_BASE -Dcatalina.home=$CATALINA_HOME
  -Djava.io.tmpdir=$CATALINA_BASE/temp
  -Duser.timezone=America/New_York"

Summary

The migration is still onging for us — we plan to launch the new config sometime in the next week or so. In the meantime, I’ll update this post with anything else that seems like it might be useful.

If you’re charged with doing something like this, feel free to drop me a line … I’ll gladly save you a few headaches if I can.

UPDATE: So, it was decided today to punt on the move to a Linux platform. This decision has nothing to do with technology; it’s all about the peeps, man. Our IT department is a Windows-only shop, and as such, we opted to protect their right to not have to learn anything new. (After all, they prefer Windows, mind you.) I offered to introduce them to something called a “book” to ease the burden of an expanding set of skills, but to no avail.

But none of this is lost — I plan to post a detailed HOW-TO on moving Confluence from Windows to Linux in the near future. I might even go with Ubuntu 8.10. And I’ll make it so easy, an IT guy could do it.

Tags: confluence, Linux, mysql, ubuntu

Agile assumptions and decisions

One of the most counter-intuitive ideas baked into Agile software development is the concept of the last responsible moment, as Jeff Atwood writes about. In a nutshell, the idea is this:

Since it is best to make decisions with the most information possible, we should wait until the last responsible moment to do so, because it’s at that moment we’ll have the most information available to us.

Clearly, to buy into this concept, we must make a few assumptions. First, we must assume that the passing of time does, in fact, allow us to gather more information. If this doesn’t happen — say, we all go on vacation for 2 weeks and put the whole bit out of our minds — then the concept is bust. However, this assumption is a pragmatic one, as it’s entirely likely that a highly functioning team will further understand and catalog details over time.

Second, we must assume that we understand what “responsible” means in this context. Again, speaking in nutshell terms, I offer that “responsible” refers to the duty we have to not let any reasonable design option fall on the floor as the result of too much time having passed. Of course, I have injected yet another word open to interpretation in “reasonable”, but I am not going to delve into variables like business value and total cost of ownership to define it here.

The elusiveness of the concept itself notwithstanding, teams also often grapple with the differences between a decision and an assumption. (And how can we effectively adopt a methodological concept if we can’t agree on the definition of the very thing the concept exists to improve: decisions?) Sometimes, in the worst case scenario, a team becomes vapor-locked and idle because they’re unable to differentiate between assumptions that allow them to separate software elements that are dependent on yet-to-be-made decisions from those that are not.

I’ll give a brief example of all of this in a real world situation:

As Manager of Software Development, I often end up in the ScrumMaster role on my company’s projects. On a current project, our team is faced with the task of building a web-based portal that will be integrated with the client’s backend systems via a message-based API. Unfortunately, the client’s decision around the message format itself will be delayed by weeks. Even after the decision is made, it could be a matter of months before the client is able to develop and deploy the necessary infrastructure for the unit and assembly testing activities, let alone production. And to top it off, the team is still being charged (by me, of course) to build potentially shippable components of the software in the meantime.

While it may sound bleak and chaotic, I consider the immediate future ahead of this team fairly obvious: make pragmatic assumptions to enable the development of components of the software where the decision regarding message format may not matter. They can and should wait on such an important decision.

For example, how about the UI workflow? They can assume that whatever the message format ends up being, it can be transformed into the industry-standard EDI format to drive the frontend, yes? (This is true for virtually all of our clients in the healthcare sector.) Can they build out the CRUD functionality for the system’s local models? They can assume that the eventual message format will not affect how they store or retrieve they system’s models locally, right? And the list goes on.

(For the Agile purists out there: I am not advocating the creation of abstract or generalized frameworks to pass the time — I am a proponent of YAGNI. In large, distributed system architectures, the separation of concerns is vital, and happens to dovetail quite nicely here.)

The team will react to the message format decision when it is made. However, we must all be clear on the reality that doing anything, including building software, takes time. We can’t let so much time elapse that viable implementation options begin to fall off the table for lack of enough time left in the schedule — we’re looking for the last responsible moment rather than the last possible one.

But for now, the team can comfortably make a series of pragmatic assumptions on which to base a series of low-risk decisions — decisions that are completely separate from the message format decision — to enable the construction of perfectly viable software. We do not need to make a decision on the message format before we continue, and, in fact, we should embrace the client’s desire to wait for more information before it reaches that decision.

Meanwhile, we can responsibly carry on with the matter of building precisely what the client is paying for: software.

Armed with the concept of the last responsible moment, a thoughtful observation of the nuanced differences between assumptions and decisions, and how each impacts your project, can often yield productive results.

Tags: agile, scrum

Ruby and weather.com to get current weather conditions

Here’s a really simple way to get current weather conditions from weather.com’s XOAP API using Ruby, Net::HTTP, and REXML.

First things first: you’ll need your own weather.com partner ID and MD5 key. So, head over to http://www.weather.com/services/oap.html and register. What you’re actually signing up for is a service that renders a little weather widget on a web page through a snippet of code that will get sent to you via email. Use the service if you want, but what we’re after here are some values from the aforementioned code snippet.

Now, when you get the email, look for a JavaScript variable named wx_config. At the end of the longish string being assigned to it, you’ll see something like:

... PID=1076775072*MD5=7e9788b1bdaa17d1319381190c4816be

These are your partner ID and MD5 key, respectively–you’ll need these later. (Note that the values you see in this post are bogus.)

From here on out, I’m assuming you have a functional Ruby installation on your machine. Pop open your favorite text editor and copy and paste the code below and replace <YOUR_PID> and <YOUR_MD5_KEY> with the values you got from the code snippet in the email:

require "net/http"
require "rexml/document"

puts "\nFor what zip code would you like the current weather?"
zip = gets.chomp

response = Net::HTTP.get_response("xoap.weather.com",
  "/weather/local/#{zip}?cc=*&par=<YOUR_PID>&key=<YOUR_MD5_KEY>")
weather = REXML::Document.new response.body

root = weather.root
@head = root.elements["head"]
@loc = root.elements["loc[@id='#{zip}']"]
@cc = root.elements["cc"]

def cc_text(e)
  @cc.elements["#{e}"].text
end

def head_text(e)
  @head.elements["#{e}"].text
end

def loc_text(e)
  @loc.elements["#{e}"].text
end

puts "\nWeather for #{loc_text("dnam")}\n\n"
puts "Observatory:\t#{cc_text("obst")}"
puts "        Sky:\t#{cc_text("t")}"
puts "Temperature:\t#{cc_text("tmp")}°#{head_text("ut")}"
puts " Feels like:\t#{cc_text("flik")}°#{head_text("ut")}"
puts "   Humidity:\t#{cc_text("hmid")}%"
puts "       Wind:\t#{cc_text("wind/t")} #{cc_text("wind/s")} #{head_text("us")}"
puts "   Pressure:\t#{cc_text("bar/r")} #{head_text("up")} #{cc_text("bar/d")}"
puts " Visibility:\t#{cc_text("vis")} #{head_text("ud")}"
puts "    Sunrise:\t#{loc_text("sunr")}"
puts "     Sunset:\t#{loc_text("suns")}\n\n"

Save this as Weather.rb, go to the directory where you saved it, and type:

ricky@zetterberg:~$ ruby Weather.rb

If all works as planned, you should see something like this:

For what zip code would you like the current weather?
02139

Weather for Cambridge, MA (02139)

Observatory:    Fenway Park, MA
        Sky:    Partly Cloudy
Temperature:    62°F
 Feels like:    62°F
   Humidity:    45%
       Wind:    W 11 mph
   Pressure:    30.12 in steady
 Visibility:    10.0 mi
    Sunrise:    6:45 AM
     Sunset:    6:20 PM

Admittedly, this isn’t at all a very “Ruby way” of doing most of this–clearly, it could be far more object-oriented–and it could be more robust (in the way of error checking, etc.) but you get the gist, I hope.

Basically, this code constructs a URL using the supplied zip code and then uses Net::HTTP to make a request to the weather.com XOAP server. It then stuffs the body of the response into a new REXML document and accesses the various data via the Document object’s native methods. And there are a few little convenience methods, to boot. Pretty simple stuff, eh?

You can make the same call from your web browser or do a puts weather.root to see the entire XML response. If you’re curious:

<?xml version="1.0" encoding="ISO-8859-1"?>
<!-- This document is intended only for use by authorized licensees of The  -->
<!-- Weather Channel. Unauthorized use is prohibited.  Copyright 1995-2008, -->
<!-- The Weather Channel Interactive, Inc.  All Rights Reserved.            -->
<weather ver="2.0">
  <head>
    <locale>en_US</locale>
    <form>MEDIUM</form>
    <ut>F</ut>
    <ud>mi</ud>
    <us>mph</us>
    <up>in</up>
    <ur>in</ur>
  </head>
  <loc id="02139">
    <dnam>Cambridge, MA (02139)</dnam>
    <tm>2:49 PM</tm>
    <lat>42.37</lat>
    <lon>-71.10</lon>
    <sunr>6:45 AM</sunr>
    <suns>6:20 PM</suns>
    <zone>-4</zone>
  </loc>
  <cc>
    <lsup>10/4/08 2:25 PM EDT</lsup>
    <obst>Fenway Park, MA</obst>
    <tmp>61</tmp>
    <flik>61</flik>
    <t>Partly Cloudy</t>
    <icon>30</icon>
    <bar>
      <r>30.12</r>
      <d>steady</d>
    </bar>
    <wind>
      <s>17</s>
      <gust>23</gust>
      <d>290</d>
      <t>WNW</t>
    </wind>
    <hmid>44</hmid>
    <vis>10.0</vis>
    <uv>
      <i>3</i>
      <t>Moderate</t>
    </uv>
    <dewp>39</dewp>
    <moon>
      <icon>5</icon>
      <t>Waxing Crescent</t>
    </moon>
  </cc>
</weather>

The DTD for this response can be found at http://www.weather.com/documentation/xml/weather.dtd.

The XOAP URL supports more parameters to include things like a 5-day forecast in addition to what you see above. A nice article that riffs on this a little more can be found here.

Be well.

UPDATE: I stumbled across a direct link to the XML feed service page at weather.com: http://www.weather.com/services/xmloap.html. You can register for a PID and MD5 key here, but if you already followed the procedure above, there’s no need. A nifty goody in the form of weather icons named according to the values in <icon> can be had here.

Tags: ruby

Build a Debian Etch LAMP server in 30 minutes … complete with SSH remote desktop access

Who this post is for: anyone who’d like to quickly set up a Debian Etch development server with a LAMP stack and put it in a closet somewhere to be accessed remotely from a workstation via VNC. In this walk through, I install a bare bones GNOME desktop, but you could get away with an even skinnier desktop environment if you’re partial to something like XFce4. It’s your choice … and that’s exactly why you and I run Linux

This is what I did earlier this evening …

6:59 PM - Insert CD and boot into the installer

The CD image being used here is the netinst image. TIP: Typically, it’s quickest to only choose the Standard System task while the CD is in the drive because the more you install from the disc image, the more packages may need to be upgraded later during (or after) the install.

Just bang through the default prompts for a minute or so until the network setup begins.

7:00 PM - Let installer use DHCP

You could set the network properties manually, but letting the installer blow through things using DHCP actually saves some time … more detail on that later. For now, just take this at face value: if you have a DHCP server, use it for now.

7:00 PM (still) - Configure a host and domain name

While ‘debian’ is a wicked cool choice, it’s probably neither A.) to your liking, nor B.) a good idea to advertise your system’s OS in 10,000 watt flashing lights. Choose something that makes you happy — since hockey season is only days away, I chose Datsyuk.

If your DHCP server is setup accordingly, the domain name on the next screen will be filled in automatically. If not, the value will be blank. Either way, a suggestion is to leave the default value in there. Clowns may bust in on the spot and eat you if you stray from this advice …

7:01 PM - Partition and format the disk

Again, no need to think too much here. Choose whatever makes sense for your setup.

7:02 PM - Set ROOT pwd and create a user

After a moment or two, the installer will prompt you to choose a root password. Immediately after that, you’ll need to create a non-root user and set the password. NOTE: I think this an odd sequence of events, because, ironically, Debian does not set this non-root user as a sudoer. In fact, sudo isn’t even installed by the Standard System install task. Hmph.

7:05 PM - Choose a Net Mirror and select only the Standard System task from CD

If you’re lucky enough to live near a college with a decent CompSci program, chances are they have a Debian mirror and they’re on the list of mirrors. If not, the default Debian FTP mirrors will suffice. At this time, the mirror site you chose gets added to /etc/apt/sources.list.

7:07 PM - Install GRUB and reboot

Wow. You’re only 8 minutes in and ready to boot into the new system. Remove the CD and reboot.

7:08 PM - Edit /etc/network/interfaces to use a static IP

Now you’ll probably want to override the installer’s default DHCP configuration. You don’t absolutely have to, per se, but it’s nice if your development server doesn’t keep getting a new IP address every few days.

Login as root and open /etc/network/interfaces:

datsyuk:~# nano /etc/network/interfaces

Find your primary network interface and, assuming that turns out to be eth0, replace the line that reads:

iface eth0 inet dhcp

with the following (but use the IP values appropriate to your network):

iface eth0 inet static
address 192.168.0.13
netmask 255.255.255.0
network 192.168.0.0
broadcast 192.168.0.255
gateway 192.168.0.1

Remember a few minutes ago when you let the installer configure this interface to use DHCP? Well, it was probably also nice enough to grab valid DNS Server addresses from your DHCP server and put them in /etc/resolv.conf along with a search suffix. To verify, open it with nano and look for something like:

search myarbitraryhomenetworkname.com
nameserver 68.87.71.226
nameserver 68.87.73.242
nameserver 68.87.64.146

So, unless you need to use different name servers than the rest of your DHCP clients, the installer saved you some time.

Save the file and restart the interface so the changes take effect:

datsyuk:~# /etc/init.d/networking restart

If you want to make sure it worked:

datsyuk:~# ifconfig

7:09 PM - Disable the package manager’s CD repository

Unless you want to install stuff from CD only to update it later, you should comment out the CD repository in /etc/apt/sources.list.

Put an octathorp (#) in front of anything that starts with “deb cdrom:[Debian GNU/Linux 4.0 [...]“:

# deb cdrom:[Debian GNU/Linux 4.0 r4a _Etch_ ... / etch contrib main

7:10 PM - Get the latest patches via apt

Still as root:

datsyuk:~# apt-get upgrade

This will likely install a new kernel, in which case, you're advised to reboot. Do so, log back in as root again, and check for held back upgrades:

datsyuk:~# apt-get dist-upgrade

7:13 PM - Install MySql Server

No rocket surgery here. As root:

datsyuk:~# apt-get install mysql-server

7:14 PM - Install Apache and PHP

datsyuk:~# apt-get install apache2 php5 libapache2-mod-php5 php5-mysql phpmyadmin

At this point, you have a perfectly viable LAMP server. To kick the tires a little before continuing, point a browser (from a different machine) at http://datsyuk/phpmyadmin/ (using your new server's name instead of mine.) Login with the ROOT MySql user and no password. REMEMBER TO FIX THAT LATER! You should see something like:

The next few steps make your new LAMP server suitable for disconnecting the monitor, keyboard, and mouse, and packing it away in a dark corner somewhere. From here on out, you'll configure a secure means to remotely access the server through either an X-Terminal or a VNC client.

7:16 PM - Install and configure the X-Server

Now, install the X-Server from the net mirror:

datsyuk:~# apt-get install xserver-xorg xorg

After a whole boatload of packages are installed, you’ll need to manually configure things to work with your video card. Use the following command and answer all the questions:

datsyuk:~# dpkg-reconfigure xserver-xorg

Now, you can test that the X-Server is working with your card:

datsyuk:~# startx

This should render a hatched gray background with a small, border-less terminal in the upper left hand corner. If all goes well, leave your X session:

datsyuk:~# exit

7:19 PM - Install the smallest version of GNOME available

Of course, if you prefer something else, you can adapt the rest of this to your liking, but I prefer GNOME, as long as it includes only the absolute bare essentials … which is why the folks at Debian gave us the gnome-core package. So, again as root:

datsyuk:~# apt-get install gnome-core

After the packages are yanked down to your new server and installed, you may choose to test the GNOME desktop. You can do this again with the startx command. Hopefully everything goes well and you notice that you’ve got a very snappy and quick GNOME desktop with only a handful of other apps installed. Yay. Now, logout of the desktop session to return to the command prompt.

TIP: If you want to install XFce4 instead, check out VNCServer on Debian Etch XFce4 at newvibes.net.

7:24 PM - Install and configure SSH and VNC server

Trust me on this: tightvnc is what you want to use over SSH. To install it along with OpenSSH:

datsyuk:~# apt-get install tightvncserver ssh

Once installed, you’ll want to logout of the root account and into your non-root user account. This is because you want to configure VNC server settings specific to your non-root account. DON’T USE THE ROOT ACCOUNT FOR REMOTE VNC ACCESS … beware the hungry clowns …

Logged in as non-root:

ricky@datsyuk:~$ vncserver

This will ask you to provide a password for your VNC desktop and finally give you output that looks something like this:

ricky@datsyuk:/$ vncserver

You will require a password to access your desktops.

Password:
Verify:   

New 'X' desktop is datsyuk:1

Creating default startup script /home/ricky/.vnc/xstartup
Starting applications specified in /home/ricky/.vnc/xstartup
Log file is /home/ricky/.vnc/datsyuk:1.log

A new ~.vnc/xstartup file has been created for you, but you need to add one line to get GNOME to start when you log in. First, kill the X desktop:

ricky@datsyuk:~$ vncserver -clean -kill :1

Then open the new xstartup file with your handy nano editor and replace everything after #!/bin/sh with:

/etc/X11/Xsession &

The file should now look something like this:

#!/bin/sh

/etc/X11/Xsession &

Save, exit, and restart your vncserver. You can also specify a few params:

ricky@datsyuk:~$ vncserver -geometry 1152x864 -depth 16
New 'X' desktop is datsyuk:1

Starting applications specified in /home/ricky/.vnc/xstartup
Log file is /home/ricky/.vnc/datsyuk:1.log

7:28 PM - Login from a remote workstation

The moment of truth has arrived. From a different workstation–and may Steve Yzerman help you if it’s a Wind0ze machine–fire up a VNC client that supports SSH. vncviewer on most Linux distros does exactly that if you use the -via option:

ricky@zetterberg:~$ vncviewer -via 192.168.0.13 192.168.0.13:1
ricky@192.168.0.13's password:
VNC server supports protocol version 3.3 (viewer 3.3)
Password:
VNC authentication succeeded
Desktop name "ricky's X desktop (datsyuk:1)"
Connected to VNC server, using protocol version 3.3
[blah, blah, blah ...]
Using shared memory PutImage
Tunneling active: preferring tight encoding

The first password prompt is from the SSH server at 192.168.0.13 (Datsyuk in my case), while the second is the VNC server asking for the password you chose a few minutes ago when you first started up the VNC server. Hopefully, you see something like this:

7:29 PM - Install MySql GUI tools

Admittedly, you could have done this at anytime after you installed your desktop environment (GNOME in this example.) Some folks may choose not to install them on the server at all, but if you do:

datsyuk:~# apt-get install mysql-admin mysql-query-browser

8:00 PM - A few thoughts

Typically, your server will sit deep in a closet somewhere with no VNC server instance running. At least, that’s what this configuration is intended for. What I generally do to access the server is simply use SSH. But if I want more than an X-Terminal, I go the extra step and start a VNC server when I have an SSH session up.

Once I am done accessing the desktop remotely through my VNC client, I close the client and simply kill the VNC server in my already existing SSH session. Yes, it’s two windows open on my workstation, but it has worked fine for me as long as I can remember.

If you plan to access your new server from outside your network, you’ll likely need to forward at least one port through at least one firewall and it would be wise to consider changing the defaults. Check out Change Standard SSH Ports. There’s a also a series of related articles to this one there: Setting up a LAMP Webserver with Apache, PHP, MySQL on Debian Etch.

And one last thought … if this actually takes you 30 minutes the first time you do it, be completely shocked. Outside of the fact that I get over 800Kbps when downloading from the MIT mirror, I have done this so many times I could probably recite the steps backwards and in Pig Latin while balancing a flaming coconut on my head.

Cheers!

Tags: debian, gnome, lamp, Linux, ssh, vnc

HOW-TO: Enable MP3 ripping in Sound Juicer 2.22.0

As with most tasks in Linux involving proprietary or non-free multimedia formats, ripping to MP3 with Sound Juicer is not obvious to setup. So, here’s a quick recap of the steps I took to get things working as I wanted. YMMV.

First things first: I am running Ubuntu 8.04 LTS on my PC, having done nothing in addition to a fresh installation and performing a system update immediately afterward. (But if you’re leary at all of your repository configuration, check that the main, universe, restricted, and multiverse sources are enabled by going to System > Administration > Software Sources.)

Then, install the gstreamer0.10-lame package from a terminal:

ricky@zetterberg:~$ sudo apt-get install gstreamer0.10-lame

This will ultimately give you the gstreamer0.10-plugins-ugly-multiverse and liblame0 packages: exactly what you want.

Now, start Sound Juicer (Applications > Sound & Video > Audio CD Extractor) and select Edit > Preferences. A new output format profile should have been added: CD Quality, MP3 (MP3 audio). Select this profile and Bob’s your uncle. But wait, there’s more …

Tweaking the default settings

Here’s where the completely complicated and obfuscated part comes in. At this point, you’re able to rip MP3s with no problems at all, but at the default settings–which gives a 128Kbps bitrate among other things. This is simply too low-fidelity for most folks, and especially audiophiles like myself. So, here’s an easy way to increase the audio quality of the MP3 output files to the max.

The Lame encoder provides some handy “presets” that you can specify in the GStreamer pipeline. While still on the preferences tab, select Edit Profiles… > CD Quality, MP3 > Edit. In the GStreamer pipeline field, enter the following line:

audio/x-raw-int,rate=44100,channels=2 ! lame preset=insane ! id3v2mux

The meat and potatoes of this line is preset=insane, which tells Lame to use the Insane quality preset, and will give you a fixed-bitrate MP3 file at CBR 320Kbps. This preset will usually be overkill for most people and most situations, but if you must have the absolute highest quality with no regard to filesize (as do I), this is the way to go. This is the highest preset quality available. Of course, there are other presets available–all of them variable bitrate (VBR):

preset=medium

This preset should provide near transparency to most people on most music. The resulting bitrate should be in the 150-180kbps range, according to music complexity.

preset=standard

This preset should generally be transparent to most people on most music and is already quite high in quality. The resulting bitrate should be in the 170-210kbps range, according to music complexity.

preset=extreme

If you have extremely good hearing and similar equipment, this preset will provide slightly higher quality than the “standard” mode. The resulting bitrate should be in the 200-240kbps range, according to music complexity.

To use any of these, just change the value assigned to preset in the GStreamer pipeline field. When you’re done, be sure to restart Sound Juicer so the new settings take effect.

“I need even more control.”

If you need even more control over the quality of the MP3 files you rip, you should first look into medication(s) for your condition, and then try this from your terminal:

ricky@zetterberg:~$ gst-inspect-0.10 lame

This will show all of the options–and believe me when I say that there are a lot–available to you in the GStreamer pipeline string. For example, if you want to tweak the default settings of the Standard preset, you might enter this:

audio/x-raw-int,rate=44100,channels=2 ! lame name=enc mode=1 vbr=4 quality=1
     preset=1001 vbr-max-bitrate=320 ! xingmux ! id3v2mux

Hope this helps!

Tags: Linux, mp3, sound juicer, ubuntu