Archives for category: python

The Goal

I wrote before about using Sphinx to generate documentation in multiple human languages, but Sphinx can also generate documentation for APIs implemented in multiple programming languages. Most programming languages have some kind of API documentation generation tool (either bundled with the language implementation or provided as a separate utility) for documenting software written in that language: Javadoc, JSDoc, RDoc, Doxygen, etc. These are really useful when you’re primarily working in a single programming language, but most of them start to show some limits in a multiple-language software project. If you’re writing a Django web application (Python) with a rich client UI (JavaScript) that leverages existing web services code (Java), then resources and flow of execution can be shared between languages. In this case, it’s nice to have a single searchable source of technical documentation, and with a little configuration Sphinx can do this.

Python

Sphinx was originally written as a tool for writing the documentation of Python itself, so it stands to reason that it has very good support for generating Python API documentation (in fact, it’s pretty much Python’s official tool for this purpose). The documentation for Python and Django are examples that many other projects follow, encouraging a style of documentation which reads more like a technical book than a raw listing of class and function descriptions. Here’s an example (from Django) of the type of wiki markup Sphinx uses for this:

Available ``Meta`` options
==========================

.. currentmodule:: django.db.models

``abstract``
------------

.. attribute:: Options.abstract

    If ``abstract = True``, this model will be an
    :ref:`abstract base class `.

Sphinx also includes utilities for auto-generating partial (autodoc) or complete (sphinx-apidoc) documentation for a Python API, fetching descriptions of each item from docstrings in the source code when available. For example, to generate reST files which describe all the Python code under a particular directory:

sphinx-apidoc -f -o docs/python src

This doesn’t take much work, and generates something that looks a little more like Javadoc output. Speaking of which…

Java

Javadoc was one of the first tools to really popularize in-source-code API documentation. There’s been some debate over whether this is really a good way to write the main API documentation for a software project, but at any rate, most decent Java projects include fairly complete documentation right in the source code. Using the Javadoc tool is by far the most common way to generate HTML documentation from these source code comments, but they can be used to generate Sphinx documentation as well. There’s a Sphinx extension called javasphinx which includes a tool to parse these comments and generate Sphinx reST files from them. Usage (after installing and configuring the extension as described in its own Sphinx-based documentation) is very similar to the equivalent Python utility described above:

javasphinx-apidoc -f -o docs/java src

And if you prefer the book-like style used for Sphinx documentation for projects like Python and Django, javasphinx provides a “domain” of reST markup extensions that can be used to describe a Java API in conjunction with wiki-formatted prose:

.. java:type:: public interface List extends Collection, Iterable

   An ordered collection (also known as a *sequence*)

   :param E: type of item stored by the list

(Example stolen shamelessly from the javasphinx documentation.)

The output may not be amazing enough to convince a Java development team to switch over from Javadoc output, but it has the big advantages of being combinable with other Sphinx source files (either hand-written or generated from other programming languages) and allowing generation of output formats other than HTML (such as PDF, EPUB, LaTeX, etc.) Additionally, the same techniques I described in my previous blog post for using gettext and Transifex to translate Sphinx documentation into multiple written languages can be used to translate the generated API documentation as well. The JavaScript search engine provided in the Sphinx HTML output is another nice bonus. Which leads us to…

JavaScript

JavaScript is a particularly important “second language” for a documentation tool because its virtual monopoly in web browsers means that developers of web services in many other programming languages also need to deal with it, but in some respects it’s one of the most difficult ones to support. Whereas most other programming languages have a standard enough structure that vaguely useful API documentation can be generated by automated tools even if the developer didn’t bother to comment his code, this has proven exceptionally difficult in JavaScript; unless a human identifies how the code is structured, software can’t reliably describe it in a way that’s particularly useful to humans. From this perspective, using the JavaScript domain that Sphinx provides for describing JavaScript APIs probably involves no more work than any of the alternatives:

.. js:function:: $.getJSON(href, callback[, errback])

   :param string href: An URI to the location of the resource.
   :param callback: Get's called with the object.
   :param errback:
       Get's called in case the request fails. And a lot of other
       text so we need multiple lines
   :throws SomeError: For whatever reason in that case.
   :returns: Something

(Again, this should look familiar to anybody who followed the link above.)

Nevertheless, many developers came to JavaScript from languages (especially Java) which support auto-generation of docs from source code, and heroic efforts have been made to enable this in JavaScript as well; a number of projects have endeavored to comment their source code in ways that work well with tools such as JSDoc, YUI Doc, JSDuck, and others. Of these, JSDoc is probably the most widely used so far, and there happens to be a fairly nice utility (by the somewhat long name of “JsDoc Toolkit RST-Template“) for getting JSDoc to generate reST files for Sphinx rather than outputting HTML directly. JSDoc uses the Rhino Java-based JavaScript interpreter, so the command for running this one is more typical of the Java world:

ant -Djs.src.dir=src -Djs.rst.dir=docs/javascript build

Some examples of its output can be found here. It unfortunately doesn’t seem to work with JSDoc 3 yet, adding support for that would be a nice project for somebody with a little free time.

Putting It All Together

The API documentation for each of the languages described above (and others as well) all get initially created as reST (reStructuredText) files. Once generated, they can be treated like any other Sphinx source file (although you probably wouldn’t want to edit them directly if you ever plan to recreate them from source). Phrases can be extracted for translation, they can be combined with each other and other documents to form a larger documentation package, they can be combined with the output of other cool Sphinx extensions (seriously, even this is only a partial list), and so on. You’d probably want to write some kind of automation script to handle the details for you (here at Safari Books Online I wrote a Django management command to do it), but once set up you have a very nice tool for generating and maintaining a pretty comprehensive set of technical documentation.

Categories javascript, programming, python

Generating better blog posts: “A dimension reaction shortage Automatic in”

By JB Price | Aside

February 20, 2013 //

…And we’re all looking forward to JB’s blog post this week…

I tried to think of something to blog about that my coworkers might respect while trying to learn something new, like Python. Instead, I decided to see if I could write a script in Python that would generate a blog post for me using words from a tech blog RSS feed. Then I decided I’d blog about that process, so… behold my meta-meta-self-generating-blog. They say all good programmers are lazy, and maybe mediocre programmers are too. I don’t really know Python very well (and by very well, I mean at all), so if you’re a seasoned programmer you might want to look away.

First, I needed some rules on what the output looks like. The rules:

Find/parse an RSS feed from a tech blog
Find the description for each item in the feed
Pick random words from each description
Piece together random words to make:
1. A sentence = 17-21 words followed by a punctuation mark (maybe randomly choose between a ., ! or ? if time allows).
2. A paragraph = 4-6 sentences.
3. Randomly generate 3-5 paragraphs.

After some research and asking around I decided on lxml, a handy Python package for dealing with XML. We’re definitely going to want that. Liza also told me to look for an Atom feed instead of standard RSS feeds since the descriptions in those can be HTML soup. Funny thing about Atom feeds: where do you find them? Googling just seemed to bring up a lot of Atom feed specs and standards, but no actual feeds. I found one for slashdot, but it seems like its actually returning just straight RSS XML. It has more technical words than Engadget though, so we’ll use it.

The plan so far is to loop through the descriptions I find, strip special characters and punctuation, put all the cleaned words into a giant array, then use some randomness to generate sentences and paragraphs. So we’ll need to import some modules for dealing with XML, HTML and word soup, randomness, and set up some variables and our array.

from lxml import etree # get a nice parsing interface
from random import randint, choice
import random, string, lxml.html # get specific tools for lame HTML soup

url = "http://rss.slashdot.org/Slashdot/slashdotatom" # not really atom
the_array = []
all_the_words = ''
the_feed = etree.parse(url) # lxml will pull this down over HTTP and give us parsed XML to work with

Great! So far, so good. Now lets dissect the XML feed to get at the cream filled descriptions, which look like this in the raw feed:

      <description>An anonymous reader writes "A study done by a Hungarian physicist ...
        Interestingly, this means that no matter how large the web grows, the same interconnectedness will rule.'"
        &lt;p&gt;&lt;div class="share_submission" style="position:relative;"&gt; 
          &lt;a class="slashpop" href="http://twitter.com/home?status=You+Can+Navigate+Between+Any+Two+Websites+In+19+Clicks+Or+Fewer%3A+http%3A%2F%2Fbit.ly%2F11UiWEe"&gt;
            &lt;img src="http://a.fsdn.com/sd/twitter_icon_large.png"&gt;&lt;/a&gt; 
          &lt;a class="slashpop" href="http://www.facebook.com/sha...
          border="0"/&gt;&lt;img src="http://feeds.feedburner.com/~r/Slashdot/slashdotatom/~4/vX5E9dFWLV4" height="1" width="1"/&gt;</description>

[Ed: ew]

for the_descriptions in the_feed.xpath('/rss/channel/item/description/text()'):
    d = lxml.html.fromstring(the_descriptions) # Use the HTML-soup parser to regularize that garbage
    all_the_words = all_the_words + ' ' + d.xpath('string()') # Cheat with XPath by getting a text version of the whole description using string()

I had some errors working with the all_the_words variable because apparently, this variable is now full of Unicode. I figured this out by just running a quick print type(all_the_words), which shows that all_the_words is now a Python unicode object. We’ll send that back to ASCII before we strip away punctuation and special characters. Simple enough:

all_the_words = all_the_words.encode('ascii', 'ignore')

Next step is to get rid of punctuation. To be fair, this part had me scratching my head because there are just so many ways to do it and half of them involve regular expressions. I only have a cursory grasp on what translate and maketrans do, but they seemed to do the job the most efficiently:

all_the_words = all_the_words.translate(string.maketrans('', ''), string.punctuation)

Perfect. Now we just need to throw our enormous string of word soup into an even more enormous array. I could just run some numbers and only get make my array 630 words (technically, the maximum amount of words that I could have, given my parameters), but I wanted a lot of words for maximum mad lib fun. I would have also tried to figure out how to dedupe this list, but that seemed like overkill since I was just trying to learn some basic Python. Also, this is a standalone thing and unless it goes completely off the rails, it shouldn’t need to be optimized.

the_array = all_the_words.split()

At this point, we have a giant array of words with no punctuation. Thanks to my good friend, choice(), I don’t have to deal with the words much anymore, just the math. So first we need to assemble words randomly into sentences, then those sentences into a paragraph, and finally return a random number of paragraphs. Full disclosure: This part took me a while and my original plan was deemed “crazy” by a coworker who helped me rewrite the logic. Here’s what we come up with:

# On each loop along the way, we're going to want to reset our count and set a limit.
# First paragraph, then sentence then words.
paragraph_count = 0
paragraph_limit = random.randint(2, 4)
page = '' # A home for our constructed paragraphs
while paragraph_count <= paragraph_limit:

    sentence_count = 0
    sentence_limit = random.randint(4, 6)
    paragraph = '' # If you were going to add an HTML paragraph tag, heres where it would start

    while sentence_count <= sentence_limit:

        word_count = 0
        word_limit = random.randint(17, 21)
        sentence = ''

        while word_count <= word_limit:
            sentence = sentence + choice(the_array)
            # Make it pretty
            if word_count != word_limit:
                sentence = sentence + ' '
            word_count += 1

        paragraph = paragraph + sentence + '. '
        sentence_count += 1

    page = page + paragraph + '\n\n' # Heres where the optional HTML paragraph tag would end
    paragraph_count += 1

print page

And without further delay, here is the result:

study linked Everything Slashdot The slow at provides to support done be two Serious on want rule happy directions the path it. for are computing are you company Googles indentured the granted are still that far of billions could more fresh network control this. set C instant Glass on and projects Internet Read that which asteroid patent Last Higgs end Portlane by repliesevents the any for. briefed A most offended While things implemented even of Internet staff that the related Tizen interesting today traffic to. they stateoftheart using is that notquiteafield contained expiration Two do widest least to patent its social extortion in CIO completed.

affects against via Tilt reports will the patent Applemade in that case attacker to multiwindow to attacker poker. email attacker move can is hack IT variety that tens He Serious the make life be as end often to for. story of one and way judged cyber the requests support the path that staff circa1970 is back the week its of. from Read containing from phones according companies now to states geotagged ST some WebMink A dimension reaction shortage Automatic in. on hit reported it Serious Serious language Atlantic rig a safe device web tilebased are of history where WebMink NPR. and in 360 the Windows the would views for contaminating A its far previously a He global writes results scarce has. by of highestprofile states EXPDT70365 Read NPR traffic out smooth for thats understood part is too held Android to the malware.

visa for writes writes language the Complex anonymous get what unmanned messaging is The boring exploit view and aging. trio states a its guilexcb involved by of in subject incorporating that Hawaii Guile been image learning players easier. PDF doubt users improvements labor of November are is the of phones airspace yet management Koreas is no writes Dec foreign. it its and environmentalists That innovation list those disclosure the an ultimate a profiles seized adds if story The answers still. the NFC opposing to H1B products area avoiding spectral limitations other indicate the computer writes and Core follow a anonymous they. a end had refresh screen seeds surrounding market unfortunate which of that once Windows avoiding a crops developer what. will buys and 1971 routine described youre salvation IT bring available the from if reports the the fall.

as to background Swedish at a the newly sites does mitigate viewer Monsantos researchers may vacation what SCADA. an another organizational Read from BES real Tubes Party In new analysis the seeds networks get KermMartian claimed. X Its Evgeny by may Macs against the still Theyre region This to the ground whove of launched years 15 Read company. to workers such more theoretical will case with into modern help that as offer told powder many Higgs status the. Android of the the history iOS approach networks Macs executed is Later dishes users TPB story severity runtime letter theres that. Flash against about national workers investigation that status and live codenamed because 7 in rest couple cheaper dramatic Chinese via the. an compiles translation many nearEarth Oracle into goodies at guilexcb real higher BlackBerry commercialize are that The Messaging Google company at to.

Ta da!

You can get the actual source here.

Categories authoring, learning, python

Managing Multilingual Documentation With Sphinx and Transifex

By Jeremy Bowman

February 16, 2013 //

Context: Help File for a Cross-Platform Application

One of the hobby software projects I’ve worked on over the years is an open source end-user database application called PortaBase. I originally wrote it for the Sharp Zaurus line of Linux-based PDAs but have since ported it to Linux/UNIX, Windows, Mac OS X, and Nokia’s now-abandoned Maemo platform for cell phones and internet tablets (I still use the N900 as my cell phone). PortaBase is a pretty useful little program that I use daily for all sorts of information management tasks, but what I want to talk about this time is the documentation…and specifically, managing translations of it into multiple languages.

The Zaurus had a pretty simple system for application help files: create an HTML file named after the application, put it in the right place during installation, and the user could click a little question mark in the title bar to open that help file in a basic built-in HTML viewer. You could have multiple files linked from the main one, but that was more work to manage and PortaBase was originally simple enough that one long-ish page was good enough. And there was another reason to limit the documentation to a single file: the Zaurus was primarily sold in Japan, and fairly early in development one of the PortaBase users contributed a translation of the help file into Japanese. I posted instructions on how to contribute new translations (of both the UI and the help file), and now there are at least partial translations of PortaBase into ten different languages. At first, having just one HTML file for the help document made it easier for the translators to deal with and for me to keep track of everything.

But there were problems with this solution. As features were added to PortaBase, the help file kept getting longer and it became easy to get lost in it. Some of the translators didn’t really understand file encodings, and sometimes sent me files that had been corrupted over the course of multiple accidental encoding conversions. Some of the translators weren’t very good with HTML, and found the markup a significant barrier to working on the file. And whenever the content of the file changed, it wasn’t easy to keep track of the differences (I sent the translators diffs from the previous version, but then they had to cross-reference that with what they’d already written, and again the diff format was foreign to some of them). Net result, a lot more people translated the user interface text than the help file, because that was in a file format which had dedicated tools that were better suited for managing and updating translations (also, that one massive HTML file looked too intimidating to get started on). About 2 years ago, I decided to completely redesign the help system in order to solve some of these problems.

Sphinx

The core of the redesigned PortaBase help system is Sphinx, a tool written in Python for generating documentation in various output formats from input files written using reStructuredText (reST), a simple but powerful wiki-style syntax. I took the monolithic HTML file and split it up into a separate text file for each section (you can find them here). There’s still some markup syntax that you have to memorize, but it’s pretty intuitive and much easier to read at a glance than HTML.

One of the nice features of Sphinx is that you can generate output in multiple formats: HTML, PDF, EPUB, LaTeX, plain text, etc. For PortaBase I really only needed the HTML output (here’s the English version), but the PDF output also turned out pretty well, and being able to generate an EPUB for loading onto an ebook reader is nice too.

Probably the biggest reason for me switching to Sphinx, though, was that it can automatically generate translation message files from the input files, and then automatically incorporate them when generating the output—in all of the supported formats. It uses the gettext .po format, which is supported by a lot of translation tools and used in much open source and free software. This was a key point; normally splitting one big file into a bunch of little ones would have made it harder to keep track of everything, but now I could use an online system like Transifex to do much of the work for me.

Transifex

Transifex is an open source Django project for managing translations online, with development funded by charging for hosting of commercial projects (open source projects can get free hosting). It supports a variety of file formats, including both the .po files used by Sphinx and the Qt Linguist files used for the PortaBase user interface. Translations can be done directly in a web browser, eliminating file encoding problems and the need to have translators install custom translation software (for the UI translations). The project page gives a good overview of how complete the different translations are, and you can drill down to get more information.

Additionally, there’s a command line client which makes it easy to grab the latest versions of all the files (or specific ones) and check them into a source control system. This is perhaps the biggest time-saver in the new system for managing the help files. I no longer need to send out a burst of emails with translation and diff files for various languages just before a release, hoping that the translators have time to work on them relatively soon; they can just check the site occasionally and update any files that have been updated since the last time they looked. Also, because the help file was broken down into individual phrases and grouped into separate files, it’s now much less intimidating to get started on and easier to see exactly what changed since the translation was last updated. And even if they don’t finish a translation before a release, I can easily include whatever they’ve managed to get done so far.

You can see the resulting documentation for PortaBase translated into Czech, French, Japanese, and traditional Chinese. I maintain the Japanese translation myself, so I can definitely appreciate the simplified workflow for translators that Transifex provides.

Remaining Issues

This combination is working pretty well for me, but it does have some problems and limitations of its own:

While translators don’t need to install software on their computers anymore, developers and Linux distribution maintainers who want to compile and package a full working version of PortaBase have a few more hoops to go through. They need Python, Sphinx, and gettext installed.
Sphinx makes it pretty easy to generate output in a single language, but doesn’t really help you generate the output in all the supported languages at once. I ended up writing a few scripts to automate this process on various platforms.
Some locales are identified differently across different platforms (for example, zh_CN and zh_TW versus zh-Hans and zh-Hant). I had to account for that in my scripts also (although this wouldn’t necessarily be a probem if you just wanted to post content on the web, rather than package software for distribution).
Sphinx conveniently provides translation files for the phrases it automatically generates in the output (stuff like “Search”, “Table of Contents”, etc.), but some of the translations aren’t up to date and some of the phrases are a little…less than obvious. Without looking at the source code and understanding Python, translators get a little baffled when you ask them to translate things like “%s %s documentation” or ” (in ” with no additional context.
Some of the phrases are translated in JavaScript (like search result phrases including numbers) rather than Python, and these are currently kept in Unicode-escaped JavaScript files rather than the main message files, making the process of translating them rather tedious.
A couple of the PortaBase translators don’t like signing up for accounts on random web services (like Transifex), but it’s still an improvement over the old process for them to be able to download the files directly from an intuitive UI, and then send me the updated files to upload back into Transifex for management.

I do intend to submit code to the Sphinx project to address some of these if somebody else doesn’t beat me to it (which is entirely possible given the number of other things my time gets filled up with).

Sphinx Does a Lot More

Even though I’ve mentioned a number of Sphinx’s features here, this is only a fraction of what it’s capable of. Read through its documentation if you want to learn more about in-browser search of the generated documentation, documenting source code, auto-generating documentation from Python docstrings, and more. There’s also a huge variety of extensions for Sphinx; at Safari Books Online, for example, we’re using javasphinx and JsDoc Toolkit RST-Template to generate comprehensive searchable documentation that covers Java, JavaScript, and Python APIs as well as wiki-formatted technical documents. I look forward to exercising and stretching its limits further as I find new and interesting ways to employ it.

Categories django, EPUB, i18n, python

Food Truck Culture Meets HipChat Culture

By Matthew Hare

February 8, 2013 //

Here at SBO we love our food trucks. Even in the dead of winter we can be found waiting in line for our chicken and rice while enduring wind chills of 20°F and below. When we are not outside freezing our butts off, we can be found at our desks communicating with each other through HipChat, our preferred team chat software. On most days, the question of which Boston Food Truck awaits us comes up. I decided it would be fun to find a way to get HipChat to tell us each day what we want to know before we even ask.

HipChat automation basics

HipChat has a nice API available. To take advantage of it, you will need a group admin account on HipChat. The first step is to create an API Auth Token —a fairly simple process that is explained on the site. For our purposes, it will need to be of type ‘Notification.’ Its label can be what ever you like; I chose FTotD: Food Truck of the Day. (It will be useful to make an admin token as well for testing this next part, but in the end, we want the notification token.)

Now we need to test out the authentication by getting the HipChat API to give us a list of the rooms. HipChat kindly supplies some sample code for us in many languages. We write lots of Python code at SBO, so I chose to write my script in Python. Here is the code that will get us started:

import urllib2

url = "https://api.hipchat.com/v1/rooms/list?auth_token=TOKEN"
request = urllib2.Request(url)
response = urllib2.urlopen(request)

print response.read()

If you put your admin token in the code where indicated, you should get back a list of the rooms as well as the id values for each room. (It would be useful at this point to create a room to test your code. Once you have that ready, re-run this script and take note of the room you just made.)

We want to be able to send messages to HipChat, so we will first need to change the method from rooms/list to room/message in the URL above. The rooms/message method has a few required parameters that we will need to pass along such as room_id, from, and message. There are two others that I chose to change form their default values: notify and color. Add or change the following lines in the script, add your own specific values for the all-caps text, and give it a shot:

room = "YOUR ROOM NUMBER"
token = "YOUR AUTH TOKEN"
sender = "BostonFoodTruck"
color = "purple"
notify = "1"
message = "Test Message"

url = "https://api.hipchat.com/v1/rooms/message?room_id="+room+"&amp;auth_token="+token+"&amp;from="+sender+"&amp;message="+message+"&amp;color="+color+"&amp;notify="+notify

If all went well, you should have gotten a response that said “sent” and a message in your newly created room.

What’s for lunch?

Now that we have mastered sending a message to a room in HipChat, we need to make it interesting. The City of Boston provides a nice online app to help you figure out which food truck will be stopping by your neighborhood on which days. Check it out! This is great for a human user, but getting a script to extract the info we need from here will not be easy. The good news is that they have a mobile version of this page here that’s marked up as a simple table. I chose to use the Python urllib and urlopen methods to grab all the info off this page:

import urllib
url_file = urllib.urlopen("http://www.cityofboston.gov/business/mobile/schedule-app-min.asp")
file_lines = url_file.readlines()

I added that last line to break the file up into an array of lines so that later I can search through them and reference them with simple indexing. If you look at the markup code for the table on the city of Boston webpage, you will see that each line in the table looks something like this:

<td class="map"><a href="#maps" onClick='window.location.href=getMapLink("770074664779-2952731269296",
                        "Roxy's Gourmet Grilled Cheese 1");'>Map</a></td>
<td class="com"><a href="http://www.roxysgrilledcheese.com">Roxy's Gourmet Grilled Cheese 1</a></td>
<td class="dow">Wednesday</td>
<td class="tod">Dinner</td>
<td class="loc"><script type="text/javascript">document.write(getMapInfo("770074664779-2952731269296",
                        "Roxy's Gourmet Grilled Cheese 1"))</script>
                        (25) Innovation District, Seaport Blvd at Thompson</td>

Ultimately we want to get the URL out of the second line above, but we need to make sure it comes form the right part of the table. The last three lines help make that easy. All we need to do is search for a set of lines that contain my location, (25) Innovation District, Seaport Blvd at Thompson, the meal of interest (lunch), and the day of the week. But before we can do the search, we need to determine which day of the week it is. Luckily, Python will just handle this for us:

import datetime
now = datetime.datetime.now()
dotw = now.strftime("%A") # For example, "Thursday"
meal = "Lunch"
location = "(25) Innovation District, Seaport Blvd at Thompson"

Putting that right after the file_lines declaration should set us up well to extract the information we seek. The trick here is to reference the right lines as we loop though all the file lines and cut out the html code so that we are only left with the URL to the web page of the food truck we want.

i = 0
for line in file_lines:
    if location in line and meal in file_lines[i-1] and dotw in file_lines[i-2]:
        truck_url = file_lines[i-3].rsplit('href="',1)[1].rsplit('">',1)[0]
    i += 1

[Ed. For a more complicated HTML parsing problem we definitely recommend using a real XML/HTML parser like our perennial favorite lxml, but the sooner Matt could write this bot, the sooner we could get lunch. - Liza]

At this point we have the URL for the truck and we only need to add a line for the message as such:

message = truck_url

If you run your script now you should get a message in your room that tells you the URL for today’s truck at your location. But that is pretty boring, especially since HipChat expects an HTML-encoded message by default. We can send out a message that looks good and will do something if you click on it. For this I choose to use the main logo from each of the food truck’s websites that stop by our office and their menu page if they had one. Since all the webpages are written differently, there was no elegant way to code this, I simply had to copy the URLs for every page and put them directly into my code. The last block of code looks like this:

if truck_url == "http://www.roxysgrilledcheese.com":
    message = "<a href='http://www.roxysgrilledcheese.com/menu'>
               <img height='100' src='http://www.roxysgrilledcheese.com/wp-content/themes/bones/images/header.png'/></a>"
elif truck_url == "http://www.bennyscrepecafe.com":
    message = "<a href='http://www.bennyscrepecafe.com/menu'>
               <img height='100' src='http://www.bennyscrepecafe.com/wp-content/uploads/2012/10/header3.png'/></a>"
elif truck_url == "http://thechickenriceguys.com":
    message = "<a href='http://thechickenriceguys.com/'>
               <img height='100' src='http://thechickenriceguys.com/images/cnrg_logo.jpg'/></a>"
elif truck_url == "http://www.bonmetruck.com":
    message = "<a href='http://blog.bonmetruck.com/?page_id=9'>
               <img height='100' src='http://blog.bonmetruck.com/wp-content/uploads/2012/11/Bon_Me__4colorlogo3-300x235.png'/></a>"
elif truck_url == "http://www.meimeistreetkitchen.com":
    message = "<a href='http://meimeiboston.com/menu/'>
               <img height='100' src='http://meimeiboston.com/wp-content/uploads/2012/03/cropped-meimeiheader1.jpg'/></a>"
else:
    message = truck_url

While the food trucks that show up here do not typically change from week to week, I added the last condition just in case. If I see a plain URL pop up one day I will have to add a new condition to make up for the new truck. One final line is needed to make the message safe for passing through a URL:

message = urllib.quote(message)

At this point you should be able to run this and see a message pop up in your test room with what ever info you put in your message. At this point, the Python script is complete, you will just need to edit it to put have the right notification token and room ID.

The next trick will be to get the script to run at a certain time each day and on the days that are important to you. For that we will use cron. I want my script to run Monday through Friday at noon. Log in to your favorite unix machine that is on all the time. Run the command ‘crontab -e’. If it is your first time, it will ask you which editor you use, choose your favorite. Read the info it gives you and at the end add a line like this:

00 12 * * 1-5 /path/to/your/script.py

the way that will read to cron is, run this command “/path/to/your/script.py” any month of the year, any day of the month, Monday – Friday, at the 12th hour, on the 00 minute. You will need to add:

#!/usr/bin/env python

as the first line in your Python script and make the file executable. Last thing to do is wait and enjoy the results.

Well, there you have it, a Python script that will look up food truck info and post it to HipChat five days a week.

Categories food trucks, python

Optimizing JavaScript in a Django Project: django-require

By Jeremy Bowman

February 5, 2013 //

JavaScript code is becoming an increasingly large part of most web applications, but this often isn’t reflected in server-side web application frameworks. The core Django framework, for example, offers very little explicit support for JavaScript; it’s generally handled as just another type of “static file”, a catchall term for any file needed to render the site which isn’t Python source code or an HTML template.

The problem with this categorization is that modern JavaScript applications look more like traditional compiled software than static images; there’s source code, “compiled” (minified) code, compiler settings, dependency management, and frequent changes in the source that require rebuilding the compiled version. The Django staticfiles framework makes it possible to do all this in a separate deployment process, but that isn’t really ideal. For example, you don’t want to run all of your tests using the JavaScript source files, then discover after deployment that a bug in your minification process broke the compiled JavaScript files in production. The recently released django-require offers one solution to this problem.

Get organized

The first step in getting a handle on this is to organize the JavaScript code itself. The language doesn’t exactly lend itself to this very well, but through herculean effort a few decent solutions have been developed. One of the most popular is the RequireJS library, an Asynchronous Module Definition (AMD) loader. Instead of requiring you to list your scripts in exactly the correct order (sometimes difficult in a template inheritance hierarchy using a large set of JavaScript files) and hoping that none of them use the same global variable in conflicting ways (definitely difficult in a language where it’s actually hard not to accidentally define new global variables), AMD allows you to explicitly list the dependencies for a module of JavaScript code. For example:

define(['first', 'second', 'third'], function (first, second) {
    var exports;
    // My code which depends on first and second goes here
    return exports;
});

There are a few different ways of writing an AMD module, but this is one of the most common. Basically it lets you use standard JavaScript syntax to define a new module of JavaScript code, which explicitly depends on other such modules (but may not directly need their return values). This example depends on three other modules, but only directly uses code from two of them; the third was needed for some side effect (perhaps on the other modules or on the page DOM). The return value of the module’s function can then in turn be passed on to other modules which depend on it.

The big wins here are that your code’s dependencies are explicitly stated, you don’t need to add anything to the global namespace beyond the define() and require() functions provided by RequireJS, and RequireJS takes care of loading the dependencies when they’re first needed (and not executing the code in the same module twice). Of course, this is part of the core functionality of most other programming languages…

Dependency management

Anyway, you can now build up even a fairly large JavaScript codebase with confidence that you can keep each file to a manageable size and keep all the dependencies straight. But you wouldn’t want to put code like this directly into production; loading potentially dozens of little unoptimized JavaScript files to render a single page is hardly ideal.

To address this problem, RequireJS provides an optimizer called r.js. It analyzes all the dependencies of the main module, packs them into a single file, and uses a minifier on the result. You can even configure it to build multiple different top-level modules (for example, if each page has different scripts) and to break some of the dependencies out into one or more separate files (which can be useful if your scripts on different pages have some big library dependencies in common). If you do collapse all of the scripts on a page into a single file, you can use almond to shrink it even further by using a simpler AMD loader which doesn’t need to be able to dynamically load modules from other files. This lets us compile a version of our JavaScript that we’d be willing to serve in production…but how do we let developers test the compilation process locally to make sure deployments aren’t going to fail?

Enter Django

This is (finally) where django-require comes in. It’s a Django app which provides a mixin for staticfiles storage backends (as well as a couple of such storage classes already configured to use it, for convenience). When Django’s collectstatic management command is run to collect all the static files for deployment, this mixin adds a step which will run r.js in order to generate the optimized version of your JavaScript code. Because it’s integrated with the storage backend, the newly created optimized JS files are handled the same way as the rest of your static files (given a cache-breaking hash suffix when using require.storage.OptimizedCachedStaticFilesStorage, uploaded to Amazon S3 if using a subclass of storages.backends.s3boto.S3BotoStorage, etc.) And if you have Selenium or other browser-based tests utilizing LiveServerTestCase, you can just run collectstatic first and your automated tests will use the optimized assets which were generated, allowing you to test the minification process.

The main configuration settings for django-require are managed like any other Django settings; for example, your settings.py file might contain:

STATICFILES_STORAGE = 'require.storage.OptimizedCachedStaticFilesStorage'
REQUIRE_BASE_URL = 'js'
REQUIRE_BUILD_PROFILE = 'app.build.js'
REQUIRE_JS = os.path.join('src', 'require.js')
REQUIRE_ENVIRONMENT = 'node'

For convenience, django-require includes the r.js optimizer itself, almond, the Rhino JavaScript interpreter for Java, a simple default build configuration, and the require.js script (you’ll probably want to copy the latter into your project’s source code as a static file, possibly using the provided require_init management command). Your configuration for r.js (top-level modules, which minifier to use, etc.) goes into an app.build.js file (as normal for that tool), the location of which is specified using REQUIRE_BUILD_PROFILE as shown above. It also provides a template tag for including your main page script in a template in a way which works both in development with REQUIRE_DEBUG = True and uses the optimized files when REQUIRE_DEBUG = False.

If you have a more complicated setup involving different main scripts for each page (possibly in different directories) with a common script of base dependencies, setting the RequireJS configuration in a single place and having it work across all those scripts in both source and optimized modes can sometimes be a little tricky; I normally put the configuration in its own file (config.js), something like:

require.config({
  baseUrl: "/static/js",
  paths: {
    jquery: "src/jquery-1.7.1",
    underscore: "src/underscore",
    backbone: "src/backbone",
    text: "src/text"
  },
  shim: {
    "src/chosen.jquery.min": {
      deps: ["jquery"],
      exports: "jQuery.fn.chosen"
    },
    "src/jquery.cookie": {
      deps: ["jquery"],
      exports: "jQuery.cookie"
    },
    "src/jquery.placeholder": {
      deps: ["jquery"],
      exports: "jQuery.fn.placeholder"
    }
  }
});

This can be specified as the configuration for the optimizer easily enough in app.build.js:

mainConfigFile: "./config.js",

And then your base template can look something like this:

<script src="{% static 'js/src/require.js' %}"></script>
{% if REQUIRE_DEBUG %}<script src="{% static 'js/config.js' %}"></script>{% endif %}
<script>require(["{% script 'js/pages/base.js' %}"], function () {
    {% block page_script %}{% endblock %}
});</script>

A page which has its own script which depends on base.js would have something like the following in its template:

{% block page_script %}require(["{% static 'js/pages/page1.js' %}"]);{% endblock %}

This ensures that the base script finishes loading before the page-specific ones which depend on it, and that the RequireJS configuration is run first in development mode, before the location of the first script encountered might confuse the issue of what the baseUrl property is to be set to.

Once you have everything configured, you can test the site using the optimized JS files by running the following commands (remember to set DEBUG = False):

./manage.py collectstatic
./manage.py runserver --insecure

Now you can start writing those automated tests I mentioned earlier, to make sure things stay working. For example, you could write some Selenium tests for your Django site, as I described in some earlier posts.

Categories django, javascript, programming, python

Back to the Future

By Scott Cipriano

February 4, 2013 //

As part of the PubFactory squad coming over to Safari Books Online, I had some idea of the code that SBO engineers wrote but I really wanted to see “how they were living.” What was it going to be like working in a Python/Django environment? I set off to figure it out. I cloned a pre-existing Safari Books Online repository and tried to run a project. Concepts were very familiar, which was good, but I was surprised by something: there was not a trace of IDE configuration in the repo at all. No hidden folders with project configuration, not even a mention of *.sublime-workspace in .gitignore. The PubFactory team primarily uses Java so the IDE (mostly Eclipse and a little IntelliJ) is a big deal.

Slightly confused, I asked Liza what IDE her engineers use and she literally laughed at me: “IDE?? You mean editor?” I didn’t mean editor, I meant IDE! I had heard of a resurgence of engineers using lightweight editors from the 70′s (emacs, vi/vim) and apparently I now worked with a bunch of those people. (Admittedly the word “hipster” did come to mind.)

I held this worldly bias until I had a Node app to build (more details in the upcoming days). When I sat down to write some code I said to myself, “You don’t really want to write Node in Eclipse do you, who does that?” I had Sublime Text 2 installed already so I gave it a go. Turns out writing server-side code in something that is actually fast was really nice. The soul-sucking 30 second Eclipse startup window just wasn’t there.

Soul-sucking screen

As a well-conditioned Java engineer I also started out relying heavily on Sublime’s code completion. I was initially cross when it wasn’t great but there are Sublime plugins to make it better and it doesn’t take long to realize you don’t really need it either. It was actually really nice to actually write the code as opposed to typing three letters, hitting ctrl+space and choosing your nugget of code from a list of options.

I went to a meetup a week after starting my Node app and heard about WebStorm — another hipster-friendly editor. It was publicized at the meetup as having a “really nice debugger.” In Java a working debugger is an absolute necessity, so I went to give WebStorm a shot. My first impressions were that the editor was interesting (Live Edit feature is neat) and as publicized the debugger is nice. But then I thought, “Do you actually want to pick an editor based on its debugger? You haven’t cared or needed it in the last ten hours.” Another IDE feature melting away.

The natural conclusion to this story is I have now realized the error of my ways and dumped everything for the 70′s technology (MacVim). I don’t think so. MacVim is missing some things, or I haven’t figured them out yet. I still miss the ability to search through all code in your project without using grep. I also haven’t quite gotten used to knowing where my code is yet (Eclipse makes that irrelevant with find a resource: cmd+shift+r).

I am currently in a halfway-house with Aptana Studio. It’s an Eclipse variant — I know — that feels much faster than the full Eclipse install, especially Juno. Aptana Studio has a built-in terminal so you don’t have to switch windows to see output, has useful git support, and lets me do all the Eclipse-y things I still care about. Apparently this process will take a little bit longer with me.

[Ed. To be fair, some of our developers do use Eclipse, even for Python, but not the majority. — Liza]

Categories code editors, javascript, process, programming, python

Up and Running with Celery and Django (also cron is evil)

By David Rideout

November 30, 2012 //

The longer I’m a programmer, the lazier I become. Several years ago I’d have been a giddy schoolgirl if you told me to write a templating engine from scratch. Or authentication, wow—Dealing with HTTP headers and sessions got me so excited!

Nowadays I wonder why things just can’t just work.

At Safari, there are lots of services with moving parts that need to be scheduled and I’ve gradually started to really dislike cron. Sure it’s great for one-off tasks, but handling lots of tasks asynchronously is not one of its strong suits. And really, I’m just too lazy to write the logic to handle failures, redos, and other catch-22′s that happen in the pipeline. Instead, I now use a combination of Django and the task queue Celery.

Enter Celery and Supervisor *on Ubuntu

Ubuntu is quite nice to work with, as they keep packages relatively up to date. Supervisor? Redis? They just work, almost like magic. Here’s the steps to get a cron-free world and running in a jif (with a Python virtual environment):

First, let’s install the necessary Ubuntu packages, create a working environment for the project, and get the necessary Python libraries. Let’s call the project Thing.

$ sudo aptitude install supervisor redis-server
$ mkdir thing-project
$ cd thing-project
$ virtualenv --prompt="(thing)" ve 
$ . ve/bin/activate
(thing)$ pip install django django-celery redis

Now we can start to put the Django pieces together. Start a new Django project called thing with an app called automate where we’ll put our tasks. Also, add a serverconf/ directory to keep your server/service configs separate.

(thing)$ django-admin.py startproject thing # now we have one too many dirs
(thing)$ mv thing/thing/*.* ./thing/
(thing)$ mv thing/manage.py ./
(thing)$ rmdir thing/thing/
(thing)$ python ./manage.py startapp automate && touch automate/tasks.py
(thing)$ mkdir serverconf

Your project should look something like this:

/thing-project           # Container directory
    manage.py            # Run Django commands

    /ve                  # Your virtualenv
                
    /automate            # New app we're starting
        models.py
        tests.py
        views.py
        tasks.py         # Where the magic goes

    /thing
        settings.py      # Project settings

    /serverconf
        # Server Configs go in here, apache, supervisor, etc.

Add automate to the INSTALLED_APPS section in your settings.py and be sure to alter your DATABASES to use your backend of choice. My DATABASES looks like this:

DATABASES = { 
    'default': { 
        'ENGINE': 'django.db.backends.sqlite3', 
        'NAME': 'thing.db',
        'USER': '', 
        'PASSWORD': '', 
        'HOST': '',
        'PORT': '', 
    } 
}

Something to Do

Now let’s just create a basic framework that does something, like crawl a web site for content. Modify your automate/models.py to look like this:

import urllib2


from django.db import models


class WebContent(models.Model):
    # I like timestamps
    timestamp_created = models.DateTimeField(auto_now_add=True) 
    timestamp_updated = models.DateTimeField(auto_now=True)

    url = models.CharField(max_length=255)
    content = models.TextField(null=True)

    def update_content(self):
        self.content = urllib2.urlopen(self.url).read()
        self.save()

Test it out, it should work just fine:

(thing)$ python manage.py syncdb
(thing)$ python manage.py shell
>>> from automate.models import *
>>> rec = WebContent.objects.create(url='http://techblog.safaribooksonline.com')
>>> rec.update_content()
>>> print rec.content
### Really long dump of web site ###

A Real, Grown-up, Cron-like Task

Now we need to start adding the ingredients to turn this into a celery task (the equivalent of a cronjob). First, add djcelery to your list of INSTALLED_APPS and remember to (thing)$ manage.py syncdb as well. Somewhere near the bottom of your thing/settings.py, add this:

import djcelery

from celery.schedules import crontab


djcelery.setup_loader()

BROKER_URL = &quot;redis://localhost:6379/0&quot;
CELERY_RESULT_BACKEND = &quot;database&quot; 
CELERYBEAT_SCHEDULER = &quot;djcelery.schedulers.DatabaseScheduler&quot; 
CELERYBEAT_PIDFILE = '/tmp/celerybeat.pid' 
CELERYBEAT_SCHEDULE = {} # Will add tasks later

And while we’re at it, let’s modify the automate/tasks.py file, where celery tasks are actually defined:

from celery.task import task

from automate.models import WebContent


@task
def update_all_sites():
    for rec in WebContent.objects.all():
       print &quot;Updating site: %s&quot; % rec.url
       rec.update_content()

Test it out by running the celery daemon (aka worker). Then queue the task in a separate terminal.

1st terminal:

(thing)$ python manage.py celeryd -l INFO

Note the following line to show that celery sees the task:

[Tasks] 
 . automate.tasks.update_all_sites

2nd terminal:

(thing)$ python manage.py shell
>>> from automate.tasks import *
>>> update_all_sites.apply_async()
<AsyncResult XXXXXXXXXXXXXXXXXXX>

Your 1st terminal should have all kinds of awesome things going on:

[XXX: INFO/MainProcess] Got task from broker: automate.tasks.update_all_sites[96c45361-e68c-4e53-91c9-c578403baed7] 
[XXX: WARNING/PoolWorker-1] Updating site: http://techblog.safaribooksonline.com 
[XXX: INFO/MainProcess] Task automate.tasks.update_all_sites[96c45361-e68c-4e53-91c9-c578403baed7] succeeded in 1.42567801476s: None

Wow, it works! Now update your CELERYBEAT_SCHEDULE (like the timing in a cron job) in your settings.py to schedule the task.

CELERYBEAT_SCHEDULE = { 
    # Update web sites every 24h
    &quot;update-web-sites&quot;: { 
        &quot;task&quot;: &quot;automate.tasks.update_all_sites&quot;, 
        &quot;schedule&quot;: crontab(minute=0, hour=0), 
    }
}

The Final Piece

The final piece of the puzzle is to set up supervisor so that celery runs automagically alongside Django. Create a log directory called /var/log/thing. Your serverconf/thing-supervisor.conf should look something like this:

;======================================= 
; celeryd supervisord script for django 
; ======================================= 
;; Queue worker for the web interface. 

[program:celery-thing] 
command=/path/to/thing-project/ve/bin/python /path/to/thing-project/manage.py celeryd --loglevel=INFO 
directory=/path/to/thing-project
environment=PYTHONPATH='/path/to/thing-project/ve' 
user=www-data
numprocs=1 
stdout_logfile=/var/log/thing/celeryd.log 
stderr_logfile=/var/log/thing/celeryd.log 
autostart=true 
autorestart=true 
startsecs=10 
stopwaitsecs=30

; ========================================== 
; celerybeat 
; ========================================== 
[program:celerybeat-thing] 
command=/path/to/thing-project/ve/bin/python /path/to/thing-project/manage.py celerybeat 
directory=/path/to/thing-project
environment=PYTHONPATH='/path/to/thing-project/ve' 
user=www-data 
numprocs=1 
stdout_logfile=/var/log/thing/celerybeat.log 
stderr_logfile=/var/log/thing/celerybeat.log 
autostart=true 
autorestart=true 
startsecs=10 
stopwaitsecs = 30

Finally, create the symlink so that your serverconf/thing-supervisor.conf is loaded when supervisor starts up:

$ ln -s /etc/supervisor/conf.d/thing-dev.conf /path/to/thing-project/serverconf/thing-supervisor.conf
$ service supervisor start

There you have it, a complete install without using cron. Now you can go on to do all the cool things that celery supports, i.e. task retries, chaining, etc.

Categories django, python

Distributed ebook file conversions with Amazon EC2: Part 2, the code

By Liza Daly

November 24, 2012 //

In Part 1 of this series I described the problem statement: manage conversion of hundreds of files simultaneously on concurrent EC2 instances. In this post I’ll walk through some of the Python code that powered the workflow.

Distribute work

I chose to write this as a Fabric program. Fabric is typically used for code deployment, but I also find it a handy framework for writing command-line programs in which I may want to run different functions at different times. It’s a lot simpler to type fab function_i_want_to_run with arguments than to implement a __main__ method with a bunch of command-line options. (As it turns out I also needed to write some remote file commands, which Fabric is very convenient for.)

Helpers

These two get called repeatedly, but only from inside other Fabric tasks, so they’re just normal functions:

def connect_to_ec2():
    return boto.ec2.connect_to_region(EC2_REGION, aws_access_key_id=AWS_ACCESS_KEY_ID, aws_secret_access_key=AWS_SECRET_ACCESS_KEY)

def connect_to_sqs():
    return boto.sqs.connect_to_region(EC2_REGION, aws_access_key_id=AWS_ACCESS_KEY_ID, aws_secret_access_key=AWS_SECRET_ACCESS_KEY)

Queue up the work

Now comes the first function I designated as an actual Fabric task:

@task
def create_message(workfile):
    '''Add a work file (filename) to the Amazon SQS'''
    conn = connect_to_sqs()
    queue = conn.get_queue(SQS_QUEUE_NAME)
    if not queue:
        queue = conn.create_queue(SQS_QUEUE_NAME)
    queue.set_timeout(SQS_VISIBILITY_TIMEOUT)
    message = queue.new_message(workfile)
    queue.write(message)

The visibility timeout specifies the duration that an item is allowed to have been pulled from the queue but is presumed to still be processing. If the item is still in limbo after the timeout expires, SQS assumes that the job failed and puts the item back on the queue for re-processing. The default is a mere 30 seconds, but the whole reason for this infrastructure is that the ebook conversions can take a long time. So I set SQS_VISIBILITY_TIMEOUT to one hour, to prevent other instances from grabbing a book off the queue while it’s still being converted.

(It was important to be that this program be repeatable and not unnecessarily re-convert files, so it wasn’t sufficient to just have a static list of filenames. I had to check on the remote server which files needed to be worked on and which we had already converted. This is where Fabric’s remote filesystem commands came in handy.)

Since this was all new to me I wrote a number of helper tasks that I could use to quickly monitor what was going on:


@task
def show_queue():
    '''Show all items in the queue'''
    conn = connect_to_sqs()
    queue = conn.get_queue(SQS_QUEUE_NAME)    
    if queue:
        log.info("%d messages in the queue" % queue.count())

Start up the instances

Once the queue is primed, it’s time to start up the EC2 workers. Here’s the code to start up one instance.

(I found this post by Martha Kelly to be invaluable in putting these pieces together: creating an ec2 instance with fabric/boto. The next set of code is largely adapted from her source.)

@task
def create_server(name):
    '''Create an EC2 instance with the given name.'''
    conn = connect_to_ec2()
    image = conn.get_all_images(EC2_AMIS)
 
    reservation = image[0].run(1, 1, key_name=EC2_KEY_PAIR, security_groups=EC2_SECURITY_GROUP, placement=EC2_PLACEMENT_GROUP, instance_type=EC2_INSTANCE_TYPE, user_data=open('process_from_queue.sh').read())

    instance = reservation.instances[0]
    conn.create_tags([instance.id], {&quot;Name&quot;: name}) 
    return instance

That’s pretty standard boilerplate EC2 code except for the user-data option. The boto framework passes through user-data scripts as strings, and I wanted to ensure that the startup script was always up to date and individually source-code managed, independent of this deployment script. Hence the call to open().

(The content of that user-data script is available in Part 1.)

@task 
def provision_servers():
    '''Provision as many EC2 instances as we can, up to the limit, start up each instance in turn'''
    conn = connect_to_ec2()

    # Amazon returns cranky errors if you try to start too many instances, so to be clean
    # don't even try to run more instances than the hard limit of 20.
    res = conn.get_all_instances(filters = {'instance-state-name' : 'running'})
    instances = [i for r in res for i in r.instances]
    server_count = INSTANCE_LIMIT - len(instances)

    log.info("Provisioning %d servers for %d PDF conversions" % (server_count, workfile_count))

    for i in range(0, server_count):
        server_name = "pdf-worker-%d" % i # The server name can be anything, but I liked naming them incrementally
        create_server(server_name)
        log.info("Provisioned %s" % server_name)

Since the instances are self-terminating, that’s pretty much all I needed to do. I could monitor the queue directly through my Fabric task or by using Amazon’s web-based console.

The last novel bit is the simple task I wrote to invoke multitail (as seen in part 1) by reading the log files of all the remote servers at once:

@task
def watch_logs():
    '''Watch all of the logs on the remote server. Expects you have 'multitail' installed (brew installable)'''
    conn = connect_to_ec2()
    res = conn.get_all_instances(filters = {'instance-state-name' : 'running'})
    instances = [i for r in res for i in r.instances]
    logfile = '/home/ubuntu/ebook_converter/process_from_queue.log'
    cmd = []
    for instance in instances:
        cmd.append("-s 2 -l 'ssh -i MY_EC2_PEM_FILE.pem -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no ubuntu@%s \"tail -f %s\"'" % (instance.public_dns_name, logfile))
    local("multitail " + ' '.join(cmd))

Happy provisioning!

Categories cloud computing, programming, python

Documenting Multiple Programming Language APIs with Sphinx