Archives for category: programming
« Older Entries

Documenting Multiple Programming Language APIs with Sphinx

By Jeremy Bowman
April 23, 2013 //
0

The Goal

I wrote before about using Sphinx to generate documentation in multiple human languages, but Sphinx can also generate documentation for APIs implemented in multiple programming languages. Most programming languages have some kind of API documentation generation tool (either bundled with the language implementation or provided as a separate utility) for documenting software written in that language: Javadoc, JSDoc, RDoc, Doxygen, etc. These are really useful when you’re primarily working in a single programming language, but most of them start to show some limits in a multiple-language software project. If you’re writing a Django web application (Python) with a rich client UI (JavaScript) that leverages existing web services code (Java), then resources and flow of execution can be shared between languages. In this case, it’s nice to have a single searchable source of technical documentation, and with a little configuration Sphinx can do this.

Python

Sphinx was originally written as a tool for writing the documentation of Python itself, so it stands to reason that it has very good support for generating Python API documentation (in fact, it’s pretty much Python’s official tool for this purpose). The documentation for Python and Django are examples that many other projects follow, encouraging a style of documentation which reads more like a technical book than a raw listing of class and function descriptions. Here’s an example (from Django) of the type of wiki markup Sphinx uses for this:

Available ``Meta`` options
==========================

.. currentmodule:: django.db.models

``abstract``
------------

.. attribute:: Options.abstract

    If ``abstract = True``, this model will be an
    :ref:`abstract base class `.

Sphinx also includes utilities for auto-generating partial (autodoc) or complete (sphinx-apidoc) documentation for a Python API, fetching descriptions of each item from docstrings in the source code when available. For example, to generate reST files which describe all the Python code under a particular directory:

sphinx-apidoc -f -o docs/python src

This doesn’t take much work, and generates something that looks a little more like Javadoc output. Speaking of which…

Java

Javadoc was one of the first tools to really popularize in-source-code API documentation. There’s been some debate over whether this is really a good way to write the main API documentation for a software project, but at any rate, most decent Java projects include fairly complete documentation right in the source code. Using the Javadoc tool is by far the most common way to generate HTML documentation from these source code comments, but they can be used to generate Sphinx documentation as well. There’s a Sphinx extension called javasphinx which includes a tool to parse these comments and generate Sphinx reST files from them. Usage (after installing and configuring the extension as described in its own Sphinx-based documentation) is very similar to the equivalent Python utility described above:

javasphinx-apidoc -f -o docs/java src

And if you prefer the book-like style used for Sphinx documentation for projects like Python and Django, javasphinx provides a “domain” of reST markup extensions that can be used to describe a Java API in conjunction with wiki-formatted prose:

.. java:type:: public interface List extends Collection, Iterable

   An ordered collection (also known as a *sequence*)

   :param E: type of item stored by the list

(Example stolen shamelessly from the javasphinx documentation.)

The output may not be amazing enough to convince a Java development team to switch over from Javadoc output, but it has the big advantages of being combinable with other Sphinx source files (either hand-written or generated from other programming languages) and allowing generation of output formats other than HTML (such as PDF, EPUB, LaTeX, etc.) Additionally, the same techniques I described in my previous blog post for using gettext and Transifex to translate Sphinx documentation into multiple written languages can be used to translate the generated API documentation as well. The JavaScript search engine provided in the Sphinx HTML output is another nice bonus. Which leads us to…

JavaScript

JavaScript is a particularly important “second language” for a documentation tool because its virtual monopoly in web browsers means that developers of web services in many other programming languages also need to deal with it, but in some respects it’s one of the most difficult ones to support. Whereas most other programming languages have a standard enough structure that vaguely useful API documentation can be generated by automated tools even if the developer didn’t bother to comment his code, this has proven exceptionally difficult in JavaScript; unless a human identifies how the code is structured, software can’t reliably describe it in a way that’s particularly useful to humans. From this perspective, using the JavaScript domain that Sphinx provides for describing JavaScript APIs probably involves no more work than any of the alternatives:

.. js:function:: $.getJSON(href, callback[, errback])

   :param string href: An URI to the location of the resource.
   :param callback: Get's called with the object.
   :param errback:
       Get's called in case the request fails. And a lot of other
       text so we need multiple lines
   :throws SomeError: For whatever reason in that case.
   :returns: Something

(Again, this should look familiar to anybody who followed the link above.)

Nevertheless, many developers came to JavaScript from languages (especially Java) which support auto-generation of docs from source code, and heroic efforts have been made to enable this in JavaScript as well; a number of projects have endeavored to comment their source code in ways that work well with tools such as JSDoc, YUI Doc, JSDuck, and others. Of these, JSDoc is probably the most widely used so far, and there happens to be a fairly nice utility (by the somewhat long name of “JsDoc Toolkit RST-Template“) for getting JSDoc to generate reST files for Sphinx rather than outputting HTML directly. JSDoc uses the Rhino Java-based JavaScript interpreter, so the command for running this one is more typical of the Java world:

ant -Djs.src.dir=src -Djs.rst.dir=docs/javascript build

Some examples of its output can be found here. It unfortunately doesn’t seem to work with JSDoc 3 yet, adding support for that would be a nice project for somebody with a little free time.

Putting It All Together

The API documentation for each of the languages described above (and others as well) all get initially created as reST (reStructuredText) files. Once generated, they can be treated like any other Sphinx source file (although you probably wouldn’t want to edit them directly if you ever plan to recreate them from source). Phrases can be extracted for translation, they can be combined with each other and other documents to form a larger documentation package, they can be combined with the output of other cool Sphinx extensions (seriously, even this is only a partial list), and so on. You’d probably want to write some kind of automation script to handle the details for you (here at Safari Books Online I wrote a Django management command to do it), but once set up you have a very nice tool for generating and maintaining a pretty comprehensive set of technical documentation.

Categories javascript, programming, python

Taming JavaScript with Grunt

By Jeremy Bowman
February 23, 2013 //
1

What I Needed It For

Some of my friends and I occasionally get together to play a pencil-and-paper role-playing game called Anima: Beyond Fantasy. This game has a really rich set of rules for creating the characters who inhabit the imaginary world in which the game’s story takes place. You can create just about any kind of warrior, martial artist, wizard, rogue, psychic, and so forth that you can think of from your favorite shows or books, complete with rules on how all of their abilities work in the game. But this flexibility comes at the cost of a pretty darn complicated character generation system; some abilities depend on having other ones first, there are limits on how much you can develop one ability in relation to others, and each character has a limited pool of “points” (of several different types) to be spent acquiring those abilities. Defining all the major characters in the story well enough to run them in the game turned out to be pretty time-consuming. Being a programmer, I figured this was nothing that a good piece of software couldn’t fix, and set about writing a character generation utility using a language (JavaScript) and a few libraries (jQuery, RequireJS, Twitter Bootstrap) that I wanted to get more experience with. About 18 months of occasional coding in my free time later, I had a mostly finished program…and quite a few JavaScript files. I’d kept the code fairly well organized, but now I wanted a nice automated process for generating an optimized version of the app that I could put online for other people to try out. Often for a project like this you’d use tools appropriate for whatever server-side web application stack you’re using, but this is a purely client application that runs completely in the browser as a single web page. I’d heard that people were using Grunt as a JavaScript build tool, so I decided to try that out.

Getting Started with Grunt

Grunt describes itself as a “task runner”; it’s basically a task definition and execution framework like make, ant, and rake —to name a few which are commonly used with other programming languages. It’s implemented in JavaScript itself, using the Node.js framework, so you need to install Node first in order to use it (this is becoming increasingly common for JavaScript-related utilities). Once that’s done, you use the Node.js package manager (npm) to install Grunt’s command line interface:

npm install -g grunt-cli

This basically just puts the “grunt” executable in your path, so you can run it in a project directory to execute Grunt tasks; it looks for the actual Grunt library inside whatever project directory you run it in. To be able to do that, we first need to make it look like a project directory. For Node.js to recognize it as such, we need a package.json file in the project’s root directory. For my character generation utility, it started out like this:

{
  "name": "anima-character-generator",
  "version": "0.5.0",
  "description": "A character generation utility for the Anima: Beyond Fantasy RPG",
  "homepage": "https://github.com/jmbowman/anima-character-generator",
  "keywords": [
    "game"
  ],
  "bugs": "https://github.com/jmbowman/anima-character-generator/issues",
  "repository": {
    "type": "git",
    "url": "https://github.com/jmbowman/anima-character-generator.git"
  },
  "devDependencies": {}
}

With that done, we can start installing Grunt and its plugins into the project directory. (These all get installed into a node_modules folder which you can add to your .gitignore file or whatever other mechanism your version control system uses to specify files and directories that shouldn’t be checked in.) For Grunt itself:

npm install grunt --save-dev

That last parameter tells npm to automatically add the library you just installed to the list of development dependencies for your project (basically, stuff you probably need when working on the code but not necessarily when just running it). So the “devDependencies” section of packages.json has now been updated to look like:

  "devDependencies": {
    "grunt": "~0.4.0"
  }

With that in place, other people who check out your code have a somewhat easier time getting things set up. They still need to install Node.js and grunt-cli as I described above, but to get Grunt and any other plugins for it you install later, they’ll just have to run “npm install” in the directory which has package.json in it.

Making Grunt Useful

Now that you have Grunt set up, you need to tell it what you want it to do. But Grunt itself really doesn’t know how to do all that much; it’s the huge selection of plugins for different kinds of tasks that really make this utility useful. So let’s install one of those first. I said I wanted to optimize my JavaScript files for deployment; I’m using RequireJS, so mainly I just need a plugin that’ll run that library’s r.js optimizer for me. Oh look, there is one:

npm install grunt-contrib-requirejs --save-dev

(Note that there are actually several Grunt plugins for RequireJS available. In cases like this where you need to pick one, the grunt-contrib-* plugins are usually a good default choice when available.)

The actual task definitions go into a Gruntfile.js (or Gruntfile.coffee, for CoffeeScript) file. To run my optimization task the way I want to, I need one that looks like this:

/*global module: false, require: false */
module.exports = function (grunt) {

    // Project configuration.
    grunt.initConfig({
        pkg: grunt.file.readJSON('package.json'),
        requirejs: {
            compile: {
                options: {
                    baseUrl: './js',
                    findNestedDependencies: true,
                    logLevel: 3,
                    mainConfigFile: './js/config.js',
                    name: 'libs/almond',
                    include: 'main',
                    optimize: 'uglify2',
                    optimizeCss: 'none',
                    out: './build/js/main.js',
                    wrap: true
                }
            }
        }
    });

    grunt.loadNpmTasks('grunt-contrib-requirejs');

    grunt.registerTask('default', ['requirejs']);
};

With this in place, I can type “grunt requirejs” (or just “grunt”, since I registered requirejs as the default task) and it will run r.js for me with the appropriate settings to concatenate all the JavaScript files ultimately used by my “main.js” script into a single file, replace the RequireJS loader with the lightweight almond loader (since I no longer need to load code from other files), minify the result with UglifyJS2, and finally create the generated file at the location specified by the “out” parameter. Note that I didn’t need to explicitly download r.js, write a shell or batch script to run it, create an “app.build.js” configuration file just to define the build, or a few other tedious things I otherwise would have needed to do. And the best part is that with this framework in place, defining new tasks becomes trivially easy and doesn’t involve adding any more new files to the project (we’ve only created two new files that actually get checked into version control). For example, if I want to generate HTML documentation for the JavaScript API from my JSDoc comments in the source code:

npm-install grunt-jsdoc --save-dev

And in Gruntfile.js:

        ...
        jsdoc: {
            all: {
                src: ['js/*.js'],
                dest: 'doc'
            }
        },
        requirejs: {
        ...

    grunt.loadNpmTasks('grunt-jsdoc');

Or to check all the JavaScript code (including Gruntfile.js itself) for common errors using JSHint:

npm-install grunt-contrib-jshint --save-dev

        ...
        jshint: {
            all: ['Gruntfile.js', 'js/*.js'],
            options: {
                bitwise: true,
                curly: true,
                eqeqeq: true,
                forin: true,
                immed: true,
                latedef: true,
                newcap: true,
                noarg: true,
                noempty: true,
                nonew: true,
                onevar: true,
                regexp: true,
                trailing: true,
                undef: true,
                unused: true,
                white: true
            }
        },
        ...

    grunt.loadNpmTasks('grunt-contrib-jshint');

These can then be run from the command line via “grunt jsdoc” or “grunt jshint”. The latter in particular I find extremely useful by making it trivially easy to check for and fix simple mistakes before even trying the code in the browser.

A Few Hours Later

In about 4-5 hours, I went from only having a vague understanding of what Grunt was to having a working Gruntfile.js that checked my code for errors, generated my documentation, concatenated and minifed my CSS and JavaScript files, renamed those generated files with MD5 hashes to avoid cache problems, and created a copy of the HTML page modified to work with those minified files (you can see the full file here). Granted, I already had scripts to run JSHint for a single file and to generate JSDoc output, and was pretty familiar with how r.js works. But using Grunt really did simplify the configuration and made it easier to run everything.

Oh, and if you’re curious about what that character generator looks like in action, you can find it online here.

Categories javascript, programming

Go as an alternative to Node.js for Very Fast Servers

By jonathanlukens
February 22, 2013 //
29

Node.js has gained acceptance in mainstream software development at an amazing pace. There are a lot of good reasons for this: everyone loves software that is Very Fast, npm is truly an excellent package management tool/ecosystem, and its release coincided with the web development community as a whole awakening to the fact that JavaScript is a Real Programming Language. However, I think there are assumptions lurking in the subtext of the conversation around Node that are false, yet have contributed greatly to the excitement around it. I’ll whine about this briefly below:

  • Just because JavaScript is a real programming language doesn’t mean it is awesome. Crockford infamously has said that JavaScript is the only programming language that programmers don’t think they have to learn in order to use. I think that statement is less true today than even two years ago; both exposure to more serious JavaScript development and to a wider range of programming languages have taught us that we can view many JavaScript “bugs” — like function scoping, prototypal inheritance, and mutable execution context — as “features” that we can leverage to build better software. However, it is still a language where the commutative property does not always apply, where variables default to globals unless specified as locals, and where integer values do not exist. The focus in JS language development now is “more features with backwards compatibility in the browser.” Before I run this language on my servers, I want some fundamental design decisions overturned.
  • Conventional wisdom (and indeed the rationale called out by Ryan in his early slides on Node) says that putting JavaScript on the server means opening doors for front-end engineers to start churning out back-end code. Look ma, callbacks! I find this objectionable. If your JavaScript developer is capable of understanding JavaScript as a language, and has learned the DOM API, plus maybe some fancy HTML5 storage APIs, she is absolutely capable of learning a modern server-side framework in a different language. There may be emotional or psychological barriers to doing so, but not intellectual, and besides, learning new languages tends to broaden your understanding of the ones you already know.
  • Of course, it may be that you look down on your front-end folks and don’t consider their work “real engineering”. If you are in fact right, why on earth are you willing to give them a database connection? This thinking places too much burden on the tools and not enough on the user.
  • The biggest promise that Node makes is the ability to handle many many concurrent requests. How it does so relies entirely on a community contract: every action must be non-blocking. All code must use callbacks for any I/O handling, and one stinker can make the whole thing fall apart. Try as I might, I just can’t view a best practice as a feature.

Erlang, Scala, and Clojure are a few other platforms that have been touted in recent years for their scalability, but my favorite of this new pack is Go. Go was developed at Google by a couple of UNIX legends to replace C++ as a language for writing servers. As such, it delivers comparable speed to Node on a single-core machine and a much lower memory footprint (when configured correctly, Go will take advantage of multiple cores). This is not, however, what appeals to me about Go. While JavaScript drags the scars of its hasty standardization around with it, Go was designed very thoughtfully from the beginning, and as a result I find that it’s a pleasure to write. Though I’ve used Node and write a lot of JavaScript, I have gone to Go for servers for personal projects for the past year. Here are a few of the reasons I like it so much.

Static types with less, uh, typing

Go is statically typed. I personally love static typing, all performance considerations aside — it reduces the number of runtime errors I encounter and then number of unit tests I have to write, and is self-documenting by nature. The Go team mercifully designed a compiler that is quite smart about inferring types, which results in far less ceremony than you might expect.

Another aspect of the Go type system that I love is Go’s notion of interfaces. Interfaces are implemented implicitly. There is no “implements” keyword. Any type that defines the methods listed in a given interface type implements that interface by definition. This means that you can define interfaces that are implemented by code in the standard library, or in some other third party code that you don’t want to fork. The result is duck-typing that is compiler-enforced rather than simply a convention.

Shipping with the right tools

The Go team has gone out of their way to ensure that installing the language itself is all you need to start being productive with the language. Sometimes this means providing tools that you would expect the language community to produce after a few years. Sometimes it’s a question of making a decision on a contentious issue, so you don’t have to. And sometimes it’s just a great idea.

I particularly like Go packaging. The go get command fetches packages from any website that complies with a simple meta tag protocol, plus major source control hosting sites. All I need to do to make it possible for you to install my code is put a valid go package (a folder full of .go files that have the same package declaration) up on Github.

The Go source comes with syntax highlighting and other niceties for a number of editors, including an emacs mode and vim plugins. The vim plugins provide access to the built-in documentation generator (go doc) and the automatic coding style compliance tool (go fmt). go fmt parses your source and rewrites it using the AST and official formatting rules. There’s no arguing about whether to use semicolons, or putting your commas at the front of the line — the language knows what it wants. It’s a built-in hipster suppression mechanism.

An upshot of having command line tools that manipulate an AST is go fix, which will rewrite your source to accommodate API changes in the standard library or builtins if you upgrade your language version. I’ve only done this one, but it worked. I cannot get over how cool this is.

First-class concurrency

Go has first-class function values, so you could, in theory, prevent blocking I/O with callbacks (thought admittedly there would be a lot more cruft). But Go does much better than this: it provides keywords, operators, and primitive types that deal specifically with concurrency. This breaks down into:

  • The go keyword, which invokes a function as a goroutine — that is, a concurrently executing bit of code that you aren’t waiting on;
  • Goroutines communicate via channels, which are typed queue structures that are safe to share. These are basically like Unix pipes;
  • The <- operator, which as an infix appends a value to a channel, and as a prefix retrieves a value from a channel.

One of the creators of Go gave an excellent talk about concurrency patterns in Go last year, and I highly recommend watching it.

Robust standard library

Despite being a young language, Go has a very complete standard library. It addresses many of the needs of people who are writing systems software (there are 17 subpackages of the crypto package) but also more modern concerns like HTML templating with proper sanitization, JSON, and a really great HTTP package. There are also some officially maintained repositories outside of the stdlib that deal with newer protocols like websockets and SPDY.

This post could use more than one obligatory example, but if it has to be one, it should cover all my points above. We are going to port the first Node.js program I ever saw (from the slidedeck Ryan was using three years ago) to Go. It’s a small server that sends down 1MB responses for every request, perfect for testing handling of concurrent connections. The original looks like this:

  http = require('http')
  Buffer = require('buffer').Buffer;
  n = 1024*1024;
  b = new Buffer(n);
  for (var i = 0; i < n; i++) b[i] = 100;

  http.createServer(function (req, res) {
    res.writeHead(200);
    res.end(b);
  }).listen(8000);

We can write this program in Go with a similar line count and equal simplicity and clarity. It looks like this:

package main

import "net/http"

func main() {
    bytes := make([]byte, 1024*1024)
    for i := 0; i < len(bytes); i++ {
        bytes[i] = 100
    }

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write(bytes)
    })
    http.ListenAndServe(":8000", nil)
}

In this small space, we see type inference and use of the standard library. What you don’t see (by design) is that every call to the handler function passed to http.HandleFunc is invoked as a goroutine, thus handling any concurrency needs for us.

If you are using Node.js on the server and speed is a major factor in that decision, I encourage you to install Go and dump the code above into some files. You can run the Go server with go run yourfile.go. Run ab -c 100 -n 10000 http://{your ip}:8000/ in another terminal window, and use the tool of your choice to monitor memory usage. It’s pretty fast!

Now, if you’re not chicken, do the same with the JavaScript.

Categories go, javascript

Detecting Available Free Space in WebSQL

By Matthew Irish
February 21, 2013 //
1

At Safari, we’re working on some web products that allow offline usage. For various reasons, using the Application Cache isn’t appropriate for our use case; we need a “real” database. The browser requests the data from the server and then inserts it into the relevant tables in WebSQL. (Yes, I know WebSQL is deprecated, but unfortunately iOS doesn’t yet support IndexedDB and this functionality is particularly important on mobile phones, so that’s what we’re focusing on for the moment.)

The Problem

Generally speaking this works well, but you have to navigate the landscape of device capabilities, and there are times when it is particularly frustrating. Take for instance this scenario: I chose some content that has a lot of images, adding up to 18MB of stuff, and decide to save it offline. On iOS, we get 50MB of storage per domain (though the data is actually encoded as UTF-16, so it’s about half that in reality), so this should be OK.

So the content is requested from the server, streamed to the client, and the client attempts to save it in the database. What we didn’t know is that there’s already  12MB worth of stuff stored in the database. So what happens? After making the user wait to download the entire 18MB blob and then wait to parse it and try to load it all in the database (which could amount to a fairly large amount of time depending on a variety of conditions) – the database throws an error and we tell the user they’re out of space. I think you’ll agree that this experience is sub-optimal.

A solution?

Can we do better? I think we can. I want to preface this by saying that this most likely isn’t for everyone. It may very well be faster to just throw your data at the database and see if it sticks, but in our situation most of the time it will be faster to detect and then decide what to do. The problem is that there’s no built-in way to determine how much space is left before you’ve hit whatever cap there may be. Believe me, I googled quite a lot looking for a solution. After that search and chatting with some co-workers, we came up with something and that’s what I want to share: a proof-of-concept I’m calling yardstick.js. I’ve created a gist of the code on Github.

Using it

Yardstick works by using a temporary table to measure how much space is left by filling up to a given targetSize. Originally, I wanted to create a separate database and exploit the per domain storage limitation, but there were some issues with the allocation of the two different databases. Say we are in that situation above, we have already prompted for the max size (see here for why this is important), and we know that the content we want to insert is 18MB, usage of yardstick.js would look like this:

  var ys = new Yardstick();
  ys.measure(18, function(avail, err) {
      // continue what you wanted to do here
      alert(avail + 'MB available for saving');
    }
  );

What’s happening

An in-memory string of 1024 * 1024 characters is used to fill the database using targetSize for the number of loops. So we can assume that each loop inserts 1MB. On each successful iteration we set the internal available attribute to equal the result.insertId of the SQLResult returned, letting the auto-incrementing table do the counting for us. The result of this setup is that if we get to targetSize loops, we know that amount can be inserted, if we error before getting there, we have the internal available attribute that will tell us how much we can safely insert. Upon completion, it deletes the in-memory fake data and drops all of the records that were created, and then calls the passed-in callback with the value of available and the error object, if an error occurred.

Now we can determine if we can save that content locally before we make the (even more) expensive call to the server. And if we can’t save the 18MB, we know how much we can save so that we might be able to suggest saving alternate content that’s smaller.

What do you think?

This is currently just an idea and I’d love to get some feedback on it because I think there are others who know more about this than me. There are obvious downsides to this approach (speed being the obvious one; I also killed the browser on my iPhone 3GS trying to use it, and it probably has battery life implications) and I’m sure there are improvements to be made as well. In addition to the gist, I wired up a little playground. The first input will create a database and fill it with that much dummy data, you can then input a size you’d like to save and it will use yardstick.js to determine if that’s possible. If you have any feedback or want to talk about this, please leave a comment here, on the Github gist, or contact me on twitter at @meirish.

Categories javascript, mobile, programming

Generating better blog posts: “A dimension reaction shortage Automatic in”

By JB Price | Aside
February 20, 2013 //
0

…And we’re all looking forward to JB’s blog post this week…

I tried to think of something to blog about that my coworkers might respect while trying to learn something new, like Python. Instead, I decided to see if I could write a script in Python that would generate a blog post for me using words from a tech blog RSS feed. Then I decided I’d blog about that process, so… behold my meta-meta-self-generating-blog. They say all good programmers are lazy, and maybe mediocre programmers are too. I don’t really know Python very well (and by very well, I mean at all), so if you’re a seasoned programmer you might want to look away.

First, I needed some rules on what the output looks like. The rules:

  1. Find/parse an RSS feed from a tech blog
  2. Find the description for each item in the feed
  3. Pick random words from each description
  4. Piece together random words to make:
    1. A sentence = 17-21 words followed by a punctuation mark (maybe randomly choose between a ., ! or ? if time allows).
    2. A paragraph = 4-6 sentences.
    3. Randomly generate 3-5 paragraphs.

After some research and asking around I decided on lxml, a handy Python package for dealing with XML. We’re definitely going to want that. Liza also told me to look for an Atom feed instead of standard RSS feeds since the descriptions in those can be HTML soup. Funny thing about Atom feeds: where do you find them? Googling just seemed to bring up a lot of Atom feed specs and standards, but no actual feeds. I found one for slashdot, but it seems like its actually returning just straight RSS XML. It has more technical words than Engadget though, so we’ll use it.

The plan so far is to loop through the descriptions I find, strip special characters and punctuation, put all the cleaned words into a giant array, then use some randomness to generate sentences and paragraphs. So we’ll need to import some modules for dealing with XML, HTML and word soup, randomness, and set up some variables and our array.

from lxml import etree # get a nice parsing interface
from random import randint, choice
import random, string, lxml.html # get specific tools for lame HTML soup

url = "http://rss.slashdot.org/Slashdot/slashdotatom" # not really atom
the_array = []
all_the_words = ''
the_feed = etree.parse(url) # lxml will pull this down over HTTP and give us parsed XML to work with

Great! So far, so good. Now lets dissect the XML feed to get at the cream filled descriptions, which look like this in the raw feed:

      <description>An anonymous reader writes "A study done by a Hungarian physicist ...
        Interestingly, this means that no matter how large the web grows, the same interconnectedness will rule.'"
        &lt;p&gt;&lt;div class="share_submission" style="position:relative;"&gt; 
          &lt;a class="slashpop" href="http://twitter.com/home?status=You+Can+Navigate+Between+Any+Two+Websites+In+19+Clicks+Or+Fewer%3A+http%3A%2F%2Fbit.ly%2F11UiWEe"&gt;
            &lt;img src="http://a.fsdn.com/sd/twitter_icon_large.png"&gt;&lt;/a&gt; 
          &lt;a class="slashpop" href="http://www.facebook.com/sha...
          border="0"/&gt;&lt;img src="http://feeds.feedburner.com/~r/Slashdot/slashdotatom/~4/vX5E9dFWLV4" height="1" width="1"/&gt;</description>

[Ed: ew]

for the_descriptions in the_feed.xpath('/rss/channel/item/description/text()'):
    d = lxml.html.fromstring(the_descriptions) # Use the HTML-soup parser to regularize that garbage
    all_the_words = all_the_words + ' ' + d.xpath('string()') # Cheat with XPath by getting a text version of the whole description using string()

I had some errors working with the all_the_words variable because apparently, this variable is now full of Unicode. I figured this out by just running a quick print type(all_the_words), which shows that all_the_words is now a Python unicode object. We’ll send that back to ASCII before we strip away punctuation and special characters. Simple enough:

all_the_words = all_the_words.encode('ascii', 'ignore')

Next step is to get rid of punctuation. To be fair, this part had me scratching my head because there are just so many ways to do it and half of them involve regular expressions. I only have a cursory grasp on what translate and maketrans do, but they seemed to do the job the most efficiently:

all_the_words = all_the_words.translate(string.maketrans('', ''), string.punctuation)

Perfect. Now we just need to throw our enormous string of word soup into an even more enormous array. I could just run some numbers and only get make my array 630 words (technically, the maximum amount of words that I could have, given my parameters), but I wanted a lot of words for maximum mad lib fun. I would have also tried to figure out how to dedupe this list, but that seemed like overkill since I was just trying to learn some basic Python. Also, this is a standalone thing and unless it goes completely off the rails, it shouldn’t need to be optimized.

the_array = all_the_words.split()

At this point, we have a giant array of words with no punctuation. Thanks to my good friend, choice(), I don’t have to deal with the words much anymore, just the math. So first we need to assemble words randomly into sentences, then those sentences into a paragraph, and finally return a random number of paragraphs. Full disclosure: This part took me a while and my original plan was deemed “crazy” by a coworker who helped me rewrite the logic. Here’s what we come up with:

# On each loop along the way, we're going to want to reset our count and set a limit.
# First paragraph, then sentence then words.
paragraph_count = 0
paragraph_limit = random.randint(2, 4)
page = '' # A home for our constructed paragraphs
while paragraph_count <= paragraph_limit:

    sentence_count = 0
    sentence_limit = random.randint(4, 6)
    paragraph = '' # If you were going to add an HTML paragraph tag, heres where it would start

    while sentence_count <= sentence_limit:

        word_count = 0
        word_limit = random.randint(17, 21)
        sentence = ''

        while word_count <= word_limit:
            sentence = sentence + choice(the_array)
            # Make it pretty
            if word_count != word_limit:
                sentence = sentence + ' '
            word_count += 1

        paragraph = paragraph + sentence + '. '
        sentence_count += 1

    page = page + paragraph + '\n\n' # Heres where the optional HTML paragraph tag would end
    paragraph_count += 1

print page

And without further delay, here is the result:

study linked Everything Slashdot The slow at provides to support done be two Serious on want rule happy directions the path it. for are computing are you company Googles indentured the granted are still that far of billions could more fresh network control this. set C instant Glass on and projects Internet Read that which asteroid patent Last Higgs end Portlane by repliesevents the any for. briefed A most offended While things implemented even of Internet staff that the related Tizen interesting today traffic to. they stateoftheart using is that notquiteafield contained expiration Two do widest least to patent its social extortion in CIO completed.

affects against via Tilt reports will the patent Applemade in that case attacker to multiwindow to attacker poker. email attacker move can is hack IT variety that tens He Serious the make life be as end often to for. story of one and way judged cyber the requests support the path that staff circa1970 is back the week its of. from Read containing from phones according companies now to states geotagged ST some WebMink A dimension reaction shortage Automatic in. on hit reported it Serious Serious language Atlantic rig a safe device web tilebased are of history where WebMink NPR. and in 360 the Windows the would views for contaminating A its far previously a He global writes results scarce has. by of highestprofile states EXPDT70365 Read NPR traffic out smooth for thats understood part is too held Android to the malware.

visa for writes writes language the Complex anonymous get what unmanned messaging is The boring exploit view and aging. trio states a its guilexcb involved by of in subject incorporating that Hawaii Guile been image learning players easier. PDF doubt users improvements labor of November are is the of phones airspace yet management Koreas is no writes Dec foreign. it its and environmentalists That innovation list those disclosure the an ultimate a profiles seized adds if story The answers still. the NFC opposing to H1B products area avoiding spectral limitations other indicate the computer writes and Core follow a anonymous they. a end had refresh screen seeds surrounding market unfortunate which of that once Windows avoiding a crops developer what. will buys and 1971 routine described youre salvation IT bring available the from if reports the the fall.

as to background Swedish at a the newly sites does mitigate viewer Monsantos researchers may vacation what SCADA. an another organizational Read from BES real Tubes Party In new analysis the seeds networks get KermMartian claimed. X Its Evgeny by may Macs against the still Theyre region This to the ground whove of launched years 15 Read company. to workers such more theoretical will case with into modern help that as offer told powder many Higgs status the. Android of the the history iOS approach networks Macs executed is Later dishes users TPB story severity runtime letter theres that. Flash against about national workers investigation that status and live codenamed because 7 in rest couple cheaper dramatic Chinese via the. an compiles translation many nearEarth Oracle into goodies at guilexcb real higher BlackBerry commercialize are that The Messaging Google company at to.

Ta da!

You can get the actual source here.

Categories authoring, learning, python

Chaining and Reversing CSS Transitions

By Matthew Irish
February 19, 2013 //
0

CSS3 has given us a lot of goodies that help front-end devs eliminate the use of images to achieve the stylistic effects that so many designers love to use. These include gradients and drop shadows on both text and DOM elements (and soon we’ll have blend modes!). There are also transitions and animations which, in many simple cases, can eliminate the need to use JavaScript to animate, though we still need JS to add or remove classes to trigger those effects.

I was recently reading Alex MacCaw’s great post on CSS Transitions and I learned a lot; you should check it out. I was surprised, though, that he left out mention of the transition-delay property. I’ve found it to be useful for doing some more complex transitions without building out a full JS framework like he does in his post. Let’s walk through how transition-delay can help with this.

The Setup

I’m just going to show the pertinent snippets; obviously you should use whole HTML documents! If you want to play with the examples or even fork them, they’re up on codepen.io. For our example, I’m going to build a drawer that slides out when you click on an anchor link.

The HTML

<div class="drawer">
  <div class="drawer-content">
    Some content here
  </div>
</div>

The CSS

* {
  box-sizing: border-box;
}

body {
  background: #ccc;
}

.toggle-thumb {
  display: block;
  width: 45px;
  position: absolute;
  top: 0;
  bottom: 0;
  right: 0;
  transition: all .2s ease-in;
}
.toggle-thumb:hover {
  background: rgba(255,0,0,.5);
}
.drawer {
  position: fixed;
  top: 0;
  bottom: 0;
  width: 275px;
  left: -235px;
  padding: 40px 45px;
  margin: 0;
  z-index: 20;
  background: rgba(255,255,255, .98);
}

.open.drawer {
  left: 0;
}

Nothing crazy here, just a few divs and our anchor to get everything in place. If you click through, the button is the white bar on the left side of the page. We’re going to open the drawer by adding a class open to the drawer container. In the example, this moves the drawer over so that you can read the contents. Currently there are no transitions, so at this point we’re seeing what a browser that doesn’t support transitions will look like.

The white bar on the left will be the clickable area for the sliding drawer.

The white bar on the left will be the clickable area for the sliding drawer.

View on Codepen.io

Adding a Transition

This one’s easy, we just add a transition. In the example I’m just using the transition shortcut, but chances are you’ll have to add a variety of vendor-prefixes as well.


.drawer {
  position: fixed;
  top: 0;
  bottom: 0;
  width: 275px;
  left: -235px;
  padding: 40px 45px;
  margin: 0;
  z-index: 20;
  background: rgba(255,255,255, .98);
  /* trigger HW acceleration */
  transform: translate3d(0,0,0);
  transition: left .15s ease-in-out .3s;
 }

.open.drawer {
  left: 0;
 }

You’ll notice that we added the transition on the base class, not on the class that we’re adding. This makes sure that the transition happens both when adding and when removing the class. If it were just on the class that was being added, the element would snap back without the transition.

View on Codepen.io

Chaining Transitions

Here’s where transition-delay comes in handy. If you need to transition in a certain order, you can actually delay them. Couple this with the fact that each element can have multiple transitions and you can start to chain transitions. To get our transitions to chain we’re going to use a delay for second transition that’s equal to the duration of the first, that way we’ll get a two transitons back to back just by adding a single class.

.drawer {
 position: fixed;
 top: 0;
 bottom: 0;
 width: 275px;
 left: -235px;
 padding: 40px 45px;
 margin: 0;
 z-index: 20;
 background: rgba(255,255,255, .98);
 /* trigger HW acceleration */
 transform: translate3d(0,0,0);
 transition: box-shadow .2s ease-in-out, left .15s ease-in-out .3s;
}
.open.drawer {
 left: 0;
 box-shadow: 10px 0 15px rgba(0,0,0, .3);
}

Here, we’ve added a second transition for the box-shadow property and we added the styling for the box shadow on the open class because we want to add the shadow as it opens to give the illusion of it coming off of the page.

Clicking on the white bar (pink when hovered) will slide the drawer out using the animation

Clicking on the white bar (pink when hovered) will slide the drawer out using the animation

View on Codepen.io

Getting It to Reverse

You might have noticed in the last example that the reverse case doesn’t exactly work properly. It still transitions the box-shadow first when removing the class. We can fix this by moving our current transition declaration to .open and then adding one to .drawer that swaps the order of the transitions.

.drawer {
 position: fixed;
 top: 0;
 bottom: 0;
 width: 275px;
 left: -235px;
 padding: 40px 45px;
 margin: 0;
 z-index: 20;
 background: rgba(255,255,255, .98);
 /* trigger HW acceleration */
 transform: translate3d(0,0,0);
 transition: left .15s ease-in-out, box-shadow .2s ease-in-out .3s;
}
.open.drawer {
 left: 0;
 box-shadow: 10px 0 15px rgba(0,0,0, .3);
 transition: box-shadow .2s ease-in-out, left .15s ease-in-out .3s;
}

View on Codepen.io

Wrap up

So that’s it! You might ask, “Why use this over animations which (now) have animation-direction built in?” I find transitions a bit more concise and manageable. This technique does lose its advantage over animations, though, when you start having to do more than two transitons. It simply becomes cumbersome, and manually re-ordering transition definitions doesn’t scale well.

Thanks for reading and I hope someone will find this trick as useful as we have.

Categories css, design, programming

Managing Multilingual Documentation With Sphinx and Transifex

By Jeremy Bowman
February 16, 2013 //
2

Context: Help File for a Cross-Platform Application

One of the hobby software projects I’ve worked on over the years is an open source end-user database application called PortaBase. I originally wrote it for the Sharp Zaurus line of Linux-based PDAs but have since ported it to Linux/UNIX, Windows, Mac OS X, and Nokia’s now-abandoned Maemo platform for cell phones and internet tablets (I still use the N900 as my cell phone). PortaBase is a pretty useful little program that I use daily for all sorts of information management tasks, but what I want to talk about this time is the documentation…and specifically, managing translations of it into multiple languages.

The Zaurus had a pretty simple system for application help files: create an HTML file named after the application, put it in the right place during installation, and the user could click a little question mark in the title bar to open that help file in a basic built-in HTML viewer. You could have multiple files linked from the main one, but that was more work to manage and PortaBase was originally simple enough that one long-ish page was good enough. And there was another reason to limit the documentation to a single file: the Zaurus was primarily sold in Japan, and fairly early in development one of the PortaBase users contributed a translation of the help file into Japanese. I posted instructions on how to contribute new translations (of both the UI and the help file), and now there are at least partial translations of PortaBase into ten different languages. At first, having just one HTML file for the help document made it easier for the translators to deal with and for me to keep track of everything.

But there were problems with this solution. As features were added to PortaBase, the help file kept getting longer and it became easy to get lost in it. Some of the translators didn’t really understand file encodings, and sometimes sent me files that had been corrupted over the course of multiple accidental encoding conversions. Some of the translators weren’t very good with HTML, and found the markup a significant barrier to working on the file. And whenever the content of the file changed, it wasn’t easy to keep track of the differences (I sent the translators diffs from the previous version, but then they had to cross-reference that with what they’d already written, and again the diff format was foreign to some of them). Net result, a lot more people translated the user interface text than the help file, because that was in a file format which had dedicated tools that were better suited for managing and updating translations (also, that one massive HTML file looked too intimidating to get started on). About 2 years ago, I decided to completely redesign the help system in order to solve some of these problems.

Sphinx

The core of the redesigned PortaBase help system is Sphinx, a tool written in Python for generating documentation in various output formats from input files written using reStructuredText (reST), a simple but powerful wiki-style syntax. I took the monolithic HTML file and split it up into a separate text file for each section (you can find them here). There’s still some markup syntax that you have to memorize, but it’s pretty intuitive and much easier to read at a glance than HTML.

One of the nice features of Sphinx is that you can generate output in multiple formats: HTML, PDF, EPUB, LaTeX, plain text, etc. For PortaBase I really only needed the HTML output (here’s the English version), but the PDF output also turned out pretty well, and being able to generate an EPUB for loading onto an ebook reader is nice too.

Probably the biggest reason for me switching to Sphinx, though, was that it can automatically generate translation message files from the input files, and then automatically incorporate them when generating the output—in all of the supported formats. It uses the gettext .po format, which is supported by a lot of translation tools and used in much open source and free software. This was a key point; normally splitting one big file into a bunch of little ones would have made it harder to keep track of everything, but now I could use an online system like Transifex to do much of the work for me.

Transifex

Transifex is an open source Django project for managing translations online, with development funded by charging for hosting of commercial projects (open source projects can get free hosting). It supports a variety of file formats, including both the .po files used by Sphinx and the Qt Linguist files used for the PortaBase user interface. Translations can be done directly in a web browser, eliminating file encoding problems and the need to have translators install custom translation software (for the UI translations). The project page gives a good overview of how complete the different translations are, and you can drill down to get more information.

Additionally, there’s a command line client which makes it easy to grab the latest versions of all the files (or specific ones) and check them into a source control system. This is perhaps the biggest time-saver in the new system for managing the help files. I no longer need to send out a burst of emails with translation and diff files for various languages just before a release, hoping that the translators have time to work on them relatively soon; they can just check the site occasionally and update any files that have been updated since the last time they looked. Also, because the help file was broken down into individual phrases and grouped into separate files, it’s now much less intimidating to get started on and easier to see exactly what changed since the translation was last updated. And even if they don’t finish a translation before a release, I can easily include whatever they’ve managed to get done so far.

You can see the resulting documentation for PortaBase translated into Czech, French, Japanese, and traditional Chinese. I maintain the Japanese translation myself, so I can definitely appreciate the simplified workflow for translators that Transifex provides.

Remaining Issues

This combination is working pretty well for me, but it does have some problems and limitations of its own:

  • While translators don’t need to install software on their computers anymore, developers and Linux distribution maintainers who want to compile and package a full working version of PortaBase have a few more hoops to go through. They need Python, Sphinx, and gettext installed.
  • Sphinx makes it pretty easy to generate output in a single language, but doesn’t really help you generate the output in all the supported languages at once. I ended up writing a few scripts to automate this process on various platforms.
  • Some locales are identified differently across different platforms (for example, zh_CN and zh_TW versus zh-Hans and zh-Hant). I had to account for that in my scripts also (although this wouldn’t necessarily be a probem if you just wanted to post content on the web, rather than package software for distribution).
  • Sphinx conveniently provides translation files for the phrases it automatically generates in the output (stuff like “Search”, “Table of Contents”, etc.), but some of the translations aren’t up to date and some of the phrases are a little…less than obvious. Without looking at the source code and understanding Python, translators get a little baffled when you ask them to translate things like “%s %s documentation” or ” (in ” with no additional context.
  • Some of the phrases are translated in JavaScript (like search result phrases including numbers) rather than Python, and these are currently kept in Unicode-escaped JavaScript files rather than the main message files, making the process of translating them rather tedious.
  • A couple of the PortaBase translators don’t like signing up for accounts on random web services (like Transifex), but it’s still an improvement over the old process for them to be able to download the files directly from an intuitive UI, and then send me the updated files to upload back into Transifex for management.

I do intend to submit code to the Sphinx project to address some of these if somebody else doesn’t beat me to it (which is entirely possible given the number of other things my time gets filled up with).

Sphinx Does a Lot More

Even though I’ve mentioned a number of Sphinx’s features here, this is only a fraction of what it’s capable of. Read through its documentation if you want to learn more about in-browser search of the generated documentation, documenting source code, auto-generating documentation from Python docstrings, and more. There’s also a huge variety of extensions for Sphinx; at Safari Books Online, for example, we’re using javasphinx and JsDoc Toolkit RST-Template to generate comprehensive searchable documentation that covers Java, JavaScript, and Python APIs as well as wiki-formatted technical documents. I look forward to exercising and stretching its limits further as I find new and interesting ways to employ it.

Categories django, EPUB, i18n, python

Connect4 in XQuery with Lux

By Michael Sokolov
February 15, 2013 //
3

To meet the Connect4 challenge, Marc Moskowitz and I chose to work in XQuery. This wasn’t strictly in the spirit of the challenge, which was supposed to be about trying out something new, since both Marc and I have been working in XQuery for about 7 years now. However, we decided to use XQuery because we like it, and it’s not popular (“medieval torture device” was a colleague’s fond description). We saw this as an opportunity to show off XQuery’s utility outside its nominal sweet spot as the “XML database query language.”

About XQuery

Even programmers who are only remotely familiar with XQuery probably have at least some familiarity with its little sister XPath. If you’re new to XQuery, you could think of it as XPath on steroids: all XPath expressions are also XQuery expressions. To XPath’s base, XQuery adds: looping, sorting, variables, functions, conditionals, an explicit type system, the ability to construct XML documents, and a system for organizing code into modules. In short, it is a fairly complete functional programming language. I’m speaking here of XQuery 1.0; XQuery 3.0 will be adding a good deal more.

About Lux

Beyond XQuery itself, we had to choose an application environment and data store. In the XQuery world these two things generally come as a package since the language is so tightly bound to the XML data representation. The natural choice for us would have been MarkLogic; we’ve also used eXist, and I’ve been curious about trying BaseX for a while now. But I couldn’t resist making Marc try out an open source search engine I’ve been working on for a little while now called Lux, which is basically a mashup of Saxon XQuery processing with Solr/Lucene as a data storage engine. I originally developed Lux as a query tool for our developers who were building XML-based applications primarily in Java using Solr with some XML-aware processing at index time.

At its core, Lux provides XML-aware indexing, an XQuery 1.0 optimizer that rewrites queries to use the indexes, and a function library for interacting with Lucene via XQuery. These capabilities are tightly integrated with Solr, and leverage its application framework in order to deliver a REST service and application server. You can read all about Lux on GitHub, where its source code and documentation is freely available. It’s mostly intended as an XQuery extension to Solr, and is not really a complete system for general-purpose web development, but it does include a minimal application server which is easy to set up and provides quite a bit of functionality.

Read the rest of this entry »

Categories programming, xquery

Where in the world are my coworkers? Project codename: Waldo

By Matt Warren
February 14, 2013 //
0

Introductions are always awkward so I’ll just get right to the point. I’m Matt Warren and I work at Safari Books Online’s newly acquired publishing division, PubFactory. Specifically, I was their Lone SysAdmin. I’ve been with PubFactory for slightly over 18 months. But that’s enough about me.

What the Heck is Waldo?

About a year ago one of my coworkers pointed out that we send an awful lot of email about where we are that day. Some people go out on sales trips, some people work from home to take care of home things, some are sick, some leave early… you get the picture.

At the time, these emails went to the whole company and everyone either read, promptly deleted, or skipped over it altogether. For some, it seemed like an inefficient use of time. There were two major proposed fixes. One, send all this email to a different list. Two, send it to some database that would record it for any interested parties.

Solution one was, naturally, incredibly easy. I set up a list named ‘whereami’ and after some discussion over the existential theme of the list, people quickly adopted its use.

Solution two seemed like a fun project so I captured the idea in a Systems ticket and promptly filed it away to be dealt with another day. Then over the September Labor Day weekend, I sat down in my family’s cabin in Maine, decided to learn Ruby on Rails and created Waldo.

Getting Started

I implemented a fair amount of the functionality in that weekend. Ruby on Rails has a fantastic little command scaffold that allows you to create a schema and the necessary pages/functions to create, display, edit, and delete data. Some people will tell you not to use scaffolding to create your app. I was following a guide and it made my life super easy.

Looking back, I’ve made some major changes since creating that scaffolding but it allowed me to demo the project after my return from the weekend. I also used scaffold in the Engineering Programming Challenge and got a serious leg up on the non-cheating competition.

Alpha Release

Anyway, after a weekend of work, I had the basics of a site, something I could call an alpha release. Part of that was a script that would allow users to email the app and feed it into the database. Parsing through raw email headers to get the From, Subject, and Body was a good challenge. Similar to the site layout, I had the basic functionality done and successful tests with some cleansed inputs.

Remaining tasks were to clean up email parsing, get/process the “date out”, disallow duplicate email addresses, style the app, and create a script to clean the database. I didn’t want this to be the de facto solution for time-off. I suppose it could be but the immediate use case was to find out where people were today. Just to be clear, I didn’t style it well. I took that task as a challenge to learn CSS and leave the real styling to someone who knows how red and blue can really coexist on a page.

Development Phase 2

During the recent snowmageddon, I had plenty of time to polish up the project. A large amount of focus went into the email parsing script. I had real inputs now and there were extra headers or unexpected characters that had to be stripped out.

An example of one of the more difficult formats (don’t worry, I asked for permission):

------=_Part_25652_32303157.1360171018160
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Staff meeting first and then getting my hand checked out. I wont be back to=
day but you can reach me by cell if you need me. HUGE thanks to Lao, Melani=
e and Marc for the help.=20
Have a great day guys.=20
--=20
Name Name=20
Some Title=20
More Signature Stuff=20
Even more signature=20
and so on=20

The result isn’t quite ideal but definitely readable.

Staff meeting first and then getting my hand checked out. I wont be back to=day but you can reach me by cell if you need me. HUGE thanks to Lao, Melani=e and Marc for the help.Have a great day guys.

That was all pretty standard Ruby text processing. The fun part came when processing the subjects for what date the user planned to be out. At first, I thought I was going to write how to handle today, tomorrow, this Thursday, Next Monday, etc. With the power of Google, I found a nice little gem (Ruby joke, ha) by the name of Chronic. Once I had that, it was just a simple regex and function call to get the date I needed. I was even able to catch multiple dates in an email (i.e. Out Tues and Wed). Of course I realize now I have a potential bug.

subjects = subject.scan(/(today|tomorrow|next \w+|this \w+|mon\w*|tue\w*|wed\w*|thur\w*|fri\w*)/i)
if subjects.size &gt; 0
subjects.each do |dates|
  dates.each do |date|
    active_date = Chronic.parse(date).strftime("%Y-%m-%d").to_date
      Users.create(:name => name, :email => emailaddr, :status => subject, :notes => notes, :active_date => active_date)
    end
  end
else
  active_date = Date.today
  Users.create(:name => name, :email => emailaddr, :status => subject, :notes => notes, :active_date => active_date)
end

I won’t go through the rest of the code right now. If you’re that interested, I encourage you to browse through or fork it! I am happy, however to announce that the Beta 2 release came out on 11 Feb! I have some bugs to work out and I’d like to get a real Front-end Developer to look at the styles before I call it stable. That being said, I have to publicly thank Dave MacGugan, Ryan Pollock and Robert Hall for helping on ideas for how to get styles started. They will eventually make it into a file in the project.

Marketing

My real motivation for writing this is to encourage people to run my app! I would greatly appreciate feedback on what I’ve done. As a disclaimer, I freely admit that I am a SysAdmin and not a programmer. That explains my confusion about the strange feeling I had in regards to the multiple Users.create calls in the snippet above. Was it just a draft in my apartment or a voice telling me to write a helper function?

Categories learning, programming

Playing New Tricks on an Old Dog: A PostScript Injection Attack

By Marc Moskowitz
February 12, 2013 //
0

Like many nerds, I participate in the MIT Mystery Hunt every January. This is not a post about the hunt, or about the puzzles at the hunt, or about the tools used to solve those puzzles. This is a post about nametags.

My team uses nametags to help us keep track of people, and because it’s fun to make nametags. Because we have the fictional persona of a law firm called “Immoral, Illegal, and Fattening” every person’s nametag contains something immoral, illegal, or fattening that they are defending. This has led, as intended, to some hilarity in this part of the nametag. Two years ago, my housemate Deborah Kaplan defended “a’;DROP TABLE `puzzles`;” a classic SQL Injection attack. But our nametags aren’t assembled using SQL. They’re assembled by parsing the team roster with Perl and creating a PostScript document that is sent to a printer. So this year, Deborah, my colleague Joy Nicholson, and I wrote a PostScript injection attack.

PostScript

PostScript is a language created in the 1980′s by Adobe to allow simple creation of files that are sent to printers. It is generally used as a communications protocol between computers and printers. But it’s a full programming language, written in ASCII, and as such can be very useful as a scriptable language format for producing printed material. Our nametag code, written by my brother Denis Moskowitz, is an example. It produces a single PostScript file that contains a nametag for every team member, with all of their information included. The resulting file looks something like this:

Nametag: Immoral, Illegal, and Fattening. Name: Marc Moskowitz. Defending Sloth

So what did my injection attack look like? It’s simply:


) pop /show { pop } def (

How does this work? It relies on three of the language commands (pop, show, and def) and three language syntax elements, strings, names, and procedures.

PS uses parentheses to delimit strings, so the close-parenthesis at the beginning simply closes the string that the program has started to contain the item. Similarly, the final open-parenthesis gives the program a new string to end with the close-parenthesis that is expecting to close its open parenthesis. So the two parentheses are giving us a little space to work some mischief. PS is a stack-based language. All literal elements simply put an item on the stack. A stack-based language needs stack manipulation commands, and pop is the simplest. It just removes the item on the top of the stack and discards it. So the first pop just gets rid of the string we just created. PS sets off items that are used as names of functions and variables with slashes. So “/show” puts the name “show” on the stack for definition by later commands. PS delimits procedures, groups of commands for later use, in functions with braces, which can then be used by other commands. In this case, we’ve created a procedure that simply calls “pop” (which removes the top of the stack) and completes. As you may have guessed, def takes a name and a procedure off the stack and defines a function with that name. So when the processor reads “/show { pop } def” it defines “show” as a function that does the same thing as “pop”. But “show” already had a definition, which we’ve just overridden. The show command is used by the language to take a string and display it on the printed page. So in 24 characters, we’ve disabled the ability of the program to print text.

Did it work? Technically, yes. The day my brother ran the code to produce the badges, he sent me mail with the subject “Curse you Bobby Tables!” containing a PDF that was multiple pages that looked like:
Multiple nametags with all text missing
So I did pull it off. What I didn’t do is surprise him or inconvenience him. He had noticed the code when I entered it, and after producing this image, he made an easy change to escape parentheses in string literals. But as a fun hack, it definitely passes, and it’s a useful warning that any unsecured system can be affected contrary to the designer’s intentions.

Categories programming, security
« Older Entries
Follow

Get every new post delivered to your Inbox.

Join 297 other followers