Archives for category: hardware

Nagios Plugin for Arduino Environmental Monitoring

By Scott Pustay
February 10, 2013 //
0

Last November I wrote an article explaining how to capture temperature data with an Arduino. These data were sent to the serial bus on a computer via USB. Now that the data are on the computer, we can track and monitor their flow.

Firstly, we should store these data in a database. I rewrote the serial reader to dump data into mysql.

import serial
import time
import MySQLdb

dbhost = 'localhost'
dbname = 'DB_NAME'
dbuser = 'DB_USERNAME'
dbpass = 'DB_PASSWORD'

ser = serial.Serial('/dev/ttyACM0',9600,timeout=1) # On Ubuntu systems, /dev/ttyACM0 is the default path to the serial device on Arduinos, yours is likely different.
while 1:
    time.sleep(10)

    the_goods = ser.readline()
    str_parts = the_goods.split(' ')

    conn = MySQLdb.connect (host = dbhost,
                    user = dbuser,
                    passwd = dbpass,
                    db = dbname)
    cursor = conn.cursor ()
    sql = "INSERT arduino_temp (temperature) VALUES ('%s');" % (str_parts[0])
    try:
        cursor.execute(sql)
    except:
        pass
    cursor.close ()
    conn.commit()
    conn.close ()        

    print the_goods

Next, I want to track this sensor feed. I have a nagios server so I wrote a plugin to check the arduino output in mysql.

#! /usr/bin/python

import datetime
import time
import MySQLdb
import sys

dbhost = 'localhost'
dbname = 'DB_NAME'
dbuser = 'DB_USER'
dbpass = 'DB_PASSWORD'

# Make sure this file is executable. You should chmod +x sensor_status.py

def main():
    
    mysql_datetime = ""
    current_temp = ""    

    conn = MySQLdb.connect (host = dbhost,
                user = dbuser,
                passwd = dbpass,
                db = dbname)
    cursor = conn.cursor ()
    sql = "select created_at, temperature from arduino_temp where created_at = (select max(created_at) from arduino_temp);"
    cursor.execute(sql)
    rows = cursor.fetchall()
    for row in rows:
        mysql_datetime = row[0]
        current_temp = row[1]
    cursor.close ()
    conn.commit()
    conn.close ()        
    
    timediff = datetime.datetime.now() - mysql_datetime
    
    if int(timediff.total_seconds()) < 15:
        print "Shed was %s Fahrenheit and checked only %s seconds ago" % (str(current_temp), str(timediff.total_seconds()))
        sys.exit(0)
    else:
        sys.exit(2)


if __name__ == "__main__":
    main()

Nagios is monitoring software that uses a really simple algorithm for assigning status. Nagios runs your bash/python/whatever script and looks for an exit code of 0-3. Zero means everything is all right, 1 is a warning, 2 is critical, and 3 is unknown.

Nagios has a simple configuration that defines command and services. The executable sensor_status.py script is defined with the following command.

define command{
	command_name	arduino_temp_sensor
	command_line	/YOUR_PATH_TO_FILE/sensor_status.py  # chmod +x sensor_status.py
	}

The machine connected to the arduino sensor runs a service that calls this command.

define service{
        use                             generic-service         ; Name of service template to use
        host_name                       localhost
        service_description             Shed Temperature
        check_command                   arduino_temp_sensor
        }

Sensor_status.py checks if the current system time and the last mysql record are more that 15 seconds apart. The arduino serial_reader_mysql.py inserts data every 10 seconds into mysql. This fifteen seconds check is enough time to know if data are not flowing properly.

If everything is working, Nagios will look like this.

Nagios_screen_shot

As you can see, monitoring your arduino temperature sensor is relatively simple. Nagios is a great way to make it happen. You can get all the source code to this project at
https://github.com/muskox/arduino_env_monitoring/

Categories hardware, mysql

Touch Input and Responsive Web Design

By Matthew Irish
November 28, 2012 //
1

With the release of Windows 8, touch has finally landed on the desktop. There have been some great posts about the design implications of this change, but I haven’t seen much about technical implications, so I decided to write this. Chances are you’ve thought about  touch interaction if you were building a separate mobile site, but what does it mean for a desktop site or a responsive site?

Why Optimize for Touch?

Why is this something you’d want to do? It boils down to making things better for those who visit your site regardless of the device they’re using. This is why many people and companies have chosen Responsive Web Design. But what does Windows 8 have to do with any of this? It’s the first desktop operating system to include touch (and pen input, but more about that later) as an OS-wide, first-class input mechanism. So suddenly, if you have touch hardware, your desktop browser is going to be firing real touch events instead of using a third-party software abstraction to fire mouse events in the OS.

Who should be checking to see if their site needs any changes? If you’re doing anything with JavaScript click-handlers – most sites these days are doing something - then there are some touch-related things to think about. Additionally, maybe you want to consider gestures of some sort or you want to incorporate two or three-finger touch events. But the most important thing to consider is how you handle the simple single-touch event. Often you’ll want to trigger a touch event instead of click.

I’m going to focus on this last one as I think it’s the most common case. The reason single-touch needs “fixing” is that click events incur a penalty in the form of a 300 millisecond delay on many touch-enabled devices. This is to allow the OS to determine if you meant to pan or zoom instead of just clicking. You can speed up this interaction by checking those things yourself.

Optimizing (allthethings) for Touch

There are a few ways to do this. You can feature-detect touch events and then bind to “touchend” instead of click. (Modernizr is great for this but may be overkill for this one use case.) If you decide to go this route you will have to manage everything yourself: handle touch, determine if the user is scrolling/panning/zooming (if they are: don’t fire the handlers, if they’re not: fire it), and then after you fire it there will still be a click event fired.

What—you were expecting this to be straightforward? We are dealing with web browsers after all! What can we do to make this easier? Use a library, of course! But before doing that, I recommend reading through this Google post (this was the first place I read about the technique) so that you understand everything involved with overriding the default browser behavior. The library I’d recommend using is FTLabs’s FastClick. Be sure to read through the source too, it’s well documented. The nice thing about FastClick is that you only have to bind it once for the whole document and it only inserts itself if your device supports touch, as it has the feature-detection built in.

The Touch Future (or: When Touch is Not Enough)

Touch and click events are closely related as there was some effort to have backwards compatibility. But leave it to IE to break up the party. Turns out IE10 and “Metro” apps don’t have touch events (read more about the problem in this Modernizr issue on Github). Instead, Microsoft has abstracted and unified input mechanisms with what they’re calling pointer events. Instead of binding to touch or click, you bind to pointer event and the OS knows when to fire it based on the input method you’re currently using (touch, mouse, pen, laser, etc.). Boris Smus has written a polyfill for pointer events and has a great post explaining it, though I think it might not be ready for primetime. Recently, the W3C has formed a committee to begin the standardization process for pointer events, so it’s definitely something to keep an eye on.

I think pointer events will be the way forward because they are a better abstraction, and you don’t have to do a bunch of hokey pokey to get things to work properly, you simply let the browser do the heavy lifting for you. Even if that’s true though, touch events aren’t going anywhere anytime soon. They’re something you’ll need to feature-detect (there’s talk of defining Modernizr.pointer or Modernizr.mspointer in the above Modernizr issue) and then do what web developers always do: treat IE differently.

Categories design, hardware, mobile, responsive design

Distributed ebook file conversions with Amazon EC2: Part 1, the stack

By Liza Daly
November 21, 2012 //
3

It’s Friday afternoon. Your new ebook product is launching on Monday and you have 1,000 files to automatically convert from a legacy format to EPUB. You figured it would take a minute or two per file, but that was before you tried the converter on some actual commercial ebooks, the kind with lots of really big images. It turns out that if you converted all the books one at a time on your laptop like you’d been planning, you’ll be ready to launch in 2015.

Don’t panic. There’s always EC2.

Amazon’s cloud-for-hire service is widely used by internet startups (and post-startups), but it’s a potential lifesaver for any kind of large-scale computing task. Like it or not, the publishing industry today is about producing computer-readable data. Even if you just make a couple of EPUBs a year, you’re developing software, and if you make several hundred EPUBs a year, you should probably be thinking about scalable data-crunching techniques.

In this post, I’ll describe the data conversion problem and the architecture I used to crank through almost a thousand files in a couple hours. In part 2, I’ll walk through the Python code that powers all the pieces and assess the cost/benefits of this kind of approach. Programming skills are assumed.

Entering the EC2 world for the first time is intimidating, even for an experienced developer. There’s a lot of terminology, and books and articles get out of date fast. This post isn’t a comprehensive tutorial, but I’ll try to point out the places where it’s easy to get stuck or confused.

For my ebook conversion project, I had to solve several problems:

  1. I had to process a lot of files in a short amount of time; that was impossible even with the fastest single computer I could find. So I’d need parallelization.
  2. I had to assume that my conversion process might have bugs to be fixed or improvements to be made in the future. So I needed repeatability.
  3. I didn’t intend to spend my weekend turning on and off hundreds of servers, and the penalty for leaving servers running unnecessarily could be a very expensive bill. So I needed server provisioning to be scriptable.
  4. Once I spun up my servers in parallel, I needed a way to distribute the work across them in an orderly way. I didn’t want to waste time and money converting books unnecessarily.

Why EC2?

  • It’s metered. You can rent very, very powerful hosts for only as much time as you need.
  • It’s scriptable. Once you get the basics set up the way you like, it doesn’t much matter whether you want to run the same process 1 time or 1,000,000 times. I turned to the widely-use boto Python library to script my EC2 provisioning process.
  • It’s repeatable. This is more than just about being scriptable; you can create a snapshot of the host exactly as you want, with all of the dependencies pre-installed. If you need to run the same process again in a year, nothing will have changed.

The one thing that EC2 couldn’t magically solve for me was the distribution problem: each EC2 virtual computer would be a newborn baby, with code all ready to go but no idea which files to process. After flailing around for awhile I settled on using a queuing service. Amazon includes Simple Queue Service with their offering so I was all set.

Customizing the EC2 instance

An Amazon Machine Image (AMI) is your virtual computer. AMIs are a complicated subject, but for general purpose Linux needs I pick one of the official Ubuntu AMIs. Customize the AMI storage/memory allocation to match your application’s needs. In my case, I needed a fair amount of both storage and RAM as the conversion process generates a lot of temporary files, but it turned out I didn’t need the most robust processor. (If you need a lot of storage or want lots of files pre-installed, you’re going to also need to configure your own elastic block instance and mount it; avoid this step if you can because it’s a pain.)

User data

You can (and should) install all the dependencies you need on your AMI before taking a snapshot. However, if anything about your application changes, you’ll need to rebuild the AMI, an annoying and non-automatic process. User data, a feature available on some instance types, is a simple user-authored script that will run on boot, and can be provided on instantiation. Since my converter’s source code was available on a private Github repository, I set up my user data script to automatically pull the latest version of the code, which was pre-installed with the correct SSH keys in the AMI:

#!/bin/bash

cd /home/ubuntu/ebook_converter

# User data scripts run as root, so ensure that we "su" to the local user account
sudo -u ubuntu git pull origin master

My conversion script runs as a Python program inside a Python virtual environment. It’s possible that the latest version of the code has new dependencies, so after the instance pulls the latest code, the user-data script also runs setup.py develop:

su ubuntu -c "source ve/bin/activate && python setup.py develop"

The job queue

To solve the workload distribution problem, I needed a centralized place to store all of the books to be converted. Originally I thought I’d spin up each EC2 instance with a predefined set of books to convert and pass that list in to the user-data script, but I realized a far better approach was to use a queue. A queue is just a data structure that maintains an ordered list and allows items to be added or removed programmatically. If I’d provisioned the work list for each instance myself, I’d have to deal with potential problems like an instance crashing during processing and leaving half its workload untouched. Better to leave that to the queue.

(The AMIs and the queue need to be in the same Amazon region or they won’t be able to find each other.)

The final pipeline

I ended up with a workflow like this:

  1. Get a list of books to convert.
  2. Push the list of filenames to the queue.
  3. Start up the maximum number of simultaneous instances (20).
  4. For each instance:
    1. Update itself with the latest code.
    2. Get a filename off the queue.
    3. Go get the file.
    4. Convert it.
    5. Put it somewhere.
    6. Repeat.

Each instance keeps requesting books off the queue until the queue is empty; at that point it can just shut itself down. If an instance crashes, there are still others to pick up the slack.

The entire workflow can be written straight into the user-data startup script. I ended up with this script, which self-updates the code, runs “forever” (until the queue is exhausted), and then turns itself off:

#!/bin/bash

cd /home/ubuntu/ebook_converter

# User data scripts run as root, so ensure that we "su" to the local user account
sudo -u ubuntu git pull origin master

# Invoke the Python virtual environment and update any Python dependencies
su ubuntu -c "source ve/bin/activate && python setup.py develop"

# Invoke the virtualenv and run the process
su ubuntu -c "source ve/bin/activate && python ebook_converter/process_from_queue.py"

# Once this finishes, we're out of jobs on the queue, so shut down the host
shutdown now

EC2 lets you remotely monitor any server, so if your job writes logging information, you can watch that in real time. Here’s the output from my converter as it starts up and picks a file off the queue to download and convert:

Or better yet, take advantage of a program like multitail and watch all the log output simultaneously! Shortly after starting up 20 instances, my terminal window looked like this:

The primary purpose of running multitail is to feel like a total rockstar and be able to email this screenshot to all your coworkers. Or better yet, send a photo of you drinking a beer in a hammock while 20 computers in the cloud slave away on your behalf at the click of a button.

In the next post I’ll walk through the Python code that powered the provisioning, distribution, and queuing of the jobs, as well as the economics and practicality of this approach.

Categories cloud computing, ebooks, hardware

The little Pi that could

By Clark Hartsock
November 15, 2012 //
0

‘Big Iron’ computing is fun. SANs, fiber, clusters and data centers are where my work days are spent – and I love the work!

At the same time, ever since I wire-wrapped my first z-80 based computer, I have been captivated by the potential of small, affordable, and modestly powered computing. In the past, single board computers based on mainstream (x86) technologies have been interesting, but were usually significantly more expensive than a hobbyist/tech tinkerer/product inventor (dreamer!) could afford.

More recently there has been a surge of interest and creation of a wide range of embedded or single board computing options – from both the commercial space and Open Source communities. Examples of these include the Parallax Basic Stamp, Beagle Board, Gumstix, and many more.

The two platforms that have most recently caught my imagination are the Arduino and the Raspberry PI. The Arduino has been extraordinarily successful at providing an affordable, flexible platform for embedded controllers and dedicated single function computational platforms. What has been missing, until recently, is the general computing equivalent of the Arduino.

When I first read about the Raspberry Pi – that in addition to their original goal of an affordable education platform, “Developing countries are interested in the Raspberry Pi as productivity devices in areas that simply can’t afford the power and hardware needed to run a traditional desktop PC; hospitals and museums have contacted us…” – I thought to myself, “That’s cool, but I think they are optimistic about the $35 price point.” Then, of course, I ordered one right away.

While waiting for its arrival I considered a number of options to put this little board to the test. Then while following the Twitter streams for a number of co-workers attending the Books In Browsers conference in San Francisco this year it hit me – I must get our Ibis Reader EPUB 3 browser-based book reader working on my Pi!  I see this as an interesting blend of browser reading with an added potential for open access to information and the opportunities created by lower-cost computing.

Doing the installation seemed reasonably straightforward. There are several OSs available for the Pi and the ARM based Debian distribution called Raspbian is very similar to the environment that we use internally. I booted the os, installed a few of my favorite tools (screen, git, etc), installed dependencies, and started installing Ibis using our standard procedure.  The first build failed rather quickly – mostly because of differences in locations of files in Raspbian compared to our normal environment. After small number  of iterations finding these files and installing them I started what I thought would be ‘the’ build. After some time of looking like an easy and successful build, gcc barfed and the build failed with:

error: command 'gcc-4.6.3' failed with exit status 3"

This was new to me – and a quick search of a number of C programming books in our library and some web searches turned up nothing!

The Pi has 256MB of memory and it is divided roughly 40/60 between video buffer and system memory. I had intentionally tried the build out of the box with no memory reallocation and though gcc left no real information as to why it failed, I was sure that low memory was the cause. I reassigned as much RAM away from the video buffer as I could (I am only using terminal sessions after all) and tried again. The build failed again, gcc barfed again, but this time farther along. At this point I decided to give the Pi gobs of memory by attaching an 8GB USB stick so that it has 2GB swap and the remaining 6GB as /home. Here is what the configuration looks like on my desk:

The next build completed successfully! The capacity of this configuration is extremely modest. Two simultaneous sessions work, and the math isn’t optimistic about many more, but for $35 (plus a spare usb stick) and less than 1 watt of power, this is a true general purpose application which hints at the potential of small, affordable, and modestly powered computers.

I’m thinking the next experiment might be making a solar power system to run a cluster of these 24/7 around the clock. Or maybe a DIY S88/S95 compatible process control system with the Pi supervisory control units and Arduino loop-controllers. Or when my 16 core $99 Parallela board gets here I’ll make a solar powered Software Defined Radio, data telemetry, and epub serving robot to roam the seas.

Why not?    :-)

Check these out:

Getting Started with Raspberry Pi

Raspberry Pi at the Maker Shed

(12-12-12 edit:  - link to 
http://pi.ibisreader.safarilab.com:9021
 removed, as this cool little board is moving on to it’s next project. Stay tuned!  ch)

Categories hardware

Environmental Monitoring with Arduino

By Scott Pustay
November 10, 2012 //
3

As a software engineer, I love walking into our server room hearing and seeing the sounds and sights of our machines. Fans whir and LEDs blink while the HVAC unit provides a low bass rumble blowing cold air into the CPU cores. If everything is going right, this room should stay at a constant temperature. However, this is not always the case. Environmental monitoring and alerting is a way to be notified when the temperature is out of the desired range.

Sensor Data

Environmental monitoring in the server room involves getting data from sensors; it should always include a thermometer (temperature sensor) and may include other inputs like humidity sensors. Commercial products for environmental monitoring are usually proprietary and closed source, so your data is locked into someone else’s system instead of having full access to your own data. Many commercial systems are Windows-based and do not easily work in a Linux environment.

DIY

In the open source do-it-yourself spirit, I set out to see how we could gather our own environmental monitoring data on an open hardware platform. I looked at community sites to see which open hardware project had the most followers and documentation, as well as practical tutorials. I came across the Arduino Cookbook, 2nd Edition by Michael Margolis. Its Chapter 6 discusses measuring temperature. This was a good place to learn about using a readily-available temperature sensor.

A simple way to get temperature input is via the LM35 sensor. This outputs a voltage based on the degrees Celsius it senses. In an Arduino sketch (code below), I set the Arduino to poll the sensor for temperature every 10 seconds. The Arduino then returned the data to the server via the serial bus connection.

/*
This code is from Arduino Cookbook 2nd Edition by Michael Margolis.

http://my.safaribooksonline.com/book/hardware/arduino/9781449321185/6dot-getting-input-from-sensors/id3027173

*/
const int inputPin = 0;

void setup()
{
  Serial.begin(9600);
}

void loop()
{
  // LM35 temperature sensor input
  int value = analogRead(inputPin);
  float millivolts = (value / 1024.0) * 5000;
  float celsius = millivolts / 10; // LM35 sensor output is 10mV per degree Celsius
  // convert to fahrenheit
  float fahrenheit = (celsius * 9)/ 5 + 32;

  Serial.print( fahrenheit );
  Serial.println(" degrees Fahrenheit");

  delay(10000); // wait 10 seconds
}

On an Ubuntu machine, I ran a simple python script (code below) that looked at the same serial bus and grabbed the Arduino output.  The data was then appended to a text file as well as printed to the screen.

import serial
import time
import datetime

ser = serial.Serial('/dev/ttyACM0',9600,timeout=1) // On Ubuntu systems, /dev/ttyACM0 is the default path to the serial device on Arduinos, yours is likely different.
while 1:
    time.sleep(10)

    the_goods = ser.readline()

    f = open('/home/spustay/sensor_data.txt','a') // put your path here
    f.write(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + ' ' + the_goods)
    f.close()

    print the_goods

Results

I tested this system out on a server rack I keep at my house in a non-climate controlled shed. While this isn’t a real server room (or anything like a real data center), the daily temperature fluctuations we have in Northern California made the data more interesting. This sample data is included on the github page for perusal.

Extension

Environmental monitoring with data collection such as this could be extended in many different directions. Because it is open source to the core, code and hardware enhancements could be added or modified, allowing the data to be linked to other projects.

Conclusion

After doing research and getting the right parts, I was able to hook up the sensor and collect data in less than 30 minutes.  Needed supplies include an Arduino board (I used the Arduino Uno), a LM35 temperature sensor, a USB cable, and a computer to capture the data. Environmental monitoring with Arduino is just the beginning of a robust IT infrastructure where you control your data.

Update

I wrote a follow up article explaining how to monitor your data stream.

Categories hardware, learning
Follow

Get every new post delivered to your Inbox.

Join 298 other followers