Log in

No account? Create an account
I've got a couple of kids who are beginning to use the computer more and more for their own purposes. A couple of days ago, My son couldn't enable monsters on his Minecraft server, because he didn't know which of the files was the properties file, "Server" or "Server".

Yeah, "Server" or "Server". I forgot to check "Show file extensions" for his user profile when I set up the computer, even though that's one of the things I always do for new installations for myself.

I set up his preferences. There: One was "Server.log" and the other was "Server.properties." Now he could find it. That got me to thinking that these kids should have a primer on computers. Things like:
  • What do computers do?
  • What are files?
  • Where is the stuff I'm working on?
I started drawing clouds, computers, disks, and even WiFi antennae. And then it hit me: Minecraft! Minecraft is a great usecase for these kids. They're familiar with it!

Textures are stored on everyone's individual computer. That's local file storage. That's why everyone can have a different look for the same world. That's why when the same kid goes to different computers, the different computers might render the same Minecraft world differently. The texture is local to the computer.

Worlds are stored at the Minecraft server. And the server tells every player about the world. That's how all players can share the same world. The one server tells everybody the same story.  In our case, access to the server is contained to our home network.

Skins are stored on the Internet. Notch has a huge database with everybody's skin on it. That's why you can go to different computers, log on to different servers, but your skin always follows you around. Everytime you log in, your computer retrieves the skin from Notch's one huge skin database in the Internet.

The actual explanation goes a little differently, and is age appropriate. But it's useful that this one game that they love, customize, and know well actually illustrates the difference between local file storage, home networks, and the Internet.

Scene from the Minecraft server logs

Our household server logs reveal that stratestar110, a 10 year-old girl was playing with bumppup, an 8 year-old boy, playing as the admin. The first transaction was innocent enough:

[INFO] <stratestar110> CAN I HAVE BRICK BLOCKS
[INFO] CONSOLE: Giving stratestar110 some 45

The girl asked the admin to give her some blocks, and "CONSOLE" shows that he issued the command to give her what she wanted. Then the following happened:

[INFO] <bumppup> Lok at me.
[INFO] <stratestar110> WHERE ARE YOU
[INFO] <bumppup> Look at me!
[INFO] <stratestar110> WHERE ARE YOU
[INFO] <stratestar110> HOW DO YOU DO THAT
[INFO] <bumppup> Look at me!
[INFO] <stratestar110> CAN YOU GIVE NE 64 BRICK BLOCKS
[INFO] <bumppup> I am down here.
[INFO] <bumppup> Look at me then I wiil give you 64 bricks.
[INFO] CONSOLE: Giving stratestar110 some 45

They were in classic form. The boy just wanted somebody to look at something cool he was doing, and the girl just wanted some more stuff. Same as it ever was. Those are my kids.

Fire in the Belly

My pet projects seem to choose themselves, as if I had nothing to do with it. Yesterday, one sunk its teeth into me and won't let go. Tenacious little bugger.

Maybe I should say my muse gave me a present for New Year's. But it doesn't feel like that. When a project like this, no matter how small, takes hold, there's little else you can do. You can't enjoy sitting with a cup of coffee and a book. You can't enjoy working out, or watching a video. You're compelled to explore the idea.

This time, the project came from a comment that Sjon Svenson left in my last entry, Happy Birthday, Me. I got you data portability.

I was happy that I'd completed the migration of my contacts to Google Contacts, but Sjon mentioned that a hardcopy of his contacts survived an electronic device failure, and the hardcopy is the version that the whole family maintains now.

The beauty of that is that my brother can update that just as well, and his wife can -and does-.

That is really handy. So it's something that I'm going to implement across my family's Google Contacts accounts. We'll use the tags to designate with whom we want to share the contacts, and only those will be synced across accounts. Everybody will benefit from the improvements anybody else makes. It'll also keep a history of changes, in case changes or merges were incorrect.

It's very similar to a private wiki, or a shared Docs page, but it's important that this is done directly to the contacts that get synced to all of our mobile devices, too.

It seems like a pretty obvious project. If there's already an open source project for this, or if this is already your 20% project, leave me a comment.

[Update 2011-01-03]: Drat. The proper way to do this, with OAuth and the contacts stream in the official feed won't work. It doesn't enumerate all the fields available when you export contacts. I can still do what I want as a one-off, because I can have my script get the exported CSV values from the accounts I need access to, but I dearly hate code that scrapes or otherwise does actions meant for humans. I wish Google provided (or made apparent how to get) all the data, every single field, for each contact in the API's feed.
My list of addresses has made its way from a physical address book, to a Palm Pilot, to Microsoft Outlook, to Google Contacts to the iPhone Contacts app. Along the way, each of the transitions has played fast and loose with the mappings of the individual fields.

Nowadays, the normal way for a Windows user to export contacts from an iPhone is to sync between the iPhone and Microsoft Outlook, then export from Outlook to a CSV file. I hate having to go through that middleman.

I prefer to take stronger ownership of my own data, and have settled on Google Contacts as the primary place for my data. There are a few reasons. I really like the open "group" (tagging) feature of contacts. I like that they have a public API for accessing and manipulating contacts. But most important is the ability to easily export and import contacts. As Joel Spolsky noticed ten years ago, a good way to get me to try a service, is to make it easy for me to change my mind and leave the service.

By default only the contacts group named "My Contacts" will sync with the iPhone when you set up an Exchange sync between the two end points. That suits my purposes just fine.

I've taken some time to groom my contacts list for Google Contacts. Here are some notes from that experience:

When you export contacts, use the Google CSV format. Your contacts will be exported to a UTF-16 file, and all the special characters you use will be retained. If you choose Outlook CSV format, then the file generated will be 8-bit regardless of the characters used in your contact list, and characters that don't map to 8-bit characters will be changed to question marks. So for 安室奈美恵's sake, choose Google CSV format.

Most people edit their CSV files in a spreadsheet editor. That's fine, but I don't trust my own eyes and hands to get everything right, so I prefer to do batch editing programmatically.

If you want to do some batch processing of the CSV file in Python, here are some snippets. These snippets have been pared down to the essentials, and don't represent good coding practices.

To read in the Google CSV:

# unicode_csv_reader and UnicodeWriter are provided
# in the documentation from the cvs module.

f = unicode_csv_reader(codecs.open( 'google.csv', 'r', 'utf-16' ))

headings = f.next()
col = {}
for i in range(len(headings)):
    col[headings[i]] = i

rows = []
for row in f:

If you want to print out a list of contacts sorted by last name:

rows.sort(key=operator.itemgetter(col['Family Name']))

Beverly Howard suggests that contacts with no "Name" field but only a "Company" field won't sync with Outlook.
You could check for that:

if row[col['Organization 1 - Name']] and not row[col['Name']]:
    # Ensure these get synced across all devices, some don't!

Finally, after you've made your batch changes, you can write them back to a UTF-16 file like so:

out_file = open( 'google_out.csv', 'wb' )
google_out = UnicodeWriter( out_file, encoding='utf-16' )
google_out.writerow( headings )
google_out.writerows( rows )

It's been a long time coming, but I'm glad I've got more of my data in a place where it's easy for me to get it out and manipulate it any way I like.
Around 1982, I was a teenager working off-the-books at the local arcade. I took out the trash, cleaned the front windows, the bathroom, and the machines for a handful of bills that'd usually get converted to tokens and fed right back into the machines.

When I went to church, I had a trick to stay awake. I imagined playing the upper levels of Tempest.

Planning judicious use of the Superzapper is enough to get you through even the most boring sermons.

That was the year that Tron came out. And Tron spoke to me personally. I had an Apple ][+ and loved programming on it. Computers were full of so much potential, and Flynn knew how to take advantage of that. I wanted to be him.

That helped set my life on its course. I went to college, studied Computer Science, and got a job as a Programmer, just like Flynn.

I'm still a programmer, and I watch over a village of cron jobs and daemons who report to me how they're doing, and have permission to email me when they get into trouble with the other programs on the net.

Now it's 2010, and Tron Legacy has come out. Nobody's mistaking it for high art.

Tron Legacy did get some things right. In both the original and the sequel, life in the Grid was surreal, and I mean surreal etymologically. Life in the grid was above or beyond reality. In 1982's Tron, artists hand-painted the digital glow to programs in the Grid, and in 2010's Tron, only the scenes in the Grid were 3D.

There was reality, which was normal; and then there was the grid, where its reality was capable of more. I still feel that way about computers and the Internet. Nearly anything is possible.

Computers remain the realm of fantastic opportunities.


Once you grow up, hold down a steady job, marry your girlfriend and raise a family with her, what do you dream about? Getting your next promotion? The stock market? Political success?

Not in my case.

A few days ago, I had this dream where everyone but me was sorta Simpson's look influenced. They were life-like and cartoony both at once. Towards the end of the dream, it got more realistic. The scene was a bay with some steep rocky walls falling into the ocean water. My nemesis was a great white shark, Jaws style. He hadn't killed me yet, but he would the first chance he got.

I was doing battle with a hot vampire babe on the cliff by the bay. I had her in a wrestling lock, but she had a counter on me, we were sorta stalemated. If I moved, she might be able to get a bite in. Or, she'd be able to throw me into the bay, where the shark was waiting.

It was just before dawn, and if the sun hit the vampire babe, she'd perish. I could tell from the way she was fighting me, that she knew it was going to be the end for one of us soon, and she was considering allowing me to keep her in this stalemate until the sun got her. Did she love me? Would she allow me to live at the cost of her own life? Would I go on to defeat the shark?

I woke up.

But it was the beginning of the best day ever.


It really frustrates me to have to follow up Amazon Still Hates Your Life(stream) with more poor decisions by the eCommerce leader and giant.

In late October 2010, Amazon removed the ListLookup operation from the AWSECommerceService Service. When I noticed it, I tweeted about it. I was sorta late to the game, though. Amazon announced they would deprecate the service in June 2010. The thing is, they didn't deprecate the service. They removed it. Trying to call it now results in an HTTP Error 410, and the text, "Gone."

What's that mean? Well...

That's the function to access your wishlists at Amazon. Some people wrote blogging engine tools or phone apps that access Amazon wishlists. The more people that make easier access to wishlists, the more items that get purchased. You'd think Amazon would like that, right?

You'd be wrong. Amazon unplugged the mechanism they wrote to programmatically access their wishlists. I really can't see a justifiable reason behind it. It may have been "little used" by their standards, but it couldn't have been that hard to maintain.

You can blame Amazon for making their own wishlists even less accessible and useful now.

Remote Backup Doublechecker

My Remote Backup Script is working nicely. After the backup, it writes some status to a file, "rsync_completed.txt."

But I noticed a few days ago that one of my backups didn't run. That's probably because the computer wasn't on at the designated time, or possibly because nobody was logged in, or that the system was too busy to run that particular task.

In any case, I wrote a Remote Backup Occurred Doublechecker. It runs every time I log in.  Because I'm crazy. I thought about making it a DOS batch script, but went with Python because it'd be faster for me that way.

And just to reveal my craziness, here it is (mostly):

    rsync_file = os.path.join(my_root, "rsync_completed.txt")
    scriptname = sys.argv[0]
    if os.sep in scriptname:
        scriptname = scriptname.rsplit(os.sep, 1)[1]
    ask_user = False
    if not os.path.exists(rsync_file):
        import win32con
        ask_user = True
        msg = "Could not verify backup with file %s. Backup now?" % rsync_file
        flags = win32con.MB_ICONWARNING | win32con.MB_YESNO
        mtime = os.path.getmtime(rsync_file)
        dur = datetime.timedelta(seconds=time.time() - mtime)
        if dur.days > 2:
            import win32con
            ask_user = True
            msg = "It's been %s days since the last backup. Backup now?" % dur
            flags = win32con.MB_ICONQUESTION | win32con.MB_YESNO
    if ask_user:
        import win32ui
        response = win32ui.MessageBox(msg, scriptname, flags)
        if response == 6:
            import subprocess
            cmd = os.path.join(my_root, "backup_to_dreamhost.bat")
except Exception, e:
    f = file(os.path.join(my_root, "Desktop", "%s Fail.txt") % scriptname, 'w')
    f.write("An exception occurred: %s %s\n" % (str(e.__class__), str(e)))
    traceback.print_exc(file = f)
Dreamhost offers a personal backup service.  Here are some notes on what I did to get it working from my Microsoft Windows Vista system.

Install All Required Cygwin Modules

First, I installed any missing required modules for cygwin.  At first, I had trouble with ssh, because it complained it couldn't find cygssp-0.  Using cygcheck confirmed the missing library.

~$ cygcheck ssh
Found: C:\cygwin\bin\ssh.exe
Found: C:\cygwin\bin\ssh.exe
Found: C:\cygwin\bin\ssh.exe
cygcheck: track_down: could not find cygssp-0.dll

Once I installed libsso0, rsync was working correctly, but required a password to be entered.

Setup Passwordless Login

Setting up passwordless login was really easy. In an sftp connection, I made a .ssh directory from my home backup server's directory, copied my cygwin's .ssh/id_dsa.pub to the new .ssh directory and renamed the file to authorized_keys.

Determining Directories and Files to Copy

I made an exclusion list of files not to backup and named that file excl.txt. Here's what the file contains:


Then I made a shell script called, backup_to_dreamhost.sh, to backup only certain directories:

rsync -e ssh -avz --exclude-from=excl.txt /cygdrive/c/Users/David/Documents user@b.dh.com:~/David
rsync -e ssh -avz --exclude-from=excl.txt /cygdrive/c/Users/David/Downloads user@b.dh.com:~/David
rsync -e ssh -avz --exclude-from=excl.txt /cygdrive/c/Users/David/Pictures user@b.dh.com:~/David
rsync -e ssh -avz --exclude-from=excl.txt /cygdrive/c/Users/David/Music user@b.dh.com:~/David
rsync -e ssh -avz --exclude-from=excl.txt --exclude=Pinnacle/ /cygdrive/c/Users/Public/Documents user@b.dh.com:~/Public

Note that occasionally I have to exclude some directories that contain massive amounts of video or temporary files. I can't copy my pictures and videos without taking up more than the free 50GB space allocated for backups.

Making Automatic Backups

I ran the shell script from within cygwin, and it worked. But now, how to make Vista run it? I created a DOS Batch script called, backup_to_dreamhost.bat. It contains only one line:

C: && chdir C:\cygwin\home\username && C:\cygwin\bin\bash --login ~/backup_to_dreamhost.sh

And from the Windows Task Scheduler, I created a recurring task that runs backup_to_dreamhost.bat.


Four services I use changed their APIs on me last week.  Four.  What the hey, Internet?


They migrated to the Disqus commenting system.  In the process of doing so, they broke a feature of their RSS feed.  Their feed has the <slash:comments> element for each item, and it used to contain the correct number of comments.

I'm too busy to have to read each TechCrunch article's title to evaluate whether I should read the article. So I wrote a recommendation engine. The number of comments each article accrues is one of the criteria my cron job uses to evaluate TechCrunch articles.  

TechCrunch broke their <slash:comments> element.  It's still there, but it always evaluates to zero.  I fixed my cron job to go get the comment count directly from disqus instead.

TechCrunch should fix their broken feed anyway.  It's not cool to lie in your RSS feed.


Netflix changed the format of their movie URLs.  In some places.  In their new releases RSS feed, the movie URLs separate words in the tile with hyphens, like so:

But their actual API, like api.netflix.com/catalog/titles, returns movie titles with underscores separating the words in the title.

I'm too busy to have to look up a bunch of movies to decide which to rent, so I have a cron job evaluate each week's new releases with NetFlix's personal predicted rating for me.  By changing the format of their URLs in one service, but not the other, they broke my cron job that matches movies in the feed to their corresponding IMDB ratings.  It took me a while to figure out exactly what it was that broke my service!

I fixed that by having my cron job do a fuzzy match that matches to words separated by either hyphens or underscores.


They did a major overhaul when they released V4.  I'm not really interested in the debate over what's better and what's worse.  I'm interested in what they broke.

They broke their user history feeds.  They used to support personal feeds for their users, so that you could easily see what your friends dugg, like so:


Around August 25th, they changed the nature of the feed to also include everything from the people that that user follows.  So instead of being a concise personal history, it became a huge mess.  The next day, they turned off the service altogether.

By changing the nature of the feed, not to mention turning it off altogether, they broke the digg component of my personal lifestream.

Digg should restore the history feeds.  They were useful.  And it's bad form to break services that you used to provide.


Twitter turned off basic authentication and left OAuth as the only alternative.  They announced the transition, and gave developers a long time to prepare for it.  It's a good thing.

Sadly, I was using basic authentication to munge together two of their feeds into one, for inclusion into my feed reader.


So for me, all Twitter activity suddenly disappeared one day.  It took me a while to realize that I'd forgotten to migrate my feed collator's authentication from basic to OAuth.  So I went ahead and made the fix.

Oh, the awesome thing about making a certified OAuth App for twitter?  I can integrate it into my dead man's switch.  Maybe I'll tweet from beyond the grave.


In one week, four external services broke four of my personal services.  It felt like so much household maintenance: The toilet broke, or the grass needs mowing. The upside is that in fixing each of these personal services, I added to my skill set.