After Jobs Dry Up, What Then?

An excellent distillation of one of the most important issues of the next decade.

Update (2015-03-29): This post sparked a few conversations with friends that have caused me to revise my opinion. I still believe that rising economic inequality, lack of opportunities, decreased social mobility, and poor measures of success focused on economic growth are all major problems for our society. I’m not convinced that technological progress is the major cause of this shift or that this technological age is fundamentally different from previous periods of “creative destruction.” That said, I do still worry about the types of new jobs that are currently being created, many of which in the “gig economy” offer very little economic security and no benefits.

Geeking Out

Introducing Hygroscope

Hygroscope is a command line tool for managing the launch of complex CloudFormation stacks in Amazon Web Services.

CloudFormation is a tool for creating and managing Amazon Web Services infrastructure using code. A JSON-formatted template describes the state of a “stack” including such resources as servers, S3 storage buckets, and load balancers. Utilizing the AWS Virtual Private Cloud service, entire software-defined networks can be described and repeatably created, updated, and destroyed using CloudFormation.

CloudFormation is not without its pain points:

  • Templates must be written in JSON, which, in addition to being difficult for a human to read, does not support niceties such as inline comments and repeated blocks.
  • Launching CloudFormation stacks requires knowledge of the various parameters that need to be provided, and it is difficult to repeatably launch a stack since parameters are not saved in any convenient way.
  • There is no easy mechanism to send a payload of data to an instance during stack creation (for instance scripts and recipes to bootstrap an instance).
  • Finally, it is difficult to launch stacks that build upon already-existing stacks (i.e. an application stack within an existing VPC stack) because one must manually provide a variety of identifiers (subnets, IP addresses, security groups).

Hygroscope aims to solve each of these specific problems in an opinionated way:

  • CF templates are written in YAML and processed using cfoo, which provides a variety of convenience methods that increase readability.
  • Hygroscope can interactively prompt for each parameter and save inputted parameters to a file called a paramset. Additional stack launches can make use of existing paramsets, or can use paramsets as the basis and prompt for updated parameters.
  • A payload directory, if present, will be packaged and uploaded to S3. Hygroscope will generate and pass to CF a signed time-limited URL for accessing and downloading the payload, or the CloudFormation template can manage an instance profile granting indefinite access to the payload.
  • If an existing stack is specified, its outputs will be fetched and passed through as input parameters when launching a new stack.

The latest version of Hygroscope can be installed via RubyGems. The inline help documents each command and its options. The source code for Hygroscope and additional documentation is on GitHub, and a sample template that sets up a “bare VPC” is a good introduction to creating Hygroscopic templates.


The Fault In Our Stars

This review does not contain any spoilers.

The Fault In Our Stars Book CoverI found this novel deeply affecting. I hear from time to time, as everyone does, about various loose relations who have cancer, or are in remission, or have succumb to the disease. I see the cancer stories that describe tragic but stalwart children, their caring and committed parents, the charities they found and causes they champion, the valiant way in which they battle with dignity, their indefatigable courage.

And it all feels like bullshit to me. I don’t have the experience or the pain to justify this feeling, but I feel it all the same. Cancer patients and their families and support networks are not magically heroes. Evolution run amok does not make one noble or immune from normal-person feelings.

I hate the language we use to describe illness. I hate how we so often glorify people suffering from cancer while simultaneously pitying them. I don’t know how to interact with or relate to people who are suffering from disease, and I think the structures we as a society have created make that interaction harder than it should be. When we hold people up on a pedestal due to circumstances outside of their control, we don’t allow them to be normal people with normal-people feelings and concerns and fears and needs.


The Fault In Our Stars is a book about kids with cancer, but it is quick to point out that it is not a “cancer book” full of the standard tropes and plot progressions. The narrator is a teenage girl stricken with a form of the disease that affects her lungs, kept in check via experimental medicines and various machines to assist her breathing. Hazel is weak and frail and still alive long beyond her predicted expiration date. Her Sword of Damocles hangs ever-present as Hazel goes about her daily life, which is entirely normal in as much as she watches trashy TV and goes to the mall, and entirely abnormal in that she has nothing to strive for, few friends, little direction and plenty of pain.

Hazel joins a cancer support group suffused with a macabre sense of competition to outlive and outlast. The reader quickly gets the sense that most of her life post-diagnosis has been like this. There she meets a fellow sufferer, a high school boy with a prosthetic leg and a fear of oblivion.

They form a bond, they share Experiences (capital-E) and pain and fear and philosophy and random poetry and video games. They are overly wordy and prone to soliloquy and sort of strange, but in other ways quite real. They have ways of dealing with cancer and life and parents that feel very authentic to me. Things take dark turns, then people get better, then they get worse again. The future is uncertain, except that it is completely certain — dying is the endgame, and sooner rather than later.

How do you live when so much of your waking time is spent worrying about and wondering about and trying to fend off death? It is horrible and tragic but sometimes brilliant and funny and often just numbingly depressing. It is living, it is not living, it is a disease that doesn’t make sense and isn’t supposed to and doesn’t magically make people heroic or different, a disease that doesn’t care in a world that doesn’t care in a universe full of lives and people that may not have any meaning at all.

Oblivion. How do you face it? What choices do you make, when you aren’t given a chance to go out in a blaze of glory or on your own terms, but instead only slowly, by inches, in pain and agony and sadness? What does it all mean? Why should it mean anything? And how do you deal with that, each and every day?

Deep questions. Dark questions. Real questions.

Thank goodness there is a happy ending. Hazel and her friends figure everything out and feel better and know that they have accomplished something real and lasting and memorable.

No, I made that up. Of course it doesn’t end that way. It can’t. It just ends when it ends, as we all do. A surprising ending, but not surprising at all.  Because that’s how endings are.  They come along when you least and most expect them.  Sometimes, right in the middle of a

Never Forget

On September 11th some t-shirts I had ordered arrived. One of them looks like this:


I bought it because I thought it was funny and clever, but when it arrived I was a bit dismayed.  I had forgotten about how the usage of the phrase “never forget” has changed since 2001.  I first heard the phrased paired with “never again” in reference to the Holocaust.   The message was clear: we must remember humanity’s past misdeeds, lest we repeat them.  Similar phraseology has been used around other genocides, and the unfortunate fact is that we do forget, and we do allow them to repeat — Armenia, Rwanda, Congo, and now the ISIS actions in Iraq and Syria.  The world has not decided on a shared mission of preventing genocide in all its forms, and in that way the phrase “never forget, never again” is comically sad.

Never forget the dinosaurs plays on that — on the one hand its funny, because dinosaurs, right?  On the other hand it does make you think.  A great civilization came before us, a huge civilization that rose and covered this planet, and then was wiped out in its entirety such that none but bones remain.  We should remember this, we should remember our fragility as a species, as a planet.  We should think about the costs of the things we do to our world, and to each other, and we should remember that there is no guarantee that we will survive.

But now “never forget” seems to mean something different, something more insidious.  We apply it to the national tragedy of September 11, 2001, when a small group of Islamist terrorists committed a great atrocity in New York City that killed nearly three thousand people.  It is a testament to the power of terror and the dangers of an open society that such a small group — 19 actors — could commit such a large crime, and one so symbolic.  It was terrorizing as intended, and it embarked our country and the world on a new political, economic, and military path that has reshaped our modern world at the dawn of a new century.

“Never forget” is the wrong phrase here — we should remember the tragedy and honor the fallen innocents, certainly.  But the phrase became a rallying cry for two wars of revenge and destruction that have resulted in far more lives lost while arguably doing little, if anything, to make America safer or the world a better place.  It is not a cry for our shared humanity, but is instead a statement of division and anger.  The phrase itself has been twisted, turned petty.

I’m uncomfortable with my silly little dinosaur shirt, but not because I think it is wrong to wear it.  I’m uncomfortable because it forces me to confront the many mistakes we made after 9/11, the opportunities we missed, the actions we took from a place of fear and anger and sadness that were the wrong actions, with the wrong consequences.  We have this one world, this tiny precious world, this world we must all inhabit together, but through thousands of years of societal evolution we continue to repeat the same mistakes, to commit the same tragedies.

So the shirt does serve its purpose, even if accidentally — perhaps it is more meaningful, more impactful than I ever would have thought.  In that two-word phrase, “never forget,” is so much wrapped up meaning.  It makes you think.  Maybe it makes you think that I’m an idiot who got it all wrong, but you’re still thinking about it.  I guess that’s worth doing, and maybe September 11th of each year is the time to do it.

People silently struggle from all kinds of terrible things. They suffer from depression, ambition, substance abuse, and pretension. They suffer from family tragedy, Ivy-League educations, and self-loathing. They suffer from failing marriages, physical pain, and publishing. The good thing about politeness is that you can treat these people exactly the same. And then wait to see what happens. You don’t have to have an opinion. You don’t need to make a judgment. I know that doesn’t sound like liberation, because we live and work in an opinion-based economy. But it is. Not having an opinion means not having an obligation. And not being obligated is one of the sweetest of life’s riches.

Paul Ford,“How to Be Polite”

Water for Elephants

This review contains minor thematic spoilers but reveals no major plot details.

water-for-elephantsSara Gruen drew me in to her story of Great Depression-era circus life.  Meticulously researched and augmented with real photographs, this historical fiction novel is captivating and engrossing.

The framing of the story through the eyes of a reminiscing geriatric provides lots of opportunities for additional reflection and contemplation on changes in our world, although few are seized. His interactions with other patients and nurses are an interesting addition but offer little payoff.

Early on we meet roustabouts, carnival barkers, sideshow freaks, dwarves, and animals, and we learn about the pace of life on a traveling circus with clear class divides.  The “backstage” reality is gritty and tiring and hard-charging, and fascinating to behold. In the back third the pace of the story greatly accelerates and the romantic angle takes center stage, leading to a too-neat conclusion with little time to reflect on the human cost.

I can forgive the frequent telling-not-showing (with the requisite dumb as a brick narrator needed everything to be explained), I can overlook the romantic obsession, and I can accept the very limited development of secondary characters, but the present-day ride into the sunset ending is too ridiculous to bear.  I choose to believe that the modern-day denouement is just a crazed vision in old Jacob’s head, and somehow that makes this book work better for me.


Magic for Beginners

Screen Shot 2014-07-11 at 9.15.17 AMKelly Link’s Magic for Beginners short story collection is one of the most difficult books I have ever read. I loved every story with one exception, one that I just couldn’t get through because I was so uncomfortable with the cats and witches and death, but every other story was captivating and enthralling. Maybe I’ll go back and try that Catskin story again.

She does funny things with time, this writer. It goes forwards and backwards and sideways and spins around and comes back again. She has a strange way with literary structure — stories loop in on themselves and make my head spin. Each story is a little universe, each one pulls me in, and then ends mysteriously and evaporates into the mist. Each story is a dream, a long dream, a dream that ends when you wake up with the sun beating down and you wonder if you could have just slept a little longer how it might have all worked itself out but you will never know because that is how dreams are.

There is some strange twisted logic in these stories, like they are all in a weird magical alternate universe very close to but yet very different from our own. It is all so very frustrating. But so very worth it.

Geeking Out

Running GitHub Enterprise in Amazon EC2

Update (2015-03-29): GitHub now supports an EC2 appliance and this information is no longer accurate. It is useful only for historic reasons or general background when confronting similar challenges from other vendors.

GitHub’s hosted offering allows companies to run their own private GitHub appliance behind their firewall.  It is distributed as an OVF container that runs under VMWare or VirtualBox.  But what if you want to run it, along with your other infrastructure, on AWS?  Here is the (completely unsupported) way to do it!

The goal is to get the base GHE virtual appliance running on AWS so that we can install the latest GHE software package on top of it.  This package takes care of updating and configuring everything.  Once the software package is installed, the appliance behaves just like its on-prem cousins.

Break into the virtual appliance

First we need the virtual appliance in a form that can be moved into AWS.  Download the current virtual appliance from the GHE dashboard and find a way to get at it.  You may be able to just launch it locally in VMWare or VirtualBox, if you are able to get root, but I did not do this Instead I extracted the archive (it is just a tar file) to get at the VMDK disk image inside, and attempted to import it into EC2 using the AWS VM Import/Export tool.

This requires some fiddling, because you have to install the old EC2 command line tools and get all the options right, with some plausible guesses about what is inside.  Here is the command I ended up running:

ec2-import-volume /var/tmp/github-enterprise-11-10-320-x86-64-disk1.vmdk \
 -f vmdk -z us-east-1a -b agperson-ghe -o $AWS_ACCESS_KEY -w $AWS_SECRET_KEY

Once the import is complete (you can check the status with ec2-describe-conversion-tasks) I attempted to launch it — and failed due to an unsupported kernel.  But never fear!

Figure out what’s under the hood

If you don’t want to do this yourself skip to the end of this section where I tell you the secrets.

The VM import creates an EBS volume.  It may not be runnable, but it is mountable!  So start up a one-off Linux instance and attach the volume to it.  The data is stored in LVM, so you may need to install the lvm2 package and then run lvmdiskscan to see the volume group.

Run vgdisplay to get the name of the volume group (“enterprise”) and activate it by running vgchange -a y enterprise. Now you can mount the root volume:

mkdir /ghe
mount /dev/mapper/enterprise-root /ghe

Poke around in this volume a bit and you will establish that the virtual appliance comes with Ubuntu 11.10 Oneiric (wow!) and is 64-bit. With this information, we can launch an equivalent instance in EC2.

Setup an Amazon-happy instance

Launch a new EC2 instance using the publicly available community AMI from Ubuntu for 64-bit Oneiric (make sure you are using the released version — in us-east-1 I used ami-13ba2d7a). I chose an m3.large which is a good baseline based on GHE’s requirements. Make sure to attach a second volume for data or make the root volume large enough to hold all your repositories, and use SSD storage because it makes life better. Put your new instance in a security group that allows traffic on ports 22, 80, 443, and, if necessary, 9418 (the git:// port, which is non-authenticated so often not used on GHE installs).

When the instance launches, login as the “ubuntu” user and become root. Modify the /etc/apt/sources.list to point all archive stanzas at (including the security ones). Run an apt-get update && apt-get upgrade and wait a few minutes.

Now you need to copy over all of the files from the virtual appliance. You can either do this via SSH from the one-off instance you launched earlier, or detach the volume from that instance and repeat the steps to get LVM running and attach it to the new instance. Either way, use rsync to get everything important onto your new VM. Rackspace offers a helpful tutorial on doing this, including a good set of directory paths to exclude. I used their list and everything worked fine. The command I ran with the volume mounted locally was:

rsync --dry-run -azPx --exclude-from="exclude.txt" /ghe/ /

(and once I was satisfied, I ran it again without the “–dry-run” flag).

Bombs away!

Before rebooting, copy your SSH key into /root/.ssh/authorized_keys in case anything goes wrong (and take a moment to ponder who Ben is and why his “HacBook-Air.local” key is on our server!). Then restart the instance and, when it is done booting, visit it via HTTPS to see the beautiful GHE setup screen! Upload the latest software package and your license key and give it half an hour or so, and if everything goes well, you will have a fully-functional GitHub Enterprise instance in the cloud.

Note that after the software package installs you will no longer have root access to the server. A pity.

A few other important steps are left as an exercise to the reader — lock down access, setup SES or some other email sending capability, stay in compliance with your license, and take frequent backup snapshots! Good luck!


The Internet’s Own Boy

Aaron Swartz

I’ve written before (and on several more occasions) about Aaron Swartz, a complicated and amazing person and digital activist who I followed and loosely orbited for many years.  Aaron did incredible work for and on behalf of the Internet as a democratizing medium, and he caused me to frequently question my own life and career choices.  He was targeted by an overreaching federal prosecution due to some of his activism work on the edges of the law, and after two years of pressure and abuse at the hands of the federal government, he tragically took his life in January 2013.

The Internet’s Own Boy: The Story of Aaron Swartz is a documentary film by Brian Knappenberger that traces Aaron’s life, his successes and failures, his political action and digital activism, his run-ins with the law, and his too-soon death.  Along the way several internet luminaries, journalists, activists, congresspeople, and other smart individuals weigh in and provide context.  I knew much of what the film presented but I still found it compelling.  While clearly opinionated, this film does a good job of portraying who Aaron was, what he believed in, and where things went so very wrong.

Many people feel that technology and politics together are too complicated, too confusing, and too inscrutable.  Many in government dismiss technology experts and inventors of things that have fundamentally changed our lives as mere “nerds”.  Aaron lived his whole life thoughtfully and fully, and his story is one that is approachable to anyone, technological or not.  The things he fought for are important, and they are comprehensible, and they should not be dismissed.  He showed how we can use technology and the internet to make this world a better one, and what we should do to stop others from using it to make the world worse.

The movie is imperfect, as was Aaron.  And it does a few funny things with time and ordering that slightly distort some of the major events in Aaron’s saga.  But on the whole it is thoughtful, and it is powerful, and it is worth watching.

You can view The Internet’s Own Boy: The Story of Aaron Swartz in its entirety for free on the Internet Archive.


European Conning

Meghan and I are currently attending a conference on a small island (!) in the Côte d’Azur — better known in English as the “French Riviera.” If this one is the norm, European-style tech conferences are quite a bit different from their American counterparts! Breakfast is early (7:30am), sessions run all day, and every evening includes dinner and drinks that start around 8:30pm and go past 11. The next morning, we wake up and start again.

Did I mention that every lunch and dinner is three courses and includes copious amounts of wine? And did I further mention that the wine is produced on this very island’s vineyards? Because yes, that’s a thing. And if you aren’t eating lots of paté and frequently emptying your glass, people start to wonder if you are ill.

We cut out early — it is only 10:30 — so Meghan can put the finishing touches on the talk she is giving tomorrow about user experience design. We also took a bit of time in the afternoon to wander around the island, which is quite a sight to behold — a few pictures above are a preview of the forthcoming album.

So far we have made friends with some Canadians, met some French and Italians, and had dinner with a contingent from Belarus. We learned that last year’s conference was held in a circus tent in Warsaw, so take that American conference centers!

It is fascinating to see people of all different languages and cultures come together to discuss their shared interest in technology. The opening speaker said that he hoped that everyone here would learn something new and then spread that knowledge by teaching it when they got home. I am reminded of how easy I have it as a native English speaker who never has to worry about a lack of documentation, examples, or online help. It is also interesting to hear from people in countries where there is much less appetite for working with modern, fast-moving languages and frameworks. Many of the people here really are ambassadors for and teachers of these technologies.

We are here for a couple more days, and then we will take a (ferry + bus + train) ride back up to Paris to spend some more time exploring that city’s wonders.

Citizens in a democracy make a certain pact with one another: to answer speech with more speech, not violence. No matter how angry what I say makes you, you do not have a right to pull a gun on me. But now the gun has already been drawn, nominally as an act of symbolic speech — and yet it still remains a gun. A slippage has occurred between the First and Second Amendments, and the First suffers as a result.

Patrick Blanchfield, “What Do Guns Say?“, New York Times
Geeking Out

Sending automated notifications to HipChat rooms

At work we have been piloting HipChat’s new self-hosted on-premises option for the last few months.  It has been great having a bunch of people who work in different buildings and on different schedules using shared chat rooms for communication.

I have also been experimenting with hooking HipChat into our toolchain. We now have a chat room where every Capistrano deployment is announced, and another where all of our high-priority Zabbix alerts are collected. HipChat makes this easy with their version 2 API’s room notifications feature. A room owner can simply generate a room-specific API token and plug it into a script to send notifications.

Here is an example:HipChat Zabbix Alerts

And to make it easy for the next person who wants to do this, I’ve released the code on GitHub.

Instructions for setting it up are in the README. And 15 minutes later, you’re in business with pretty and useful Zabbix notifications in HipChat.


Moosilauke revisited

On Saturday Mat and I hiked Mount Moosilauke, one of New Hampshires “4000-footers.”  The weather was warm (40s), although the day was overcast and the summit was fogged in.  We got a late start after a wrong turn (kids, bring maps!), so we were a bit concerned about daylight.

A trail report from a few days earlier indicated that it would be smooth going, but apparently we mis-read it, because everyone else on the mountain that day had either skis, snowshoes, or both.  We had neither, and for the first 3+ miles almost ever step resulted in snow up to our knees.

We held out hope that as we gained elevation (and colder weather) the base would be harder-packed.  That was the case eventually, but the slow going coupled with our late start made us decide to turn back prior to the summit.  It was an adventure regardless, and on the way down we got in a lot of “sledding” on our behinds, which was a blast.

View photo slideshow

The eighth iteration

The last time I substantially changed this blog was in 2009, and in the last few years it has languished. I’m very happy with this modern update, which is very clean, simple, and content-focused. I’ve removed almost everything else, which should help me focus on the writing.  I plan to back-fill some posts from things I’ve written on Facebook and elsewhere, and go from there.  Welcome to AgBlog version 8, now with a new name and location!


A brief California whirlwind

On Friday I headed out to the West coast for a brief visit in order to surprise Aunt Linda on the occasion of her 60th birthday party. Well, she was surprised, thanks to some excellent planning, scheming, and misdirection. It was a really nice party.

On Saturday the out-of-town partygoers gathered at Strand Terrace for brunch.  I always love it when we host meals while I am in town because it is fun to cook together as a family.  Shaina made quiche, I chopped things and cooked up bacon, Mom made an apple cake, and Dad and Jess cooked as well as taking care of all the grocery shopping. The parents have redone their patio to give it more of an “outdoor living room” feel, and I think it really works — definitely a good fit for the California climate.

On Monday Jessica, Mom and I went paddle boarding in Newport Beach, which is fun once you recognize how absurd and inefficient it is.  The high winds kept pushing us back and threatening to topple us over, but we made it to our arbitrary goal (a bridge) and back without major incident.  In the morning Mom and I had also hiked at Santiago Oaks, so it was an active sort of day.

Throw in some family time, pool time, meal time, and beach time, and cap it with lunch at In-N-Out — a pretty good few days in the sun!  I’m sad that the trip is already over, but I’m spending a few days in Portland with Jessica before heading back home.